Search Results for author:
Found 9 papers, 1 papers with code
MADA: Meta-Adaptive Optimizers through hyper-gradient Descent
Since Adam was introduced, several novel adaptive optimizers for deep learning have been proposed.
Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training
In contrast to existing solutions, the implementation of the SageMaker library is much more generic and flexible, in that it can automatically partition and run pipeline parallelism over arbitrary model architectures with minimal code change, and also offers a general and extensible framework for tensor parallelism, which supports a wider range of use cases, and is modular enough to be easily applied to new training scripts.
Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations
Communication bottleneck has been identified as a significant issue in distributed optimization of large-scale learning models.
Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations
Communication bottleneck has been identified as a significant issue in distributed optimization of large-scale learning models.
Densifying Assumed-sparse Tensors: Improving Memory Efficiency and MPI Collective Performance during Tensor Accumulation for Parallelized Training of Neural Machine Translation Models
Neural machine translation - using neural networks to translate human language - is an area of active research exploring new neuron types and network topologies with the goal of dramatically improving machine translation performance.
Differentially Private Consensus-Based Distributed Optimization
Consensus-based optimization consists of a set of computational nodes arranged in a graph, each having a local objective that depends on their local data, where in every step nodes take a linear combination of their neighbors' messages, as well as taking a new gradient step.
Privacy-Utility Trade-off of Linear Regression under Random Projections and Additive Noise
Data privacy is an important concern in machine learning, and is fundamentally at odds with the task of training useful learning models, which typically require the acquisition of large amounts of private user data.
Redundancy Techniques for Straggler Mitigation in Distributed Optimization and Learning
Performance of distributed optimization and learning systems is bottlenecked by "straggler" nodes and slow communication links, which significantly delay computation.
Straggler Mitigation in Distributed Optimization Through Data Encoding
Slow running or straggler tasks can significantly reduce computation speed in distributed computation.
