Search Results for author:
Found 10 papers, 3 papers with code
PartIR: Composing SPMD Partitioning Strategies for Machine Learning
Training of modern large neural networks (NN) requires a combination of parallelization strategies encompassing data, model, or optimizer sharding.
Efficient and Sound Differentiable Programming in a Functional Array-Processing Language
Automatic differentiation (AD) is a technique for computing the derivative of a function represented by a program.
Automatic Discovery of Composite SPMD Partitioning Strategies in PartIR
Large neural network models are commonly trained through a combination of advanced parallelism strategies in a single program, multiple data (SPMD) paradigm.
Automap: Towards Ergonomic Automated Parallelism for ML Models
The rapid rise in demand for training large neural network architectures has brought into focus the need for partitioning strategies, for example by using data, model, or pipeline parallelism.
Memory-efficient array redistribution through portable collective communication
In this paper we address the problem of redistributing multi-dimensional array data in SPMD computations, the most prevalent form of parallelism in deep learning.
Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learning
We present a novel characterization of the mapping of multiple parallelism forms (e. g. data and model parallelism) onto hierarchical accelerator systems that is hierarchy-aware and greatly reduces the space of software-to-hardware mapping.
Gradient Forward-Propagation for Large-Scale Temporal Video Modelling
How can neural networks be trained on large-volume temporal data efficiently?
Efficient Differentiable Programming in a Functional Array-Processing Language
We present a system for the automatic differentiation of a higher-order functional array-processing language.
AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks
Through an implementation on multi-core CPUs, we show that AMP training converges to the same accuracy as conventional synchronous training algorithms in a similar number of epochs, but utilizes the available hardware more efficiently even for small minibatch sizes, resulting in significantly shorter overall training times.
Measuring Neural Net Robustness with Constraints
Despite having high accuracy, neural nets have been shown to be susceptible to adversarial examples, where a small perturbation to an input can cause it to become mislabeled.
