Search Results for author:
Found 6 papers, 2 papers with code
Rockmate: an Efficient, Fast, Automatic and Generic Tool for Re-materialization in PyTorch
We show through experiments on many models that Rockmate is as fast as Rotor and as efficient as Checkmate, and that it allows in many cases to obtain a significantly lower memory consumption for activations (by a factor of 2 to 5) for a rather negligible overhead (of the order of 10% to 20%).
Survey on Large Scale Neural Network Training
Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training.
Efficient Combination of Rematerialization and Offloading for Training DNNs
Rematerialization and offloading are two well known strategies to save memory during the training phase of deep neural networks, allowing data scientists to consider larger models, batch sizes or higher resolution data.
Geometric Deep Reinforcement Learning for Dynamic DAG Scheduling
In this paper, we propose a reinforcement learning approach to solve a realistic scheduling problem, and apply it to an algorithm commonly executed in the high performance computing community, the Cholesky factorization.
Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory
This paper introduces a new activation checkpointing method which allows to significantly decrease memory usage when training Deep Neural Networks with the back-propagation algorithm.
Training on the Edge: The why and the how
Edge computing is the natural progression from Cloud computing, where, instead of collecting all data and processing it centrally, like in a cloud computing environment, we distribute the computing power and try to do as much processing as possible, close to the source of the data.
Distributed, Parallel, and Cluster Computing
