Search Results for author:
Found 11 papers, 4 papers with code
Better Schedules for Low Precision Training of Deep Neural Networks
From these experiments, we discover alternative CPT schedules that offer further improvements in training efficiency and model performance, as well as derive a set of best practices for choosing CPT schedules.
Cold Start Streaming Learning for Deep Networks
To mitigate these shortcomings, we propose Cold Start Streaming Learning (CSSL), a simple, end-to-end approach for streaming learning with deep networks that uses a combination of replay and data augmentation to avoid catastrophic forgetting.
PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communication
Notably, little is known regarding the convergence rate of GCN training with both stale features and stale feature gradients.
i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery
We propose a novel, structured pruning algorithm for neural networks -- the iterative, Sparse Structured Pruning algorithm, dubbed as i-SpaSP.
How much pre-training is enough to discover a good subnetwork?
Aiming to mathematically analyze the amount of dense network pre-training needed for a pruned network to perform well, we discover a simple theoretical bound in the number of gradient descent pre-training iterations on a two-layer, fully-connected network, beyond which pruning via greedy forward selection [61] yields a subnetwork that achieves good training error.
Exceeding the Limits of Visual-Linguistic Multi-Task Learning
By leveraging large amounts of product data collected across hundreds of live e-commerce websites, we construct 1000 unique classification tasks that share similarly-structured input data, comprised of both text and images.
ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
Thus, ResIST reduces the per-iteration communication, memory, and time requirements of ResNet training to only a fraction of the requirements of full-model training.
GIST: Distributed Training for Large-Scale Graph Convolutional Networks
The graph convolutional network (GCN) is a go-to solution for machine learning on graphs, but its training is notoriously difficult to scale both in terms of graph size and the number of model parameters.
E-Stitchup: Data Augmentation for Pre-Trained Embeddings
In this work, we propose data augmentation methods for embeddings from pre-trained deep learning models that take a weighted combination of a pair of input embeddings, as inspired by Mixup, and combine such augmentation with extra label softening.
Distributed Learning of Deep Neural Networks using Independent Subnet Training
These properties of IST can cope with issues due to distributed data, slow interconnects, or limited device memory, making IST a suitable approach for cases of mandatory distribution.
Functional Generative Design of Mechanisms with Recurrent Neural Networks and Novelty Search
However, while static designs are easily produced with 3D printing, functional designs with moving parts are more difficult to generate: The search space is too high-dimensional, the resolution of the 3D-printed parts is not adequate, and it is difficult to predict the physical behavior of imperfect 3D-printed mechanisms.
