Search Results for author:
Found 14 papers, 7 papers with code
Reducing the Cost of Quantum Chemical Data By Backpropagating Through Density Functional Theory
For comparison, Sch\"utt et al. (2019) spent 626 hours creating a dataset on which they trained their NN for 160h, for a total of 786h; our method achieves comparable performance within 31h.
SparQ Attention: Bandwidth-Efficient LLM Inference
The computational difficulties of large language model (LLM) inference remain a significant obstacle to their widespread deployment.
Generating QM1B with PySCF$_{\text{IPU}}$
Similar benefits are yet to be unlocked for quantum chemistry, where the potential of deep learning is constrained by comparatively small datasets with 100k to 20M training examples.
Training and inference of large language models using 8-bit floating point
FP8 formats are gaining popularity to boost the computational efficiency for training and inference of large deep learning models.
Unit Scaling: Out-of-the-Box Low-Precision Training
We present unit scaling, a paradigm for designing deep learning models that simplifies the use of low-precision number formats.
BESS: Balanced Entity Sampling and Sharing for Large-Scale Knowledge Graph Completion
We present the award-winning submission to the WikiKG90Mv2 track of OGB-LSC@NeurIPS 2022.
8-bit Numerical Formats for Deep Neural Networks
Given the current trend of increasing size and complexity of machine learning architectures, it has become of critical importance to identify new approaches to improve the computational efficiency of model training.
Towards Structured Dynamic Sparse Pre-Training of BERT
Identifying algorithms for computational efficient unsupervised training of large language models is an important and active area of research.
GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
Attention based language models have become a critical component in state-of-the-art natural language processing systems.
Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
Much recent research has been dedicated to improving the efficiency of training and inference for image classification.
Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
We investigate the reasons for the performance degradation incurred with batch-independent normalization.
Parallel Training of Deep Networks with Local Updates
Deep learning models trained on large data sets have been widely successful in both vision and language domains.
Improving Neural Network Training in Low Dimensional Random Bases
Stochastic Gradient Descent (SGD) has proven to be remarkably effective in optimizing deep neural networks that employ ever-larger numbers of parameters.
Revisiting Small Batch Training for Deep Neural Networks
Modern deep neural network training is typically based on mini-batch stochastic gradient optimization.
