Search Results for author:
Found 10 papers, 7 papers with code
Mesh-TensorFlow: Deep Learning for Supercomputers
We use Mesh-TensorFlow to implement an efficient data-parallel, model-parallel version of the Transformer sequence-to-sequence model.
Ranked #10 on
Language Modelling
on One Billion Word
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Scaling up deep neural network capacity has been known as an effective approach to improving model quality for several different machine learning tasks.
Ranked #4 on
Fine-Grained Image Classification
on Birdsnap
(using extra training data)
Image Classification at Supercomputer Scale
Deep learning is extremely computationally intensive, and hardware vendors have responded by building faster accelerators in large clusters.
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
High Resolution Medical Image Analysis with Spatial Partitioning
It is infeasible to train CNN models directly on such high resolution images, because neural activations of a single image do not fit in the memory of a single GPU/TPU, and naive data and model parallelism approaches do not work.
Talking-Heads Attention
We introduce "talking-heads attention" - a variation on multi-head attention which includes linearprojections across the attention-heads dimension, immediately before and after the softmax operation. While inserting only a small number of additional parameters and a moderate amount of additionalcomputation, talking-heads attention leads to better perplexities on masked language modeling tasks, aswell as better quality when transfer-learning to language comprehension and question answering tasks.
Toward Annotator Group Bias in Crowdsourcing
We conduct experiments on both synthetic and real-world datasets.
Enhanced Exploration in Neural Feature Selection for Deep Click-Through Rate Prediction Models via Ensemble of Gating Layers
Inspired by gradient-based neural architecture search (NAS) and network pruning methods, people have tackled the NFS problem with Gating approach that inserts a set of differentiable binary gates to drop less informative features.
CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPU
Our experiments show that previous scaling rules fail in the training of CTR prediction neural networks.
Monolith: Real Time Recommendation System With Collisionless Embedding Table
In this paper, we present Monolith, a system tailored for online training.
