Search Results for author:
Found 15 papers, 7 papers with code
Token Dropping for Efficient BERT Pretraining
Transformer-based models generally allocate the same amount of computation for each token in a given sequence.
Auto-scaling Vision Transformers without Training
The motivation comes from two pain spots: 1) the lack of efficient and principled methods for designing and scaling ViTs; 2) the tremendous computational cost of training ViT that is much heavier than its convolution counterpart.
A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation
In this paper, we comprehensively study three architecture design choices on ViT -- spatial reduction, doubled channels, and multiscale features -- and demonstrate that a vanilla ViT architecture can fulfill this goal without handcrafting multiscale features, maintaining the original ViT design philosophy.
Speeding up Deep Model Training by Sharing Weights and Then Unsharing
Our approach exploits the special structure of BERT that contains a stack of repeated modules (i. e., transformer encoders).
A new communication paradigm: from bit accuracy to semantic fidelity
In this article, we deploy semantics to solve the spectrum and power bottleneck and propose a first understanding and then transmission framework with high semantic fidelity.
Networking and Internet Architecture
Speeding up Deep Learning Training by Sharing Weights and Then Unsharing
It has been widely observed that increasing deep learning model sizes often leads to significant performance improvements on a variety of natural language processing and computer vision tasks.
Efficient Scale-Permuted Backbone with Learned Resource Distribution
Furthermore, SpineNet is built with a uniform resource distribution over operations.
Go Wide, Then Narrow: Efficient Training of Deep Thin Networks
This is achieved by layerwise imitation, that is, forcing the thin network to mimic the intermediate outputs of the wide network from layer to layer.
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Then, we conduct knowledge transfer from this teacher to MobileBERT.
Ranked #20 on
Semantic Textual Similarity
on MRPC
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
Without extra retraining or post-processing steps, we are able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs.
Ranked #30 on
Neural Architecture Search
on ImageNet
SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization
We propose SpineNet, a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search.
Ranked #9 on
Image Classification
on iNaturalist
Scaling Up Neural Architecture Search with Big Single-Stage Models
In this work, we propose BigNAS, an approach that simplifies this workflow and scales up neural architecture search to target a wide range of model sizes simultaneously.
High Resolution Medical Image Analysis with Spatial Partitioning
It is infeasible to train CNN models directly on such high resolution images, because neural activations of a single image do not fit in the memory of a single GPU/TPU, and naive data and model parallelism approaches do not work.
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches.
Ranked #11 on
Question Answering
on SQuAD1.1 dev
(F1 metric)
A practical convolutional neural network as loop filter for intra frame
It aims to design a single CNN model with low redundancy to adapt to decoded frames with different qualities and ensure consistency.
Multimedia
