Search Results for author:
Found 26 papers, 18 papers with code
What’s Hidden in a One-layer Randomly Weighted Transformer?
Hidden within a one-layer randomly weighted Transformer, we find that subnetworks that can achieve 29. 45/17. 29 BLEU on IWSLT14/WMT14.
Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
Since hardware resources are limited, the objective of training deep learning models is typically to maximize accuracy subject to the time and memory constraints of training and inference.
K-LITE: Learning Transferable Visual Models with External Knowledge
In this paper, we propose K-LITE (Knowledge-augmented Language-Image Training and Evaluation), a simple strategy to leverage external knowledge to build transferable visual systems: In training, it enriches entities in natural language with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that can understand both visual concepts and their knowledge; In evaluation, the natural language is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models.
One Parameter Defense -- Defending against Data Inference Attacks via Differential Privacy
The experimental results show the method to be an effective and timely defense against both membership inference and model inversion attacks with no reduction in accuracy.
Staged Training for Transformer Language Models
As an alternative, we consider a staged training setup that begins with a small model and incrementally increases the amount of compute used for training by applying a "growth operator" to increase the model depth and width.
Implicit Transformer Network for Screen Content Image Continuous Super-Resolution
Nowadays, there is an explosive growth of screen contents due to the wide application of screen sharing, remote cooperation, and online education.
Discovering Non-monotonic Autoregressive Orderings with Variational Inference
Permutations then serve as target generation orders for training an insertion-based Transformer language model.
Multitask Prompted Training Enables Zero-Shot Task Generalization
Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020).
What's Hidden in a One-layer Randomly Weighted Transformer?
Hidden within a one-layer randomly weighted Transformer, we find that subnetworks that can achieve 29. 45/17. 29 BLEU on IWSLT14/WMT14.
How Much Can CLIP Benefit Vision-and-Language Tasks?
Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world.
Ranked #5 on
Visual Entailment
on SNLI-VE val
(using extra training data)
Learned Token Pruning for Transformers
We extensively test the performance of LTP on GLUE tasks and show that our method outperforms the prior state-of-the-art token pruning methods by up to ~2. 5% higher accuracy with the same amount of FLOPs.
LEAP: Learnable Pruning for Transformer-based Models
Moreover, in order to reduce hyperparameter tuning, a novel adaptive regularization coefficient is deployed to control the regularization penalty adaptively.
Discovering Autoregressive Orderings with Variational Inference
One strategy to recover this information is to decode both the content and location of tokens.
Reservoir Transformers
We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated.
From Distributed Machine Learning To Federated Learning: In The View Of Data Privacy And Security
Federated learning is an improved version of distributed machine learning that further offloads operations which would usually be performed by a central server.
Distributed, Parallel, and Cluster Computing
MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
Phrase localization is a task that studies the mapping from textual phrases to regions of an image.
Noisy Self-Knowledge Distillation for Text Summarization
In this paper we apply self-knowledge distillation to text summarization which we argue can alleviate problems with maximum-likelihood training on single reference and noisy datasets.
Differentially Private Multi-Agent Planning for Logistic-like Problems
To the best of our knowledge, this paper is the first to apply differential privacy to the field of multi-agent planning as a means of preserving the privacy of agents for logistic-like problems.
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning
We introduce ADAHESSIAN, a second order stochastic optimization algorithm which dynamically incorporates the curvature of the loss function via ADAptive estimates of the HESSIAN.
PowerNorm: Rethinking Batch Normalization in Transformers
To address this, we propose Power Normalization (PN), a novel normalization scheme that resolves this issue by (i) relaxing zero-mean normalization in BN, (ii) incorporating a running quadratic mean instead of per batch statistics to stabilize fluctuations, and (iii) using an approximate backpropagation for incorporating the running statistics in the forward pass.
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
Since hardware resources are limited, the objective of training deep learning models is typically to maximize accuracy subject to the time and memory constraints of training and inference.
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
In particular, we propose a new group-wise quantization scheme, and we use a Hessian based mix-precision method to compress the model further.
An annotated dataset of literary entities
We present a new dataset comprised of 210, 532 tokens evenly drawn from 100 different English-language literary texts annotated for ACE entity categories (person, location, geo-political entity, facility, organization, and vehicle).
Pragmatically Informative Text Generation
We improve the informativeness of models for conditional text generation using techniques from computational pragmatics.
Ranked #1 on
Data-to-Text Generation
on E2E NLG Challenge
Abstractive Text Summarization
Conditional Text Generation
+3
On the Generation of Medical Question-Answer Pairs
Question answering (QA) has achieved promising progress recently.
Emoji-Powered Representation Learning for Cross-Lingual Sentiment Classification
To tackle this problem, cross-lingual sentiment classification approaches aim to transfer knowledge learned from one language that has abundant labeled examples (i. e., the source language, usually English) to another language with fewer labels (i. e., the target language).
