Search Results for author:
Found 19 papers, 12 papers with code
Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Image as a Foreign Language: BEiT Pretraining for Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is emerging.
A Unified View of Masked Image Modeling
Masked image modeling has demonstrated great potential to eliminate the label-hungry problem of training large-scale vision Transformers, achieving impressive performance on various downstream tasks.
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is emerging.
Ranked #1 on
Visual Reasoning
on NLVR2 Test
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
The large-size BEiT v2 obtains 87. 3% top-1 accuracy for ImageNet-1K (224 size) fine-tuning, and 56. 7% mIoU on ADE20K for semantic segmentation.
Ranked #27 on
Self-Supervised Image Classification
on ImageNet
VL-BEiT: Generative Vision-Language Pretraining
Our minimalist solution conducts masked prediction on both monomodal and multimodal data with a shared Transformer.
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
As more and more pre-trained language models adopt on-cloud deployment, the privacy issues grow quickly, mainly for the exposure of plain-text user data (e. g., search history, medical record, bank account).
Corrupted Image Modeling for Self-Supervised Visual Pre-Training
Given this corrupted image, an enhancer network learns to either recover all the original image pixels, or predict whether each visual token is replaced by a generator sample or not.
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network.
Ranked #2 on
Image Retrieval
on PhotoChat
s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Pretrained bidirectional Transformers, such as BERT, have achieved significant improvements in a wide variety of language understanding tasks, while it is not straightforward to directly apply them for natural language generation.
Learning to Sample Replacements for ELECTRA Pre-Training
Moreover, we propose to use a focal loss for the generator in order to relieve oversampling of correct tokens as replacements.
BEiT: BERT Pre-Training of Image Transformers
We first "tokenize" the original image into visual tokens.
Ranked #10 on
Document Layout Analysis
on PubLayNet val
Attention Temperature Matters in Abstractive Summarization Distillation
In this paper, we find simply manipulating attention temperatures in Transformers can make pseudo labels easier to learn for student models.
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers.
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Ranked #4 on
Question Generation
on SQuAD1.1
(using extra training data)
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher).
Ranked #8 on
Zero-shot Text Search
on BEIR
Inspecting Unification of Encoding and Matching with Transformer: A Case Study of Machine Reading Comprehension
Most machine reading comprehension (MRC) models separately handle encoding and matching with different network architectures.
Neural Melody Composition from Lyrics
In this paper, we study a novel task that learns to compose music from natural language.
Neural Question Generation from Text: A Preliminary Study
Automatic question generation aims to generate questions from a text passage where the generated questions can be answered by certain sub-spans of the given passage.
Ranked #13 on
Question Generation
on SQuAD1.1
