Search Results for author: Furu Wei

Found 169 papers, 73 papers with code

Pseudo-Label Guided Unsupervised Domain Adaptation of Contextual Embeddings

no code implementations EACL (AdaptNLP) 2021 Tianyu Chen, Shaohan Huang, Furu Wei, JianXin Li

In unsupervised domain adaptation, we aim to train a model that works well on a target domain when provided with labeled source samples and unlabeled target samples.

Language Modelling Masked Language Modeling +2

MarkupLM: Pre-training of Text and Markup Language for Visually Rich Document Understanding

no code implementations ACL 2022 Junlong Li, Yiheng Xu, Lei Cui, Furu Wei

Multimodal pre-training with text, layout, and image has made significant progress for Visually Rich Document Understanding (VRDU), especially the fixed-layout documents such as scanned document images.

Towards Making the Most of Cross-Lingual Transfer for Zero-Shot Neural Machine Translation

1 code implementation ACL 2022 Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei

When applied to zero-shot cross-lingual abstractive summarization, it produces an average performance gain of 12. 3 ROUGE-L over mBART-ft. We conduct detailed analyses to understand the key ingredients of SixT+, including multilinguality of the auxiliary parallel data, positional disentangled encoder, and the cross-lingual transferability of its encoder.

Abstractive Text Summarization Cross-Lingual Abstractive Summarization +3

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

no code implementations Findings (ACL) 2022 Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, JianXin Li, Furu Wei

As more and more pre-trained language models adopt on-cloud deployment, the privacy issues grow quickly, mainly for the exposure of plain-text user data (e. g., search history, medical record, bank account).

XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding

no code implementations Findings (ACL) 2022 Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei

Multimodal pre-training with text, layout, and image has achieved SOTA performance for visually rich document understanding tasks recently, which demonstrates the great potential for joint learning across different modalities.

Pseudo-Masked Language Models for Unified Language Model Pre-Training

1 code implementation ICML 2020 Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, Hsiao-Wuen Hon

We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).

Language Modelling Natural Language Understanding

Visually-Augmented Language Modeling

1 code implementation20 May 2022 Weizhi Wang, Li Dong, Hao Cheng, Haoyu Song, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei

With the visually-augmented context, VaLM uses a visual knowledge fusion layer to enable multimodal grounded language modeling by attending on both text context and visual knowledge in images.

Image Retrieval Language Modelling

Prototypical Calibration for Few-shot Learning of Language Models

no code implementations20 May 2022 Zhixiong Han, Yaru Hao, Li Dong, Furu Wei

In-context learning of GPT-like models has been recognized as fragile across different hand-crafted templates, and demonstration permutations.

Few-Shot Learning

Lossless Acceleration for Seq2seq Generation with Aggressive Decoding

1 code implementation20 May 2022 Tao Ge, Heming Xia, Xin Sun, Si-Qing Chen, Furu Wei

We study lossless acceleration for seq2seq generation with a novel decoding algorithm -- Aggressive Decoding.

Abstractive Text Summarization Grammatical Error Correction +3

Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization

no code implementations ACL 2022 Ruipeng Jia, Xingxing Zhang, Yanan Cao, Shi Wang, Zheng Lin, Furu Wei

In zero-shot multilingual extractive text summarization, a model is typically trained on English summarization dataset and then applied on summarization datasets of other languages.

Extractive Summarization Extractive Text Summarization

Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?

no code implementations27 Apr 2022 Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Zhuo Chen, Peidong Wang, Gang Liu, Jinyu Li, Jian Wu, Xiangzhan Yu, Furu Wei

Recently, self-supervised learning (SSL) has demonstrated strong performance in speaker recognition, even if the pre-training objective is designed for speech recognition.

Self-Supervised Learning Speaker Recognition +2

On the Representation Collapse of Sparse Mixture of Experts

1 code implementation20 Apr 2022 Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei

We also present a comprehensive analysis on the representation and routing behaviors of our models.

Language Modelling

StableMoE: Stable Routing Strategy for Mixture of Experts

1 code implementation ACL 2022 Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei

We point out that existing learning-to-route MoE methods suffer from the routing fluctuation issue, i. e., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference.

Language Modelling Machine Translation

LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

1 code implementation18 Apr 2022 Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei

In this paper, we propose LayoutLMv3 to pre-train multimodal Transformers for Document AI with unified text and image masking.

Document AI Document Image Classification +8

Lossless Speedup of Autoregressive Translation with Generalized Aggressive Decoding

2 code implementations30 Mar 2022 Heming Xia, Tao Ge, Furu Wei, Zhifang Sui

Different from previous work accelerating translation at the cost of quality loss, we propose Generalized Aggressive Decoding (GAD) -- a novel decoding paradigm for lossless speedup of autoregressive translation, through the collaboration of autoregressive and non-autoregressive translation (NAT) of the Transformer.

Abstractive Text Summarization Translation

CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment

no code implementations ACL 2022 Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei

We first evaluate CLIP's zero-shot performance on a typical visual question answering task and demonstrate a zero-shot cross-modality transfer capability of CLIP on the visual entailment task.

Question Answering Visual Entailment +2

DiT: Self-supervised Pre-training for Document Image Transformer

3 code implementations4 Mar 2022 Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei

Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.)

Document AI Document Image Classification +3

DeepNet: Scaling Transformers to 1,000 Layers

4 code implementations1 Mar 2022 Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Furu Wei

In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers.

Translation

Controllable Natural Language Generation with Contrastive Prefixes

no code implementations Findings (ACL) 2022 Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen

We propose a novel supervised method and also an unsupervised method to train the prefixes for single-aspect control while the combination of these two methods can achieve multi-aspect control.

Language Modelling Pretrained Language Models +1

Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt

no code implementations23 Feb 2022 Lianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei, Houfeng Wang

To collocate with the unified prompt, we propose a new initialization method for the target label word to further improve the model's transferability across languages.

Pretrained Language Models Zero-Shot Cross-Lingual Transfer

A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models

no code implementations17 Feb 2022 Da Yin, Li Dong, Hao Cheng, Xiaodong Liu, Kai-Wei Chang, Furu Wei, Jianfeng Gao

With the increasing of model capacity brought by pre-trained language models, there emerges boosting needs for more knowledgeable natural language processing (NLP) models with advanced functionalities including providing and making flexible use of encyclopedic and commonsense knowledge.

Language Modelling

EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation

1 code implementation16 Feb 2022 Tao Ge, Furu Wei

We propose EdgeFormer -- a parameter-efficient Transformer of the encoder-decoder architecture for on-device seq2seq generation, which is customized under strict computation and memory constraints.

Grammatical Error Correction Knowledge Distillation +2

Corrupted Image Modeling for Self-Supervised Visual Pre-Training

no code implementations7 Feb 2022 Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, Furu Wei

CIM is a general and flexible visual pre-training framework that is suitable for various network architectures.

Image Classification Semantic Segmentation

A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model

no code implementations26 Jan 2022 Xin Sun, Tao Ge, Shuming Ma, Jingjing Li, Furu Wei, Houfeng Wang

Synthetic data construction of Grammatical Error Correction (GEC) for non-English languages relies heavily on human-designed and language-specific rules, which produce limited error-corrected patterns.

Grammatical Error Correction Language Modelling +1

Kformer: Knowledge Injection in Transformer Feed-Forward Layers

1 code implementation15 Jan 2022 Yunzhi Yao, Shaohan Huang, Ningyu Zhang, Li Dong, Furu Wei, Huajun Chen

Knowledge-Enhanced Model have developed a diverse set of techniques for knowledge integration on different knowledge sources.

Language Modelling Question Answering

PromptBERT: Improving BERT Sentence Embeddings with Prompts

1 code implementation12 Jan 2022 Ting Jiang, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Liangjie Zhang, Qi Zhang

To this end, we propose a prompt based sentence embeddings method which can reduce token embeddings biases and make the original BERT layers more effective.

Denoising Semantic Similarity +2

Phrase-level Adversarial Example Generation for Neural Machine Translation

no code implementations6 Jan 2022 Juncheng Wan, Jian Yang, Shuming Ma, Dongdong Zhang, Weinan Zhang, Yong Yu, Furu Wei

In this paper, we propose a phrase-level adversarial example generation (PAEG) method to enhance the robustness of the model.

Machine Translation Translation

SMDT: Selective Memory-Augmented Neural Document Translation

no code implementations5 Jan 2022 Xu Zhang, Jian Yang, Haoyang Huang, Shuming Ma, Dongdong Zhang, Jinlong Li, Furu Wei

Existing document-level neural machine translation (NMT) models have sufficiently explored different context settings to provide guidance for target generation.

Document Level Machine Translation Document Translation +2

Distilled Dual-Encoder Model for Vision-Language Understanding

no code implementations16 Dec 2021 Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, Furu Wei

We propose a cross-modal attention distillation framework to train a dual-encoder model for vision-language understanding tasks, such as visual reasoning and visual question answering.

Question Answering Visual Entailment +2

Swin Transformer V2: Scaling Up Capacity and Resolution

6 code implementations18 Nov 2021 Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo

Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images.

Action Classification Image Classification +3

Document AI: Benchmarks, Models and Applications

no code implementations16 Nov 2021 Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei

Document AI, or Document Intelligence, is a relatively new research topic that refers to the techniques for automatically reading, understanding, and analyzing business documents.

Document AI Document Image Classification +3

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

no code implementations3 Nov 2021 Wenhui Wang, Hangbo Bao, Li Dong, Furu Wei

We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network.

Visual Question Answering Visual Reasoning +1

s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning

1 code implementation26 Oct 2021 Hangbo Bao, Li Dong, Wenhui Wang, Nan Yang, Furu Wei

Pretrained bidirectional Transformers, such as BERT, have achieved significant improvements in a wide variety of language understanding tasks, while it is not straightforward to directly apply them for natural language generation.

Abstractive Text Summarization Question Generation +1

Improving Non-autoregressive Generation with Mixup Training

1 code implementation21 Oct 2021 Ting Jiang, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Liangjie Zhang, Qi Zhang

While pre-trained language models have achieved great success on various natural language understanding tasks, how to effectively leverage them into non-autoregressive generation tasks remains a challenge.

Natural Language Understanding Paraphrase Generation +1

MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding

1 code implementation16 Oct 2021 Junlong Li, Yiheng Xu, Lei Cui, Furu Wei

Multimodal pre-training with text, layout, and image has made significant progress for Visually Rich Document Understanding (VRDU), especially the fixed-layout documents such as scanned document images.

Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation

1 code implementation16 Oct 2021 Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei

When applied to zero-shot cross-lingual abstractive summarization, it produces an average performance gain of 12. 3 ROUGE-L over mBART-ft. We conduct detailed analyses to understand the key ingredients of SixT+, including multilinguality of the auxiliary parallel data, positional disentangled encoder, and the cross-lingual transferability of its encoder.

Abstractive Text Summarization Cross-Lingual Abstractive Summarization +3

SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing

1 code implementation ACL 2022 Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei

Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning.

Automatic Speech Recognition Quantization +5

TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models

2 code implementations21 Sep 2021 Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei

Existing approaches for text recognition are usually built based on CNN for image understanding and RNN for char-level text generation.

Handwritten Text Recognition Language Modelling +2

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

1 code implementation EMNLP 2021 Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

We find that many languages are under-represented in recent cross-lingual language models due to the limited vocabulary capacity.

Language Modelling

Sequence Level Contrastive Learning for Text Summarization

no code implementations8 Sep 2021 Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei

In this paper, we propose a contrastive learning model for supervised abstractive text summarization, where we view a document, its gold summary and its model generated summaries as different views of the same mean representation and maximize the similarities between them during training.

Abstractive Text Summarization Contrastive Learning +2

Multilingual Agreement for Multilingual Neural Machine Translation

no code implementations ACL 2021 Jian Yang, Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Zhoujun Li, Furu Wei

Although multilingual neural machine translation (MNMT) enables multiple language translations, the training process is based on independent multilingual objectives.

Machine Translation Translation

SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation

no code implementations ACL 2021 Shuo Ren, Long Zhou, Shujie Liu, Furu Wei, Ming Zhou, Shuai Ma

While pre-training techniques are working very well in natural language processing, how to pre-train a decoder and effectively use it for neural machine translation (NMT) still remains a tricky issue.

Machine Translation Translation

UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset

no code implementations12 Jul 2021 Chengyi Wang, Yu Wu, Shujie Liu, Jinyu Li, Yao Qian, Kenichi Kumatani, Furu Wei

Recently, there has been a vast interest in self-supervised learning (SSL) where the model is pre-trained on large scale unlabeled data and then fine-tuned on a small labeled dataset.

Self-Supervised Learning Speech Recognition

Learning to Sample Replacements for ELECTRA Pre-Training

no code implementations Findings (ACL) 2021 Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei

Moreover, we propose to use a focal loss for the generator in order to relieve oversampling of correct tokens as replacements.

Language Modelling Masked Language Modeling

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders

1 code implementation25 Jun 2021 Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei

While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG).

Abstractive Text Summarization Machine Translation +4

Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding

1 code implementation ACL 2021 Xin Sun, Tao Ge, Furu Wei, Houfeng Wang

In this paper, we propose Shallow Aggressive Decoding (SAD) to improve the online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC).

Grammatical Error Correction

Attention Temperature Matters in Abstractive Summarization Distillation

1 code implementation ACL 2022 Shengqiang Zhang, Xingxing Zhang, Hangbo Bao, Furu Wei

In this paper, we find simply manipulating attention temperatures in Transformers can make pseudo labels easier to learn for student models.

Abstractive Text Summarization

Knowledge Neurons in Pretrained Transformers

3 code implementations ACL 2022 Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei

In this paper, we present preliminary studies on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons.

Pretrained Language Models

LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding

4 code implementations18 Apr 2021 Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei

In this paper, we present LayoutXLM, a multimodal pre-trained model for multilingual document understanding, which aims to bridge the language barriers for visually-rich document understanding.

Document Image Classification

MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

1 code implementation EMNLP 2021 Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang Xian-Ling Mao, Heyan Huang, Furu Wei

Multilingual T5 (mT5) pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks.

Abstractive Text Summarization Machine Translation +4

UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data

2 code implementations19 Jan 2021 Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang

In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both unlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive self-supervised learning are conducted in a multi-task learning manner.

Multi-Task Learning Representation Learning +2

Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting

1 code implementation EMNLP 2021 Wangchunshu Zhou, Tao Ge, Canwen Xu, Ke Xu, Furu Wei

In this paper, we generalize text infilling (e. g., masked language models) by proposing Sequence Span Rewriting (SSR) as a self-supervised sequence-to-sequence (seq2seq) pre-training objective.

Text Infilling

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

1 code implementation Findings (ACL) 2021 Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei

We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers.

LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding

4 code implementations ACL 2021 Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou

Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents.

Document Image Classification Document Layout Analysis +4

Investigating Learning Dynamics of BERT Fine-Tuning

no code implementations Asian Chapter of the Association for Computational Linguistics 2020 Yaru Hao, Li Dong, Furu Wei, Ke Xu

The recently introduced pre-trained language model BERT advances the state-of-the-art on many NLP tasks through the fine-tuning approach, but few studies investigate how the fine-tuning process improves the model performance on downstream tasks.

Language Modelling

Unsupervised Fine-tuning for Text Clustering

no code implementations COLING 2020 Shaohan Huang, Furu Wei, Lei Cui, Xingxing Zhang, Ming Zhou

Fine-tuning with pre-trained language models (e. g. BERT) has achieved great success in many language understanding tasks in supervised settings (e. g. text classification).

Text Classification Text Clustering

Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction

no code implementations EMNLP 2020 Mengyun Chen, Tao Ge, Xingxing Zhang, Furu Wei, Ming Zhou

We propose a novel language-independent approach to improve the efficiency for Grammatical Error Correction (GEC) by dividing the task into two subtasks: Erroneous Span Detection (ESD) and Erroneous Span Correction (ESC).

Grammatical Error Correction

Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph

1 code implementation EMNLP 2020 Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Xiaoyan Zhu, Minlie Huang

Despite the success of generative pre-trained language models on a series of text generation tasks, they still suffer in cases where reasoning over underlying commonsense knowledge is required during generation.

Text Generation

Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths

no code implementations Asian Chapter of the Association for Computational Linguistics 2020 Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Minlie Huang

Commonsense explanation generation aims to empower the machine's sense-making capability by generating plausible explanations to statements against commonsense.

Explanation Generation

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

2 code implementations NAACL 2021 Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, He-Yan Huang, Ming Zhou

In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.

Contrastive Learning Cross-Lingual Transfer +1

BERT Loses Patience: Fast and Robust Inference with Early Exit

1 code implementation NeurIPS 2020 Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, Furu Wei

In this paper, we propose Patience-based Early Exit, a straightforward yet effective inference method that can be used as a plug-and-play technique to simultaneously improve the efficiency and robustness of a pretrained language model (PLM).

Language Modelling

DocBank: A Benchmark Dataset for Document Layout Analysis

1 code implementation COLING 2020 Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou

DocBank is constructed using a simple yet effective way with weak supervision from the \LaTeX{} documents available on the arXiv. com.

Document Layout Analysis

Harvesting and Refining Question-Answer Pairs for Unsupervised QA

1 code implementation ACL 2020 Zhongli Li, Wenhui Wang, Li Dong, Furu Wei, Ke Xu

Our approach outperforms previous unsupervised approaches by a large margin and is competitive with early supervised models.

Few-Shot Learning Question Answering

TableBank: Table Benchmark for Image-based Table Detection and Recognition

1 code implementation LREC 2020 Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li

We present TableBank, a new image-based table detection and recognition dataset built with novel weak supervision from Word and Latex documents on the internet.

Table Detection

Scheduled DropHead: A Regularization Method for Transformer Models

1 code implementation Findings of the Association for Computational Linguistics 2020 Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou

In this paper, we introduce DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of transformer, a state-of-the-art model for various NLP tasks.

Machine Translation Text Classification +1

Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

2 code implementations23 Apr 2020 Yaru Hao, Li Dong, Furu Wei, Ke Xu

The great success of Transformer-based models benefits from the powerful multi-head self-attention mechanism, which learns token dependencies and encodes contextual information from the input.

At Which Level Should We Extract? An Empirical Analysis on Extractive Document Summarization

no code implementations COLING 2020 Qingyu Zhou, Furu Wei, Ming Zhou

In this work, we show that unnecessity and redundancy issues exist when extracting full sentences, and extracting sub-sentential units is a promising alternative.

Constituency Parsing Document Summarization +3

Learning to Summarize Passages: Mining Passage-Summary Pairs from Wikipedia Revision Histories

no code implementations6 Apr 2020 Qingyu Zhou, Furu Wei, Ming Zhou

In this paper, we propose a method for automatically constructing a passage-to-summary dataset by mining the Wikipedia page revision histories.

Pre-training for Abstractive Document Summarization by Reinstating Source Text

no code implementations EMNLP 2020 Yanyan Zou, Xingxing Zhang, Wei Lu, Furu Wei, Ming Zhou

The main idea is that, given an input text artificially constructed from a document, a model is pre-trained to reinstate the original document.

Abstractive Text Summarization Document Summarization

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

2 code implementations28 Feb 2020 Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon

We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).

Ranked #3 on Question Generation on SQuAD1.1 (using extra training data)

Abstractive Text Summarization Language Modelling +2

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

1 code implementation NeurIPS 2020 Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou

The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher).

Zero-shot Text Search

Multimodal Matching Transformer for Live Commenting

no code implementations7 Feb 2020 Chaoqun Duan, Lei Cui, Shuming Ma, Furu Wei, Conghui Zhu, Tiejun Zhao

In this work, we aim to improve the relevance between live comments and videos by modeling the cross-modal interactions among different modalities.

Text Generation

Self-Adversarial Learning with Comparative Discrimination for Text Generation

no code implementations ICLR 2020 Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou

Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples.

Text Generation

Fact-aware Sentence Split and Rephrase with Permutation Invariant Training

no code implementations16 Jan 2020 Yinuo Guo, Tao Ge, Furu Wei

To overcome the challenges, we first propose the Fact-aware Sentence Encoding, which enables the model to learn facts from the long sentence and thus improves the precision of sentence split; then we introduce Permutation Invariant Training to alleviate the effects of order variance in seq2seq learning for this task.

Split and Rephrase

LayoutLM: Pre-training of Text and Layout for Document Image Understanding

12 code implementations31 Dec 2019 Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou

In this paper, we propose the \textbf{LayoutLM} to jointly model interactions between text and layout information across scanned document images, which is beneficial for a great number of real-world document image understanding tasks such as information extraction from scanned documents.

Document AI Document Image Classification +2

Transforming Wikipedia into Augmented Data for Query-Focused Summarization

no code implementations8 Nov 2019 Haichao Zhu, Li Dong, Furu Wei, Bing Qin, Ting Liu

The manual construction of a query-focused summarization corpus is costly and timeconsuming.

Data Augmentation

Improving Grammatical Error Correction with Machine Translation Pairs

1 code implementation Findings of the Association for Computational Linguistics 2020 Wangchunshu Zhou, Tao Ge, Chang Mu, Ke Xu, Furu Wei, Ming Zhou

The poor translation model resembles the ESL (English as a second language) learner and tends to generate translations of low quality in terms of fluency and grammatical correctness, while the good translation model generally generates fluent and grammatically correct translations.

Grammatical Error Correction Language Modelling +2

Video Dialog via Progressive Inference and Cross-Transformer

no code implementations IJCNLP 2019 Weike Jin, Zhou Zhao, Mao Gu, Jun Xiao, Furu Wei, Yueting Zhuang

Video dialog is a new and challenging task, which requires the agent to answer questions combining video information with dialog history.

Answer Generation Question Answering +3

Cross-Lingual Natural Language Generation via Pre-Training

1 code implementation23 Sep 2019 Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao, He-Yan Huang

In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages.

Abstractive Text Summarization Machine Translation +4

Visualizing and Understanding the Effectiveness of BERT

no code implementations IJCNLP 2019 Yaru Hao, Li Dong, Furu Wei, Ke Xu

Language model pre-training, such as BERT, has achieved remarkable results in many NLP tasks.

Language Modelling

BERT-based Lexical Substitution

no code implementations ACL 2019 Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou

Our approach first applies dropout to the target word{'}s embedding for partially masking the word, allowing BERT to take balanced consideration of the target word{'}s semantics and contexts for proposing substitute candidates, and then validates the candidates based on their substitution{'}s influence on the global contextualized representation of the sentence.

Learning to Ask Unanswerable Questions for Machine Reading Comprehension

no code implementations ACL 2019 Haichao Zhu, Li Dong, Furu Wei, Wenhui Wang, Bing Qin, Ting Liu

We also present a way to construct training data for our question generation models by leveraging the existing reading comprehension dataset.

Data Augmentation Machine Reading Comprehension +1

HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

no code implementations ACL 2019 Xingxing Zhang, Furu Wei, Ming Zhou

Neural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods.

Document Summarization Extractive Summarization +1

Unified Language Model Pre-training for Natural Language Understanding and Generation

8 code implementations NeurIPS 2019 Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon

This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks.

Ranked #2 on Generative Question Answering on CoQA (using extra training data)

Abstractive Text Summarization Document Summarization +6

TableBank: A Benchmark Dataset for Table Detection and Recognition

2 code implementations LREC 2020 Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li

We present TableBank, a new image-based table detection and recognition dataset built with novel weak supervision from Word and Latex documents on the internet.

Table Detection

Text Morphing

no code implementations30 Sep 2018 Shaohan Huang, Yu Wu, Furu Wei, Ming Zhou

In this paper, we introduce a novel natural language generation task, termed as text morphing, which targets at generating the intermediate sentences that are fluency and smooth with the two input sentences.

Text Generation

Unsupervised Machine Commenting with Neural Variational Topic Model

no code implementations13 Sep 2018 Shuming Ma, Lei Cui, Furu Wei, Xu sun

To fully exploit the unpaired data, we completely remove the need for parallel data and propose a novel unsupervised approach to train an automatic article commenting model, relying on nothing but unpaired articles and comments.

Neural Melody Composition from Lyrics

no code implementations12 Sep 2018 Hangbo Bao, Shaohan Huang, Furu Wei, Lei Cui, Yu Wu, Chuanqi Tan, Songhao Piao, Ming Zhou

In this paper, we study a novel task that learns to compose music from natural language.

Retrieval-Enhanced Adversarial Training for Neural Response Generation

no code implementations ACL 2019 Qingfu Zhu, Lei Cui, Wei-Nan Zhang, Furu Wei, Ting Liu

Dialogue systems are usually built on either generation-based or retrieval-based approaches, yet they do not benefit from the advantages of different models.

Response Generation

Attention-Guided Answer Distillation for Machine Reading Comprehension

no code implementations EMNLP 2018 Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, Ming Zhou

Despite that current reading comprehension systems have achieved significant advancements, their promising performances are often obtained at the cost of making an ensemble of numerous models.

Knowledge Distillation Machine Reading Comprehension

Neural Latent Extractive Document Summarization

no code implementations EMNLP 2018 Xingxing Zhang, Mirella Lapata, Furu Wei, Ming Zhou

Extractive summarization models require sentence-level labels, which are usually created heuristically (e. g., with rule-based methods) given that most summarization datasets only have document-summary pairs.

Document Summarization Extractive Document Summarization +2

Sequential Copying Networks

1 code implementation6 Jul 2018 Qingyu Zhou, Nan Yang, Furu Wei, Ming Zhou

Copying mechanism shows effectiveness in sequence-to-sequence based neural network models for text generation tasks, such as abstractive sentence summarization and question generation.

Question Generation Sentence Summarization +1

Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study

1 code implementation3 Jul 2018 Tao Ge, Furu Wei, Ming Zhou

Neural sequence-to-sequence (seq2seq) approaches have proven to be successful in grammatical error correction (GEC).

Grammatical Error Correction

Multiway Attention Networks for Modeling Sentence Pairs

1 code implementation IJCAI 2018 Chuanqi Tan, Furu Wei, Wenhui Wang, Weifeng Lv, Ming Zhou

Modeling sentence pairs plays the vital role for judging the relationship between two sentences, such as paraphrase identification, natural language inference, and answer sentence selection.

Natural Language Inference Paraphrase Identification

Fluency Boost Learning and Inference for Neural Grammatical Error Correction

no code implementations ACL 2018 Tao Ge, Furu Wei, Ming Zhou

Most of the neural sequence-to-sequence (seq2seq) models for grammatical error correction (GEC) have two limitations: (1) a seq2seq model may not be well generalized with only limited error-corrected data; (2) a seq2seq model may fail to completely correct a sentence with multiple errors through normal seq2seq inference.

Grammatical Error Correction

Dictionary-Guided Editing Networks for Paraphrase Generation

no code implementations21 Jun 2018 Shaohan Huang, Yu Wu, Furu Wei, Ming Zhou

An intuitive way for a human to write paraphrase sentences is to replace words or phrases in the original sentence with their corresponding synonyms and make necessary changes to ensure the new sentences are fluent and grammatically correct.

Paraphrase Generation

Response Generation by Context-aware Prototype Editing

3 code implementations19 Jun 2018 Yu Wu, Furu Wei, Shaohan Huang, Yunli Wang, Zhoujun Li, Ming Zhou

Open domain response generation has achieved remarkable progress in recent years, but sometimes yields short and uninformative responses.

Informativeness Response Generation

Neural Open Information Extraction

no code implementations ACL 2018 Lei Cui, Furu Wei, Ming Zhou

Conventional Open Information Extraction (Open IE) systems are usually built on hand-crafted patterns from other NLP tools such as syntactic parsing, yet they face problems of error propagation.

Open Information Extraction

Improv Chat: Second Response Generation for Chatbot

no code implementations10 May 2018 Furu Wei

Existing research on response generation for chatbot focuses on \textbf{First Response Generation} which aims to teach the chatbot to say the first response (e. g. a sentence) appropriate to the conversation context (e. g. the user's query).

Chatbot Response Generation

Faithful to the Original: Fact Aware Neural Abstractive Summarization

no code implementations13 Nov 2017 Ziqiang Cao, Furu Wei, Wenjie Li, Sujian Li

While previous abstractive summarization approaches usually focus on the improvement of informativeness, we argue that faithfulness is also a vital prerequisite for a practical abstractive summarization system.

Abstractive Text Summarization Extractive Summarization +3

S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension

no code implementations15 Jun 2017 Chuanqi Tan, Furu Wei, Nan Yang, Bowen Du, Weifeng Lv, Ming Zhou

We build the answer extraction model with state-of-the-art neural networks for single passage reading comprehension, and propose an additional task of passage ranking to help answer extraction in multiple passages.

Answer Generation Machine Reading Comprehension +1

Reinforced Mnemonic Reader for Machine Reading Comprehension

3 code implementations8 May 2017 Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, Furu Wei, Ming Zhou

In this paper, we introduce the Reinforced Mnemonic Reader for machine reading comprehension tasks, which enhances previous attentive readers in two aspects.

Machine Reading Comprehension Question Answering +1

Selective Encoding for Abstractive Sentence Summarization

2 code implementations ACL 2017 Qingyu Zhou, Nan Yang, Furu Wei, Ming Zhou

We propose a selective encoding model to extend the sequence-to-sequence framework for abstractive sentence summarization.

Sentence Summarization

Entity Linking for Queries by Searching Wikipedia Sentences

no code implementations EMNLP 2017 Chuanqi Tan, Furu Wei, Pengjie Ren, Weifeng Lv, Ming Zhou

The key idea is to search sentences similar to a query from Wikipedia articles and directly use the human-annotated entities in the similar sentences as candidate entities for the query.

Entity Linking Word Embeddings

Neural Question Generation from Text: A Preliminary Study

4 code implementations6 Apr 2017 Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, Ming Zhou

Automatic question generation aims to generate questions from a text passage where the generated questions can be answered by certain sub-spans of the given passage.

Question Generation

Learning to Generate Product Reviews from Attributes

no code implementations EACL 2017 Li Dong, Shaohan Huang, Furu Wei, Mirella Lapata, Ming Zhou, Ke Xu

This paper presents an attention-enhanced attribute-to-sequence model to generate product reviews for given attribute information, such as user, product, and rating.

Review Generation Sentiment Analysis +1

A Redundancy-Aware Sentence Regression Framework for Extractive Summarization

no code implementations COLING 2016 Pengjie Ren, Furu Wei, Zhumin Chen, Jun Ma, Ming Zhou

Existing sentence regression methods for extractive summarization usually model sentence importance and redundancy in two separate processes.

Document Summarization Extractive Summarization +1

Improving Multi-Document Summarization via Text Classification

no code implementations28 Nov 2016 Ziqiang Cao, Wenjie Li, Sujian Li, Furu Wei

Developed so far, multi-document summarization has reached its bottleneck due to the lack of sufficient training data and diverse categories of documents.

Classification Document Summarization +3

Unsupervised Word and Dependency Path Embeddings for Aspect Term Extraction

no code implementations25 May 2016 Yichun Yin, Furu Wei, Li Dong, Kaimeng Xu, Ming Zhang, Ming Zhou

In this paper, we develop a novel approach to aspect term extraction based on unsupervised learning of distributed representations of words and dependency paths.

Term Extraction

AttSum: Joint Learning of Focusing and Summarization with Neural Attention

no code implementations COLING 2016 Ziqiang Cao, Wenjie Li, Sujian Li, Furu Wei, Yan-ran Li

Query relevance ranking and sentence saliency ranking are the two main tasks in extractive query-focused summarization.

Multi-Document Summarization via Discriminative Summary Reranking

no code implementations8 Jul 2015 Xiaojun Wan, Ziqiang Cao, Furu Wei, Sujian Li, Ming Zhou

However, according to our quantitative analysis, none of the existing summarization models can always produce high-quality summaries for different document sets, and even a summarization model with good overall performance may produce low-quality summaries for some document sets.

Document Summarization Multi-Document Summarization