Search Results for author:
Found 169 papers, 73 papers with code
Pseudo-Label Guided Unsupervised Domain Adaptation of Contextual Embeddings
In unsupervised domain adaptation, we aim to train a model that works well on a target domain when provided with labeled source samples and unlabeled target samples.
MarkupLM: Pre-training of Text and Markup Language for Visually Rich Document Understanding
Multimodal pre-training with text, layout, and image has made significant progress for Visually Rich Document Understanding (VRDU), especially the fixed-layout documents such as scanned document images.
Towards Making the Most of Cross-Lingual Transfer for Zero-Shot Neural Machine Translation
When applied to zero-shot cross-lingual abstractive summarization, it produces an average performance gain of 12. 3 ROUGE-L over mBART-ft. We conduct detailed analyses to understand the key ingredients of SixT+, including multilinguality of the auxiliary parallel data, positional disentangled encoder, and the cross-lingual transferability of its encoder.
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+3
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
As more and more pre-trained language models adopt on-cloud deployment, the privacy issues grow quickly, mainly for the exposure of plain-text user data (e. g., search history, medical record, bank account).
XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding
Multimodal pre-training with text, layout, and image has achieved SOTA performance for visually rich document understanding tasks recently, which demonstrates the great potential for joint learning across different modalities.
Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Visually-Augmented Language Modeling
With the visually-augmented context, VaLM uses a visual knowledge fusion layer to enable multimodal grounded language modeling by attending on both text context and visual knowledge in images.
Prototypical Calibration for Few-shot Learning of Language Models
In-context learning of GPT-like models has been recognized as fragile across different hand-crafted templates, and demonstration permutations.
Lossless Acceleration for Seq2seq Generation with Aggressive Decoding
We study lossless acceleration for seq2seq generation with a novel decoding algorithm -- Aggressive Decoding.
Abstractive Text Summarization
Grammatical Error Correction
+3
Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization
In zero-shot multilingual extractive text summarization, a model is typically trained on English summarization dataset and then applied on summarization datasets of other languages.
Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?
Recently, self-supervised learning (SSL) has demonstrated strong performance in speaker recognition, even if the pre-training objective is designed for speech recognition.
On the Representation Collapse of Sparse Mixture of Experts
We also present a comprehensive analysis on the representation and routing behaviors of our models.
StableMoE: Stable Routing Strategy for Mixture of Experts
We point out that existing learning-to-route MoE methods suffer from the routing fluctuation issue, i. e., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference.
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
In this paper, we propose LayoutLMv3 to pre-train multimodal Transformers for Document AI with unified text and image masking.
Ranked #1 on
Key information extraction
on CORD
Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data
In this way, the decoder learns to reconstruct original speech information with codes before learning to generate correct text.
Lossless Speedup of Autoregressive Translation with Generalized Aggressive Decoding
Different from previous work accelerating translation at the cost of quality loss, we propose Generalized Aggressive Decoding (GAD) -- a novel decoding paradigm for lossless speedup of autoregressive translation, through the collaboration of autoregressive and non-autoregressive translation (NAT) of the Transformer.
CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment
We first evaluate CLIP's zero-shot performance on a typical visual question answering task and demonstrate a zero-shot cross-modality transfer capability of CLIP on the visual entailment task.
DiT: Self-supervised Pre-training for Document Image Transformer
Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.)
Ranked #2 on
Document Layout Analysis
on PubLayNet val
DeepNet: Scaling Transformers to 1,000 Layers
In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers.
Controllable Natural Language Generation with Contrastive Prefixes
We propose a novel supervised method and also an unsupervised method to train the prefixes for single-aspect control while the combination of these two methods can achieve multi-aspect control.
Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt
To collocate with the unified prompt, we propose a new initialization method for the target label word to further improve the model's transferability across languages.
A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models
With the increasing of model capacity brought by pre-trained language models, there emerges boosting needs for more knowledgeable natural language processing (NLP) models with advanced functionalities including providing and making flexible use of encyclopedic and commonsense knowledge.
EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation
We propose EdgeFormer -- a parameter-efficient Transformer of the encoder-decoder architecture for on-device seq2seq generation, which is customized under strict computation and memory constraints.
Corrupted Image Modeling for Self-Supervised Visual Pre-Training
CIM is a general and flexible visual pre-training framework that is suitable for various network architectures.
A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model
Synthetic data construction of Grammatical Error Correction (GEC) for non-English languages relies heavily on human-designed and language-specific rules, which produce limited error-corrected patterns.
Kformer: Knowledge Injection in Transformer Feed-Forward Layers
Knowledge-Enhanced Model have developed a diverse set of techniques for knowledge integration on different knowledge sources.
PromptBERT: Improving BERT Sentence Embeddings with Prompts
To this end, we propose a prompt based sentence embeddings method which can reduce token embeddings biases and make the original BERT layers more effective.
Phrase-level Adversarial Example Generation for Neural Machine Translation
In this paper, we propose a phrase-level adversarial example generation (PAEG) method to enhance the robustness of the model.
SMDT: Selective Memory-Augmented Neural Document Translation
Existing document-level neural machine translation (NMT) models have sufficiently explored different context settings to provide guidance for target generation.
Distilled Dual-Encoder Model for Vision-Language Understanding
We propose a cross-modal attention distillation framework to train a dual-encoder model for vision-language understanding tasks, such as visual reasoning and visual question answering.
Swin Transformer V2: Scaling Up Capacity and Resolution
Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images.
Ranked #1 on
Instance Segmentation
on COCO minival
Document AI: Benchmarks, Models and Applications
Document AI, or Document Intelligence, is a relatively new research topic that refers to the techniques for automatically reading, understanding, and analyzing business documents.
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network.
Ranked #1 on
Visual Question Answering
on VQA v2 test-dev
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation.
Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Multi-talker conversational speech processing has drawn many interests for various applications such as meeting transcription.
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks.
s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Pretrained bidirectional Transformers, such as BERT, have achieved significant improvements in a wide variety of language understanding tasks, while it is not straightforward to directly apply them for natural language generation.
Improving Non-autoregressive Generation with Mixup Training
While pre-trained language models have achieved great success on various natural language understanding tasks, how to effectively leverage them into non-autoregressive generation tasks remains a challenge.
MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
Multimodal pre-training with text, layout, and image has made significant progress for Visually Rich Document Understanding (VRDU), especially the fixed-layout documents such as scanned document images.
Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
When applied to zero-shot cross-lingual abstractive summarization, it produces an average performance gain of 12. 3 ROUGE-L over mBART-ft. We conduct detailed analyses to understand the key ingredients of SixT+, including multilinguality of the auxiliary parallel data, positional disentangled encoder, and the cross-lingual transferability of its encoder.
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+3
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning.
UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
We integrate the proposed methods into the HuBERT framework.
Jointly Learning to Repair Code and Generate Commit Message
We propose a novel task of jointly repairing program codes and generating commit messages.
TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
Existing approaches for text recognition are usually built based on CNN for image understanding and RNN for char-level text generation.
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
We find that many languages are under-represented in recent cross-lingual language models due to the limited vocabulary capacity.
Sequence Level Contrastive Learning for Text Summarization
In this paper, we propose a contrastive learning model for supervised abstractive text summarization, where we view a document, its gold summary and its model generated summaries as different views of the same mean representation and maximize the similarities between them during training.
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Recent studies on compression of pretrained language models (e. g., BERT) usually use preserved accuracy as the metric for evaluation.
LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Reading order detection is the cornerstone to understanding visually-rich documents (e. g., receipts and forms).
Multilingual Agreement for Multilingual Neural Machine Translation
Although multilingual neural machine translation (MNMT) enables multiple language translations, the training process is based on independent multilingual objectives.
xMoCo: Cross Momentum Contrastive Learning for Open-Domain Question Answering
Dense passage retrieval has been shown to be an effective approach for information retrieval tasks such as open domain question answering.
SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation
While pre-training techniques are working very well in natural language processing, how to pre-train a decoder and effectively use it for neural machine translation (NMT) still remains a tricky issue.
UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
Recently, there has been a vast interest in self-supervised learning (SSL) where the model is pre-trained on large scale unlabeled data and then fine-tuned on a small labeled dataset.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training.
Ranked #1 on
Zero-Shot Cross-Lingual Transfer
on XTREME
Learning to Sample Replacements for ELECTRA Pre-Training
Moreover, we propose to use a focal loss for the generator in order to relieve oversampling of correct tokens as replacements.
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG).
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
In this paper, we present a general approach to developing small, fast and effective pre-trained models for specific domains.
Consistency Regularization for Cross-Lingual Fine-Tuning
Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others.
BEiT: BERT Pre-Training of Image Transformers
We first "tokenize" the original image into visual tokens.
Ranked #3 on
Document Layout Analysis
on PubLayNet val
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
The cross-lingual language models are typically pretrained with masked language modeling on multilingual text or parallel sentences.
VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
Video transcript summarization is a fundamental task for video understanding.
Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
In this paper, we propose Shallow Aggressive Decoding (SAD) to improve the online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC).
Attention Temperature Matters in Abstractive Summarization Distillation
In this paper, we find simply manipulating attention temperatures in Transformers can make pseudo labels easier to learn for student models.
Memory-Efficient Differentiable Transformer Architecture Search
To this end, we propose a multi-split reversible network and combine it with DARTS.
Knowledge Neurons in Pretrained Transformers
In this paper, we present preliminary studies on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons.
LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
In this paper, we present LayoutXLM, a multimodal pre-trained model for multilingual document understanding, which aims to bridge the language barriers for visually-rich document understanding.
Ranked #10 on
Document Image Classification
on RVL-CDIP
MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Multilingual T5 (mT5) pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks.
Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
In this paper, we focus on a zero-shot cross-lingual transfer task in NMT.
Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
Cant is important for understanding advertising, comedies and dog-whistle politics.
UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both unlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive self-supervised learning are conducted in a multi-task learning manner.
Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
In this paper, we generalize text infilling (e. g., masked language models) by proposing Sequence Span Rewriting (SSR) as a self-supervised sequence-to-sequence (seq2seq) pre-training objective.
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Multilingual machine translation enables a single model to translate between different languages.
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers.
LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents.
Ranked #1 on
Key information extraction
on SROIE
UnihanLM: Coarse-to-Fine Chinese-Japanese Language Model Pretraining with the Unihan Database
Chinese and Japanese share many characters with similar surface morphology.
Investigating Learning Dynamics of BERT Fine-Tuning
The recently introduced pre-trained language model BERT advances the state-of-the-art on many NLP tasks through the fine-tuning approach, but few studies investigate how the fine-tuning process improves the model performance on downstream tasks.
Unsupervised Fine-tuning for Text Clustering
Fine-tuning with pre-trained language models (e. g. BERT) has achieved great success in many language understanding tasks in supervised settings (e. g. text classification).
Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers
We also find in experiments that our model is less dependent on sentence positions.
Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction
We propose a novel language-independent approach to improve the efficiency for Grammatical Error Correction (GEC) by dividing the task into two subtasks: Erroneous Span Detection (ESD) and Erroneous Span Correction (ESC).
Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph
Despite the success of generative pre-trained language models on a series of text generation tasks, they still suffer in cases where reasoning over underlying commonsense knowledge is required during generation.
Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths
Commonsense explanation generation aims to empower the machine's sense-making capability by generating plausible explanations to statements against commonsense.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.
Ranked #12 on
Zero-Shot Cross-Lingual Transfer
on XTREME
BERT Loses Patience: Fast and Robust Inference with Early Exit
In this paper, we propose Patience-based Early Exit, a straightforward yet effective inference method that can be used as a plug-and-play technique to simultaneously improve the efficiency and robustness of a pretrained language model (PLM).
DocBank: A Benchmark Dataset for Document Layout Analysis
DocBank is constructed using a simple yet effective way with weak supervision from the \LaTeX{} documents available on the arXiv. com.
Harvesting and Refining Question-Answer Pairs for Unsupervised QA
Our approach outperforms previous unsupervised approaches by a large margin and is competitive with early supervised models.
Ranked #185 on
Question Answering
on SQuAD1.1
TableBank: Table Benchmark for Image-based Table Detection and Recognition
We present TableBank, a new image-based table detection and recognition dataset built with novel weak supervision from Word and Latex documents on the internet.
Scheduled DropHead: A Regularization Method for Transformer Models
In this paper, we introduce DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of transformer, a state-of-the-art model for various NLP tasks.
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
The great success of Transformer-based models benefits from the powerful multi-head self-attention mechanism, which learns token dependencies and encodes contextual information from the input.
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks.
Ranked #1 on
Text-Image Retrieval
on COCO (image as query)
At Which Level Should We Extract? An Empirical Analysis on Extractive Document Summarization
In this work, we show that unnecessity and redundancy issues exist when extracting full sentences, and extracting sub-sentential units is a promising alternative.
Learning to Summarize Passages: Mining Passage-Summary Pairs from Wikipedia Revision Histories
In this paper, we propose a method for automatically constructing a passage-to-summary dataset by mining the Wikipedia page revision histories.
Pre-training for Abstractive Document Summarization by Reinstating Source Text
The main idea is that, given an input text artificially constructed from a document, a model is pre-trained to reinstate the original document.
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Ranked #3 on
Question Generation
on SQuAD1.1
(using extra training data)
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher).
Ranked #5 on
Zero-shot Text Search
on BEIR
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
Our approach first divides the original BERT into several modules and builds their compact substitutes.
Multimodal Matching Transformer for Live Commenting
In this work, we aim to improve the relevance between live comments and videos by modeling the cross-modal interactions among different modalities.
Self-Adversarial Learning with Comparative Discrimination for Text Generation
Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples.
Fact-aware Sentence Split and Rephrase with Permutation Invariant Training
To overcome the challenges, we first propose the Fact-aware Sentence Encoding, which enables the model to learn facts from the long sentence and thus improves the precision of sentence split; then we introduce Permutation Invariant Training to alleviate the effects of order variance in seq2seq learning for this task.
LayoutLM: Pre-training of Text and Layout for Document Image Understanding
In this paper, we propose the \textbf{LayoutLM} to jointly model interactions between text and layout information across scanned document images, which is beneficial for a great number of real-world document image understanding tasks such as information extraction from scanned documents.
Ranked #11 on
Document Image Classification
on RVL-CDIP
Can Monolingual Pretrained Models Help Cross-Lingual Classification?
Multilingual pretrained language models (such as multilingual BERT) have achieved impressive results for cross-lingual transfer.
Transforming Wikipedia into Augmented Data for Query-Focused Summarization
The manual construction of a query-focused summarization corpus is costly and timeconsuming.
Improving Grammatical Error Correction with Machine Translation Pairs
The poor translation model resembles the ESL (English as a second language) learner and tends to generate translations of low quality in terms of fluency and grammatical correctness, while the good translation model generally generates fluent and grammatically correct translations.
Video Dialog via Progressive Inference and Cross-Transformer
Video dialog is a new and challenging task, which requires the agent to answer questions combining video information with dialog history.
Inspecting Unification of Encoding and Matching with Transformer: A Case Study of Machine Reading Comprehension
Most machine reading comprehension (MRC) models separately handle encoding and matching with different network architectures.
Cross-Lingual Natural Language Generation via Pre-Training
In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages.
Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting
We study sequence-to-sequence (seq2seq) pre-training with data augmentation for sentence rewriting.
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short).
Ranked #1 on
Visual Question Answering
on VCR (Q-A) dev
Visualizing and Understanding the Effectiveness of BERT
Language model pre-training, such as BERT, has achieved remarkable results in many NLP tasks.
BERT-based Lexical Substitution
Our approach first applies dropout to the target word{'}s embedding for partially masking the word, allowing BERT to take balanced consideration of the target word{'}s semantics and contexts for proposing substitute candidates, and then validates the candidates based on their substitution{'}s influence on the global contextualized representation of the sentence.
Automatic Grammatical Error Correction for Sequence-to-sequence Text Generation: An Empirical Study
Sequence-to-sequence (seq2seq) models have achieved tremendous success in text generation tasks.
Learning to Ask Unanswerable Questions for Machine Reading Comprehension
We also present a way to construct training data for our question generation models by leveraging the existing reading comprehension dataset.
HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
Neural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods.
Ranked #7 on
Extractive Text Summarization
on CNN / Daily Mail
Unified Language Model Pre-training for Natural Language Understanding and Generation
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks.
Ranked #2 on
Generative Question Answering
on CoQA
(using extra training data)
Formality Style Transfer with Hybrid Textual Annotations
Its challenge is the lack of large-scale sentence-aligned parallel data.
TableBank: A Benchmark Dataset for Table Detection and Recognition
We present TableBank, a new image-based table detection and recognition dataset built with novel weak supervision from Word and Latex documents on the internet.
Fine-grained Coordinated Cross-lingual Text Stream Alignment for Endless Language Knowledge Acquisition
This paper proposes to study fine-grained coordinated cross-lingual text stream alignment through a novel information network decipherment paradigm.
Text Morphing
In this paper, we introduce a novel natural language generation task, termed as text morphing, which targets at generating the intermediate sentences that are fluency and smooth with the two input sentences.
Unsupervised Machine Commenting with Neural Variational Topic Model
To fully exploit the unpaired data, we completely remove the need for parallel data and propose a novel unsupervised approach to train an automatic article commenting model, relying on nothing but unpaired articles and comments.
LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
We introduce the task of automatic live commenting.
Neural Melody Composition from Lyrics
In this paper, we study a novel task that learns to compose music from natural language.
Retrieval-Enhanced Adversarial Training for Neural Response Generation
Dialogue systems are usually built on either generation-based or retrieval-based approaches, yet they do not benefit from the advantages of different models.
Attention-Guided Answer Distillation for Machine Reading Comprehension
Despite that current reading comprehension systems have achieved significant advancements, their promising performances are often obtained at the cost of making an ensemble of numerous models.
Neural Latent Extractive Document Summarization
Extractive summarization models require sentence-level labels, which are usually created heuristically (e. g., with rule-based methods) given that most summarization datasets only have document-summary pairs.
Ranked #10 on
Extractive Text Summarization
on CNN / Daily Mail
Read + Verify: Machine Reading Comprehension with Unanswerable Questions
Machine reading comprehension with unanswerable questions aims to abstain from answering when no answer can be inferred.
Ranked #12 on
Question Answering
on SQuAD2.0 dev
Sequential Copying Networks
Copying mechanism shows effectiveness in sequence-to-sequence based neural network models for text generation tasks, such as abstractive sentence summarization and question generation.
Neural Document Summarization by Jointly Learning to Score and Select Sentences
In this paper, we present a novel end-to-end neural network framework for extractive document summarization by jointly learning to score and select sentences.
Ranked #8 on
Extractive Text Summarization
on CNN / Daily Mail
Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study
Neural sequence-to-sequence (seq2seq) approaches have proven to be successful in grammatical error correction (GEC).
Ranked #1 on
Grammatical Error Correction
on Unrestricted
Multiway Attention Networks for Modeling Sentence Pairs
Modeling sentence pairs plays the vital role for judging the relationship between two sentences, such as paraphrase identification, natural language inference, and answer sentence selection.
Ranked #11 on
Paraphrase Identification
on Quora Question Pairs
(Accuracy metric)
Fluency Boost Learning and Inference for Neural Grammatical Error Correction
Most of the neural sequence-to-sequence (seq2seq) models for grammatical error correction (GEC) have two limitations: (1) a seq2seq model may not be well generalized with only limited error-corrected data; (2) a seq2seq model may fail to completely correct a sentence with multiple errors through normal seq2seq inference.
Retrieve, Rerank and Rewrite: Soft Template Based Neural Summarization
Most previous seq2seq summarization systems purely depend on the source text to generate summaries, which tends to work unstably.
Ranked #19 on
Text Summarization
on GigaWord
Dictionary-Guided Editing Networks for Paraphrase Generation
An intuitive way for a human to write paraphrase sentences is to replace words or phrases in the original sentence with their corresponding synonyms and make necessary changes to ensure the new sentences are fluent and grammatically correct.
Response Generation by Context-aware Prototype Editing
Open domain response generation has achieved remarkable progress in recent years, but sometimes yields short and uninformative responses.
Neural Open Information Extraction
Conventional Open Information Extraction (Open IE) systems are usually built on hand-crafted patterns from other NLP tools such as syntactic parsing, yet they face problems of error propagation.
Improv Chat: Second Response Generation for Chatbot
Existing research on response generation for chatbot focuses on \textbf{First Response Generation} which aims to teach the chatbot to say the first response (e. g. a sentence) appropriate to the conversation context (e. g. the user's query).
Faithful to the Original: Fact Aware Neural Abstractive Summarization
While previous abstractive summarization approaches usually focus on the improvement of informativeness, we argue that faithfulness is also a vital prerequisite for a practical abstractive summarization system.
Ranked #18 on
Text Summarization
on GigaWord
Gated Self-Matching Networks for Reading Comprehension and Question Answering
We first match the question and passage with gated attention-based recurrent networks to obtain the question-aware passage representation.
Ranked #36 on
Question Answering
on SQuAD1.1 dev
S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
We build the answer extraction model with state-of-the-art neural networks for single passage reading comprehension, and propose an additional task of passage ranking to help answer extraction in multiple passages.
Reinforced Mnemonic Reader for Machine Reading Comprehension
In this paper, we introduce the Reinforced Mnemonic Reader for machine reading comprehension tasks, which enhances previous attentive readers in two aspects.
Ranked #17 on
Question Answering
on TriviaQA
Selective Encoding for Abstractive Sentence Summarization
We propose a selective encoding model to extend the sequence-to-sequence framework for abstractive sentence summarization.
Ranked #8 on
Text Summarization
on DUC 2004 Task 1
Entity Linking for Queries by Searching Wikipedia Sentences
The key idea is to search sentences similar to a query from Wikipedia articles and directly use the human-annotated entities in the similar sentences as candidate entities for the query.
Neural Question Generation from Text: A Preliminary Study
Automatic question generation aims to generate questions from a text passage where the generated questions can be answered by certain sub-spans of the given passage.
Ranked #10 on
Question Generation
on SQuAD1.1
Learning to Generate Product Reviews from Attributes
This paper presents an attention-enhanced attribute-to-sequence model to generate product reviews for given attribute information, such as user, product, and rating.
A Redundancy-Aware Sentence Regression Framework for Extractive Summarization
Existing sentence regression methods for extractive summarization usually model sentence importance and redundancy in two separate processes.
Improving Multi-Document Summarization via Text Classification
Developed so far, multi-document summarization has reached its bottleneck due to the lack of sufficient training data and diverse categories of documents.
Unsupervised Word and Dependency Path Embeddings for Aspect Term Extraction
In this paper, we develop a novel approach to aspect term extraction based on unsupervised learning of distributed representations of words and dependency paths.
AttSum: Joint Learning of Focusing and Summarization with Neural Attention
Query relevance ranking and sentence saliency ranking are the two main tasks in extractive query-focused summarization.
TGSum: Build Tweet Guided Multi-Document Summarization Dataset
Both informativeness and readability of the collected summaries are verified by manual judgment.
A Dependency-Based Neural Network for Relation Classification
Previous research on relation classification has verified the effectiveness of using dependency shortest paths or subtrees.
Ranked #5 on
Relation Classification
on SemEval 2010 Task 8
Multi-Document Summarization via Discriminative Summary Reranking
However, according to our quantitative analysis, none of the existing summarization models can always produce high-quality summaries for different document sets, and even a summarization model with good overall performance may produce low-quality summaries for some document sets.