Search Results for author:
Found 230 papers, 77 papers with code
Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Analytical Reasoning of Text
In this paper, we study the challenge of analytical reasoning of text and collect a new dataset consisting of questions from the Law School Admission Test from 1991 to 2016.
XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation
In this paper, we introduce XGLUE, a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora, and evaluate their performance across a diverse set of cross-lingual tasks.
AI-Driven Reinvention of Hydrological Modeling for Accurate Predictions and Interpretation to Transform Earth System Modeling
Traditional equation-driven hydrological models often struggle to accurately predict streamflow in challenging regional Earth systems like the Tibetan Plateau, while hybrid and existing algorithm-driven models face difficulties in interpreting hydrological behaviors.
CLSP: High-Fidelity Contrastive Language-State Pre-training for Agent State Representation
Then, we deploy contrastive learning to train the CLSP encoder to effectively represent precise state information.
Task-level Distributionally Robust Optimization for Large Language Model-based Dense Retrieval
In this paper, we propose a new task-level Distributionally Robust Optimization (tDRO) algorithm for LLM-DR fine-tuning, targeted at improving the universal domain generalization ability by end-to-end reweighting the data distribution of each task.
Category-Aware Dynamic Label Assignment with High-Quality Oriented Proposal
To address this problem, an OBB representation based on the complex plane is introduced in the oriented detection framework, and a trigonometric loss function is proposed.
Ranked #1 on
Object Detection In Aerial Images
on HRSC2016
(using extra training data)
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale
In recent years, Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence.
TCAN: Text-oriented Cross Attention Network for Multimodal Sentiment Analysis
Motivated by these insights, we introduce a Text-oriented Cross-Attention Network (TCAN), emphasizing the predominant role of the text modality in MSA.
Building Open-Ended Embodied Agent via Language-Policy Bidirectional Adaptation
Building embodied agents on integrating Large Language Models (LLMs) and Reinforcement Learning (RL) have revolutionized human-AI interaction: researchers can now leverage language instructions to plan decision-making for open-ended tasks.
LLMaAA: Making Large Language Models as Active Annotators
Recently, the superior few-shot performance of large language models (LLMs) has propelled the development of dataset generation, where the training data are solely synthesized from LLMs.
Two is Better Than One: Answering Complex Questions by Multiple Knowledge Sources with Generalized Links
Incorporating multiple knowledge sources is proven to be beneficial for answering complex factoid questions.
On Realization of Intelligent Decision-Making in the Real World: A Foundation Decision Model Perspective
The pervasive uncertainty and dynamic nature of real-world environments present significant challenges for the widespread implementation of machine-driven Intelligent Decision-Making (IDM) systems.
Instance Regularization for Discriminative Language Model Pre-training
They treat training instances equally throughout the training process, with little attention on the individual contribution of those instances.
HORIZON: High-Resolution Semantically Controlled Panorama Synthesis
Panorama synthesis endeavors to craft captivating 360-degree visual landscapes, immersing users in the heart of virtual worlds.
Pre-Training a Graph Recurrent Network for Language Representation
Transformer-based pre-trained models have gained much advance in recent years, becoming one of the most important backbones in natural language processing.
Improving Task Generalization via Unified Schema Prompt
Task generalization has been a long standing challenge in Natural Language Processing (NLP).
ProQA: Structural Prompt-based Pre-training for Unified Question Answering
Furthermore, ProQA exhibits strong ability in both continual learning and transfer learning by taking the advantages of the structural prompt.
Adversarial Fine-tune with Dynamically Regulated Adversary
Adversarial training is an effective method to boost model robustness to malicious, adversarial attacks.
UniXcoder: Unified Cross-Modal Pre-training for Code Representation
Furthermore, we propose to utilize multi-modal contents to learn representation of code fragment with contrastive learning, and then align representations among programming languages using a cross-modal generation task.
A density peaks clustering algorithm with sparse search and K-d tree
It is proved that the algorithm can effectively reduce the computational complexity of the original DPC from $O(n^2K)$ to $O(n(n^{1-1/K}+k))$.
Efficient Policy Space Response Oracles
Policy Space Response Oracle methods (PSRO) provide a general solution to learn Nash equilibrium in two-player zero-sum games but suffer from two drawbacks: (1) the computation inefficiency due to the need for consistent meta-game evaluation via simulations, and (2) the exploration inefficiency due to finding the best response against a fixed meta-strategy at every epoch.
Generative Adversarial Exploration for Reinforcement Learning
Exploration is crucial for training the optimal reinforcement learning (RL) policy, where the key is to discriminate whether a state visiting is novel.
Reasoning over Hybrid Chain for Table-and-Text Open Domain QA
CARP utilizes hybrid chain to model the explicit intermediate reasoning process across table and text for question answering.
Ranked #2 on
Question Answering
on OTT-QA
A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models
In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text.
Applications of Generative Adversarial Networks in Anomaly Detection: A Systematic Literature Review
The goal of this review paper is to analyze and summarize: (1) which anomaly detection techniques can benefit from certain types of GANs, and how, (2) in which application domains GAN-assisted anomaly detection techniques have been applied, and (3) which datasets and performance metrics have been used to evaluate these techniques.
Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese
Although pre-trained models (PLMs) have achieved remarkable improvements in a wide range of NLP tasks, they are expensive in terms of time and resources.
Jointly Learning to Repair Code and Generate Commit Message
We propose a novel task of jointly repairing program codes and generating commit messages.
Discovering Representation Sprachbund For Multilingual Pre-Training
Then we cluster all the target languages into multiple groups and name each group as a representation sprachbund.
From LSAT: The Progress and Challenges of Complex Reasoning
Complex reasoning aims to draw a correct inference based on complex rules.
Control Image Captioning Spatially and Temporally
Moreover, the controllability and explainability of LoopCAG are validated by analyzing spatial and temporal sensitivity during the generation process.
Ranked #1 on
Image Captioning
on Localized Narratives
Compare to The Knowledge: Graph Neural Fake News Detection with External Knowledge
Specifically, we first construct a \textit{directed heterogeneous document graph} for each news incorporating topics and entities.
SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation
While pre-training techniques are working very well in natural language processing, how to pre-train a decoder and effectively use it for neural machine translation (NMT) still remains a tricky issue.
Learning to Ask Conversational Questions by Optimizing Levenshtein Distance
Conversational Question Simplification (CQS) aims to simplify self-contained questions into conversational ones by incorporating some conversational characteristics, e. g., anaphora and ellipsis.
MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning
Our framework is comprised of three key components: (1) a centralized task dispatching model, which supports the self-generated tasks and scalable training with heterogeneous policy combinations; (2) a programming architecture named Actor-Evaluator-Learner, which achieves high parallelism for both training and sampling, and meets the evaluation requirement of auto-curriculum learning; (3) a higher-level abstraction of MARL training paradigms, which enables efficient code reuse and flexible deployments on different distributed computing paradigms.
Smart-Start Decoding for Neural Machine Translation
Most current neural machine translation models adopt a monotonic decoding order of either left-to-right or right-to-left.
CoSQA: 20,000+ Web Queries for Code Search and Question Answering
Finding codes given natural language query isb eneficial to the productivity of software developers.
Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text
Logical reasoning of text requires understanding critical logical information in the text and performing inference over them.
Ranked #7 on
Reading Comprehension
on ReClor
Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts
We specify the dynamics sample complexity and the opponent sample complexity in MARL, and conduct a theoretic analysis of return discrepancy upper bound.
Multi-agent Reinforcement Learning
Reinforcement Learning (RL)
AR-LSAT: Investigating Analytical Reasoning of Text
Analytical reasoning is an essential and challenging task that requires a system to analyze a scenario involving a set of particular circumstances and perform reasoning over it to make conclusions.
BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation
Standard automatic metrics, e. g. BLEU, are not reliable for document-level MT evaluation.
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Benchmark datasets have a significant impact on accelerating research in programming language tasks.
Ranked #1 on
Cloze Test
on CodeXGLUE - CT-maxmin
Regioned Episodic Reinforcement Learning
Goal-oriented reinforcement learning algorithms are often good at exploration, not exploitation, while episodic algorithms excel at exploitation, not exploration.
BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation.
Unsupervised Fine-tuning for Text Clustering
Fine-tuning with pre-trained language models (e. g. BERT) has achieved great success in many language understanding tasks in supervised settings (e. g. text classification).
GLGE: A New General Language Generation Evaluation Benchmark
Multi-task benchmarks such as GLUE and SuperGLUE have driven great progress of pretraining and transfer learning in Natural Language Processing (NLP).
ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training
This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism.
Machine Reasoning: Technology, Dilemma and Future
Machine reasoning research aims to build interpretable AI systems that can solve problems or draw conclusions from what they are told (i. e. facts and observations) and already know (i. e. models, common sense and knowledge) under certain constraints.
ProphetNet-Ads: A Looking Ahead Strategy for Generative Retrieval Models in Sponsored Search Engine
We build a dataset from a real-word sponsored search engine and carry out experiments to analyze different generative retrieval models.
SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
We open-source the SMARTS platform and the associated benchmark tasks and evaluation metrics to encourage and empower research on multi-agent learning for autonomous driving.
Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers
We also find in experiments that our model is less dependent on sentence positions.
Neural Deepfake Detection with Factual Structure of Text
To address this, we propose a graph-based model that utilizes the factual structure of a document for deepfake detection of text.
Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction
We propose a novel language-independent approach to improve the efficiency for Grammatical Error Correction (GEC) by dividing the task into two subtasks: Erroneous Span Detection (ESD) and Erroneous Span Correction (ESC).
Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space
In this paper, we propose a novel data augmentation method, referred to as Controllable Rewriting based Question Data Augmentation (CRQDA), for machine reading comprehension (MRC), question generation, and question-answering natural language inference tasks.
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
Evaluation metrics play a vital role in the growth of an area as it defines the standard of distinguishing between good and bad models.
GraphCodeBERT: Pre-training Code Representations with Data Flow
Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables.
Ranked #3 on
Type prediction
on ManyTypes4TypeScript
Continuous Speech Separation with Conformer
Continuous speech separation plays a vital role in complicated speech related tasks such as conversation transcription.
Ranked #1 on
Speech Separation
on LibriCSS
(using extra training data)
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.
Ranked #16 on
Zero-Shot Cross-Lingual Transfer
on XTREME
A Simple and Effective Unified Encoder for Document-Level Machine Translation
Most of the existing models for document-level machine translation adopt dual-encoder structures.
A Graph-based Coarse-to-fine Method for Unsupervised Bilingual Lexicon Induction
To deal with those issues, in this paper, we propose a novel graph-based paradigm to induce bilingual lexicons in a coarse-to-fine way.
Bilingual Lexicon Induction
Cross-Lingual Word Embeddings
+2
MIND: A Large-scale Dataset for News Recommendation
News recommendation is an important technique for personalized news service.
A Retrieve-and-Rewrite Initialization Method for Unsupervised Machine Translation
The commonly used framework for unsupervised machine translation builds initial translation models of both translation directions, and then performs iterative back-translation to jointly boost their translation performance.
Improving Neural Machine Translation with Soft Template Prediction
Although neural machine translation (NMT) has achieved significant progress in recent years, most previous NMT models only depend on the source text to generate translation.
Graph Neural News Recommendation with Unsupervised Preference Disentanglement
Furthermore, the learned representations are disentangled with latent preference factors by a neighborhood routing algorithm, which can enhance expressiveness and interpretability.
Evidence-Aware Inferential Text Generation with Vector Quantised Variational AutoEncoder
Generating inferential texts about an event in different perspectives requires reasoning over different contexts that the event occurs.
Ranked #1 on
Common Sense Reasoning
on Event2Mind test
(BLEU metric)
DocBank: A Benchmark Dataset for Document Layout Analysis
DocBank is constructed using a simple yet effective way with weak supervision from the \LaTeX{} documents available on the arXiv. com.
Statistical Issues and Recommendations for Clinical Trials Conducted During the COVID-19 Pandemic
The COVID-19 pandemic has had and continues to have major impacts on planned and ongoing clinical trials.
Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension
Natural Questions is a new challenging machine reading comprehension benchmark with two-grained answers, which are a long answer (typically a paragraph) and a short answer (one or more entities inside the long answer).
TableBank: Table Benchmark for Image-based Table Detection and Recognition
We present TableBank, a new image-based table detection and recognition dataset built with novel weak supervision from Word and Latex documents on the internet.
Leveraging Declarative Knowledge in Text and First-Order Logic for Fine-Grained Propaganda Detection
We study the detection of propagandistic text fragments in news articles.
Scheduled DropHead: A Regularization Method for Transformer Models
In this paper, we introduce DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of transformer, a state-of-the-art model for various NLP tasks.
LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Network
The graph is used to obtain graph-enhanced contextual representations of words in Transformer-based architecture.
A Heterogeneous Graph with Factual, Temporal and Logical Knowledge for Question Answering Over Dynamic Contexts
We study question answering over a dynamic textual environment.
Curriculum Pre-training for End-to-End Speech Translation
End-to-end speech translation poses a heavy burden on the encoder, because it has to transcribe, understand, and learn cross-lingual semantics simultaneously.
Pre-training Text Representations as Meta Learning
In this work, we introduce a learning algorithm which directly optimizes model's ability to learn text representations for effective learning of downstream tasks.
MuTual: A Dataset for Multi-Turn Dialogue Reasoning
Non-task oriented dialogue systems have achieved great success in recent years due to largely accessible conversation data and the development of deep learning techniques.
Inferential Text Generation with Multiple Knowledge Sources and Meta-Learning
In this work, we use multiple knowledge sources as fuels for the model.
Learning to Summarize Passages: Mining Passage-Summary Pairs from Wikipedia Revision Histories
In this paper, we propose a method for automatically constructing a passage-to-summary dataset by mining the Wikipedia page revision histories.
At Which Level Should We Extract? An Empirical Analysis on Extractive Document Summarization
In this work, we show that unnecessity and redundancy issues exist when extracting full sentences, and extracting sub-sentential units is a promising alternative.
Pre-training for Abstractive Document Summarization by Reinstating Source Text
The main idea is that, given an input text artificially constructed from a document, a model is pre-trained to reinstate the original document.
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks.
Low Latency End-to-End Streaming Speech Recognition with a Scout Network
The attention-based Transformer model has achieved promising results for speech recognition (SR) in the offline mode.
Audio and Speech Processing
XGPT: Cross-modal Generative Pre-Training for Image Captioning
While many BERT-based cross-modal pre-trained models produce excellent results on downstream understanding tasks like image-text retrieval and VQA, they cannot be applied to generation tasks directly.
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Ranked #4 on
Question Generation
on SQuAD1.1
(using extra training data)
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher).
Ranked #2 on
Sentence Retrieval
on PeerQA
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation tasks.
Ranked #1 on
Code Documentation Generation
on CodeSearchNet - Go
UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
However, most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks.
Ranked #2 on
Action Segmentation
on COIN
(using extra training data)
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
Our approach first divides the original BERT into several modules and builds their compact substitutes.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa.
Ranked #1 on
Entity Typing
on Open Entity
Self-Adversarial Learning with Comparative Discrimination for Text Generation
Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples.
ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism.
Ranked #6 on
Question Generation
on SQuAD1.1
(using extra training data)
Multi-Agent Interactions Modeling with Correlated Policies
In this paper, we cast the multi-agent interactions modeling problem into a multi-agent imitation learning framework with explicit modeling of correlated policies by approximating opponents' policies, which can recover agents' policies that can regenerate similar interactions.
LayoutLM: Pre-training of Text and Layout for Document Image Understanding
In this paper, we propose the \textbf{LayoutLM} to jointly model interactions between text and layout information across scanned document images, which is beneficial for a great number of real-world document image understanding tasks such as information extraction from scanned documents.
Ranked #9 on
Relation Extraction
on FUNSD
Semantic Mask for Transformer based End-to-End Speech Recognition
Attention-based encoder-decoder model has achieved impressive results for both automatic speech recognition (ASR) and text-to-speech (TTS) tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Improving Grammatical Error Correction with Machine Translation Pairs
The poor translation model resembles the ESL (English as a second language) learner and tends to generate translations of low quality in terms of fluency and grammatical correctness, while the good translation model generally generates fluent and grammatically correct translations.
Inspecting Unification of Encoding and Matching with Transformer: A Case Study of Machine Reading Comprehension
Most machine reading comprehension (MRC) models separately handle encoding and matching with different network architectures.
Asking Clarification Questions in Knowledge-Based Question Answering
We provide representative baselines for these tasks and further introduce a coarse-to-fine model for clarification question generation.
Multi-Agent Reinforcement Learning for Order-dispatching via Order-Vehicle Distribution Matching
Improving the efficiency of dispatching orders to vehicles is a research hotspot in online ride-hailing systems.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
End-to-end speech translation, a hot topic in recent years, aims to translate a segment of audio into a specific language with an end-to-end model.
Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting
We study sequence-to-sequence (seq2seq) pre-training with data augmentation for sentence rewriting.
Signal Instructed Coordination in Cooperative Multi-agent Reinforcement Learning
Although existing works formulate this problem into a centralized learning with decentralized execution framework, which avoids the non-stationary problem in training, their decentralized execution paradigm limits the agents' capability to coordinate.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Reasoning Over Semantic-Level Graph for Fact Checking
We evaluate our system on FEVER, a benchmark dataset for fact checking, and find that rich structural information is helpful and both our graph-based mechanisms improve the accuracy.
Ranked #2 on
Fact Verification
on FEVER
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
On XNLI, 1. 8% averaged accuracy improvement (on 15 languages) is obtained.
Cross-Lingual Natural Language Inference
Cross-Lingual Question Answering
+2
Explicit Cross-lingual Pre-training for Unsupervised Machine Translation
Pre-training has proven to be effective in unsupervised machine translation due to its ability to model deep context information in cross-lingual scenarios.
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner.
Ranked #5 on
Image-to-Text Retrieval
on MS COCO
(Recall@10 metric)
Pathologist-Level Grading of Prostate Biopsies with Artificial Intelligence
We additionally evaluated grading performance on 87 biopsies individually graded by 23 experienced urological pathologists from the International Society of Urological Pathology.
BERT-based Lexical Substitution
Our approach first applies dropout to the target word{'}s embedding for partially masking the word, allowing BERT to take balanced consideration of the target word{'}s semantics and contexts for proposing substitute candidates, and then validates the candidates based on their substitution{'}s influence on the global contextualized representation of the sentence.
Automatic Grammatical Error Correction for Sequence-to-sequence Text Generation: An Empirical Study
Sequence-to-sequence (seq2seq) models have achieved tremendous success in text generation tasks.
Dense Procedure Captioning in Narrated Instructional Videos
Understanding narrated instructional videos is important for both research and real-world web applications.
A Tensorized Transformer for Language Modeling
In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD).
Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing
In this paper, we present an approach to incorporate retrieved datapoints as supporting evidence for context-dependent semantic parsing, such as generating source code conditioned on the class environment.
CoRide: Joint Order Dispatching and Fleet Management for Multi-Scale Ride-Hailing Platforms
How to optimally dispatch orders to vehicles and how to trade off between immediate and future returns are fundamental questions for a typical ride-hailing platform.
Multiagent Systems
HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
Neural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods.
Ranked #7 on
Extractive Text Summarization
on CNN / Daily Mail
Unified Language Model Pre-training for Natural Language Understanding and Generation
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks.
Ranked #2 on
Generative Question Answering
on CoQA
(using extra training data)
TableBank: A Benchmark Dataset for Table Detection and Recognition
We present TableBank, a new image-based table detection and recognition dataset built with novel weak supervision from Word and Latex documents on the internet.
Unsupervised Neural Machine Translation with SMT as Posterior Regularization
To address this issue, we introduce phrase based Statistic Machine Translation (SMT) models which are robust to noisy data, as posterior regularizations to guide the training of unsupervised NMT models in the iterative back-translation process.
Dialog-to-Action: Conversational Question Answering Over a Large-Scale Knowledge Base
We present an approach to map utterances in conversation to logical forms, which will be executed on a large-scale knowledge base.
Bidirectional Generative Adversarial Networks for Neural Machine Translation
To address this issue and stabilize the GAN training, in this paper, we propose a novel Bidirectional Generative Adversarial Network for Neural Machine Translation (BGAN-NMT), which aims to introduce a generator model to act as the discriminator, whereby the discriminator naturally considers the entire translation space so that the inadequate training problem can be alleviated.
Fine-grained Coordinated Cross-lingual Text Stream Alignment for Endless Language Knowledge Acquisition
This paper proposes to study fine-grained coordinated cross-lingual text stream alignment through a novel information network decipherment paradigm.
Text Morphing
In this paper, we introduce a novel natural language generation task, termed as text morphing, which targets at generating the intermediate sentences that are fluency and smooth with the two input sentences.
Neural Speech Synthesis with Transformer Network
Although end-to-end neural text-to-speech (TTS) methods (such as Tacotron2) are proposed and achieve state-of-the-art performance, they still suffer from two problems: 1) low efficiency during training and inference; 2) hard to model long dependency using current recurrent neural networks (RNNs).
Ranked #9 on
Text-To-Speech Synthesis
on LJSpeech
(using extra training data)
Neural Melody Composition from Lyrics
In this paper, we study a novel task that learns to compose music from natural language.
Factorized Q-Learning for Large-Scale Multi-Agent Systems
Deep Q-learning has achieved a significant success in single-agent decision making tasks.
Multiagent Systems
Improving Question Answering by Commonsense-Based Pre-Training
Although neural network approaches achieve remarkable success on a variety of NLP tasks, many of them struggle to answer questions that require commonsense knowledge.
Approximate Distribution Matching for Sequence-to-Sequence Learning
Then, we interpret sequence-to-sequence learning as learning a transductive model to transform the source local latent distributions to match their corresponding target distributions.
Style Transfer as Unsupervised Machine Translation
Language style transferring rephrases text with specific stylistic attributes while preserving the original attribute-independent content.
Ranked #3 on
Unsupervised Text Style Transfer
on GYAFC
Attention-Guided Answer Distillation for Machine Reading Comprehension
Despite that current reading comprehension systems have achieved significant advancements, their promising performances are often obtained at the cost of making an ensemble of numerous models.
Neural Latent Extractive Document Summarization
Extractive summarization models require sentence-level labels, which are usually created heuristically (e. g., with rule-based methods) given that most summarization datasets only have document-summary pairs.
Ranked #11 on
Extractive Text Summarization
on CNN / Daily Mail
Question Generation from SQL Queries Improves Neural Semantic Parsing
We study how to learn a semantic parser of state-of-the-art accuracy with less supervised training data.
Regularizing Neural Machine Translation by Target-bidirectional Agreement
Although Neural Machine Translation (NMT) has achieved remarkable progress in the past several years, most NMT systems still suffer from a fundamental shortcoming as in other sequence generation tasks: errors made early in generation process are fed as inputs to the model and can be quickly amplified, harming subsequent sequence generation.
Sequential Copying Networks
Copying mechanism shows effectiveness in sequence-to-sequence based neural network models for text generation tasks, such as abstractive sentence summarization and question generation.
Neural Document Summarization by Jointly Learning to Score and Select Sentences
In this paper, we present a novel end-to-end neural network framework for extractive document summarization by jointly learning to score and select sentences.
Ranked #9 on
Extractive Text Summarization
on CNN / Daily Mail
Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study
Neural sequence-to-sequence (seq2seq) approaches have proven to be successful in grammatical error correction (GEC).
Ranked #1 on
Grammatical Error Correction
on Unrestricted
Fluency Boost Learning and Inference for Neural Grammatical Error Correction
Most of the neural sequence-to-sequence (seq2seq) models for grammatical error correction (GEC) have two limitations: (1) a seq2seq model may not be well generalized with only limited error-corrected data; (2) a seq2seq model may fail to completely correct a sentence with multiple errors through normal seq2seq inference.
Multiway Attention Networks for Modeling Sentence Pairs
Modeling sentence pairs plays the vital role for judging the relationship between two sentences, such as paraphrase identification, natural language inference, and answer sentence selection.
Ranked #11 on
Paraphrase Identification
on Quora Question Pairs
(Accuracy metric)
Dictionary-Guided Editing Networks for Paraphrase Generation
An intuitive way for a human to write paraphrase sentences is to replace words or phrases in the original sentence with their corresponding synonyms and make necessary changes to ensure the new sentences are fluent and grammatically correct.
Learning to Collaborate for Question Answering and Asking
Secondly, directly applying GAN that regards all the generated questions as negative instances could not improve the accuracy of the QA model.
Generative Bridging Network for Neural Sequence Prediction
In order to alleviate data sparsity and overfitting problems in maximum likelihood estimation (MLE) for sequence prediction tasks, we propose the Generative Bridging Network (GBN), in which a novel bridge module is introduced to assist the training of the sequence prediction model (the generator network).
Table-to-Text: Describing Table Region with Natural Language
The model maps a row from a table to a continuous vector and then generates a natural language sentence by leveraging the semantics of a table.
Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
Normalization techniques such as Batch Normalization have been applied successfully for training deep neural networks.
R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering
To better utilize semantic knowledge in images, we propose a novel framework to learn visual relation facts for VQA.
Triangular Architecture for Rare Language Translation
Neural Machine Translation (NMT) performs poor on the low-resource language pair $(X, Z)$, especially when $Z$ is a rare language.
Neural Open Information Extraction
Conventional Open Information Extraction (Open IE) systems are usually built on hand-crafted patterns from other NLP tools such as syntactic parsing, yet they face problems of error propagation.
Learning Matching Models with Weak Supervision for Response Selection in Retrieval-based Chatbots
We propose a method that can leverage unlabeled data to learn a matching model for response selection in retrieval-based chatbots.
Semantic Parsing with Syntax- and Table-Aware SQL Generation
We present a generative model to map natural language questions into SQL queries.
Ranked #4 on
Code Generation
on WikiSQL
Achieving Human Parity on Automatic Chinese to English News Translation
Machine translation has made rapid advances in recent years.
Ranked #3 on
Machine Translation
on WMT 2017 English-Chinese
Joint Training for Neural Machine Translation Models with Monolingual Data
Monolingual data have been demonstrated to be helpful in improving translation quality of both statistical machine translation (SMT) systems and neural machine translation (NMT) systems, especially in resource-poor or domain adaptation tasks where parallel data are not rich enough.
Mean Field Multi-Agent Reinforcement Learning
Existing multi-agent reinforcement learning methods are limited typically to a small number of agents.
Assertion-based QA with Question-Aware Open Information Extraction
We present assertion based question answering (ABQA), an open domain question answering task that takes a question and a passage as inputs, and outputs a semi-structured assertion consisting of a subject, a predicate and a list of arguments.
Neural Response Generation with Dynamic Vocabularies
We study response generation for open domain conversation in chatbots.
A Sequential Matching Framework for Multi-turn Response Selection in Retrieval-based Chatbots
The task requires matching a response candidate with a conversation context, whose challenges include how to recognize important parts of the context, and how to model the relationships among utterances in the context.
Stack-based Multi-layer Attention for Transition-based Dependency Parsing
Although sequence-to-sequence (seq2seq) network has achieved significant success in many NLP tasks such as machine translation and text summarization, simply applying this approach to transition-based dependency parsing cannot yield a comparable performance gain as in other state-of-the-art methods, such as stack-LSTM and head selection.
Question Generation for Question Answering
This paper presents how to generate questions from given passages using neural networks, where large scale QA pairs are automatically crawled and processed from Community-QA website, and used as training data.
Beihang-MSRA at SemEval-2017 Task 3: A Ranking System with Neural Matching Features for Community Question Answering
This paper presents the system in SemEval-2017 Task 3, Community Question Answering (CQA).
Gated Self-Matching Networks for Reading Comprehension and Question Answering
We first match the question and passage with gated attention-based recurrent networks to obtain the question-aware passage representation.
Ranked #35 on
Question Answering
on SQuAD1.1 dev
Chunk-based Decoder for Neural Machine Translation
The chunk-level decoder models global dependencies while the word-level decoder decides the local word order in a chunk.
Sequence-to-Dependency Neural Machine Translation
Nowadays a typical Neural Machine Translation (NMT) model generates translations from left to right as a linear sequence, during which latent syntactic structures of the target sentences are not explicitly concerned.
Generative Bridging Network in Neural Sequence Prediction
In order to alleviate data sparsity and overfitting problems in maximum likelihood estimation (MLE) for sequence prediction tasks, we propose the Generative Bridging Network (GBN), in which a novel bridge module is introduced to assist the training of the sequence prediction model (the generator network).
S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
We build the answer extraction model with state-of-the-art neural networks for single passage reading comprehension, and propose an additional task of passage ranking to help answer extraction in multiple passages.
Content-Based Table Retrieval for Web Queries
Understanding the connections between unstructured text and semi-structured table is an important yet neglected problem in natural language processing.
Question Answering and Question Generation as Dual Tasks
On one side, the QA model judges whether the generated question of a QG model is relevant to the answer.
Reinforced Mnemonic Reader for Machine Reading Comprehension
In this paper, we introduce the Reinforced Mnemonic Reader for machine reading comprehension tasks, which enhances previous attentive readers in two aspects.
Ranked #17 on
Question Answering
on SQuAD1.1 dev
Selective Encoding for Abstractive Sentence Summarization
We propose a selective encoding model to extend the sequence-to-sequence framework for abstractive sentence summarization.
Ranked #8 on
Text Summarization
on DUC 2004 Task 1
Entity Linking for Queries by Searching Wikipedia Sentences
The key idea is to search sentences similar to a query from Wikipedia articles and directly use the human-annotated entities in the similar sentences as candidate entities for the query.
Neural Question Generation from Text: A Preliminary Study
Automatic question generation aims to generate questions from a text passage where the generated questions can be answered by certain sub-spans of the given passage.
Ranked #13 on
Question Generation
on SQuAD1.1
Hierarchical Recurrent Attention Network for Response Generation
With the word level attention, hidden vectors of a word level encoder are synthesized as utterance vectors and fed to an utterance level encoder to construct hidden representations of the context.
Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots
Existing work either concatenates utterances in context or matches a response with a highly abstract context vector finally, which may lose relationships among utterances or important contextual information.
Ranked #7 on
Conversational Response Selection
on RRS
A Redundancy-Aware Sentence Regression Framework for Extractive Summarization
Existing sentence regression methods for extractive summarization usually model sentence importance and redundancy in two separate processes.
Event Detection with Burst Information Networks
Retrospective event detection is an important task for discovering previously unidentified events in a text stream.
Constraint-Based Question Answering with Knowledge Graph
WebQuestions and SimpleQuestions are two benchmark data-sets commonly used in recent knowledge-based question answering (KBQA) work.
Improving Attention Modeling with Implicit Distortion and Fertility for Machine Translation
In neural machine translation, the attention mechanism facilitates the translation process by producing a soft alignment between the source sentence and the target sentence.
Knowledge Enhanced Hybrid Neural Network for Text Matching
Long text brings a big challenge to semantic matching due to their complicated semantic and syntactic structures.
Detecting Context Dependent Messages in a Conversational Environment
While automatic response generation for building chatbot systems has drawn a lot of attention recently, there is limited understanding on when we need to consider the linguistic context of an input text in the generation process.
Aligning Coordinated Text Streams through Burst Information Network Construction and Decipherment
We introduce a novel Burst Information Network (BINet) representation that can display the most important information and illustrate the connections among bursty entities, events and keywords in the corpus.
Topic Aware Neural Response Generation
We consider incorporating topic information into the sequence-to-sequence framework to generate informative and interesting responses for chatbots.
Unsupervised Word and Dependency Path Embeddings for Aspect Term Extraction
In this paper, we develop a novel approach to aspect term extraction based on unsupervised learning of distributed representations of words and dependency paths.
Response Selection with Topic Clues for Retrieval-based Chatbots
The message vector, the response vector, and the two topic vectors are fed to neural tensors to calculate a matching score.
Jointly Modeling Topics and Intents with Global Order Structure
Modeling document structure is of great importance for discourse analysis and related applications.
TGSum: Build Tweet Guided Multi-Document Summarization Dataset
Both informativeness and readability of the collected summaries are verified by manual judgment.
A Dependency-Based Neural Network for Relation Classification
Previous research on relation classification has verified the effectiveness of using dependency shortest paths or subtrees.
Ranked #5 on
Relation Classification
on SemEval 2010 Task 8
Multi-Document Summarization via Discriminative Summary Reranking
However, according to our quantitative analysis, none of the existing summarization models can always produce high-quality summaries for different document sets, and even a summarization model with good overall performance may produce low-quality summaries for some document sets.
Beyond Word-based Language Model in Statistical Machine Translation
Language model is one of the most important modules in statistical machine translation and currently the word-based language model dominants this community.
