ACL 2019

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

ACL 2019 huggingface/pytorch-transformers

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling.

LANGUAGE MODELLING

Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings

ACL 2019 facebookresearch/LASER

Machine translation is highly sensitive to the size and quality of the training data, which has led to an increasing interest in collecting and filtering large parallel corpora.

CROSS-LINGUAL BITEXT MINING MACHINE TRANSLATION PARALLEL CORPUS MINING SENTENCE EMBEDDINGS

Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation

ACL 2019 asyml/texar

The versatile toolkit also fosters technique sharing across different text generation tasks.

MACHINE TRANSLATION TEXT GENERATION

Multi-Task Deep Neural Networks for Natural Language Understanding

ACL 2019 namisan/mt-dnn

In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks.

DOMAIN ADAPTATION LANGUAGE MODELLING LINGUISTIC ACCEPTABILITY NATURAL LANGUAGE INFERENCE PARAPHRASE IDENTIFICATION SENTIMENT ANALYSIS

A Multiscale Visualization of Attention in the Transformer Model

ACL 2019 jessevig/bertviz

The Transformer is a sequence model that forgoes traditional recurrent architectures in favor of a fully attention-based approach.

ERNIE: Enhanced Language Representation with Informative Entities

ACL 2019 thunlp/ERNIE

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks.

ENTITY TYPING KNOWLEDGE GRAPHS NATURAL LANGUAGE INFERENCE SENTIMENT ANALYSIS

BERT Rediscovers the Classical NLP Pipeline

ACL 2019 jsalt18-sentence-repl/jiant

Pre-trained text encoders have rapidly advanced the state of the art on many NLP tasks.

NAMED ENTITY RECOGNITION

Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling

ACL 2019 jsalt18-sentence-repl/jiant

Natural language understanding has recently seen a surge of progress with the use of sentence encoders like ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2019) which are pretrained on variants of language modeling.

LANGUAGE MODELLING TRANSFER LEARNING

Adaptive Attention Span in Transformers

ACL 2019 facebookresearch/adaptive-span

We propose a novel self-attention mechanism that can learn its optimal attention span.

LANGUAGE MODELLING

Multilingual Constituency Parsing with Self-Attention and Pre-Training

ACL 2019 nikitakit/self-attentive-parser

We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions.

CONSTITUENCY PARSING