Search Results for author:
Found 174 papers, 50 papers with code
The DeepMind Chinese–English Document Translation System at WMT2020
This paper describes the DeepMind submission to the Chinese\rightarrowEnglish constrained data track of the WMT2020 Shared Task on News Translation.
Continuous diffusion for categorical data
Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement.
MAD for Robust Reinforcement Learning in Machine Translation
We introduce a new distributed policy gradient algorithm and show that it outperforms existing reward-aware training procedures such as REINFORCE, minimum risk training (MRT) and proximal policy optimization (PPO) in terms of training stability and generalization performance when optimizing machine translation models.
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale
We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics.
Enabling arbitrary translation objectives with Adaptive Tree Search
When applied to autoregressive models, our algorithm has different biases than beam search has, which enables a new analysis of the role of decoding bias in autoregressive models.
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.
Ranked #1 on
College Mathematics
on BIG-bench
(using extra training data)
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering
We model retrieval decisions as latent variables over sets of relevant documents.
Diverse Pretrained Context Encodings Improve Document Translation
We propose a new architecture for adapting a sentence-level sequence-to-sequence transformer by incorporating multiple pretrained document context signals and assess the impact on translation performance of (1) different pretraining approaches for generating these signals, (2) the quantity of parallel data for which document context is available, and (3) conditioning on source, target, or source and target contexts.
Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings
While recent work has shown that scores from models trained by the ubiquitous masked language modeling (MLM) objective effectively discriminate probable from improbable sequences, it is still an open question if these MLMs specify a principled probability distribution over the space of possible sequences.
Game-theoretic Vocabulary Selection via the Shapley Value and Banzhaf Index
We propose a vocabulary selection method that views words as members of a team trying to maximize the model{'}s performance.
Syntactic Structure Distillation Pretraining For Bidirectional Encoders
Textual representation learners trained on large amounts of data have achieved notable success on downstream tasks; intriguingly, they have also performed well on challenging tests of syntactic competence.
Learning to Segment Actions from Observation and Narration
We apply a generative segmental model of task structure, guided by narration, to action segmentation in video.
A Probabilistic Generative Model for Typographical Analysis of Early Modern Printing
We show that our approach outperforms rigid interpretable clustering baselines (Ocular) and overly-flexible deep generative models (VAE) alike on the task of completely unsupervised discovery of typefaces in mixed-font documents.
Learning Robust and Multilingual Speech Representations
Unsupervised speech representation learning has shown remarkable success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance.
Transition-Based Dependency Parsing using Perceptron Learner
Syntactic parsing using dependency structures has become a standard technique in natural language processing with many different parsing models, in particular data-driven models that can be trained on syntactically annotated corpora.
Comparing Top-Down and Bottom-Up Neural Generative Dependency Models
Recurrent neural network grammars generate sentences using phrase-structure syntax and perform very well on both parsing and language modeling.
Text Genre and Training Data Size in Human-like Parsing
Domain-specific training typically makes NLP systems work better.
Better Document-Level Machine Translation with Bayes' Rule
We show that Bayes' rule provides an effective mechanism for creating document translation models that can be learned from only parallel sentences and monolingual documents---a compelling benefit as parallel documents are not always available.
Relative Pixel Prediction For Autoregressive Image Generation
In natural images, transitions between adjacent pixels tend to be smooth and gradual, a fact that has long been exploited in image compression models based on predictive coding.
Unsupervised Learning of Efficient and Robust Speech Representations
We present an unsupervised method for learning speech representations based on a bidirectional contrastive predictive coding that implicitly discovers phonetic structure from large-scale corpora of unlabelled raw audio signals.
Putting Machine Translation in Context with the Noisy Channel Model
We show that Bayes' rule provides a compelling mechanism for controlling unconditional document language models, using the long-standing challenge of effectively leveraging document context in machine translation.
A Critical Analysis of Biased Parsers in Unsupervised Parsing
A series of recent papers has used a parsing algorithm due to Shen et al. (2018) to recover phrase-structure trees based on proxies for "syntactic depth."
Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation
Neural networks are part of many contemporary NLP systems, yet their empirical successes come at the price of vulnerability to adversarial attacks.
Shallow Syntax in Deep Water
Shallow syntax provides an approximation of phrase-syntactic structure of sentences; it can be produced with high accuracy, and is computationally cheap to obtain.
Compound Probabilistic Context-Free Grammars for Grammar Induction
We study a formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context-free grammar.
Ranked #6 on
Constituency Grammar Induction
on PTB
Scalable Syntax-Aware Language Models Using Knowledge Distillation
Prior work has shown that, on small amounts of training data, syntactic neural language models learn structurally sensitive generalisations more successfully than sequential language models.
An Empirical Investigation of Global and Local Normalization for Recurrent Neural Sequence Models Using a Continuous Relaxation to Beam Search
Globally normalized neural sequence models are considered superior to their locally normalized equivalents because they may ameliorate the effects of label bias.
Unsupervised Recurrent Neural Network Grammars
On language modeling, unsupervised RNNGs perform as well their supervised counterparts on benchmarks in English and Chinese.
Ranked #6 on
Constituency Grammar Induction
on PTB
(Max F1 (WSJ) metric)
Learning and Evaluating General Linguistic Intelligence
We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly.
Sentence Encoding with Tree-constrained Relation Networks
The meaning of a sentence is a function of the relations that hold between its words.
Learning to Discover, Ground and Use Words with Segmental Neural Language Models
We propose a segmental neural language model that combines the generalization power of neural networks with the ability to discover word-like units that are latent in unsegmented character sequences.
Unsupervised Word Discovery with Segmental Neural Language Models
We propose a segmental neural language model that combines the representational power of neural networks and the structure learning mechanism of Bayesian nonparametrics, and show that it learns to discover semantically meaningful units (e. g., morphemes and words) from unsegmented character sequences.
Syntactic Scaffolds for Semantic Structures
We introduce the syntactic scaffold, an approach to incorporating syntactic information into semantic tasks.
Neural Arithmetic Logic Units
Neural networks can learn to represent and manipulate numerical information, but they seldom generalize well outside of the range of numerical values encountered during training.
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better
Language exhibits hierarchical structure, but recent work using a subject-verb agreement diagnostic argued that state-of-the-art language models, LSTMs, fail to learn long-range syntax sensitive dependencies.
Finding Syntax in Human Encephalography with Beam Search
Model comparisons attribute the early peak to syntactic composition within the RNNG.
Relational inductive biases, deep learning, and graph networks
As a companion to this paper, we have released an open-source software library for building graph networks, with demonstrations of how to use them in practice.
Using Morphological Knowledge in Open-Vocabulary Neural Language Models
Languages with productive morphology pose problems for language models that generate words from a fixed vocabulary.
Unsupervised Text Style Transfer using Language Models as Discriminators
Binary classifiers are often employed as discriminators in GAN-based unsupervised style transfer systems to ensure that transferred sentences are similar to sentences in the target domain.
Pushing the bounds of dropout
We show that dropout training is best understood as performing MAP estimation concurrently for a family of conditional models whose objectives are themselves lower bounded by the original dropout objective.
Ranked #23 on
Language Modelling
on Penn Treebank (Word Level)
Fast Parametric Learning with Activation Memorization
Neural networks trained with backpropagation often struggle to identify classes that have been observed a small number of times.
Ranked #49 on
Language Modelling
on WikiText-103
Learning Deep Generative Models of Graphs
Graphs are fundamental data structures which concisely capture the relational structure in many important real-world domains, such as knowledge graphs, physical and social interactions, language, and chemistry.
Paraphrase-Supervised Models of Compositionality
Compositional vector space models of meaning promise new solutions to stubborn language understanding problems.
Memory Architectures in Recurrent Neural Network Language Models
We compare and analyze sequential, random access, and stack memory architectures for recurrent neural network language models.
The NarrativeQA Reading Comprehension Challenge
Reading comprehension (RC)---in contrast to information retrieval---requires integrating information and reasoning about events, entities, and their relations across a full document.
Ranked #9 on
Question Answering
on NarrativeQA
(BLEU-1 metric)
Should Neural Network Architecture Reflect Linguistic Structure?
On the generation front, I introduce recurrent neural network grammars (RNNGs), a joint, generative model of phrase-structure trees and sentences.
End-to-End Neural Segmental Models for Speech Recognition
Segmental models are an alternative to frame-based models for sequence prediction, where hypothesized path weights are based on entire segment scores rather than a single frame at a time.
A Continuous Relaxation of Beam Search for End-to-end Training of Neural Sequence Models
In experiments, we show that optimizing this new training objective yields substantially better results on two sequence tasks (Named Entity Recognition and CCG Supertagging) when compared with both cross entropy trained greedy decoding and cross entropy trained beam decoding baselines.
Ranked #3 on
Motion Segmentation
on Hopkins155
On the State of the Art of Evaluation in Neural Language Models
Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks.
Ranked #31 on
Language Modelling
on WikiText-2
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Solving algebraic word problems requires executing a series of arithmetic operations{---}a program{---}to obtain a final answer.
Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold
We present a new, efficient frame-semantic parser that labels semantic arguments to FrameNet predicates.
Dynamic Integration of Background Knowledge in Neural NLU Systems
Common-sense and background knowledge is required to understand natural language, but in most neural natural language understanding (NLU) systems, this knowledge must be acquired from training corpora during learning, and then it is static at test time.
Ranked #27 on
Question Answering
on TriviaQA
Greedy Transition-Based Dependency Parsing with Stack LSTMs
During training, dynamic oracles alternate between sampling parser states from the training data and from the model as it is being learned, making the model more robust to the kinds of errors that will be made at test time.
On-the-fly Operation Batching in Dynamic Computation Graphs
Dynamic neural network toolkits such as PyTorch, DyNet, and Chainer offer more flexibility for implementing models that cope with data of varying dimensions and structure, relative to toolkits that operate on statically declared computations (e. g., TensorFlow, CNTK, and Theano).
Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems
Solving algebraic word problems requires executing a series of arithmetic operations---a program---to obtain a final answer.
Ontology-Aware Token Embeddings for Prepositional Phrase Attachment
Type-level word embeddings use the same set of parameters to represent all instances of a word regardless of its context, ignoring the inherent lexical ambiguity in language.
Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
Fixed-vocabulary language models fail to account for one of the most characteristic statistical facts of natural language: the frequent creation and reuse of new word types.
Differentiable Scheduled Sampling for Credit Assignment
We demonstrate that a continuous relaxation of the argmax operation can be used to create a differentiable approximation to greedy decoding for sequence-to-sequence (seq2seq) models.
Generative and Discriminative Text Classification with Recurrent Neural Networks
We empirically characterize the performance of discriminative and generative LSTM models for text classification.
Multitask Learning with CTC and Segmental CRF for Speech Recognition
Segmental conditional random fields (SCRFs) and connectionist temporal classification (CTC) are two sequence labeling methods used for end-to-end training of speech recognition models.
DyNet: The Dynamic Neural Network Toolkit
In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives.
The Role of Context in Neural Morphological Disambiguation
While morphological analyzers can reduce this sparsity by providing morpheme-level analyses for words, they will often introduce ambiguity by returning multiple analyses for the same surface form.
PanPhon: A Resource for Mapping IPA Segments to Articulatory Feature Vectors
This paper contributes to a growing body of evidence that{---}when coupled with appropriate machine-learning techniques{--}linguistically motivated, information-rich representations can outperform one-hot encodings of linguistic data.
Named Entity Recognition for Linguistic Rapid Response in Low-Resource Languages: Sorani Kurdish and Tajik
This paper describes our construction of named-entity recognition (NER) systems in two Western Iranian languages, Sorani Kurdish and Tajik, as a part of a pilot study of {``}Linguistic Rapid Response{''} to potential emergency humanitarian relief situations.
Learning to Compose Words into Sentences with Reinforcement Learning
We use reinforcement learning to learn tree-structured neural networks for computing representations of natural language sentences.
What Do Recurrent Neural Network Grammars Learn About Syntax?
We investigate what information they learn, from a linguistic perspective, through various ablations to the model and the data, and by augmenting the model with an attention mechanism (GA-RNNG) to enable closer inspection.
Ranked #20 on
Constituency Parsing
on Penn Treebank
The Neural Noisy Channel
We formulate sequence to sequence transduction as a noisy channel decoding problem and use recurrent neural networks to parameterise the source and channel models.
Reference-Aware Language Models
We propose a general class of language models that treat reference as an explicit stochastic latent variable.
Ranked #1 on
Recipe Generation
on allrecipes.com
Semantic Parsing with Semi-Supervised Sequential Autoencoders
We present a novel semi-supervised approach for sequence transduction and apply it to semantic parsing.
Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser
We introduce two first-order graph-based dependency parsers achieving a new state of the art.
Ranked #18 on
Dependency Parsing
on Penn Treebank
Neural Machine Translation with Recurrent Attention Modeling
Knowing which words have been attended to in previous time steps while generating a translation is a rich source of information for predicting what words will be attended to in the future.
Greedy, Joint Syntactic-Semantic Parsing with Stack LSTMs
We present a transition-based parser that jointly produces syntactic and semantic dependencies.
Correlation-based Intrinsic Evaluation of Word Vector Representations
We introduce QVEC-CCA--an intrinsic evaluation metric for word vector representations based on correlations of learned vectors with features extracted from linguistic resources.
Generalizing and Hybridizing Count-based and Neural Language Models
Language models (LMs) are statistical models that calculate probabilities over sequences of words or other discrete symbols.
Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning
We introduce polyglot language models, recurrent neural network models trained to predict symbol sequences in many different languages using shared representations of symbols and conditioning on typological information about the language to be predicted.
Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning
We use Bayesian optimization to learn curricula for word representation learning, optimizing performance on downstream tasks that depend on the learned representations as features.
Problems With Evaluation of Word Embeddings Using Word Similarity Tasks
Our study suggests that the use of word similarity tasks for evaluation of word vectors is not sustainable and calls for further research on evaluation methods.
Bridge-Language Capitalization Inference in Western Iranian: Sorani, Kurmanji, Zazaki, and Tajik
In Sorani Kurdish, one of the most useful orthographic features in named-entity recognition {--} capitalization {--} is absent, as the language{'}s Perso-Arabic script does not make a distinction between uppercase and lowercase letters.
Cross-lingual Models of Word Embeddings: An Empirical Comparison
Despite interest in using cross-lingual knowledge to learn word embeddings for various tasks, a systematic comparison of the possible approaches is lacking in the literature.
Training with Exploration Improves a Greedy Stack-LSTM Parser
We adapt the greedy Stack-LSTM dependency parser of Dyer et al. (2015) to support a training-with-exploration procedure using dynamic oracles(Goldberg and Nivre, 2013) instead of cross-entropy minimization.
Ranked #2 on
Chinese Dependency Parsing
on Chinese Pennbank
Neural Architectures for Named Entity Recognition
State-of-the-art named entity recognition systems rely heavily on hand-crafted features and domain-specific knowledge in order to learn effectively from the small, supervised training corpora that are available.
Ranked #8 on
Named Entity Recognition (NER)
on CoNLL++
Segmental Recurrent Neural Networks for End-to-end Speech Recognition
This model connects the segmental conditional random field (CRF) with a recurrent neural network (RNN) used for feature extraction.
Ranked #16 on
Speech Recognition
on TIMIT
Recurrent Neural Network Grammars
We introduce recurrent neural network grammars, probabilistic models of sentences with explicit phrase structure.
Ranked #25 on
Constituency Parsing
on Penn Treebank
Semantic Scan: Detecting Subtle, Spatially Localized Events in Text Streams
Many methods have been proposed for detecting emerging events in text streams using topic modeling.
Massively Multilingual Word Embeddings
We introduce new methods for estimating and evaluating embeddings of words in more than fifty languages in a single shared embedding space.
Many Languages, One Parser
We train one multilingual model for dependency parsing and use it to parse sentences in several languages.
Incorporating Structural Alignment Biases into an Attentional Neural Translation Model
Neural encoder-decoder models of machine translation have achieved impressive results, rivalling traditional translation models.
Morphological Inflection Generation Using Character Sequence to Sequence Learning
Morphological inflection generation is the task of generating the inflected form of a given lemma corresponding to a particular linguistic transformation.
Modeling Dynamic Relationships Between Characters in Literary Novels
Studying characters plays a vital role in computationally representing and interpreting narratives.
Segmental Recurrent Neural Networks
Representations of the input segments (i. e., contiguous subsequences of the input) are computed by encoding their constituent tokens using bidirectional recurrent neural nets, and these "segment embeddings" are used to define compatibility scores with output labels.
Learning to Represent Words in Context with Multilingual Supervision
We present a neural network architecture based on bidirectional LSTMs to compute representations of words in the sentential contexts.
Character-based Neural Machine Translation
We introduce a neural machine translation model that views the input and output sentences as sequences of characters rather than words.
Document Context Language Models
Text documents are structured on multiple levels of detail: individual words are related by syntax, but larger units of text are related by discourse structure.
Depth-Gated LSTM
This gate is a function of the lower layer memory cell, the input to and the past memory cell of this layer.
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
We introduce a model for constructing vector representations of words by composing characters using bidirectional LSTMs.
Ranked #4 on
Part-Of-Speech Tagging
on Penn Treebank
Improved Transition-Based Parsing by Modeling Characters instead of Words with LSTMs
We present extensions to a continuous-state dependency parsing method that makes it applicable to morphologically rich languages.
Non-distributional Word Vector Representations
Data-driven representation learning for words is a technique of central importance in NLP.
Sparse Overcomplete Word Vector Representations
Current distributed representations of words show little resemblance to theories of lexical semantics.
Transition-Based Dependency Parsing with Stack Long Short-Term Memory
We propose a technique for learning representations of parser states in transition-based dependency parsers.
Unsupervised POS Induction with Word Embeddings
Unsupervised word embeddings have been shown to be valuable as features in supervised learning problems; however, their role in unsupervised problems has been less thoroughly explored.
Retrofitting Word Vectors to Semantic Lexicons
Vector space word representations are learned from distributional information of words in large corpora.
Conditional Random Field Autoencoders for Unsupervised Structured Prediction
We introduce a framework for unsupervised learning of structured predictors with overlapping, global features.
Notes on Noise Contrastive Estimation and Negative Sampling
Estimating the parameters of probabilistic models of language such as maxent models and probabilistic neural models is computationally difficult since it involves evaluating partition functions by summing over an entire vocabulary, which may be millions of word types in size.
Learning Word Representations with Hierarchical Sparse Coding
We propose a new method for learning word representations using hierarchical regularization in sparse coding inspired by the linguistic study of word meanings.
Augmenting English Adjective Senses with Supersenses
We develop a supersense taxonomy for adjectives, based on that of GermaNet, and apply it to English adjectives in WordNet using human annotation and supervised classification.
Dual Subtitles as Parallel Corpora
In this paper, we leverage the existence of dual subtitles as a source of parallel data.
A Unified Annotation Scheme for the Semantic/Pragmatic Components of Definiteness
We present a definiteness annotation scheme that captures the semantic, pragmatic, and discourse information, which we call communicative functions, associated with linguistic descriptions such as {``}a story about my speech{''}, {``}the story{''}, {``}every time I give it{''}, {``}this slideshow{''}.
Discriminative Lexical Semantic Segmentation with Gaps: Running the MWE Gamut
We present a novel representation, evaluation measure, and supervised models for the task of identifying the multiword expressions (MWEs) in a sentence, resulting in a lexical semantic segmentation.
Locally Non-Linear Learning for Statistical Machine Translation via Discretization and Structured Regularization
Linear models, which support efficient learning and inference, are the workhorses of statistical machine translation; however, linear decision rules are less attractive from a modeling perspective.
Language Modeling with Power Low Rank Ensembles
We present power low rank ensembles (PLRE), a flexible framework for n-gram language modeling where ensembles of low rank matrices and tensors are used to obtain smoothed probability estimates of words in context.
Predicting the NFL using Twitter
We study the relationship between social media output and National Football League (NFL) games, using a dataset containing messages from Twitter and NFL game statistics.
Minimum Error Rate Training and the Convex Hull Semiring
We describe the line search used in the minimum error rate training algorithm MERT as the "inside score" of a weighted proof forest under a semiring defined in terms of well-understood operations from computational geometry.
A framework for (under)specifying dependency syntax without overloading annotators
We introduce a framework for lightweight dependency syntax annotation.