Search Results for author:
Found 112 papers, 40 papers with code
The DeepMind Chinese–English Document Translation System at WMT2020
This paper describes the DeepMind submission to the Chinese\rightarrowEnglish constrained data track of the WMT2020 Shared Task on News Translation.
Learning to encode spatial relations from natural language
Natural language processing has made significant inroads into learning the semantics of words through distributional approaches, however representations learnt via these methods fail to capture certain kinds of information implicit in the real world.
A Generative Framework for Simultaneous Machine Translation
We propose a generative framework for simultaneous machine translation.
BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts
BAM makes full use of specialized dense models by not only using their FFN to initialize the MoE layers but also leveraging experts' attention parameters fully by initializing them into a soft-variant of Mixture of Attention (MoA) layers.
Separations in the Representational Capabilities of Transformers and Recurrent Architectures
Furthermore, we show that two-layer Transformers of logarithmic size can perform decision tasks such as string equality or disjointness, whereas both one-layer Transformers and recurrent models require linear size for these tasks.
Improving Reward Models with Synthetic Critiques
This offers richer signals and more robust features for RMs to assess and score on.
Aya 23: Open Weight Releases to Further Multilingual Progress
This technical report introduces Aya 23, a family of multilingual language models.
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
Recent breakthroughs in large language models (LLMs) have centered around a handful of data-rich languages.
Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions
In this work, we take a step towards answering these questions by demonstrating the following: (a) On a test-bed with a variety of Boolean function classes, we find that Transformers can nearly match the optimal learning algorithm for 'simpler' tasks, while their performance deteriorates on more 'complex' tasks.
Human Feedback is not Gold Standard
We critically analyse the use of human feedback for both training and evaluation, to verify whether it fully captures a range of crucial error criteria.
Structural Transfer Learning in NL-to-Bash Semantic Parsers
Large-scale pre-training has made progress in many fields of natural language processing, though little is understood about the design of pre-training datasets.
On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Research
This evidence-based position paper critiques current research practices within the language model pre-training literature.
Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions
(ii) When trained on Boolean functions, both Transformers and LSTMs prioritize learning functions of low sensitivity, with Transformers ultimately converging to functions of lower sensitivity.
Augmenting Multi-Turn Text-to-SQL Datasets with Self-Play
We first design a SQL-to-text model conditioned on a sampled goal query, which represents a user's intent, that then converses with a text-to-SQL semantic parser to generate new interactions.
Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization
Vision-and-language (V&L) models pretrained on large-scale multimodal data have demonstrated strong performance on various tasks such as image captioning and visual question answering (VQA).
StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models
Knowledge and language understanding of models evaluated through question answering (QA) has been usually studied on static snapshots of knowledge, like Wikipedia.
Revisiting the Compositional Generalization Abilities of Neural Sequence Models
Compositional generalization is a fundamental trait in humans, allowing us to effortlessly combine known phrases to form novel sentences.
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale
We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics.
Relational Memory Augmented Language Models
We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph.
A Systematic Investigation of Commonsense Knowledge in Large Language Models
Language models (LMs) trained on large amounts of data have shown impressive performance on many NLP tasks under the zero-shot and few-shot setup.
Counterfactual Data Augmentation for Neural Machine Translation
We propose a data augmentation method for neural machine translation.
Pretraining the Noisy Channel Model for Task-Oriented Dialogue
Direct decoding for task-oriented dialogue is known to suffer from the explaining-away effect, manifested in models that prefer short and generic responses.
Ranked #2 on
End-To-End Dialogue Modelling
on MULTIWOZ 2.0
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Hence, given the compilation of ever-larger language modelling datasets, combined with the growing list of language-model-based NLP applications that require up-to-date factual knowledge about the world, we argue that now is the right time to rethink the static way in which we currently train and evaluate our language models, and develop adaptive language models that can remain up-to-date with respect to our ever-changing and non-stationary world.
Mutual Information Constraints for Monte-Carlo Objectives
A common failure mode of density models trained as variational autoencoders is to model the data without relying on their latent variables, rendering these variables useless.
The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsets
For neural models to garner widespread public trust and ensure fairness, we must have human-intelligible explanations for their predictions.
Syntactic Structure Distillation Pretraining For Bidirectional Encoders
Textual representation learners trained on large amounts of data have achieved notable success on downstream tasks; intriguingly, they have also performed well on challenging tests of syntactic competence.
Learning to Segment Actions from Observation and Narration
We apply a generative segmental model of task structure, guided by narration, to action segmentation in video.
A Survey on Contextual Embeddings
Contextual embeddings, such as ELMo and BERT, move beyond global word representations like Word2Vec and achieve ground-breaking performance on a wide range of natural language processing tasks.
Visual Grounding in Video for Unsupervised Word Translation
Given this shared embedding we demonstrate that (i) we can map words between the languages, particularly the 'visual' words; (ii) that the shared embedding provides a good initialization for existing unsupervised text-based word translation techniques, forming the basis for our proposed hybrid visual-text mapping algorithm, MUVE; and (iii) our approach achieves superior performance by addressing the shortcomings of text-based methods -- it is more robust, handles datasets with less commonality, and is applicable to low-resource languages.
Learning Robust and Multilingual Speech Representations
Unsupervised speech representation learning has shown remarkable success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance.
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations
To increase trust in artificial intelligence systems, a promising research direction consists of designing neural models capable of generating natural language explanations for their predictions.
Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods
We aim for this framework to provide a publicly available, off-the-shelf evaluation when the feature-selection perspective on explanations is needed.
Better Document-Level Machine Translation with Bayes' Rule
We show that Bayes' rule provides an effective mechanism for creating document translation models that can be learned from only parallel sentences and monolingual documents---a compelling benefit as parallel documents are not always available.
Unsupervised Learning of Efficient and Robust Speech Representations
We present an unsupervised method for learning speech representations based on a bidirectional contrastive predictive coding that implicitly discovers phonetic structure from large-scale corpora of unlabelled raw audio signals.
Putting Machine Translation in Context with the Noisy Channel Model
We show that Bayes' rule provides a compelling mechanism for controlling unconditional document language models, using the long-standing challenge of effectively leveraging document context in machine translation.
A Critical Analysis of Biased Parsers in Unsupervised Parsing
A series of recent papers has used a parsing algorithm due to Shen et al. (2018) to recover phrase-structure trees based on proxies for "syntactic depth."
Mogrifier LSTM
Many advances in Natural Language Processing have been based upon more expressive models for how inputs interact with the context in which they occur.
WikiCREM: A Large Unsupervised Corpus for Coreference Resolution
We use a language-model-based approach for pronoun resolution in combination with our WikiCREM dataset.
Scalable Syntax-Aware Language Models Using Knowledge Distillation
Prior work has shown that, on small amounts of training data, syntactic neural language models learn structurally sensitive generalisations more successfully than sequential language models.
Learning and Evaluating General Linguistic Intelligence
We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly.
e-SNLI: Natural Language Inference with Natural Language Explanations
In order for machine learning to garner widespread public adoption, models must be able to provide interpretable and robust explanations for their decisions, as well as learn from human-provided explanations at train time.
Ranked #1 on
Natural Language Inference
on e-SNLI
Learning with Stochastic Guidance for Navigation
Due to the sparse rewards and high degree of environment variation, reinforcement learning approaches such as Deep Deterministic Policy Gradient (DDPG) are plagued by issues of high variance when applied in complex real world environments.
Robotics
Sentence Encoding with Tree-constrained Relation Networks
The meaning of a sentence is a function of the relations that hold between its words.
Learning to Discover, Ground and Use Words with Segmental Neural Language Models
We propose a segmental neural language model that combines the generalization power of neural networks with the ability to discover word-like units that are latent in unsegmented character sequences.
Transferring Physical Motion Between Domains for Neural Inertial Tracking
Inertial information processing plays a pivotal role in ego-motion awareness for mobile agents, as inertial measurements are entirely egocentric and not environment dependent.
Unsupervised Word Discovery with Segmental Neural Language Models
We propose a segmental neural language model that combines the representational power of neural networks and the structure learning mechanism of Bayesian nonparametrics, and show that it learns to discover semantically meaningful units (e. g., morphemes and words) from unsegmented character sequences.
Neural Arithmetic Logic Units
Neural networks can learn to represent and manipulate numerical information, but they seldom generalize well outside of the range of numerical values encountered during training.
Encoding Spatial Relations from Natural Language
Natural language processing has made significant inroads into learning the semantics of words through distributional approaches, however representations learnt via these methods fail to capture certain kinds of information implicit in the real world.
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better
Language exhibits hierarchical structure, but recent work using a subject-verb agreement diagnostic argued that state-of-the-art language models, LSTMs, fail to learn long-range syntax sensitive dependencies.
Neural Syntactic Generative Models with Exact Marginalization
We present neural syntactic generative models with exact marginalization that support both dependency parsing and language modeling.
Pushing the bounds of dropout
We show that dropout training is best understood as performing MAP estimation concurrently for a family of conditional models whose objectives are themselves lower bounded by the original dropout objective.
Ranked #24 on
Language Modelling
on Penn Treebank (Word Level)
Understanding Grounded Language Learning Agents
Neural network-based systems can now learn to locate the referents of words and phrases in images, answer questions about visual scenes, and even execute symbolic instructions as first-person actors in partially-observable worlds.
Memory Architectures in Recurrent Neural Network Language Models
We compare and analyze sequential, random access, and stack memory architectures for recurrent neural network language models.
The NarrativeQA Reading Comprehension Challenge
Reading comprehension (RC)---in contrast to information retrieval---requires integrating information and reasoning about events, entities, and their relations across a full document.
Ranked #9 on
Question Answering
on NarrativeQA
(BLEU-1 metric)
Understanding Early Word Learning in Situated Artificial Agents
Neural network-based systems can now learn to locate the referents of words and phrases in images, answer questions about visual scenes, and execute symbolic instructions as first-person actors in partially-observable worlds.
Oxford at SemEval-2017 Task 9: Neural AMR Parsing with Pointer-Augmented Attention
We present a neural encoder-decoder AMR parser that extends an attention-based model by predicting the alignment between graph nodes and sentence tokens explicitly with a pointer mechanism.
Ranked #27 on
AMR Parsing
on LDC2017T10
On the State of the Art of Evaluation in Neural Language Models
Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks.
Ranked #32 on
Language Modelling
on WikiText-2
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Solving algebraic word problems requires executing a series of arithmetic operations{---}a program{---}to obtain a final answer.
Grounded Language Learning in a Simulated 3D World
Trained via a combination of reinforcement and unsupervised learning, and beginning with minimal prior knowledge, the agent learns to relate linguistic symbols to emergent perceptual representations of its physical surroundings and to pertinent sequences of actions.
Discovering Discrete Latent Topics with Neural Variational Inference
Topic models have been widely explored as probabilistic generative models of documents.
Ranked #2 on
Topic Models
on 20NewsGroups
Latent Intention Dialogue Models
Developing a dialogue agent that is capable of making autonomous decisions and communicating by natural language is one of the long-term goals of machine learning research.
Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems
Solving algebraic word problems requires executing a series of arithmetic operations---a program---to obtain a final answer.
Robust Incremental Neural Semantic Graph Parsing
Parsing sentences to linguistically-expressive semantic representations is a key goal of Natural Language Processing.
Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
Fixed-vocabulary language models fail to account for one of the most characteristic statistical facts of natural language: the frequent creation and reuse of new word types.
Generative and Discriminative Text Classification with Recurrent Neural Networks
We empirically characterize the performance of discriminative and generative LSTM models for text classification.
Learning to Compose Words into Sentences with Reinforcement Learning
We use reinforcement learning to learn tree-structured neural networks for computing representations of natural language sentences.
The Neural Noisy Channel
We formulate sequence to sequence transduction as a noisy channel decoding problem and use recurrent neural networks to parameterise the source and channel models.
Reference-Aware Language Models
We propose a general class of language models that treat reference as an explicit stochastic latent variable.
Ranked #1 on
Recipe Generation
on allrecipes.com
Semantic Parsing with Semi-Supervised Sequential Autoencoders
We present a novel semi-supervised approach for sequence transduction and apply it to semantic parsing.
Online Segment to Segment Neural Transduction
We introduce an online neural sequence to sequence model that learns to alternate between encoding and decoding segments of the input as it is read.
Language as a Latent Variable: Discrete Generative Models for Sentence Compression
In this work we explore deep generative models of text in which the latent representation of a document is itself drawn from a discrete language model distribution.
Optimizing Performance of Recurrent Neural Networks on GPUs
As recurrent neural networks become larger and deeper, training times for single networks are rising into weeks or even months.
Latent Predictor Networks for Code Generation
Many language generation tasks require the production of text conditioned on both structured and unstructured inputs.
Ranked #10 on
Code Generation
on Django
Stochastic Collapsed Variational Inference for Sequential Data
Stochastic variational inference for collapsed models has recently been successfully applied to large scale topic modelling.
Stochastic Collapsed Variational Inference for Hidden Markov Models
In this paper, we propose a stochastic collapsed variational inference algorithm for hidden Markov models, in a sequential data setting.
Neural Variational Inference for Text Processing
We validate this framework on two very different text modelling applications, generative document modelling and supervised question answering.
Ranked #1 on
Question Answering
on QASent
Reasoning about Entailment with Neural Attention
We extend this model with a word-by-word neural attention mechanism that encourages reasoning over entailments of pairs of words and phrases.
Ranked #81 on
Natural Language Inference
on SNLI
A Bayesian Model for Generative Transition-based Dependency Parsing
We propose a simple, scalable, fully generative model for transition-based dependency parsing with high accuracy.
Teaching Machines to Read and Comprehend
Teaching machines to read natural language documents remains an elusive challenge.
Ranked #13 on
Question Answering
on CNN / Daily Mail
Learning to Transduce with Unbounded Memory
Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems.
Pragmatic Neural Language Modelling in Machine Translation
This paper presents an in-depth investigation on integrating neural language models in translation systems.
Bayesian Optimisation for Machine Translation
This paper presents novel Bayesian optimisation algorithms for minimum error rate training of statistical machine translation systems.
Deep Learning for Answer Sentence Selection
Answer sentence selection is the task of identifying sentences that contain the answer to a given question.
Ranked #3 on
Question Answering
on QASent
Deep Multi-Instance Transfer Learning
We present a new approach for transferring knowledge from groups to individuals that comprise them.
Modelling, Visualising and Summarising Documents with a Single Convolutional Neural Network
Capturing the compositional process which maps the meaning of words to that of documents is a central challenge for researchers in Natural Language Processing and Information Retrieval.
Compositional Morphology for Word Representations and Language Modelling
This paper presents a scalable method for integrating compositional morphological representations into a vector-based probabilistic language model.
Learning Bilingual Word Representations by Marginalizing Alignments
We present a probabilistic model that simultaneously learns alignments and distributed representations for bilingual data.
A Deep Architecture for Semantic Parsing
Many successful approaches to semantic parsing build on top of the syntactic analysis of text, and make use of distributional representations or statistical models to match parses to ontology-specific queries.
Multilingual Models for Compositional Distributed Semantics
We present a novel technique for learning semantic representations, which extends the distributional hypothesis to multilingual data and joint-space embeddings.
Cross-Lingual Document Classification
Document Classification
+2
A Convolutional Neural Network for Modelling Sentences
The ability to accurately represent sentences is central to language understanding.
Modelling the Lexicon in Unsupervised Part of Speech Induction
Automatically inducing the syntactic part-of-speech categories for words in text is a fundamental task in Computational Linguistics.
Multilingual Distributed Representations without Word Alignment
Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP.
Cross-Lingual Document Classification
Document Classification
+3
Recurrent Convolutional Neural Networks for Discourse Compositionality
The compositionality of meaning extends beyond the single sentence.
"Not not bad" is not "bad": A distributional account of negation
With the increasing empirical success of distributional models of compositional semantics, it is timely to consider the types of textual logic that such models are capable of capturing.
Unsupervised Induction of Tree Substitution Grammars for Dependency Parsing
Inducing a grammar directly from text is one of the oldest and most challenging tasks in Computational Linguistics.
Ranked #3 on
Unsupervised Dependency Parsing
on Penn Treebank
Dependency Grammar Induction
Unsupervised Dependency Parsing
Bayesian Synchronous Grammar Induction
We present a novel method for inducing synchronous context free grammars (SCFGs) from a corpus of parallel string pairs.
