Search Results for author:
Found 133 papers, 64 papers with code
Don’t Read Too Much Into It: Adaptive Computation for Open-Domain Question Answering
Most approaches to Open-Domain Question Answering consist of a light-weight retriever that selects a set of candidate passages, and a computationally expensive reader that examines the passages to identify the correct answer.
Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?
For queries requiring latent recall of countries as the intermediate answer, the best models achieve 80% latent composability, but this drops to just 5% for the recall of years.
Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
In this report, we introduce the Gemini 1. 5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio.
Ranked #1 on
Visual Question Answering
on MM-Vet
Do Large Language Models Latently Perform Multi-Hop Reasoning?
We find strong evidence of latent multi-hop reasoning for the prompts of certain relation types, with the reasoning pathway used in more than 80% of the prompts.
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Visual Question Answering (VQA)
on InfographicVQA
Strings from the Library of Babel: Random Sampling as a Strong Baseline for Prompt Optimisation
Recent prompt optimisation approaches use the generative nature of language models to produce prompts -- even rivaling the performance of human-curated prompts.
Improving Language Plasticity via Pretraining with Active Forgetting
Pretrained language models (PLMs) are today the primary model for natural language processing.
Can discrete information extraction prompts generalize across language models?
We study whether automatically-induced prompts that effectively extract information from a language model can also be used, out-of-the-box, to probe other language models for the same information.
Task-aware Retrieval with Instructions
We study the problem of retrieval with instructions, where users of a retrieval system explicitly describe their intent along with their queries.
An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks
Experiments on various knowledge-intensive tasks such as question answering and dialogue datasets show that, simply augmenting parametric models (T5-base) using our method produces more accurate results (e. g., 25. 8 -> 44. 3 EM on NQ) while retaining a high throughput (e. g., 1000 queries/s on NQ).
Ranked #4 on
Question Answering
on KILT: ELI5
Query Expansion Using Contextual Clue Sampling with Language Models
To balance these two considerations, we propose a combination of an effective filtering strategy and fusion of the retrieved documents based on the generation probability of each context.
EditEval: An Instruction-Based Benchmark for Text Improvements
Evaluation of text generation to date has primarily focused on content created sequentially, rather than improvements on a piece of text.
PEER: A Collaborative Language Model
Textual content is often the output of a collaborative writing process: We start with an initial draft, ask for suggestions, and repeatedly make changes.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Retrieval augmented models are known to excel at knowledge intensive tasks without the need for as many parameters, but it is unclear whether they work in few-shot settings.
Ranked #1 on
Question Answering
on Natural Questions
ReFactor GNNs: Revisiting Factorisation-based Models from a Message-Passing Perspective
Factorisation-based Models (FMs), such as DistMult, have enjoyed enduring success for Knowledge Graph Completion (KGC) tasks, often outperforming Graph Neural Networks (GNNs).
Improving Wikipedia Verifiability with AI
Hence, maintaining and improving the quality of Wikipedia references is an important challenge and there is a pressing need for better tools to assist humans in this effort.
EDIN: An End-to-end Benchmark and Pipeline for Unknown Entity Discovery and Indexing
This paper created the Unknown Entity Discovery and Indexing (EDIN) benchmark where unknown entities, that is entities without a description in the knowledge base and labeled mentions, have to be integrated into an existing entity linking system.
Lifting the Curse of Multilinguality by Pre-training Modular Transformers
Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages.
Open Vocabulary Extreme Classification Using Generative Models
To develop systems that simplify this process, we introduce the task of open vocabulary XMC (OXMC): given a piece of content, predict a set of labels, some of which may be outside of the known tag set.
Autoregressive Search Engines: Generating Substrings as Document Identifiers
Knowledge-intensive language tasks require NLP systems to both provide the correct answer and retrieve supporting evidence for it in a given corpus.
The Web Is Your Oyster -- Knowledge-Intensive NLP against a Very Large Web Corpus
In order to address increasing demands of real-world applications, the research for knowledge-intensive NLP (KI-NLP) should advance by capturing the challenges of a truly open-domain environment: web-scale knowledge, lack of structure, inconsistent quality and noise.
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants
We collect training datasets in twenty experimental settings and perform a detailed analysis of this approach for the task of extractive question answering (QA) for both standard and adversarial data collection.
Unsupervised Dense Information Retrieval with Contrastive Learning
In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers and show that it leads to strong performance in various retrieval settings.
Boosted Dense Retriever
DrBoost is trained in stages: each component model is learned sequentially and specialized by focusing only on retrieval mistakes made by the current ensemble.
A Few More Examples May Be Worth Billions of Parameters
We investigate the dynamics of increasing the number of model parameters versus the number of labeled examples across a wide variety of tasks.
Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations
Learning good representations on multi-relational graphs is essential to knowledge base completion (KBC).
Ranked #1 on
Link Prediction
on CoDEx Small
Contrastive Pre-training for Zero-Shot Information Retrieval
By contrast, in many other NLP tasks, conventional self-supervised pre-training based on masking leads to strong generalization with small number of training examples.
Challenges in Generalization in Open Domain Question Answering
Recent work on Open Domain Question Answering has shown that there is a large discrepancy in model performance between novel test questions and those that largely overlap with training questions.
ProoFVer: Natural Logic Theorem Proving for Fact Verification
Fact verification systems typically rely on neural network classifiers for veracity prediction which lack explainability.
Ranked #1 on
Fact Verification
on FEVER
Domain-matched Pre-training Tasks for Dense Retrieval
Pre-training on larger datasets with ever increasing model size is now a proven recipe for increased performance across almost all NLP tasks.
Ranked #2 on
Passage Retrieval
on Natural Questions
(using extra training data)
Training Adaptive Computation for Open-Domain Question Answering with Computational Constraints
Adaptive Computation (AC) has been shown to be effective in improving the efficiency of Open-Domain Question Answering (ODQA) systems.
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning.
Database Reasoning Over Text
Neural models have shown impressive performance gains in answering queries from natural language text.
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation
We further conduct a novel human-in-the-loop evaluation to show that our models are considerably more robust to new human-written adversarial examples: crowdworkers can fool our model only 8. 8% of the time on average, compared to 17. 6% for a model trained without synthetic data.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
When primed with only a handful of training samples, very large, pretrained language models such as GPT-3 have shown competitive results when compared to fully-supervised, fine-tuned, large, pretrained language models.
Dynabench: Rethinking Benchmarking in NLP
We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking.
Multilingual Autoregressive Entity Linking
Moreover, in a zero-shot setting on languages with no training data at all, mGENRE treats the target language as a latent variable that is marginalized at prediction time.
Ranked #2 on
Entity Disambiguation
on Mewsli-9
(using extra training data)
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them
We introduce a new QA-pair retriever, RePAQ, to complement PAQ.
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
Joint Verification and Reranking for Open Fact Checking Over Tables
Structured information is an important knowledge source for automatic verification of factual claims.
A Memory Efficient Baseline for Open Domain Question Answering
Recently, retrieval systems based on dense representations have led to important improvements in open-domain question answering, and related tasks.
Don't Read Too Much into It: Adaptive Computation for Open-Domain Question Answering
Most approaches to Open-Domain Question Answering consist of a light-weight retriever that selects a set of candidate passages, and a computationally expensive reader that examines the passages to identify the correct answer.
Generating Fact Checking Briefs
Fact checking at scale is difficult -- while the number of active fact checking websites is growing, it remains too small for the needs of the contemporary media ecosystem.
Neural Databases
We describe NeuralDB, a database system with no pre-defined schema, in which updates and queries are given in natural language.
Autoregressive Entity Retrieval
For instance, Encyclopedias such as Wikipedia are structured by entities (e. g., one per Wikipedia article).
Ranked #1 on
Entity Linking
on KILT: WNED-WIKI
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
We propose a simple and efficient multi-hop dense retrieval approach for answering complex open-domain questions, which achieves state-of-the-art performance on two multi-hop datasets, HotpotQA and multi-evidence FEVER.
Ranked #14 on
Question Answering
on HotpotQA
KILT: a Benchmark for Knowledge Intensive Language Tasks
We test both task-specific and general baselines, evaluating downstream performance in addition to the ability of the models to provide provenance.
Ranked #3 on
Entity Linking
on KILT: WNED-CWEB
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets
We also find that 30% of test-set questions have a near-duplicate paraphrase in their corresponding training sets.
Learning Reasoning Strategies in End-to-End Differentiable Proving
Attempts to render deep learning models interpretable, data-efficient, and robust have seen some success through hybridisation with rule-based systems, for example, in Neural Theorem Provers (NTPs).
Ranked #1 on
Relational Reasoning
on CLUTRR (k=3)
Concept Matching for Low-Resource Classification
We propose a model to tackle classification tasks in the presence of very little training data.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks.
Ranked #5 on
Question Answering
on WebQuestions
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks.
Ranked #12 on
Text-To-SQL
on spider
(Exact Match Accuracy (Dev) metric)
How Context Affects Language Models' Factual Predictions
When pre-trained on large unsupervised textual corpora, language models are able to store and retrieve factual knowledge to some extent, making it possible to use them directly for zero-shot cloze-style question answering.
AxCell: Automatic Extraction of Results from Machine Learning Papers
Tracking progress in machine learning has become increasingly difficult with the recent explosion in the number of papers.
Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training
Natural Language Inference (NLI) datasets contain annotation artefacts resulting in spurious correlations between the natural language utterances and their respective entailment classes.
Undersensitivity in Neural Reading Comprehension
Current reading comprehension models generalise well to in-distribution test sets, yet perform poorly on adversarially selected inputs.
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension
We find that training on adversarially collected samples leads to strong generalisation to non-adversarially collected datasets, yet with progressive performance deterioration with increasingly stronger models-in-the-loop.
Ranked #1 on
Reading Comprehension
on AdversarialQA
(using extra training data)
Differentiable Reasoning on Large Knowledge Bases and Natural Language
Reasoning with knowledge expressed in natural language and Knowledge Bases (KBs) is a major challenge for Artificial Intelligence, with applications in machine reading, dialogue, and question answering.
Ranked #3 on
Link Prediction
on FB122
Scalable Zero-shot Entity Linking with Dense Entity Retrieval
This paper introduces a conceptually simple, scalable, and highly effective BERT-based entity linking model, along with an extensive evaluation of its accuracy-speed trade-off.
How Decoding Strategies Affect the Verifiability of Generated Text
A generated sentence is verifiable if it can be corroborated or disproved by Wikipedia, and we find that the verifiability of generated text strongly depends on the decoding strategy.
MLQA: Evaluating Cross-lingual Extractive Question Answering
An alternative to building large monolingual training datasets is to develop cross-lingual systems which can transfer to a target language without requiring training data in that language.
MVFST-RL: An Asynchronous RL Framework for Congestion Control with Delayed Actions
This is largely an artifact of building on top of frameworks designed for RL in games (e. g. OpenAI Gym).
Soft Token Matching for Interpretable Low-Resource Classification
We propose a model to tackle classification tasks in the presence of very little training data.
Language Models as Knowledge Bases?
Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks.