Search Results for author:
Found 115 papers, 57 papers with code
Contextualized Role Interaction for Neural Machine Translation
Word inputs tend to be represented as single continuous vectors in deep neural networks.
Talk The Walk: Navigating Grids in New York City through Grounded Dialogue
We introduce `"Talk The Walk", the first large-scale dialogue dataset grounded in action and perception.
Findings of the WOAH 5 Shared Task on Fine Grained Hateful Memes Detection
We present the results and main findings of the shared task at WOAH 5 on hateful memes detection.
Findings of the WMT 2021 Shared Task on Large-Scale Multilingual Machine Translation
There were a total of 10 participating teams for the tasks, with a total of 151 intermediate model submissions and 13 final models.
What’s Hidden in a One-layer Randomly Weighted Transformer?
Hidden within a one-layer randomly weighted Transformer, we find that subnetworks that can achieve 29. 45/17. 29 BLEU on IWSLT14/WMT14.
Great Models Think Alike and this Undermines AI Oversight
We study how model similarity affects both aspects of AI oversight by proposing a probabilistic metric for LM similarity based on overlap in model mistakes.
LMUnit: Fine-grained Evaluation with Natural Language Unit Tests
As language models become integral to critical workflows, assessing their behavior remains a fundamental challenge -- human evaluation is costly and noisy, while automated metrics provide only coarse, difficult-to-interpret signals.
Nearest Neighbor Normalization Improves Multimodal Retrieval
Multimodal models leverage large-scale pre-training to achieve strong but still imperfect performance on tasks such as image captioning, visual question answering, and cross-modal retrieval.
OLMoE: Open Mixture-of-Experts Language Models
We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE).
Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
We study this and find that (i) preference data gives a better learning signal when the underlying responses are contrastive, and (ii) alignment objectives lead to better performance when they specify more control over the model during training.
Lynx: An Open Source Hallucination Evaluation Model
Retrieval Augmented Generation (RAG) techniques aim to mitigate hallucinations in Large Language Models (LLMs).
Generative Representational Instruction Tuning
Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss.
KTO: Model Alignment as Prospect Theoretic Optimization
Kahneman & Tversky's $\textit{prospect theory}$ tells us that humans perceive random variables in a biased but well-defined manner (1992); for example, humans are famously loss-averse.
I am a Strange Dataset: Metalinguistic Tests for Language Models
We also provide minimally different metalinguistic non-self-reference examples to complement the main dataset by probing for whether models can handle metalinguistic language at all.
Leveraging Diffusion Perturbations for Measuring Fairness in Computer Vision
In this work, we demonstrate that diffusion models can be leveraged to create such a dataset.
FinanceBench: A New Benchmark for Financial Question Answering
We test 16 state of the art model configurations (including GPT-4-Turbo, Llama2 and Claude2, with vector stores and long context prompts) on a sample of 150 cases from FinanceBench, and manually review their answers (n=2, 400).
Ranked #1 on
How to refund a wrong transaction in PhonePe
on How to refund a wrong transaction in PhonePe
(using extra training data)
How to refund a wrong transaction in PhonePe
Question Answering
+2
Anchor Points: Benchmarking Models with Much Fewer Examples
Moreover, just several anchor points can be used to estimate model per-class predictions on all other points in a dataset with low mean absolute error, sufficient for gauging where the model is likely to fail.
Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language
We propose LENS, a modular approach for tackling computer vision problems by leveraging the power of large language models (LLMs).
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
Large multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks.
Ranked #14 on
MMR total
on MRR-Benchmark
(using extra training data)
AfroDigits: A Community-Driven Spoken Digit Dataset for African Languages
The advancement of speech technologies has been remarkable, yet its integration with African languages remains limited due to the scarcity of African speech corpora.
Investigating Multi-source Active Learning for Natural Language Inference
We show that four popular active learning schemes fail to outperform random selection when applied to unlabelled pools comprised of multiple data sources on the task of natural language inference.
Measuring Data
We identify the task of measuring data to quantitatively characterize the composition of machine learning data and datasets.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements
We introduce Evaluate and Evaluation on the Hub --a set of tools to facilitate the evaluation of models and datasets in ML.
DataPerf: Benchmarks for Data-Centric AI Development
Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems.
Perturbation Augmentation for Fairer NLP
Unwanted and often harmful social biases are becoming ever more salient in NLP research, affecting both models and datasets.
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
We present a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning, which we call Winoground.
Ranked #41 on
Visual Reasoning
on Winoground
Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks
We introduce Dynatask: an open source system for setting up custom NLP tasks that aims to greatly lower the technical knowledge and effort required for hosting and evaluating state-of-the-art NLP models, as well as for conducting model in the loop data collection with crowdworkers.
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants
We collect training datasets in twenty experimental settings and perform a detailed analysis of this approach for the task of extractive question answering (QA) for both standard and adversarial data collection.
FLAVA: A Foundational Language And Vision Alignment Model
State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks.
Ranked #4 on
Image Retrieval
on MS COCO
Analyzing Dynamic Adversarial Training Data in the Limit
To create models that are robust across a wide range of test inputs, training datasets should include diverse examples that span numerous phenomena.
What's Hidden in a One-layer Randomly Weighted Transformer?
Hidden within a one-layer randomly weighted Transformer, we find that subnetworks that can achieve 29. 45/17. 29 BLEU on IWSLT14/WMT14.
Human-Adversarial Visual Question Answering
Human subjects interact with a state-of-the-art VQA model, and for each image in the dataset, attempt to find a question where the model's predicted answer is incorrect.
On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study
In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions.
True Few-Shot Learning with Language Models
Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning.
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform.
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation
We further conduct a novel human-in-the-loop evaluation to show that our models are considerably more robust to new human-written adversarial examples: crowdworkers can fool our model only 8. 8% of the time on average, compared to 17. 6% for a model trained without synthetic data.
Cross-Modal Retrieval Augmentation for Multi-Modal Classification
Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing.
Retrieval Augmentation Reduces Hallucination in Conversation
Despite showing increasingly human-like conversational abilities, state-of-the-art dialogue models often suffer from factual incorrectness and hallucination of knowledge (Roller et al., 2020).
Gradient-based Adversarial Attacks against Text Transformers
We propose the first general-purpose gradient-based attack against transformer models.
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent in classical NLP pipelines.
Dynabench: Rethinking Benchmarking in NLP
We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking.
Quasi-Equivalence Discovery for Zero-Shot Emergent Communication
In contrast, in this work, we present a novel problem setting and the Quasi-Equivalence Discovery (QED) algorithm that allows for zero-shot coordination (ZSC), i. e., discovering protocols that can generalize to independently trained agents.
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length
We introduce a method to determine if a certain capability helps to achieve an accurate model of given data.
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection
We provide a new dataset of ~40, 000 entries, generated and labelled by trained annotators over four rounds of dynamic data creation.
Reservoir Transformers
We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated.
DynaSent: A Dynamic Benchmark for Sentiment Analysis
We introduce DynaSent ('Dynamic Sentiment'), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis.
To what extent do human explanations of model behavior align with actual model behavior?
Given the increasingly prominent role NLP models (will) play in our lives, it is important for human expectations of model behavior to align with actual model behavior.
I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling
To quantify how well natural language understanding models can capture consistency in a general conversation, we introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues.
Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations
Indeed, emergent communication is now a vibrant field of research, with common settings involving discrete cheap-talk channels.
ANLIzing the Adversarial Natural Language Inference Dataset
We perform an in-depth error analysis of Adversarial NLI (ANLI), a recently introduced large-scale human-and-model-in-the-loop natural language inference dataset collected over multiple rounds.
Adversarial Natural Language Inference
Natural Language Inference
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
We propose a simple and efficient multi-hop dense retrieval approach for answering complex open-domain questions, which achieves state-of-the-art performance on two multi-hop datasets, HotpotQA and multi-evidence FEVER.
Ranked #14 on
Question Answering
on HotpotQA
Learning Optimal Representations with the Decodable Information Bottleneck
We address the question of characterizing and finding optimal representations for supervised learning.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks.
Ranked #4 on
Question Answering
on WebQuestions
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes.
Ranked #7 on
Meme Classification
on Hateful Memes
Multi-Dimensional Gender Bias Classification
We show our classifiers prove valuable for a variety of important applications, such as controlling for gender bias in generative models, detecting gender bias in arbitrary text, and shed light on offensive language in terms of genderedness.
Unsupervised Question Decomposition for Question Answering
We aim to improve question answering (QA) by decomposing hard questions into simpler sub-questions that existing QA systems are capable of answering.
I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents
Dialogue research tends to distinguish between chit-chat and goal-oriented tasks.
On the interaction between supervision and self-play in emergent communication
A promising approach for teaching artificial agents to use natural language involves using human-in-the-loop training.
Generating Interactive Worlds with Text
We show that the game environments created with our approach are cohesive, diverse, and preferred by human evaluators compared to other machine learning based world construction algorithms.
Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation
Models often easily learn biases present in the training data, and their predictions directly reflect this bias.
Finding Generalizable Evidence by Learning to Convince Q\&A Models
We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question-answering (QA) as a testbed.
Seeded self-play for language learning
Once the meta-learning agent is able to quickly adapt to each population of agents, it can be deployed in new populations, including populations speaking human language.
Adversarial NLI: A New Benchmark for Natural Language Understanding
We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure.
Hyperbolic Graph Neural Networks
Learning from graph-structured data is an important task in machine learning and artificial intelligence, for which Graph Neural Networks (GNNs) have shown great promise.
Generalized Inner Loop Meta-Learning
Many (but not all) approaches self-qualifying as "meta-learning" in deep learning and reinforcement learning fit a common pattern of approximating the solution to a nested optimization problem.
Finding Generalizable Evidence by Learning to Convince Q&A Models
We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question-answering (QA) as a testbed.
Countering Language Drift via Visual Grounding
Emergent multi-agent communication protocols are very different from natural language and not easily interpretable by humans.
Supervised Multimodal Bitransformers for Classifying Images and Text
Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks.
Ranked #1 on
Natural Language Inference
on V-SNLI
(using extra training data)
Why Build an Assistant in Minecraft?
In this document we describe a rationale for a research program aimed at building an open "assistant" in the game Minecraft, in order to make progress on the problems of natural language understanding and learning from dialogue.
Learning to Speak and Act in a Fantasy Text Adventure Game
We analyze the ingredients necessary for successful grounding in this setting, and how each of these factors relate to agents that can talk and act successfully.
What makes a good conversation? How controllable attributes affect human judgments
A good conversation requires balance -- between simplicity and detail; staying on topic and changing it; asking questions and answering them.
Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings
Moreover -- and in contrast with other methods -- the hierarchical nature of hyperbolic space allows us to learn highly efficient representations and to improve the taxonomic consistency of the inferred hierarchies.
The Second Conversational Intelligence Challenge (ConvAI2)
We describe the setting and results of the ConvAI2 NeurIPS competition that aims to further the state-of-the-art in open-domain chatbots.
No Training Required: Exploring Random Encoders for Sentence Classification
We explore various methods for computing sentence representations from pre-trained word embeddings without any training, i. e., using nothing but random parameterizations.
Emergent Linguistic Phenomena in Multi-Agent Communication Games
In this work, we propose a computational framework in which agents equipped with communication capabilities simultaneously play a series of referential games, where agents are trained using deep reinforcement learning.
Countering Language Drift via Grounding
While reinforcement learning (RL) shows a lot of promise for natural language processing—e. g.
Jump to better conclusions: SCAN both left and right
Lake and Baroni (2018) recently introduced the SCAN data set, which consists of simple commands paired with action sequences and is intended to test the strong generalization abilities of recurrent sequence-to-sequence models.
Talk the Walk: Navigating New York City through Grounded Dialogue
We introduce "Talk The Walk", the first large-scale dialogue dataset grounded in action and perception.
Code-Switched Named Entity Recognition with Embedding Attention
We describe our work for the CALCS 2018 shared task on named entity recognition on code-switched data.
Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry
We are concerned with the discovery of hierarchical relationships from large-scale unstructured similarity scores.
Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora
Methods for unsupervised hypernym detection may broadly be categorized according to two paradigms: pattern-based and distributional methods.
Dynamic Meta-Embeddings for Improved Sentence Representations
While one of the first steps in many NLP systems is selecting what pre-trained word embeddings to use, we argue that such a step is better left for neural networks to figure out by themselves.
Ranked #52 on
Natural Language Inference
on SNLI
SentEval: An Evaluation Toolkit for Universal Sentence Representations
We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Chit-chat models are known to have several problems: they lack specificity, do not display a consistent personality and are often not very captivating.
Ranked #5 on
Dialogue Generation
on Persona-Chat
(using extra training data)
Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descent
Contrary to most natural language processing research, which makes use of static datasets, humans learn language interactively, grounded in an environment.
Emergent Translation in Multi-Agent Communication
While most machine translation systems to date are trained on large parallel corpora, humans learn language in a different way: by being grounded in an environment and interacting with other humans.
Grasping the Finer Point: A Supervised Similarity Network for Metaphor Detection
The ubiquity of metaphor in our everyday communication makes it an important problem for natural language understanding.
Learning Visually Grounded Sentence Representations
We introduce a variety of models, trained on a supervised image captioning corpus to predict the image features for a given caption, to perform sentence representation grounding.
Automatically Generating Rhythmic Verse with Neural Networks
We propose two novel methodologies for the automatic generation of rhythmic poetry in a variety of forms.
Emergent Communication in a Multi-Modal, Multi-Step Referential Game
Inspired by previous work on emergent communication in referential games, we propose a novel multi-modal, multi-step referential game, where the sender and receiver have access to distinct modalities of an object, and their information exchange is bidirectional and of arbitrary duration.
Poincaré Embeddings for Learning Hierarchical Representations
Representation learning has become an invaluable approach for learning from symbolic data such as text and graphs.
Ranked #2 on
Link Prediction
on WordNet
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
Many modern NLP systems rely on word embeddings, previously trained in an unsupervised manner on large corpora, as base features.
Ranked #1 on
Semantic Textual Similarity
on SentEval
Cross-Lingual Natural Language Inference
Semantic Textual Similarity
+3
Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation
Recent work on evaluating representation learning architectures in NLP has established a need for evaluation protocols based on subconscious cognitive measures rather than manually tailored intrinsic similarity and relatedness tasks.
Learning to Negate Adjectives with Bilinear Models
We learn a mapping that negates adjectives by predicting an adjective{'}s antonym in an arbitrary word embedding model.
Visually Grounded and Textual Semantic Models Differentially Decode Brain Activity Associated with Concrete and Abstract Nouns
Dual coding theory considers concrete concepts to be encoded in the brain both linguistically and visually, and abstract concepts only linguistically.
Robust Text Classification for Sparsely Labelled Data Using Multi-level Embeddings
The conventional solution for handling sparsely labelled data is extensive feature engineering.
Virtual Embodiment: A Scalable Long-Term Strategy for Artificial Intelligence Research
Meaning has been called the "holy grail" of a variety of scientific disciplines, ranging from linguistics to philosophy, psychology and the neurosciences.
HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment
We introduce HyperLex - a dataset and evaluation resource that quantifies the extent of of the semantic category membership, that is, type-of relation also known as hyponymy-hypernymy or lexical entailment (LE) relation between 2, 616 concept pairs.
