Search Results for author:
Found 71 papers, 24 papers with code
Transferring Representations of Logical Connectives
In modern natural language processing pipelines, it is common practice to “pretrain” a generative language model on a large corpus of text, and then to “finetune” the created representations by continuing to train them on a discriminative textual inference task.
Do ``Undocumented Workers'' == ``Illegal Aliens''? Differentiating Denotation and Connotation in Vector Spaces
In politics, neologisms are frequently invented for partisan objectives.
“Was it “stated” or was it “claimed”?: How linguistic bias affects generative language models
People use language in subtle and nuanced ways to convey their beliefs.
NeuroSurgeon: A Toolkit for Subnetwork Analysis
Despite recent advances in the field of explainability, much remains unknown about the algorithms that neural networks learn to represent.
Language Models Implement Simple Word2Vec-style Vector Arithmetic
A primary criticism towards language models (LMs) is their inscrutability.
Testing Causal Models of Word Meaning in GPT-3 and -4
The theory posits a causal graph that relates the meanings of such words to the form, use, and history of the objects to which they refer.
Training Priors Predict Text-To-Image Model Performance
By looking at the subject-verb-object (SVO) triads that form the backbone of these prompts (e. g., "astronaut", "ride", "horse"), we find that the more often an SVO triad appears in the training data, the better the model can generate an image aligned with that triad.
Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs
To study such interactions, we propose Simfluence, a new paradigm for TDA where the goal is not to produce a single influence score per example, but instead a training run simulator: the user asks, ``If my model had trained on example $z_1$, then $z_2$, ..., then $z_n$, how would it behave on $z_{test}$?
Comparing Trajectory and Vision Modalities for Verb Representation
Given the importance of 3D space in formal models of verb semantics, we expect that these 2D images would result in impoverished representations that fail to capture nuanced differences in meaning.
Break It Down: Evidence for Structural Compositionality in Neural Networks
Many tasks can be described as compositions over subroutines.
Are Language Models Worse than Humans at Following Prompts? It's Complicated
In this study, we challenge a central assumption in such work: that humans would perform badly when given pathological instructions.
Does CLIP Bind Concepts? Probing Compositionality in Large Image Models
In this work, we focus on the ability of a large pretrained vision and language model (CLIP) to encode compositional concepts and to bind variables in a structure-sensitive way (e. g., differentiating ''cube behind sphere'' from ''sphere behind cube'').
Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hex
AlphaZero, an approach to reinforcement learning that couples neural networks and Monte Carlo tree search (MCTS), has produced state-of-the-art strategies for traditional board games like chess, Go, shogi, and Hex.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Linearly Mapping from Image to Text Space
Prior work has shown that pretrained LMs can be taught to caption images when a vision model's parameters are optimized to encode images in the language space.
Unit Testing for Concepts in Neural Networks
Many complex problems are naturally understood in terms of symbolic concepts.
Pretraining on Interactions for Learning Grounded Affordance Representations
Lexical semantics and cognitive science point to affordances (i. e. the actions that objects support) as critical for understanding and representing nouns and verbs.
Do Trajectories Encode Verb Meaning?
Distributional models learn representations of words from text, but are criticized for their lack of grounding, or the linking of text to the non-linguistic world.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?
CompMap first asks a VL model to generate primitive concept activations with text prompts, and then learns to construct a composition model that maps the primitive concept activations (e. g. the likelihood of black tail or red wing) to composite concepts (e. g. a red-winged blackbird).
A Novel Corpus of Discourse Structure in Humans and Computers
We present a novel corpus of 445 human- and computer-generated documents, comprising about 27, 000 clauses, annotated for semantic clause types and coherence relations that allow for nuanced comparison of artificial and natural discourse modes.
Mapping Language Models to Grounded Conceptual Spaces
A fundamental criticism of text-only language models (LMs) is their lack of grounding---that is, the ability to tie a word for which they have learned a representation, to its actual use in the world.
Does Vision-and-Language Pretraining Improve Lexical Grounding?
Linguistic representations derived from text alone have been criticized for their lack of grounding, i. e., connecting words to their meanings in the physical world.
Frequency Effects on Syntactic Rule Learning in Transformers
Pre-trained language models perform well on a variety of linguistic tasks that require symbolic reasoning, raising the question of whether such models implicitly represent abstract symbols and rules.
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color
Pretrained language models have been shown to encode relational information, such as the relations between entities or concepts in knowledge-bases -- (Paris, Capital, France).
Do Prompt-Based Models Really Understand the Meaning of their Prompts?
We find that models learn just as fast with many prompts that are intentionally irrelevant or even pathologically misleading as they do with instructively "good" prompts.
AND does not mean OR: Using Formal Languages to Study Language Models' Representations
A current open question in natural language processing is to what extent language models, which are trained with access only to the form of language, are able to capture the meaning of language.
The MultiBERTs: BERT Reproductions for Robustness Analysis
Experiments with pre-trained models such as BERT are often based on a single checkpoint.
Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering
Through a user preference study, we demonstrate that the oracle behavior of our proposed system that provides responses based on presupposition failure is preferred over the oracle behavior of existing QA systems.
Information-theoretic Probing Explains Reliance on Spurious Features
In this work, we test the hypothesis that the extent to which a feature influences a model's decisions can be predicted using a combination of two factors: The feature's "extractability" after pre-training (measured using information-theoretic probing techniques), and the "evidence" available during fine-tuning (defined as the feature's co-occurrence rate with the label).
Spatial Language Understanding for Object Search in Partially Observed City-scale Environments
We propose SLOOP (Spatial Language Object-Oriented POMDP), a new framework for partially observable decision making with a probabilistic observation model for spatial language.
A Visuospatial Dataset for Naturalistic Verb Learning
We introduce a new dataset for training and evaluating grounded language models.
Measuring and Reducing Gendered Correlations in Pre-trained Models
Pre-trained models have revolutionized natural language understanding.
Self-play for Data Efficient Language Acquisition
When communicating, people behave consistently across conversational roles: People understand the words they say and are able to produce the words they hear.
Are "Undocumented Workers" the Same as "Illegal Aliens"? Disentangling Denotation and Connotation in Vector Spaces
In politics, neologisms are frequently invented for partisan objectives.
Interpretability and Analysis in Neural NLP
While deep learning has transformed the natural language processing (NLP) field and impacted the larger computational linguistics community, the rise of neural networks is stained by their opaque nature: It is challenging to interpret the inner workings of neural network models, and explicate their behavior.
Robot Object Retrieval with Contextual Natural Language Queries
The model takes in a language command containing a verb, for example "Hand me something to cut," and RGB images of candidate objects and selects the object that best satisfies the task specified by the verb.
Does Data Augmentation Improve Generalization in NLP?
Neural models often exploit superficial features to achieve good performance, rather than deriving more general features.
What Happens To BERT Embeddings During Fine-tuning?
While there has been much recent work studying how linguistic information is encoded in pre-trained sentence representations, comparatively little is understood about how these models change when adapted to solve downstream tasks.
How well do NLI models capture verb veridicality?
In natural language inference (NLI), contexts are considered veridical if they allow us to infer that their underlying propositions make true claims about the real world.
INTERNAL-CONSISTENCY CONSTRAINTS FOR EMERGENT COMMUNICATION
When communicating, humans rely on internally-consistent language representations.
Using Grounded Word Representations to Study Theories of Lexical Concepts
The fields of cognitive science and philosophy have proposed many different theories for how humans represent {``}concepts{''}.
Planning with State Abstractions for Non-Markovian Task Specifications
Often times, we specify tasks for a robot using temporal language that can also span different levels of abstraction.
What do you learn from context? Probing for sentence structure in contextualized word representations
The jiant toolkit for general-purpose text understanding models
BERT Rediscovers the Classical NLP Pipeline
Pre-trained text encoders have rapidly advanced the state of the art on many NLP tasks.
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling
Work on the problem of contextualized word representation—the development of reusable neural network components for sentence understanding—has recently seen a surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo (Peters et al., 2018).
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
Our results show that pretraining on language modeling performs the best on average across our probing tasks, supporting its widespread use for pretraining state-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably.
Inherent Disagreements in Human Textual Inferences
We analyze human{'}s disagreements about the validity of natural language inferences.
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics.
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
Natural language understanding has recently seen a surge of progress with the use of sentence encoders like ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2019) which are pretrained on variants of language modeling.
Learning Scalar Adjective Intensity from Paraphrases
Adjectives like {``}warm{''}, {``}hot{''}, and {``}scalding{''} all describe temperature but differ in intensity.
WikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and Discourse
We release a corpus of 43 million atomic edits across 8 languages.
Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation
We present a large-scale collection of diverse natural language inference (NLI) datasets that help provide insight into how well a sentence representation captures distinct types of reasoning.
Identifying 1950s American Jazz Musicians: Fine-Grained IsA Extraction via Modifier Composition
We present a method for populating fine-grained classes (e. g., {``}1950s American jazz musicians{''}) with instances (e. g., Charles Mingus ).
Optimizing Statistical Machine Translation for Text Simplification
Most recent sentence simplification systems use basic machine translation models to learn lexical and syntactic paraphrases from a manually simplified parallel corpus.
Ranked #7 on
Text Simplification
on TurkCorpus
An Empirical Analysis of Formality in Online Communication
This paper presents an empirical study of linguistic formality.
The Language Demographics of Amazon Mechanical Turk
We present a large scale study of the languages spoken by bilingual workers on Mechanical Turk (MTurk).
