Search Results for author:
Found 18 papers, 8 papers with code
[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
The big changes for this year's competition are as follows: First, we replace the loose track with a paper track, which allows (for example) non-model-based submissions, novel cognitively-inspired benchmarks, or analysis techniques.
WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words
Training on multiple modalities of input can augment the capabilities of a language model.
Quantifying the redundancy between prosody and text
Using a large spoken corpus of English audiobooks, we extract prosodic features aligned to individual words and test how well they can be predicted from LLM embeddings, compared to non-contextual word embeddings.
Controlled Text Generation with Natural Language Instructions
Large language models generate fluent texts and can follow natural language instructions to solve a wide range of tasks without task-specific training.
Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
In partnership with CoNLL and CMCL, we provide a platform for approaches to pretraining with a limited-size corpus sourced from data inspired by the input to children.
A Targeted Assessment of Incremental Processing in Neural Language Models and Humans
We present a targeted, scaled-up comparison of incremental processing in humans and neural language models by collecting by-word reaction time data for sixteen different syntactic test suites across a range of structural phenomena.
Investigating Novel Verb Learning in BERT: Selectional Preference Classes and Alternation-Based Syntactic Generalization
Previous studies investigating the syntactic abilities of deep learning models have not targeted the relationship between the strength of the grammatical generalization and the amount of evidence to which the model is exposed during training.
Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models
Humans can learn structural properties about a word from minimal experience, and deploy their learned syntactic representations uniformly in different grammatical contexts.
SyntaxGym: An Online Platform for Targeted Evaluation of Language Models
Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models.
A Systematic Assessment of Syntactic Generalization in Neural Language Models
While state-of-the-art neural network models continue to achieve lower perplexity scores on language modeling benchmarks, it remains unknown whether optimizing for broad-coverage predictive performance leads to human-like syntactic knowledge.
Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study
We assess whether different neural language models trained on English and French represent phrase-level number and gender features, and use those features to drive downstream expectations.
Hierarchical Representation in Neural Language Models: Suppression and Recovery of Expectations
Deep learning sequence models have led to a marked increase in performance for a range of Natural Language Processing tasks, but it remains an open question whether they are able to induce proper hierarchical generalizations for representing natural language from linear input alone.
What Syntactic Structures block Dependencies in RNN Language Models?
Here, we provide new evidence that RNN language models are sensitive to hierarchical syntactic structure by investigating the filler--gap dependency and constraints on it, known as syntactic islands.
Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State
We deploy the methods of controlled psycholinguistic experimentation to shed light on the extent to which the behavior of neural network language models reflects incremental representations of syntactic state.
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies
State-of-the-art LSTM language models trained on large corpora learn sequential contingencies in impressive detail and have been shown to acquire a number of non-local grammatical dependencies with some success.
What do RNN Language Models Learn about Filler--Gap Dependencies?
RNN language models have achieved state-of-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn.
RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency
Recurrent neural networks (RNNs) are the state of the art in sequence modeling for natural language.
What do RNN Language Models Learn about Filler-Gap Dependencies?
RNN language models have achieved state-of-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn.
