Search Results for author:
Found 16 papers, 4 papers with code
Can Transformers Process Recursive Nested Constructions, Like Humans?
A recent study evaluated recursive processing in recurrent neural language models (RNN-LMs) and showed that such models perform below chance level on embedded dependencies within nested constructions – a prototypical example of recursion in natural language.
Meta-Learning Neural Mechanisms rather than Bayesian Priors
Most surprisingly, we find that meta-training on a single formal language can provide as much improvement to a model as meta-training on 5000 different formal languages, provided that the formal language incentivizes the learning of useful neural mechanisms.
Disentanglement and Compositionality of Letter Identity and Letter Position in Variational Auto-Encoder Vision Models
Specifically, we trained beta variational autoencoder ($\beta$-VAE) to reconstruct images of letter strings and evaluated their disentanglement performance using CompOrth - a new benchmark that we created for studying compositional learning and zero-shot generalization in visual models for orthography.
A polar coordinate system represents syntax in large language models
However, this syntactic code remains incomplete: the distance between the Structural Probe word embeddings can represent the existence but not the type and direction of syntactic relations.
What Makes Two Language Models Think Alike?
Do architectural differences significantly affect the way models represent and process language?
Metric-Learning Encoding Models Identify Processing Profiles of Linguistic Features in BERT's Representations
Together, this demonstrates the utility of Metric-Learning Encoding Methods for studying how linguistic features are neurally encoded in language models and the advantage of MLEMs over traditional methods.
Language acquisition: do children and language models follow similar learning stages?
To investigate this, we here compare the learning trajectories of deep language models to those of children.
Probing Brain Context-Sensitivity with Masked-Attention Generation
Two fundamental questions in neurolinguistics concerns the brain regions that integrate information beyond the lexical level, and the size of their window of integration.
Information-Restricted Neural Language Models Reveal Different Brain Regions' Sensitivity to Semantics, Syntax and Context
A fundamental question in neurolinguistics concerns the brain regions involved in syntactic and semantic processing during speech comprehension, both at the lexical (word processing) and supra-lexical levels (sentence and discourse processing).
Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps
Neural Language Models (NLMs) have made tremendous advances during the last years, achieving impressive performance on various linguistic tasks.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans
A recent study evaluated recursive processing in recurrent neural language models (RNN-LMs) and showed that such models perform below chance level on embedded dependencies within nested constructions -- a prototypical example of recursion in natural language.
Can RNNs learn Recursive Nested Subject-Verb Agreements?
Finally, probing the internal states of the model during the processing of sentences with nested tree structures, we found a complex encoding of grammatical agreement information (e. g. grammatical number), in which all the information for multiple words nouns was carried by a single unit.
Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans
We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing, namely the storing of grammatical number and gender information in working memory and its use in long-distance agreement (e. g., capturing the correct number agreement between subject and verb when they are separated by other phrases).
The emergence of number and syntax units in LSTM language models
Importantly, the behaviour of these units is partially controlled by other units independently shown to track syntactic structure.
Metric Learning for Phoneme Perception
This study presents a new framework for learning a metric function for perceptual distances among pairs of phonemes.
