Search Results for author:
Found 11 papers, 4 papers with code
When classifying grammatical role, BERT doesn’t care about word order... except when it matters
Because meaning can often be inferred from lexical semantics alone, word order is often a redundant cue in natural language.
Pretrain on just structure: Understanding linguistic inductive biases using transfer learning
What inductive learning biases make this learning possible?
Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models
Here we show another problem with multilingual models: grammatical structures in higher-resource languages bleed into lower-resource languages, a phenomenon we call grammatical structure bias.
When classifying grammatical role, BERT doesn't care about word order... except when it matters
Because meaning can often be inferred from lexical semantics alone, word order is often a redundant cue in natural language.
Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies
Little is known about what makes cross-lingual transfer hard, since factors like tokenization, morphology, and syntax all change at once between languages.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages.
Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT
Further examining the characteristics that our classifiers rely on, we find that features such as passive voice, animacy and case strongly correlate with classification decisions, suggesting that mBERT does not encode subjecthood purely syntactically, but that subjecthood embedding is continuous and dependent on semantic and discourse factors, as is proposed in much of the functional linguistics literature.
Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
We propose transfer learning as a method for analyzing the encoding of grammatical structure in neural language models.
