Search Results for author:
Found 22 papers, 3 papers with code
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
Massively multilingual transformers (MMTs) pretrained via language modeling (e. g., mBERT, XLM-R) have become a default paradigm for zero-shot language transfer in NLP, offering unmatched transfer performance.
Multilingual ELMo and the Effects of Corpus Sampling
Multilingual pretrained language models are rapidly gaining popularity in NLP systems for non-English languages.
Word Order Does Matter and Shuffled Language Models Know It
Recent studies have shown that language models pretrained and/or fine-tuned on randomly permuted sentences exhibit competitive performance on GLUE, putting into question the importance of word order information.
A Closer Look at Parameter Contributions When Training Neural Language and Translation Models
We analyze the learning dynamics of neural language and translation models using Loss Change Allocation (LCA), an indicator that enables a fine-grained analysis of parameter updates when optimizing for the loss function.
Word Order and World Knowledge
Word order is an important concept in natural language, and in this work, we study how word order affects the induction of world knowledge from raw text using language models.
Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings
We introduce a novel suite of state-of-the-art bilingual text embedding models that are designed to support English and another target language.
Word Order Does Matter (And Shuffled Language Models Know It)
Recent studies have shown that language models pretrained and/or fine-tuned on randomly permuted sentences exhibit competitive performance on GLUE, putting into question the importance of word order information.
The Impact of Positional Encodings on Multilingual Compression
In order to preserve word-order information in a non-autoregressive setting, transformer architectures tend to include positional knowledge, by (for instance) adding positional encodings to token embeddings.
Attention Can Reflect Syntactic Structure (If You Let It)
Since the popularization of the Transformer as a general-purpose feature encoder for NLP, many studies have attempted to decode linguistic structure from its novel multi-head attention mechanism.
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations
Large-scale pretrained language models are the major driving force behind recent improvements in performance on the Winograd Schema Challenge, a widely employed test of common sense reasoning ability.
From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
Massively multilingual transformers pretrained with language modeling objectives (e. g., mBERT, XLM-R) have become a de facto default transfer paradigm for zero-shot cross-lingual transfer in NLP, offering unmatched transfer performance.
Do Neural Language Models Show Preferences for Syntactic Formalisms?
Recent work on the interpretability of deep neural language models has concluded that many properties of natural language syntax are encoded in their representational spaces.
A Systematic Comparison of Architectures for Document-Level Sentiment Classification
Documents are composed of smaller pieces - paragraphs, sentences, and tokens - that have complex relationships between one another.
Multilingual Probing of Deep Pre-Trained Contextual Encoders
Encoders that generate representations based on context have, in recent years, benefited from adaptations that allow for pre-training on large text corpora.
Probing Multilingual Sentence Representations With X-Probe
This paper extends the task of probing sentence representations for linguistic insight in a multilingual domain.
The WMT'18 Morpheval test suites for English-Czech, English-German, English-Finnish and Turkish-English
Progress in the quality of machine translation output calls for new automatic evaluation procedures and metrics.
What can we learn from Semantic Tagging?
We investigate the effects of multi-task learning using the recently introduced task of semantic tagging.
Discriminator at SemEval-2018 Task 10: Minimally Supervised Discrimination
We participated to the SemEval-2018 shared task on capturing discriminative attributes (Task 10) with a simple system that ranked 8th amongst the 26 teams that took part in the evaluation.
A prototype dependency treebank for Breton
This paper describes the development of the first syntactically-annotated corpus of Breton.
