Search Results for author: Hinrich Schütze

Found 225 papers, 116 papers with code

Wine is not v i n. On the Compatibility of Tokenizations across Languages

no code implementations Findings (EMNLP) 2021 Antonis Maronikolakis, Philipp Dufter, Hinrich Schütze

The size of the vocabulary is a central design choice in large pretrained language models, with respect to both performance and memory requirements.

Separating Hate Speech and Offensive Language Classes via Adversarial Debiasing

1 code implementation NAACL (WOAH) 2022 Shuzhou Yuan, Antonis Maronikolakis, Hinrich Schütze

Research to tackle hate speech plaguing online media has made strides in providing solutions, analyzing bias and curating data.

Multidomain Pretrained Language Models for Green NLP

1 code implementation EACL (AdaptNLP) 2021 Antonis Maronikolakis, Hinrich Schütze

Thus, instead of training multiple models, we can train a single multidomain model saving on computational resources and training time.

Domain Adaptation

Don’t Forget Cheap Training Signals Before Building Unsupervised Bilingual Word Embeddings

no code implementations LREC (BUCC) 2022 Silvia Severini, Viktor Hangya, Masoud Jalili Sabet, Alexander Fraser, Hinrich Schütze

The two approaches we find most effective are: 1) using identical words as seed lexicons (which unsupervised approaches incorrectly assume are not available for orthographically distinct language pairs) and 2) combining such lexicons with pairs extracted by matching romanized versions of words with an edit distance threshold.

Cross-Lingual Transfer Word Embeddings

Few-Shot Text Generation with Natural Language Instructions

no code implementations EMNLP 2021 Timo Schick, Hinrich Schütze

Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion.

Headline Generation text-classification +1

GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages

2 code implementations31 Oct 2024 Amir Hossein Kargaran, François Yvon, Hinrich Schütze

The need for large text corpora has increased with the advent of pretrained language models and, in particular, the discovery of scaling laws for these models.

Language Identification

MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment

1 code implementation8 Oct 2024 Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, Hinrich Schütze

This suggests that MEXA is a reliable method for estimating the multilingual capabilities of English-centric LLMs, providing a clearer understanding of their multilingual potential and the inner workings of LLMs.

ARC Belebele +1

Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization

no code implementations3 Oct 2024 Mingyang Wang, Lukas Lange, Heike Adel, Jannik Strötgen, Hinrich Schütze

Evaluations on three model editing benchmarks show that SAUL is a practical and reliable solution for model editing outperforming state-of-the-art methods while maintaining generation quality and reducing computational overhead.

Language Modelling Model Editing +1

EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

1 code implementation26 Sep 2024 Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, Barry Haddow

In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages.

Cross-Lingual Transfer Language Modelling

LangSAMP: Language-Script Aware Multilingual Pretraining

1 code implementation26 Sep 2024 Yihong Liu, Haotian Ye, Chunlan Ma, Mingyang Wang, Hinrich Schütze

However, this removal increases the burden on token embeddings to encode all language-specific information, which may hinder the model's ability to produce more language-neutral representations.

Continual Pretraining Representation Learning +1

How Transliterations Improve Crosslingual Alignment

no code implementations25 Sep 2024 Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Ayyoob Imani, Orgest Xhelili, Haotian Ye, Chunlan Ma, François Yvon, Hinrich Schütze

However, we also show that better alignment does not always yield better downstream performance, suggesting that further research is needed to clarify the connection between alignment and performance.

Sentence Transliteration

CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation

1 code implementation3 Sep 2024 Ingo Ziegler, Abdullatif Köksal, Desmond Elliott, Hinrich Schütze

Building high-quality datasets for specialized tasks is a time-consuming and resource-intensive process that often requires specialized domain knowledge.

Question Answering Retrieval

SYNTHEVAL: Hybrid Behavioral Testing of NLP Models with Synthetic CheckLists

1 code implementation30 Aug 2024 Raoyuan Zhao, Abdullatif Köksal, Yihong Liu, Leonie Weissweiler, Anna Korhonen, Hinrich Schütze

In this work, we propose SYNTHEVAL, a hybrid behavioral testing framework that leverages large language models (LLMs) to generate a wide range of test types for a comprehensive evaluation of NLP models.

Benchmarking Sentiment Analysis

ChatZero:Zero-shot Cross-Lingual Dialogue Generation via Pseudo-Target Language

no code implementations16 Aug 2024 Yongkang Liu, Feng Shi, Daling Wang, Yifei Zhang, Hinrich Schütze

Although large language models(LLMs) show amazing capabilities, among various exciting applications discovered for LLMs fall short in other low-resource languages.

Contrastive Learning Dialogue Generation

TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish

1 code implementation17 Jul 2024 Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, Hinrich Schütze

These questions are written by curriculum experts, suitable for the high-school curricula in Turkey, covering subjects ranging from natural sciences and math questions to more culturally representative topics such as Turkish Literature and the history of the Turkish Republic.

Math Multiple-choice +1

Consistent Document-Level Relation Extraction via Counterfactuals

1 code implementation9 Jul 2024 Ali Modarressi, Abdullatif Köksal, Hinrich Schütze

We first demonstrate that models trained on factual data exhibit inconsistent behavior: while they accurately extract triples from factual data, they fail to extract the same triples after counterfactual modification.

counterfactual Document-level Relation Extraction +1

Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scripts

no code implementations2 Jul 2024 Chunlan Ma, Yihong Liu, Haotian Ye, Hinrich Schütze

Inspired by recent work that leverages transliteration in encoder-only models, we investigate whether transliteration is also effective in improving LLMs' performance for low-resource languages written in non-Latin scripts.

Decoder In-Context Learning +3

A Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models

no code implementations29 Jun 2024 Peiqin Lin, André F. T. Martins, Hinrich Schütze

Recent studies have highlighted the potential of exploiting parallel corpora to enhance multilingual large language models, improving performance in both bilingual tasks, e. g., machine translation, and general-purpose tasks, e. g., text classification.

Language Identification Machine Translation +4

Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment

1 code implementation28 Jun 2024 Orgest Xhelili, Yihong Liu, Hinrich Schütze

However, the transfer performance is often hindered when a low-resource target language is written in a different script than the high-resource source language, even though the two languages may be related or share parts of their vocabularies.

Cross-Lingual Transfer Transliteration

Learn it or Leave it: Module Composition and Pruning for Continual Learning

no code implementations26 Jun 2024 Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schütze

In real-world environments, continual learning is essential for machine learning models, as they need to acquire new knowledge incrementally without forgetting what they have already learned.

Continual Learning Transfer Learning

BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning

no code implementations25 Jun 2024 Ercong Nie, Bo Shao, Zifeng Ding, Mingyang Wang, Helmut Schmid, Hinrich Schütze

Large language models (LLMs) possess extensive parametric knowledge, but this knowledge is difficult to update with new information because retraining is very expensive and infeasible for closed-source models.

In-Context Learning knowledge editing +1

A Unified Data Augmentation Framework for Low-Resource Multi-Domain Dialogue Generation

1 code implementation14 Jun 2024 Yongkang Liu, Ercong Nie, Shi Feng, Zheng Hua, Zifeng Ding, Daling Wang, Yifei Zhang, Hinrich Schütze

We conduct experiments on Chinese dialogue datasets from five different domains and show that AMD$^2$G achieves superior performance compared to both direct training on the target domain corpus and collective training on all five domain corpora.

Data Augmentation Dialogue Generation +1

MaskLID: Code-Switching Language Identification through Iterative Masking

1 code implementation10 Jun 2024 Amir Hossein Kargaran, François Yvon, Hinrich Schütze

This method uses the LID itself to identify the features that require masking and does not rely on any external resource.

Language Identification Sentence

Joint Lemmatization and Morphological Tagging with LEMMING

no code implementations EMNLP 2015 Thomas Muller, Ryan Cotterell, Alexander Fraser, Hinrich Schütze

We present LEMMING, a modular log-linear model that jointly models lemmatization and tagging and supports the integration of arbitrary global features.

Lemmatization Morphological Tagging

TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data

1 code implementation16 May 2024 Yihong Liu, Chunlan Ma, Haotian Ye, Hinrich Schütze

We applied TransMI to three recent strong mPLMs, and our experiments demonstrate that TransMI not only preserves their ability to handle non-transliterated data, but also enables the models to effectively process transliterated data: the results show a consistent improvement of 3% to 34%, varying across different models and tasks.

Transliteration

XAMPLER: Learning to Retrieve Cross-Lingual In-Context Examples

1 code implementation8 May 2024 Peiqin Lin, André F. T. Martins, Hinrich Schütze

Thus, we introduce XAMPLER: Cross-Lingual Example Retrieval, a method tailored to tackle the challenge of cross-lingual in-context learning using only annotated English data.

In-Context Learning Language Modelling +5

MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory

no code implementations17 Apr 2024 Ali Modarressi, Abdullatif Köksal, Ayyoob Imani, Mohsen Fayyaz, Hinrich Schütze

While current large language models (LLMs) demonstrate some capabilities in knowledge-intensive tasks, they are limited by relying on their parameters as an implicit storage mechanism.

Hallucination Language Modelling +3

Labeled Morphological Segmentation with Semi-Markov Models

no code implementations CONLL 2015 Ryan Cotterell, Thomas Müller, Alexander Fraser, Hinrich Schütze

We present labeled morphological segmentation, an alternative view of morphological processing that unifies several tasks.

Segmentation TAG

Verbing Weirds Language (Models): Evaluation of English Zero-Derivation in Five LLMs

no code implementations26 Mar 2024 David R. Mortensen, Valentina Izrailevitch, Yunze Xiao, Hinrich Schütze, Leonie Weissweiler

We find that GPT-4 performs best on the task, followed by GPT-3. 5, but that the open source language models are also able to perform it and that the 7B parameter Mistral displays as little difference between its baseline performance on the natural language inference task and the non-prototypical syntactic category task, as the massive GPT-4.

Natural Language Inference

Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons

1 code implementation26 Mar 2024 Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze, David R. Mortensen, Lori Levin

In this paper, we make a contribution that can be understood from two perspectives: from an NLP perspective, we introduce a small challenge dataset for NLI with large lexical overlap, which minimises the possibility of models discerning entailment solely based on token distinctions, and show that GPT-4 and Llama 2 fail it with strong bias.

MaiBaam: A Multi-Dialectal Bavarian Universal Dependency Treebank

1 code implementation15 Mar 2024 Verena Blaschke, Barbara Kovačić, Siyao Peng, Hinrich Schütze, Barbara Plank

Despite the success of the Universal Dependencies (UD) project exemplified by its impressive language breadth, there is still a lack in `within-language breadth': most treebanks focus on standard languages.

POS POS Tagging

Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena

no code implementations11 Mar 2024 Leonie Weissweiler, Abdullatif Köksal, Hinrich Schütze

Argument Structure Constructions (ASCs) are one of the most well-studied construction groups, providing a unique opportunity to demonstrate the usefulness of Construction Grammar (CxG).

Dependency Parsing Sentence

Decomposed Prompting: Unveiling Multilingual Linguistic Structure Knowledge in English-Centric Large Language Models

no code implementations28 Feb 2024 Ercong Nie, Shuzhou Yuan, Bolei Ma, Helmut Schmid, Michael Färber, Frauke Kreuter, Hinrich Schütze

Despite the predominance of English in their training data, English-centric Large Language Models (LLMs) like GPT-3 and LLaMA display a remarkable ability to perform multilingual tasks, raising questions about the depth and nature of their cross-lingual capabilities.

Part-Of-Speech Tagging Sentence

Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models

1 code implementation26 Feb 2024 Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, Dirk Hovy

Motivated by this discrepancy, we challenge the prevailing constrained evaluation paradigm for values and opinions in LLMs and explore more realistic unconstrained evaluations.

Multiple-choice

ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks

1 code implementation29 Jan 2024 Bolei Ma, Ercong Nie, Shuzhou Yuan, Helmut Schmid, Michael Färber, Frauke Kreuter, Hinrich Schütze

However, most previous studies primarily focused on sentence-level classification tasks, and only a few considered token-level labeling tasks such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging.

Benchmarking In-Context Learning +8

HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy

1 code implementation26 Jan 2024 Yongkang Liu, Yiqun Zhang, Qian Li, Tong Liu, Shi Feng, Daling Wang, Yifei Zhang, Hinrich Schütze

As LMs grow in size, fine-tuning the full parameters of LMs requires a prohibitively large amount of GPU memory.

parameter-efficient fine-tuning

TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models

1 code implementation12 Jan 2024 Yihong Liu, Chunlan Ma, Haotian Ye, Hinrich Schütze

As a consequence, mPLMs are faced with a script barrier: representations from different scripts are located in different subspaces, which can result in crosslingual transfer involving languages of different scripts performing suboptimally.

Contrastive Learning Transliteration

MoSECroT: Model Stitching with Static Word Embeddings for Crosslingual Zero-shot Transfer

no code implementations9 Jan 2024 Haotian Ye, Yihong Liu, Chunlan Ma, Hinrich Schütze

In this paper, we introduce MoSECroT Model Stitching with Static Word Embeddings for Crosslingual Zero-shot Transfer), a novel and challenging task that is especially relevant to low-resource languages for which static word embeddings are available.

Word Embeddings

Multilingual Word Embeddings for Low-Resource Languages using Anchors and a Chain of Related Languages

no code implementations21 Nov 2023 Viktor Hangya, Silvia Severini, Radoslav Ralev, Alexander Fraser, Hinrich Schütze

In this paper, we propose to build multilingual word embeddings (MWEs) via a novel language chain-based approach, that incorporates intermediate related languages to bridge the gap between the distant source and target.

Bilingual Lexicon Induction Multilingual NLP +1

OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining

1 code implementation15 Nov 2023 Yihong Liu, Peiqin Lin, Mingyang Wang, Hinrich Schütze

Instead of pretraining multilingual language models from scratch, a more efficient method is to adapt existing pretrained language models (PLMs) to new languages via vocabulary extension and continued pretraining.

Language Modelling Multilingual Word Embeddings

GlotLID: Language Identification for Low-Resource Languages

3 code implementations24 Oct 2023 Amir Hossein Kargaran, Ayyoob Imani, François Yvon, Hinrich Schütze

Several recent papers have published good solutions for language identification (LID) for about 300 high-resource and medium-resource languages.

Dialect Identification

GradSim: Gradient-Based Language Grouping for Effective Multilingual Training

no code implementations23 Oct 2023 Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schütze

However, not all languages positively influence each other and it is an open research question how to select the most suitable set of languages for multilingual training and avoid negative interference among languages whose characteristics or data distributions are not compatible.

Sentiment Analysis

Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration

1 code implementation8 Oct 2023 Ercong Nie, Helmut Schmid, Hinrich Schütze

Pretrained multilingual encoder models can directly perform zero-shot multilingual tasks or linguistic probing by reformulating the input examples into cloze-style prompts.

Position

GlotScript: A Resource and Tool for Low Resource Writing System Identification

1 code implementation23 Sep 2023 Amir Hossein Kargaran, François Yvon, Hinrich Schütze

We present GlotScript, an open resource and tool for low resource writing system identification.

Language Modelling

Cross-Lingual Constituency Parsing for Middle High German: A Delexicalized Approach

no code implementations9 Aug 2023 Ercong Nie, Helmut Schmid, Hinrich Schütze

However, training an automatic syntactic analysis system for ancient languages solely relying on annotated parse data is a formidable task due to the inherent challenges in building treebanks for such languages.

Constituency Parsing Cross-Lingual Transfer

Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding

1 code implementation15 Jul 2023 Bolei Ma, Ercong Nie, Helmut Schmid, Hinrich Schütze

We conduct comprehensive experiments on diverse cross-lingual language understanding tasks (sentiment classification, paraphrase identification, and natural language inference) and empirically analyze the variation trends of prompt-based finetuning performance in cross-lingual transfer across different few-shot and full-data settings.

Natural Language Inference Natural Language Understanding +4

On the Copying Problem of Unsupervised NMT: A Training Schedule with a Language Discriminator Loss

1 code implementation26 May 2023 Yihong Liu, Alexandra Chronopoulou, Hinrich Schütze, Alexander Fraser

By conducting extensive experiments on different language pairs, including similar and distant, high and low-resource languages, we find that our method alleviates the copying problem, thus improving the translation performance on low-resource languages.

Machine Translation NMT +2

Evaluate What You Can't Evaluate: Unassessable Quality for Generated Response

no code implementations24 May 2023 Yongkang Liu, Shi Feng, Daling Wang, Yifei Zhang, Hinrich Schütze

There are risks in using eference-free evaluators based on LLMs to evaluate the quality of dialogue responses.

Dialogue Generation

mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models

1 code implementation23 May 2023 Peiqin Lin, Chengzhi Hu, Zheyu Zhang, André F. T. Martins, Hinrich Schütze

Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining.

Open-Ended Question Answering Zero-Shot Cross-Lingual Transfer

RET-LLM: Towards a General Read-Write Memory for Large Language Models

1 code implementation23 May 2023 Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, Hinrich Schütze

Large language models (LLMs) have significantly advanced the field of natural language processing (NLP) through their extensive parameters and comprehensive data utilization.

Question Answering

A study of conceptual language similarity: comparison and evaluation

no code implementations22 May 2023 Haotian Ye, Yihong Liu, Hinrich Schütze

An interesting line of research in natural language processing (NLP) aims to incorporate linguistic typology to bridge linguistic diversity and assist the research of low-resource languages.

Binary Classification Diversity

Language-Agnostic Bias Detection in Language Models with Bias Probing

1 code implementation22 May 2023 Abdullatif Köksal, Omer Faruk Yalcin, Ahmet Akbiyik, M. Tahir Kilavuz, Anna Korhonen, Hinrich Schütze

For nationality as a case study, we show that LABDet `surfaces' nationality bias by training a classifier on top of a frozen PLM on non-nationality sentiment detection.

Bias Detection

Taxi1500: A Multilingual Dataset for Text Classification in 1500 Languages

1 code implementation15 May 2023 Chunlan Ma, Ayyoob ImaniGooghari, Haotian Ye, Renhao Pei, Ehsaneddin Asgari, Hinrich Schütze

While natural language processing tools have been developed extensively for some of the world's languages, a significant portion of the world's over 7000 languages are still neglected.

text-classification Text Classification

A Crosslingual Investigation of Conceptualization in 1335 Languages

3 code implementations15 May 2023 Yihong Liu, Haotian Ye, Leonie Weissweiler, Philipp Wicke, Renhao Pei, Robert Zangenfeind, Hinrich Schütze

The resulting measure for the conceptual similarity of two languages is complementary to standard genealogical, typological, and surface similarity measures.

NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysis

no code implementations28 Apr 2023 Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schütze

In this work, we propose to leverage language-adaptive and task-adaptive pretraining on African texts and study transfer learning with source language selection on top of an African language-centric pretrained language model.

Language Modelling Sentiment Analysis +1

Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages

6 code implementations20 Apr 2023 Verena Blaschke, Hinrich Schütze, Barbara Plank

This can for instance be observed when finetuning PLMs on one language and evaluating them on data in a closely related language variety with no standardized orthography.

Cross-Lingual Transfer Part-Of-Speech Tagging +2

A Survey of Corpora for Germanic Low-Resource Languages and Dialects

2 code implementations19 Apr 2023 Verena Blaschke, Hinrich Schütze, Barbara Plank

In this work, we instead focus on low-resource languages and in particular non-standardized low-resource languages.

Survey

Sociocultural knowledge is needed for selection of shots in hate speech detection tasks

no code implementations4 Apr 2023 Antonis Maronikolakis, Abdullatif Köksal, Hinrich Schütze

We introduce HATELEXICON, a lexicon of slurs and targets of hate speech for the countries of Brazil, Germany, India and Kenya, to aid training and interpretability of models.

Few-Shot Learning Hate Speech Detection

MenuCraft: Interactive Menu System Design with Large Language Models

1 code implementation8 Mar 2023 Amir Hossein Kargaran, Nafiseh Nikeghbal, Abbas Heydarnoori, Hinrich Schütze

Menu system design for user interfaces is a challenging task involving many design options and various human factors.

Few-Shot Learning In-Context Learning

Construction Grammar Provides Unique Insight into Neural Language Models

no code implementations4 Feb 2023 Leonie Weissweiler, Taiqi He, Naoki Otani, David R. Mortensen, Lori Levin, Hinrich Schütze

Construction Grammar (CxG) has recently been used as the basis for probing studies that have investigated the performance of large pretrained language models (PLMs) with respect to the structure and meaning of constructions.

Position

Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages

1 code implementation19 Dec 2022 Ercong Nie, Sheng Liang, Helmut Schmid, Hinrich Schütze

Multilingual Pretrained Language Models (MPLMs) have shown their strong multilinguality in recent empirical cross-lingual transfer studies.

Cross-Lingual Transfer Natural Language Inference +3

MEAL: Stable and Active Learning for Few-Shot Prompting

1 code implementation15 Nov 2022 Abdullatif Köksal, Timo Schick, Hinrich Schütze

Few-shot classification has made great strides due to foundation models that, through priming and prompting, are highly effective few-shot learners.

Active Learning Few-Shot Learning +1

Graph-Based Multilingual Label Propagation for Low-Resource Part-of-Speech Tagging

1 code implementation18 Oct 2022 Ayyoob Imani, Silvia Severini, Masoud Jalili Sabet, François Yvon, Hinrich Schütze

An established method for training a POS tagger in such a scenario is to create a labeled training set by transferring from high-resource languages.

Graph Neural Network Part-Of-Speech Tagging +4

Federated Continual Learning for Text Classification via Selective Inter-client Transfer

1 code implementation12 Oct 2022 Yatin Chaudhary, Pranav Rai, Matthias Schubert, Hinrich Schütze, Pankaj Gupta

The objective of Federated Continual Learning (FCL) is to improve deep learning models over life time at each client by (relevant and efficient) knowledge transfer without sharing data.

Continual Learning Federated Learning +3

Modeling Content-Emotion Duality via Disentanglement for Empathetic Conversation

1 code implementation26 Sep 2022 Peiqin Lin, Jiashuo Wang, Hinrich Schütze, Wenjie Li

To solve the task, it is essential to model the content-emotion duality of a dialogue, which is composed of the content view (i. e., what personal experiences are described) and the emotion view (i. e., the feelings of the speaker on these experiences).

Disentanglement Empathetic Response Generation +1

Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions

no code implementations28 Jul 2022 Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, Yoav Goldberg

Our causal framework and our results demonstrate the importance of studying datasets and the benefits of causality for understanding NLP models.

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

4 code implementations9 Jun 2022 Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, ZiRui Wang, Ziyi Wu

BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.

Common Sense Reasoning Math +1

Don't Forget Cheap Training Signals Before Building Unsupervised Bilingual Word Embeddings

no code implementations31 May 2022 Silvia Severini, Viktor Hangya, Masoud Jalili Sabet, Alexander Fraser, Hinrich Schütze

The two approaches we find most effective are: 1) using identical words as seed lexicons (which unsupervised approaches incorrectly assume are not available for orthographically distinct language pairs) and 2) combining such lexicons with pairs extracted by matching romanized versions of words with an edit distance threshold.

Cross-Lingual Transfer Word Embeddings

Analyzing Hate Speech Data along Racial, Gender and Intersectional Axes

no code implementations NAACL (GeBNLP) 2022 Antonis Maronikolakis, Philip Baader, Hinrich Schütze

To tackle the rising phenomenon of hate speech, efforts have been made towards data curation and analysis.

Flow-Adapter Architecture for Unsupervised Machine Translation

no code implementations ACL 2022 Yihong Liu, Haris Jabbar, Hinrich Schütze

The primary novelties of our model are: (a) capturing language-specific sentence representations separately for each language using normalizing flows and (b) using a simple transformation of these latent representations for translating from one language to another.

NMT Sentence +2

CaMEL: Case Marker Extraction without Labels

1 code implementation ACL 2022 Leonie Weissweiler, Valentin Hofmann, Masoud Jalili Sabet, Hinrich Schütze

We introduce CaMEL (Case Marker Extraction without Labels), a novel and challenging task in computational morphology that is especially relevant for low-resource languages.

ECOLA: Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations

no code implementations17 Mar 2022 Zhen Han, Ruotong Liao, Jindong Gu, Yao Zhang, Zifeng Ding, Yujia Gu, Heinz Köppl, Hinrich Schütze, Volker Tresp

Since conventional knowledge embedding models cannot take full advantage of the abundant textual information, there have been extensive research efforts in enhancing knowledge embedding using texts.

Knowledge Graph Embedding Link Prediction +1

Geographic Adaptation of Pretrained Language Models

1 code implementation16 Mar 2022 Valentin Hofmann, Goran Glavaš, Nikola Ljubešić, Janet B. Pierrehumbert, Hinrich Schütze

While pretrained language models (PLMs) have been shown to possess a plethora of linguistic knowledge, the existing body of research has largely neglected extralinguistic knowledge, which is generally difficult to obtain by pretraining on text alone.

Language Identification Language Modelling +2

Semantic-Oriented Unlabeled Priming for Large-Scale Language Models

no code implementations12 Feb 2022 Yanchen Liu, Timo Schick, Hinrich Schütze

Due to the high costs associated with finetuning large language models, various recent works propose to adapt them to specific tasks without any parameter updates through in-context learning.

In-Context Learning

Towards a Broad Coverage Named Entity Resource: A Data-Efficient Approach for Many Diverse Languages

no code implementations LREC 2022 Silvia Severini, Ayyoob Imani, Philipp Dufter, Hinrich Schütze

Prior work on extracting MNE datasets from parallel corpora required resources such as large monolingual corpora or word aligners that are unavailable or perform poorly for underresourced languages.

Bilingual Lexicon Induction Transliteration

BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief

no code implementations EMNLP 2021 Nora Kassner, Oyvind Tafjord, Hinrich Schütze, Peter Clark

We show that, in a controlled experimental setting, these two mechanisms result in more consistent beliefs in the overall system, improving both the accuracy and consistency of its answers over time.

Language Modelling World Knowledge

Active Learning for Argument Mining: A Practical Approach

no code implementations28 Sep 2021 Nikolai Solmsdorf, Dietrich Trautmann, Hinrich Schütze

Despite considerable recent progress, the creation of well-balanced and diverse resources remains a time-consuming and costly challenge in Argument Mining.

Active Learning Argument Mining

Scene Graph Generation for Better Image Captioning?

no code implementations23 Sep 2021 Maximilian Mozes, Martin Schmitt, Vladimir Golkov, Hinrich Schütze, Daniel Cremers

We investigate the incorporation of visual relationships into the task of supervised image caption generation by proposing a model that leverages detected objects and auto-generated visual relationships to describe images in natural language.

Caption Generation Graph Generation +2

BERT Cannot Align Characters

no code implementations EMNLP (insights) 2021 Antonis Maronikolakis, Philipp Dufter, Hinrich Schütze

We show that the closer two languages are, the better BERT can align them on the character level.

Locating Language-Specific Information in Contextualized Embeddings

1 code implementation16 Sep 2021 Sheng Liang, Philipp Dufter, Hinrich Schütze

Multilingual pretrained language models (MPLMs) exhibit multilinguality and are well suited for transfer across languages.

Wine is Not v i n. -- On the Compatibility of Tokenizations Across Languages

no code implementations13 Sep 2021 Antonis Maronikolakis, Philipp Dufter, Hinrich Schütze

The size of the vocabulary is a central design choice in large pretrained language models, with respect to both performance and memory requirements.

Graph Algorithms for Multiparallel Word Alignment

1 code implementation EMNLP 2021 Ayyoob Imani, Masoud Jalili Sabet, Lütfi Kerem Şenel, Philipp Dufter, François Yvon, Hinrich Schütze

With the advent of end-to-end deep learning approaches in machine translation, interest in word alignments initially decreased; however, they have again become a focus of research more recently.

Link Prediction Machine Translation +3

Discrete and Soft Prompting for Multilingual Models

1 code implementation EMNLP 2021 Mengjie Zhao, Hinrich Schütze

It has been shown for English that discrete and soft prompting perform strongly in few-shot learning with pretrained language models (PLMs).

Few-Shot Learning Natural Language Inference

Continuous Entailment Patterns for Lexical Inference in Context

1 code implementation EMNLP 2021 Martin Schmitt, Hinrich Schütze

If we allow for tokens outside the PLM's vocabulary, patterns can be adapted more flexibly to a PLM's idiosyncrasies.

Few-Shot NLI Lexical Entailment +1

ParCourE: A Parallel Corpus Explorer for a Massively Multilingual Corpus

no code implementations ACL 2021 Ayyoob Imani, Masoud Jalili Sabet, Philipp Dufter, Michael Cysouw, Hinrich Schütze

With more than 7000 languages worldwide, multilingual natural language processing (NLP) is essential both from an academic and commercial perspective.

Multilingual NLP Transfer Learning

Modeling Ideological Salience and Framing in Polarized Online Groups with Graph Neural Networks and Structured Sparsity

1 code implementation Findings (NAACL) 2022 Valentin Hofmann, Xiaowen Dong, Janet B. Pierrehumbert, Hinrich Schütze

The increasing polarization of online political discourse calls for computational tools that automatically detect and monitor ideological divides in social media.

Multi-source Neural Topic Modeling in Multi-view Embedding Spaces

1 code implementation NAACL 2021 Pankaj Gupta, Yatin Chaudhary, Hinrich Schütze

Though word embeddings and topics are complementary representations, several past works have only used pretrained word embeddings in (neural) topic modeling to address data sparsity in short-text or small collection of documents.

Information Retrieval Retrieval +1

Generating Datasets with Pretrained Language Models

2 code implementations EMNLP 2021 Timo Schick, Hinrich Schütze

To obtain high-quality sentence embeddings from pretrained language models (PLMs), they must either be augmented with additional pretraining objectives or finetuned on a large set of labeled text pairs.

Semantic Textual Similarity Sentence +1

Static Embeddings as Efficient Knowledge Bases?

1 code implementation NAACL 2021 Philipp Dufter, Nora Kassner, Hinrich Schütze

Recent research investigates factual knowledge stored in large pretrained language models (PLMs).

Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP

3 code implementations28 Feb 2021 Timo Schick, Sahana Udupa, Hinrich Schütze

In this paper, we first demonstrate a surprising finding: pretrained language models recognize, to a considerable degree, their undesirable biases and the toxicity of the content they produce.

Language Modelling

Language Models for Lexical Inference in Context

1 code implementation EACL 2021 Martin Schmitt, Hinrich Schütze

Lexical inference in context (LIiC) is the task of recognizing textual entailment between two very similar sentences, i. e., sentences that only differ in one expression.

Few-Shot NLI Natural Language Inference

Improving Scene Graph Classification by Exploiting Knowledge from Texts

no code implementations9 Feb 2021 Sahand Sharifzadeh, Sina Moayed Baharlou, Martin Schmitt, Hinrich Schütze, Volker Tresp

We show that by fine-tuning the classification pipeline with the extracted knowledge from texts, we can achieve ~8x more accurate results in scene graph classification, ~3x in object classification, and ~1. 5x in predicate classification, compared to the supervised baselines with only 1% of the annotated images.

General Classification Graph Classification +7

Does He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models

no code implementations6 Feb 2021 Lutfi Kerem Senel, Hinrich Schütze

Recent progress in pretraining language models on large corpora has resulted in large performance gains on many NLP tasks.

Language Modelling

Measuring and Improving Consistency in Pretrained Language Models

1 code implementation1 Feb 2021 Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg

In this paper we study the question: Are Pretrained Language Models (PLMs) consistent with respect to factual knowledge?

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

no code implementations ACL 2021 Mengjie Zhao, Yi Zhu, Ehsan Shareghi, Ivan Vulić, Roi Reichart, Anna Korhonen, Hinrich Schütze

Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT.

Few-Shot Learning

Few-Shot Text Generation with Pattern-Exploiting Training

2 code implementations22 Dec 2020 Timo Schick, Hinrich Schütze

Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion.

Headline Generation text-classification +2

Subword Sampling for Low Resource Word Alignment

no code implementations21 Dec 2020 Ehsaneddin Asgari, Masoud Jalili Sabet, Philipp Dufter, Christopher Ringlstetter, Hinrich Schütze

This method's hypothesis is that the aggregation of different granularities of text for certain language pairs can help word-level alignment.

Bayesian Optimization Machine Translation +1

Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification

2 code implementations COLING 2020 Timo Schick, Helmut Schmid, Hinrich Schütze

A recent approach for few-shot text classification is to convert textual inputs to cloze questions that contain some form of task description, process them with a pretrained language model and map the predicted words to labels.

Few-Shot Text Classification General Classification +2

Dynamic Contextualized Word Embeddings

1 code implementation ACL 2021 Valentin Hofmann, Janet B. Pierrehumbert, Hinrich Schütze

Static word embeddings that represent words by a single vector cannot capture the variability of word meaning in different linguistic and extralinguistic contexts.

Language Modelling Word Embeddings

It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

5 code implementations NAACL 2021 Timo Schick, Hinrich Schütze

When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance.

Natural Language Understanding

Automatic Domain Adaptation Outperforms Manual Domain Adaptation for Predicting Financial Outcomes

no code implementations ACL 2019 Marina Sedinkina, Nikolas Breitkopf, Hinrich Schütze

In our experiments, we demonstrate that the automatically adapted sentiment dictionary outperforms the previous state of the art in predicting the financial outcomes excess return and volatility.

Domain Adaptation

Neural Topic Modeling with Continual Lifelong Learning

1 code implementation ICML 2020 Pankaj Gupta, Yatin Chaudhary, Thomas Runkler, Hinrich Schütze

To address the problem, we propose a lifelong learning framework for neural topic modeling that can continuously process streams of document collections, accumulate topics and guide future topic modeling tasks by knowledge transfer from several sources to better deal with the sparse data.

Data Augmentation Information Retrieval +2

Explainable and Discourse Topic-aware Neural Language Understanding

1 code implementation ICML 2020 Yatin Chaudhary, Hinrich Schütze, Pankaj Gupta

Marrying topic models and language models exposes language understanding to a broader source of document-level context beyond sentences via topics.

Document Classification Language Modelling +5

Unsupervised Embedding-based Detection of Lexical Semantic Changes

no code implementations16 May 2020 Ehsaneddin Asgari, Christoph Ringlstetter, Hinrich Schütze

This paper describes EmbLexChange, a system introduced by the "Life-Language" team for SemEval-2020 Task 1, on unsupervised detection of lexical-semantic changes.

Identifying Necessary Elements for BERT's Multilinguality

1 code implementation1 May 2020 Philipp Dufter, Hinrich Schütze

We aim to identify architectural properties of BERT and linguistic properties of languages that are necessary for BERT to become multilingual.

Masking as an Efficient Alternative to Finetuning for Pretrained Language Models

no code implementations EMNLP 2020 Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, Hinrich Schütze

We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning.

Quantifying the Contextualization of Word Representations with Semantic Class Probing

no code implementations Findings of the Association for Computational Linguistics 2020 Mengjie Zhao, Philipp Dufter, Yadollah Yaghoobzadeh, Hinrich Schütze

Pretrained language models have achieved a new state of the art on many NLP tasks, but there are still many open questions about how and why they work so well.

SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings

3 code implementations Findings of the Association for Computational Linguistics 2020 Masoud Jalili Sabet, Philipp Dufter, François Yvon, Hinrich Schütze

We find that alignments created from embeddings are superior for four and comparable for two language pairs compared to those produced by traditional statistical aligners, even with abundant parallel data; e. g., contextualized embeddings achieve a word alignment F1 for English-German that is 5 percentage points higher than eflomal, a high-quality statistical aligner, trained on 100k parallel sentences.

Machine Translation Multilingual Word Embeddings +3

350