no code implementations • Findings (EMNLP) 2021 • Antonis Maronikolakis, Philipp Dufter, Hinrich Schütze
The size of the vocabulary is a central design choice in large pretrained language models, with respect to both performance and memory requirements.
1 code implementation • NAACL (WOAH) 2022 • Shuzhou Yuan, Antonis Maronikolakis, Hinrich Schütze
Research to tackle hate speech plaguing online media has made strides in providing solutions, analyzing bias and curating data.
1 code implementation • EACL (AdaptNLP) 2021 • Antonis Maronikolakis, Hinrich Schütze
Thus, instead of training multiple models, we can train a single multidomain model saving on computational resources and training time.
no code implementations • LREC (BUCC) 2022 • Silvia Severini, Viktor Hangya, Masoud Jalili Sabet, Alexander Fraser, Hinrich Schütze
The two approaches we find most effective are: 1) using identical words as seed lexicons (which unsupervised approaches incorrectly assume are not available for orthographically distinct language pairs) and 2) combining such lexicons with pairs extracted by matching romanized versions of words with an edit distance threshold.
no code implementations • EMNLP 2021 • Timo Schick, Hinrich Schütze
Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion.
2 code implementations • 31 Oct 2024 • Amir Hossein Kargaran, François Yvon, Hinrich Schütze
The need for large text corpora has increased with the advent of pretrained language models and, in particular, the discovery of scaling laws for these models.
1 code implementation • 8 Oct 2024 • Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, Hinrich Schütze
This suggests that MEXA is a reliable method for estimating the multilingual capabilities of English-centric LLMs, providing a clearer understanding of their multilingual potential and the inner workings of LLMs.
no code implementations • 3 Oct 2024 • Mingyang Wang, Lukas Lange, Heike Adel, Jannik Strötgen, Hinrich Schütze
Evaluations on three model editing benchmarks show that SAUL is a practical and reliable solution for model editing outperforming state-of-the-art methods while maintaining generation quality and reducing computational overhead.
1 code implementation • 26 Sep 2024 • Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, Barry Haddow
In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages.
1 code implementation • 26 Sep 2024 • Yihong Liu, Haotian Ye, Chunlan Ma, Mingyang Wang, Hinrich Schütze
However, this removal increases the burden on token embeddings to encode all language-specific information, which may hinder the model's ability to produce more language-neutral representations.
no code implementations • 25 Sep 2024 • Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Ayyoob Imani, Orgest Xhelili, Haotian Ye, Chunlan Ma, François Yvon, Hinrich Schütze
However, we also show that better alignment does not always yield better downstream performance, suggesting that further research is needed to clarify the connection between alignment and performance.
1 code implementation • 19 Sep 2024 • Abdullatif Köksal, Marion Thaler, Ayyoob Imani, Ahmet Üstün, Anna Korhonen, Hinrich Schütze
Instruction tuning enhances large language models (LLMs) by aligning them with human preferences across diverse tasks.
1 code implementation • 3 Sep 2024 • Ingo Ziegler, Abdullatif Köksal, Desmond Elliott, Hinrich Schütze
Building high-quality datasets for specialized tasks is a time-consuming and resource-intensive process that often requires specialized domain knowledge.
1 code implementation • 30 Aug 2024 • Raoyuan Zhao, Abdullatif Köksal, Yihong Liu, Leonie Weissweiler, Anna Korhonen, Hinrich Schütze
In this work, we propose SYNTHEVAL, a hybrid behavioral testing framework that leverages large language models (LLMs) to generate a wide range of test types for a comprehensive evaluation of NLP models.
no code implementations • 16 Aug 2024 • Yongkang Liu, Feng Shi, Daling Wang, Yifei Zhang, Hinrich Schütze
Although large language models(LLMs) show amazing capabilities, among various exciting applications discovered for LLMs fall short in other low-resource languages.
1 code implementation • 17 Jul 2024 • Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, Hinrich Schütze
These questions are written by curriculum experts, suitable for the high-school curricula in Turkey, covering subjects ranging from natural sciences and math questions to more culturally representative topics such as Turkish Literature and the history of the Turkish Republic.
1 code implementation • 9 Jul 2024 • Ali Modarressi, Abdullatif Köksal, Hinrich Schütze
We first demonstrate that models trained on factual data exhibit inconsistent behavior: while they accurately extract triples from factual data, they fail to extract the same triples after counterfactual modification.
no code implementations • 2 Jul 2024 • Chunlan Ma, Yihong Liu, Haotian Ye, Hinrich Schütze
Inspired by recent work that leverages transliteration in encoder-only models, we investigate whether transliteration is also effective in improving LLMs' performance for low-resource languages written in non-Latin scripts.
no code implementations • 29 Jun 2024 • Peiqin Lin, André F. T. Martins, Hinrich Schütze
Recent studies have highlighted the potential of exploiting parallel corpora to enhance multilingual large language models, improving performance in both bilingual tasks, e. g., machine translation, and general-purpose tasks, e. g., text classification.
1 code implementation • 28 Jun 2024 • Orgest Xhelili, Yihong Liu, Hinrich Schütze
However, the transfer performance is often hindered when a low-resource target language is written in a different script than the high-resource source language, even though the two languages may be related or share parts of their vocabularies.
no code implementations • 26 Jun 2024 • Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schütze
In real-world environments, continual learning is essential for machine learning models, as they need to acquire new knowledge incrementally without forgetting what they have already learned.
no code implementations • 25 Jun 2024 • Ercong Nie, Bo Shao, Zifeng Ding, Mingyang Wang, Helmut Schmid, Hinrich Schütze
Large language models (LLMs) possess extensive parametric knowledge, but this knowledge is difficult to update with new information because retraining is very expensive and infeasible for closed-source models.
1 code implementation • 14 Jun 2024 • Yongkang Liu, Ercong Nie, Shi Feng, Zheng Hua, Zifeng Ding, Daling Wang, Yifei Zhang, Hinrich Schütze
We conduct experiments on Chinese dialogue datasets from five different domains and show that AMD$^2$G achieves superior performance compared to both direct training on the target domain corpus and collective training on all five domain corpora.
1 code implementation • 10 Jun 2024 • Amir Hossein Kargaran, François Yvon, Hinrich Schütze
This method uses the LID itself to identify the features that require masking and does not rely on any external resource.
no code implementations • EMNLP 2015 • Thomas Muller, Ryan Cotterell, Alexander Fraser, Hinrich Schütze
We present LEMMING, a modular log-linear model that jointly models lemmatization and tagging and supports the integration of arbitrary global features.
1 code implementation • 16 May 2024 • Yihong Liu, Chunlan Ma, Haotian Ye, Hinrich Schütze
We applied TransMI to three recent strong mPLMs, and our experiments demonstrate that TransMI not only preserves their ability to handle non-transliterated data, but also enables the models to effectively process transliterated data: the results show a consistent improvement of 3% to 34%, varying across different models and tasks.
1 code implementation • 8 May 2024 • Peiqin Lin, André F. T. Martins, Hinrich Schütze
Thus, we introduce XAMPLER: Cross-Lingual Example Retrieval, a method tailored to tackle the challenge of cross-lingual in-context learning using only annotated English data.
no code implementations • 17 Apr 2024 • Ali Modarressi, Abdullatif Köksal, Ayyoob Imani, Mohsen Fayyaz, Hinrich Schütze
While current large language models (LLMs) demonstrate some capabilities in knowledge-intensive tasks, they are limited by relying on their parameters as an implicit storage mechanism.
no code implementations • CONLL 2015 • Ryan Cotterell, Thomas Müller, Alexander Fraser, Hinrich Schütze
We present labeled morphological segmentation, an alternative view of morphological processing that unifies several tasks.
1 code implementation • 31 Mar 2024 • Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schütze
Continual learning aims at incrementally acquiring new knowledge while not forgetting existing knowledge.
no code implementations • 26 Mar 2024 • David R. Mortensen, Valentina Izrailevitch, Yunze Xiao, Hinrich Schütze, Leonie Weissweiler
We find that GPT-4 performs best on the task, followed by GPT-3. 5, but that the open source language models are also able to perform it and that the 7B parameter Mistral displays as little difference between its baseline performance on the natural language inference task and the non-prototypical syntactic category task, as the massive GPT-4.
1 code implementation • 26 Mar 2024 • Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze, David R. Mortensen, Lori Levin
In this paper, we make a contribution that can be understood from two perspectives: from an NLP perspective, we introduce a small challenge dataset for NLI with large lexical overlap, which minimises the possibility of models discerning entailment solely based on token distinctions, and show that GPT-4 and Llama 2 fail it with strong bias.
1 code implementation • 26 Mar 2024 • Leonie Weissweiler, Nina Böbel, Kirian Guiller, Santiago Herrera, Wesley Scivetti, Arthur Lorenzi, Nurit Melnik, Archna Bhatia, Hinrich Schütze, Lori Levin, Amir Zeldes, Joakim Nivre, William Croft, Nathan Schneider
The Universal Dependencies (UD) project has created an invaluable collection of treebanks with contributions in over 140 languages.
1 code implementation • 15 Mar 2024 • Verena Blaschke, Barbara Kovačić, Siyao Peng, Hinrich Schütze, Barbara Plank
Despite the success of the Universal Dependencies (UD) project exemplified by its impressive language breadth, there is still a lack in `within-language breadth': most treebanks focus on standard languages.
no code implementations • 11 Mar 2024 • Leonie Weissweiler, Abdullatif Köksal, Hinrich Schütze
Argument Structure Constructions (ASCs) are one of the most well-studied construction groups, providing a unique opportunity to demonstrate the usefulness of Construction Grammar (CxG).
no code implementations • 28 Feb 2024 • Ercong Nie, Shuzhou Yuan, Bolei Ma, Helmut Schmid, Michael Färber, Frauke Kreuter, Hinrich Schütze
Despite the predominance of English in their training data, English-centric Large Language Models (LLMs) like GPT-3 and LLaMA display a remarkable ability to perform multilingual tasks, raising questions about the depth and nature of their cross-lingual capabilities.
1 code implementation • 26 Feb 2024 • Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, Dirk Hovy
Motivated by this discrepancy, we challenge the prevailing constrained evaluation paradigm for values and opinions in LLMs and explore more realistic unconstrained evaluations.
no code implementations • 19 Feb 2024 • Verena Blaschke, Christoph Purschke, Hinrich Schütze, Barbara Plank
Natural language processing (NLP) has largely focused on modelling standardized languages.
1 code implementation • 18 Feb 2024 • Shuzhou Yuan, Ercong Nie, Michael Färber, Helmut Schmid, Hinrich Schütze
Large Language Models (LLMs) exhibit strong In-Context Learning (ICL) capabilities when prompts with demonstrations are used.
1 code implementation • 29 Jan 2024 • Bolei Ma, Ercong Nie, Shuzhou Yuan, Helmut Schmid, Michael Färber, Frauke Kreuter, Hinrich Schütze
However, most previous studies primarily focused on sentence-level classification tasks, and only a few considered token-level labeling tasks such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging.
1 code implementation • 26 Jan 2024 • Yongkang Liu, Yiqun Zhang, Qian Li, Tong Liu, Shi Feng, Daling Wang, Yifei Zhang, Hinrich Schütze
As LMs grow in size, fine-tuning the full parameters of LMs requires a prohibitively large amount of GPU memory.
no code implementations • 24 Jan 2024 • Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F. T. Martins, Hinrich Schütze
Large language models (LLMs) have advanced the state of the art in natural language processing.
1 code implementation • 12 Jan 2024 • Yihong Liu, Chunlan Ma, Haotian Ye, Hinrich Schütze
As a consequence, mPLMs are faced with a script barrier: representations from different scripts are located in different subspaces, which can result in crosslingual transfer involving languages of different scripts performing suboptimally.
no code implementations • 9 Jan 2024 • Haotian Ye, Yihong Liu, Chunlan Ma, Hinrich Schütze
In this paper, we introduce MoSECroT Model Stitching with Static Word Embeddings for Crosslingual Zero-shot Transfer), a novel and challenging task that is especially relevant to low-resource languages for which static word embeddings are available.
no code implementations • 21 Nov 2023 • Viktor Hangya, Silvia Severini, Radoslav Ralev, Alexander Fraser, Hinrich Schütze
In this paper, we propose to build multilingual word embeddings (MWEs) via a novel language chain-based approach, that incorporates intermediate related languages to bridge the gap between the distant source and target.
1 code implementation • 15 Nov 2023 • Yihong Liu, Peiqin Lin, Mingyang Wang, Hinrich Schütze
Instead of pretraining multilingual language models from scratch, a more efficient method is to adapt existing pretrained language models (PLMs) to new languages via vocabulary extension and continued pretraining.
3 code implementations • 24 Oct 2023 • Amir Hossein Kargaran, Ayyoob Imani, François Yvon, Hinrich Schütze
Several recent papers have published good solutions for language identification (LID) for about 300 high-resource and medium-resource languages.
Ranked #1 on Language Identification on GlotLID-C
no code implementations • 23 Oct 2023 • Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schütze, Kemal Oflazer, David R. Mortensen
Large language models (LLMs) have recently reached an impressive level of linguistic capability, prompting comparisons with human language skills.
no code implementations • 23 Oct 2023 • Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schütze
However, not all languages positively influence each other and it is an open research question how to select the most suitable set of languages for multilingual training and avoid negative interference among languages whose characteristics or data distributions are not compatible.
no code implementations • 18 Oct 2023 • Shengqiang Zhang, Philipp Wicke, Lütfi Kerem Şenel, Luis Figueredo, Abdeldjallil Naceri, Sami Haddadin, Barbara Plank, Hinrich Schütze
The convergence of embodied agents and large language models (LLMs) has brought significant advancements to embodied instruction following.
1 code implementation • 8 Oct 2023 • Ercong Nie, Helmut Schmid, Hinrich Schütze
Pretrained multilingual encoder models can directly perform zero-shot multilingual tasks or linguistic probing by reformulating the input examples into cloze-style prompts.
1 code implementation • 23 Sep 2023 • Amir Hossein Kargaran, François Yvon, Hinrich Schütze
We present GlotScript, an open resource and tool for low resource writing system identification.
no code implementations • 9 Aug 2023 • Ercong Nie, Helmut Schmid, Hinrich Schütze
However, training an automatic syntactic analysis system for ancient languages solely relying on annotated parse data is a formidable task due to the inherent challenges in building treebanks for such languages.
1 code implementation • 15 Jul 2023 • Bolei Ma, Ercong Nie, Helmut Schmid, Hinrich Schütze
We conduct comprehensive experiments on diverse cross-lingual language understanding tasks (sentiment classification, paraphrase identification, and natural language inference) and empirically analyze the variation trends of prompt-based finetuning performance in cross-lingual transfer across different few-shot and full-data settings.
Natural Language Inference Natural Language Understanding +4
1 code implementation • 16 Jun 2023 • Victor Steinborn, Antonis Maronikolakis, Hinrich Schütze
Non-English bias research, however, is still in its infancy with most work focusing on English.
1 code implementation • 26 May 2023 • Yihong Liu, Alexandra Chronopoulou, Hinrich Schütze, Alexander Fraser
By conducting extensive experiments on different language pairs, including similar and distant, high and low-resource languages, we find that our method alleviates the copying problem, thus improving the translation performance on low-resource languages.
no code implementations • 24 May 2023 • Yongkang Liu, Shi Feng, Daling Wang, Yifei Zhang, Hinrich Schütze
There are risks in using eference-free evaluators based on LLMs to evaluate the quality of dialogue responses.
1 code implementation • 24 May 2023 • Xinpeng Wang, Leonie Weissweiler, Hinrich Schütze, Barbara Plank
To the best of our knowledge, this is the first work comprehensively evaluating distillation objectives in both settings.
1 code implementation • 23 May 2023 • Peiqin Lin, Chengzhi Hu, Zheyu Zhang, André F. T. Martins, Hinrich Schütze
Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining.
Open-Ended Question Answering Zero-Shot Cross-Lingual Transfer
1 code implementation • 23 May 2023 • Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, Hinrich Schütze
Large language models (LLMs) have significantly advanced the field of natural language processing (NLP) through their extensive parameters and comprehensive data utilization.
no code implementations • 22 May 2023 • Haotian Ye, Yihong Liu, Hinrich Schütze
An interesting line of research in natural language processing (NLP) aims to incorporate linguistic typology to bridge linguistic diversity and assist the research of low-resource languages.
1 code implementation • 22 May 2023 • Abdullatif Köksal, Omer Faruk Yalcin, Ahmet Akbiyik, M. Tahir Kilavuz, Anna Korhonen, Hinrich Schütze
For nationality as a case study, we show that LABDet `surfaces' nationality bias by training a classifier on top of a frozen PLM on non-nationality sentiment detection.
2 code implementations • 22 May 2023 • Yihong Liu, Haotian Ye, Leonie Weissweiler, Renhao Pei, Hinrich Schütze
ColexNet's nodes are concepts and its edges are colexifications.
1 code implementation • 20 May 2023 • Ayyoob Imani, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André F. T. Martins, François Yvon, Hinrich Schütze
The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i. e., making them better for about 100 languages.
1 code implementation • 15 May 2023 • Chunlan Ma, Ayyoob ImaniGooghari, Haotian Ye, Renhao Pei, Ehsaneddin Asgari, Hinrich Schütze
While natural language processing tools have been developed extensively for some of the world's languages, a significant portion of the world's over 7000 languages are still neglected.
3 code implementations • 15 May 2023 • Yihong Liu, Haotian Ye, Leonie Weissweiler, Philipp Wicke, Renhao Pei, Robert Zangenfeind, Hinrich Schütze
The resulting measure for the conceptual similarity of two languages is complementary to standard genealogical, typological, and surface similarity measures.
no code implementations • 28 Apr 2023 • Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schütze
In this work, we propose to leverage language-adaptive and task-adaptive pretraining on African texts and study transfer learning with source language selection on top of an African language-centric pretrained language model.
6 code implementations • 20 Apr 2023 • Verena Blaschke, Hinrich Schütze, Barbara Plank
This can for instance be observed when finetuning PLMs on one language and evaluating them on data in a closely related language variety with no standardized orthography.
2 code implementations • 19 Apr 2023 • Verena Blaschke, Hinrich Schütze, Barbara Plank
In this work, we instead focus on low-resource languages and in particular non-standardized low-resource languages.
2 code implementations • 17 Apr 2023 • Abdullatif Köksal, Timo Schick, Anna Korhonen, Hinrich Schütze
We generate instructions via LLMs for human-written corpus examples using reverse instructions.
no code implementations • 4 Apr 2023 • Antonis Maronikolakis, Abdullatif Köksal, Hinrich Schütze
We introduce HATELEXICON, a lexicon of slurs and targets of hate speech for the countries of Brazil, Germany, India and Kenya, to aid training and interpretability of models.
1 code implementation • 8 Mar 2023 • Amir Hossein Kargaran, Nafiseh Nikeghbal, Abbas Heydarnoori, Hinrich Schütze
Menu system design for user interfaces is a challenging task involving many design options and various human factors.
no code implementations • 4 Feb 2023 • Leonie Weissweiler, Taiqi He, Naoki Otani, David R. Mortensen, Lori Levin, Hinrich Schütze
Construction Grammar (CxG) has recently been used as the basis for probing studies that have investigated the performance of large pretrained language models (PLMs) with respect to the structure and meaning of constructions.
1 code implementation • 19 Dec 2022 • Ercong Nie, Sheng Liang, Helmut Schmid, Hinrich Schütze
Multilingual Pretrained Language Models (MPLMs) have shown their strong multilinguality in recent empirical cross-lingual transfer studies.
no code implementations • 18 Dec 2022 • Yongkang Liu, Shi Feng, Daling Wang, Yifei Zhang, Hinrich Schütze
We investigate response generation for multi-turn dialogue in generative-based chatbots.
1 code implementation • 14 Dec 2022 • Valentin Hofmann, Janet B. Pierrehumbert, Hinrich Schütze
We propose a fully unsupervised method to detect bias in contextualized embeddings.
2 code implementations • Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing 2022 • Sajad Mirzababaei, Amir Hossein Kargaran, Hinrich Schütze, Ehsaneddin Asgari
We create Hengam in the following concrete steps: (1) we develop HengamTagger, an extensible rule-based tool that can extract temporal expressions from a set of diverse language-specific patterns for any language of interest.
Ranked #1 on Temporal Tagging on HengamCorpus
1 code implementation • 15 Nov 2022 • Abdullatif Köksal, Timo Schick, Hinrich Schütze
Few-shot classification has made great strides due to foundation models that, through priming and prompting, are highly effective few-shot learners.
no code implementations • 25 Oct 2022 • Junze Li, Mengjie Zhao, Yubo Xie, Antonis Maronikolakis, Pearl Pu, Hinrich Schütze
Humor is a magnetic component in everyday human interactions and communications.
no code implementations • 24 Oct 2022 • Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, Hinrich Schütze
Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasising the connection between syntax and semantics.
1 code implementation • 18 Oct 2022 • Ayyoob Imani, Silvia Severini, Masoud Jalili Sabet, François Yvon, Hinrich Schütze
An established method for training a POS tagger in such a scenario is to create a labeled training set by transferring from high-resource languages.
1 code implementation • 12 Oct 2022 • Yatin Chaudhary, Pranav Rai, Matthias Schubert, Hinrich Schütze, Pankaj Gupta
The objective of Federated Continual Learning (FCL) is to improve deep learning models over life time at each client by (relevant and efficient) knowledge transfer without sharing data.
1 code implementation • 12 Oct 2022 • Abdullatif Köksal, Silvia Severini, Hinrich Schütze
Word alignments are essential for a variety of NLP tasks.
1 code implementation • 26 Sep 2022 • Peiqin Lin, Jiashuo Wang, Hinrich Schütze, Wenjie Li
To solve the task, it is essential to model the content-emotion duality of a dialogue, which is composed of the content view (i. e., what personal experiences are described) and the emotion view (i. e., the feelings of the speaker on these experiences).
no code implementations • 28 Jul 2022 • Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, Yoav Goldberg
Our causal framework and our results demonstrate the importance of studying datasets and the benefits of causality for understanding NLP models.
4 code implementations • 9 Jun 2022 • Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, ZiRui Wang, Ziyi Wu
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
no code implementations • 31 May 2022 • Silvia Severini, Viktor Hangya, Masoud Jalili Sabet, Alexander Fraser, Hinrich Schütze
The two approaches we find most effective are: 1) using identical words as seed lexicons (which unsupervised approaches incorrectly assume are not available for orthographically distinct language pairs) and 2) combining such lexicons with pairs extracted by matching romanized versions of words with an edit distance threshold.
no code implementations • NAACL (GeBNLP) 2022 • Antonis Maronikolakis, Philip Baader, Hinrich Schütze
To tackle the rising phenomenon of hate speech, efforts have been made towards data curation and analysis.
no code implementations • ACL 2022 • Yihong Liu, Haris Jabbar, Hinrich Schütze
The primary novelties of our model are: (a) capturing language-specific sentence representations separately for each language using normalizing flows and (b) using a simple transformation of these latent representations for translating from one language to another.
no code implementations • 31 Mar 2022 • Marina Sedinkina, Martin Schmitt, Hinrich Schütze
The practical success of much of NLP depends on the availability of training data.
1 code implementation • ACL 2022 • Leonie Weissweiler, Valentin Hofmann, Masoud Jalili Sabet, Hinrich Schütze
We introduce CaMEL (Case Marker Extraction without Labels), a novel and challenging task in computational morphology that is especially relevant for low-resource languages.
no code implementations • 17 Mar 2022 • Zhen Han, Ruotong Liao, Jindong Gu, Yao Zhang, Zifeng Ding, Yujia Gu, Heinz Köppl, Hinrich Schütze, Volker Tresp
Since conventional knowledge embedding models cannot take full advantage of the abundant textual information, there have been extensive research efforts in enhancing knowledge embedding using texts.
no code implementations • Findings (ACL) 2022 • Ayyoob Imani, Lütfi Kerem Şenel, Masoud Jalili Sabet, François Yvon, Hinrich Schütze
First, we create a multiparallel word alignment graph, joining all bilingual word alignment pairs in one graph.
1 code implementation • 16 Mar 2022 • Valentin Hofmann, Goran Glavaš, Nikola Ljubešić, Janet B. Pierrehumbert, Hinrich Schütze
While pretrained language models (PLMs) have been shown to possess a plethora of linguistic knowledge, the existing body of research has largely neglected extralinguistic knowledge, which is generally difficult to obtain by pretraining on text alone.
no code implementations • Findings (ACL) 2022 • Sheng Liang, Mengjie Zhao, Hinrich Schütze
Recent research has made impressive progress in large-scale multimodal pre-training.
1 code implementation • ACL 2022 • Lütfi Kerem Senel, Timo Schick, Hinrich Schütze
Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks.
no code implementations • 12 Feb 2022 • Yanchen Liu, Timo Schick, Hinrich Schütze
Due to the high costs associated with finetuning large language models, various recent works propose to adapt them to specific tasks without any parameter updates through in-context learning.
no code implementations • LREC 2022 • Silvia Severini, Ayyoob Imani, Philipp Dufter, Hinrich Schütze
Prior work on extracting MNE datasets from parallel corpora required resources such as large monolingual corpora or word aligners that are unavailable or perform poorly for underresourced languages.
no code implementations • Findings (NAACL) 2022 • Mengjie Zhao, Fei Mi, Yasheng Wang, Minglei Li, Xin Jiang, Qun Liu, Hinrich Schütze
We propose LMTurk, a novel approach that treats few-shot learners as crowdsourcing workers.
no code implementations • 26 Nov 2021 • Timo Schick, Hinrich Schütze
Prompt-based approaches are strong at few-shot learning.
no code implementations • EMNLP 2021 • Nora Kassner, Oyvind Tafjord, Hinrich Schütze, Peter Clark
We show that, in a controlled experimental setting, these two mechanisms result in more consistent beliefs in the overall system, improving both the accuracy and consistency of its answers over time.
no code implementations • 28 Sep 2021 • Nikolai Solmsdorf, Dietrich Trautmann, Hinrich Schütze
Despite considerable recent progress, the creation of well-balanced and diverse resources remains a time-consuming and costly challenge in Argument Mining.
no code implementations • 23 Sep 2021 • Maximilian Mozes, Martin Schmitt, Vladimir Golkov, Hinrich Schütze, Daniel Cremers
We investigate the incorporation of visual relationships into the task of supervised image caption generation by proposing a model that leverages detected objects and auto-generated visual relationships to describe images in natural language.
no code implementations • EMNLP (insights) 2021 • Antonis Maronikolakis, Philipp Dufter, Hinrich Schütze
We show that the closer two languages are, the better BERT can align them on the character level.
1 code implementation • 16 Sep 2021 • Sheng Liang, Philipp Dufter, Hinrich Schütze
Multilingual pretrained language models (MPLMs) exhibit multilinguality and are well suited for transfer across languages.
no code implementations • 13 Sep 2021 • Antonis Maronikolakis, Philipp Dufter, Hinrich Schütze
The size of the vocabulary is a central design choice in large pretrained language models, with respect to both performance and memory requirements.
1 code implementation • EMNLP 2021 • Ayyoob Imani, Masoud Jalili Sabet, Lütfi Kerem Şenel, Philipp Dufter, François Yvon, Hinrich Schütze
With the advent of end-to-end deep learning approaches in machine translation, interest in word alignments initially decreased; however, they have again become a focus of research more recently.
1 code implementation • EMNLP 2021 • Mengjie Zhao, Hinrich Schütze
It has been shown for English that discrete and soft prompting perform strongly in few-shot learning with pretrained language models (PLMs).
1 code implementation • EMNLP 2021 • Martin Schmitt, Hinrich Schütze
If we allow for tokens outside the PLM's vocabulary, patterns can be adapted more flexibly to a PLM's idiosyncrasies.
Ranked #1 on Few-Shot NLI on SherLIiC
no code implementations • ACL 2021 • Ayyoob Imani, Masoud Jalili Sabet, Philipp Dufter, Michael Cysouw, Hinrich Schütze
With more than 7000 languages worldwide, multilingual natural language processing (NLP) is essential both from an academic and commercial perspective.
1 code implementation • 2 Jul 2021 • Luisa März, Stefan Schweter, Nina Poerner, Benjamin Roth, Hinrich Schütze
We propose new methods for in-domain and cross-domain Named Entity Recognition (NER) on historical data for Dutch and French.
1 code implementation • Findings (NAACL) 2022 • Valentin Hofmann, Xiaowen Dong, Janet B. Pierrehumbert, Hinrich Schütze
The increasing polarization of online political discourse calls for computational tools that automatically detect and monitor ideological divides in social media.
1 code implementation • NAACL 2021 • Pankaj Gupta, Yatin Chaudhary, Hinrich Schütze
Though word embeddings and topics are complementary representations, several past works have only used pretrained word embeddings in (neural) topic modeling to address data sparsity in short-text or small collection of documents.
2 code implementations • EMNLP 2021 • Timo Schick, Hinrich Schütze
To obtain high-quality sentence embeddings from pretrained language models (PLMs), they must either be augmented with additional pretraining objectives or finetuned on a large set of labeled text pairs.
Ranked #8 on Semantic Textual Similarity on SICK
1 code implementation • NAACL 2021 • Philipp Dufter, Nora Kassner, Hinrich Schütze
Recent research investigates factual knowledge stored in large pretrained language models (PLMs).
3 code implementations • 28 Feb 2021 • Timo Schick, Sahana Udupa, Hinrich Schütze
In this paper, we first demonstrate a surprising finding: pretrained language models recognize, to a considerable degree, their undesirable biases and the toxicity of the content they produce.
no code implementations • CL (ACL) 2022 • Philipp Dufter, Martin Schmitt, Hinrich Schütze
Transformers are arguably the main workhorse in recent Natural Language Processing research.
1 code implementation • EACL 2021 • Martin Schmitt, Hinrich Schütze
Lexical inference in context (LIiC) is the task of recognizing textual entailment between two very similar sentences, i. e., sentences that only differ in one expression.
Ranked #2 on Few-Shot NLI on SherLIiC
no code implementations • 9 Feb 2021 • Sahand Sharifzadeh, Sina Moayed Baharlou, Martin Schmitt, Hinrich Schütze, Volker Tresp
We show that by fine-tuning the classification pipeline with the extracted knowledge from texts, we can achieve ~8x more accurate results in scene graph classification, ~3x in object classification, and ~1. 5x in predicate classification, compared to the supervised baselines with only 1% of the annotated images.
no code implementations • 6 Feb 2021 • Lutfi Kerem Senel, Hinrich Schütze
Recent progress in pretraining language models on large corpora has resulted in large performance gains on many NLP tasks.
1 code implementation • EACL 2021 • Nora Kassner, Philipp Dufter, Hinrich Schütze
(i) Can mBERT be used as a multilingual knowledge base?
1 code implementation • 1 Feb 2021 • Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg
In this paper we study the question: Are Pretrained Language Models (PLMs) consistent with respect to factual knowledge?
1 code implementation • ACL 2021 • Valentin Hofmann, Janet B. Pierrehumbert, Hinrich Schütze
How does the input segmentation of pretrained language models (PLMs) affect their interpretations of complex words?
no code implementations • ACL 2021 • Mengjie Zhao, Yi Zhu, Ehsan Shareghi, Ivan Vulić, Roi Reichart, Anna Korhonen, Hinrich Schütze
Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT.
2 code implementations • 22 Dec 2020 • Timo Schick, Hinrich Schütze
Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion.
no code implementations • 21 Dec 2020 • Ehsaneddin Asgari, Masoud Jalili Sabet, Philipp Dufter, Christopher Ringlstetter, Hinrich Schütze
This method's hypothesis is that the aggregation of different granularities of text for certain language pairs can help word-level alignment.
2 code implementations • COLING 2020 • Timo Schick, Helmut Schmid, Hinrich Schütze
A recent approach for few-shot text classification is to convert textual inputs to cloze questions that contain some form of task description, process them with a pretrained language model and map the predicted words to labels.
1 code implementation • ACL 2021 • Valentin Hofmann, Janet B. Pierrehumbert, Hinrich Schütze
Static word embeddings that represent words by a single vector cannot capture the variability of word meaning in different linguistic and extralinguistic contexts.
1 code implementation • Findings of the Association for Computational Linguistics 2020 • Yatin Chaudhary, Pankaj Gupta, Khushbu Saxena, Vivek Kulkarni, Thomas Runkler, Hinrich Schütze
Our work thus focuses on optimizing the computational cost of fine-tuning for document classification.
5 code implementations • NAACL 2021 • Timo Schick, Hinrich Schütze
When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance.
3 code implementations • EMNLP (NLP4ConvAI) 2021 • Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, Iryna Gurevych
We show that the PLMs BART and T5 achieve new state-of-the-art results and that task-adaptive pretraining strategies improve their performance even further.
Ranked #1 on KG-to-Text Generation on WebNLG (All)
no code implementations • ACL 2019 • Marina Sedinkina, Nikolas Breitkopf, Hinrich Schütze
In our experiments, we demonstrate that the automatically adapted sentiment dictionary outperforms the previous state of the art in predicting the financial outcomes excess return and volatility.
1 code implementation • ICML 2020 • Pankaj Gupta, Yatin Chaudhary, Thomas Runkler, Hinrich Schütze
To address the problem, we propose a lifelong learning framework for neural topic modeling that can continuously process streams of document collections, accumulate topics and guide future topic modeling tasks by knowledge transfer from several sources to better deal with the sparse data.
1 code implementation • ICML 2020 • Yatin Chaudhary, Hinrich Schütze, Pankaj Gupta
Marrying topic models and language models exposes language understanding to a broader source of document-level context beyond sentences via topics.
1 code implementation • CONLL 2020 • Nora Kassner, Benno Krojer, Hinrich Schütze
How can pretrained language models (PLMs) learn factual knowledge from the training set?
no code implementations • NAACL (TextGraphs) 2021 • Martin Schmitt, Leonardo F. R. Ribeiro, Philipp Dufter, Iryna Gurevych, Hinrich Schütze
We present Graformer, a novel Transformer-based encoder-decoder architecture for graph-to-text generation.
Ranked #5 on KG-to-Text Generation on AGENDA
no code implementations • 16 May 2020 • Ehsaneddin Asgari, Christoph Ringlstetter, Hinrich Schütze
This paper describes EmbLexChange, a system introduced by the "Life-Language" team for SemEval-2020 Task 1, on unsupervised detection of lexical-semantic changes.
1 code implementation • EMNLP 2020 • Valentin Hofmann, Janet B. Pierrehumbert, Hinrich Schütze
Can pretrained language models (PLMs) generate derivationally complex words?
1 code implementation • Findings of the Association for Computational Linguistics 2020 • Nora Kassner, Hinrich Schütze
Khandelwal et al. (2020) use a k-nearest-neighbor (kNN) component to improve language model performance.
1 code implementation • 1 May 2020 • Philipp Dufter, Hinrich Schütze
We aim to identify architectural properties of BERT and linguistic properties of languages that are necessary for BERT to become multilingual.
no code implementations • EMNLP 2020 • Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, Hinrich Schütze
We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning.
no code implementations • Findings of the Association for Computational Linguistics 2020 • Mengjie Zhao, Philipp Dufter, Yadollah Yaghoobzadeh, Hinrich Schütze
Pretrained language models have achieved a new state of the art on many NLP tasks, but there are still many open questions about how and why they work so well.
3 code implementations • Findings of the Association for Computational Linguistics 2020 • Masoud Jalili Sabet, Philipp Dufter, François Yvon, Hinrich Schütze
We find that alignments created from embeddings are superior for four and comparable for two language pairs compared to those produced by traditional statistical aligners, even with abundant parallel data; e. g., contextualized embeddings achieve a word alignment F1 for English-German that is 5 percentage points higher than eflomal, a high-quality statistical aligner, trained on 100k parallel sentences.