Search Results for author: Ivan Vulić

Found 117 papers, 69 papers with code

Data Augmentation and Learned Layer Aggregation for Improved Multilingual Language Understanding in Dialogue

no code implementations Findings (ACL) 2022 Evgeniia Razumovskaia, Ivan Vulić, Anna Korhonen

Scaling dialogue systems to a multitude of domains, tasks and languages relies on costly and time-consuming data annotation for different domain-task-language configurations.

Data Augmentation Natural Language Understanding

Multi-SimLex: A Large-Scale Evaluation of Multilingual and Crosslingual Lexical Semantic Similarity

no code implementations CL (ACL) 2020 Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, Roi Reichart, Anna Korhonen

We introduce Multi-SimLex, a large-scale lexical resource and evaluation benchmark covering data sets for 12 typologically diverse languages, including major languages (e. g., Mandarin Chinese, Spanish, Russian) as well as less-resourced ones (e. g., Welsh, Kiswahili).

Representation Learning Semantic Similarity +2

BAD-X: Bilingual Adapters Improve Zero-Shot Cross-Lingual Transfer

1 code implementation NAACL 2022 Marinela Parović, Goran Glavaš, Ivan Vulić, Anna Korhonen

Adapter modules enable modular and efficient zero-shot cross-lingual transfer, where current state-of-the-art adapter-based approaches learn specialized language adapters (LAs) for individual languages.

Zero-Shot Cross-Lingual Transfer

MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer

no code implementations Findings (EMNLP) 2021 Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, Anna Korhonen

While offering (1) improved fine-tuning efficiency (by a factor of around 50 in our experiments), (2) a smaller parameter budget, and (3) increased language coverage, MAD-G remains competitive with more expensive methods for language-specific adapter training across the board.

Dependency Parsing named-entity-recognition +4

Natural Language Processing for Multilingual Task-Oriented Dialogue

no code implementations ACL 2022 Evgeniia Razumovskaia, Goran Glavaš, Olga Majewska, Edoardo Ponti, Ivan Vulić

In this tutorial, we will thus discuss and demonstrate the importance of (building) multilingual ToD systems, and then provide a systematic overview of current research gaps, challenges and initiatives related to multilingual ToD systems, with a particular focus on their connections to current research and challenges in multilingual and low-resource NLP.

Semantic Data Set Construction from Human Clustering and Spatial Arrangement

no code implementations CL (ACL) 2021 Olga Majewska, Diana McCarthy, Jasper J. F. van den Bosch, Nikolaus Kriegeskorte, Ivan Vulić, Anna Korhonen

We demonstrate how the resultant data set can be used for fine-grained analyses and evaluation of representation learning models on the intrinsic tasks of semantic clustering and semantic similarity.

Clustering Representation Learning +3

Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?

no code implementations4 Mar 2024 Evgeniia Razumovskaia, Ivan Vulić, Anna Korhonen

Supervised fine-tuning (SFT), supervised instruction tuning (SIT) and in-context learning (ICL) are three alternative, de facto standard approaches to few-shot learning.

Few-Shot Learning In-Context Learning +1

Self-Augmented In-Context Learning for Unsupervised Word Translation

no code implementations15 Feb 2024 Yaoyiran Li, Anna Korhonen, Ivan Vulić

Recent work has shown that, while large language models (LLMs) demonstrate strong word translation or bilingual lexicon induction (BLI) capabilities in few-shot setups, they still cannot match the performance of 'traditional' mapping-based approaches in the unsupervised scenario where no seed translation pairs are available, especially for lower-resource languages.

Bilingual Lexicon Induction Cross-Lingual Word Embeddings +9

Unmemorization in Large Language Models via Self-Distillation and Deliberate Imagination

1 code implementation15 Feb 2024 Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, Ivan Vulić

Our results demonstrate the usefulness of this approach across different models and sizes, and also with parameter-efficient fine-tuning, offering a novel pathway to addressing the challenges with private and sensitive data in LLM applications.

Natural Language Understanding

Scaling Sparse Fine-Tuning to Large Language Models

2 code implementations29 Jan 2024 Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, Edoardo M. Ponti

We experiment with instruction-tuning of LLMs on standard dataset mixtures, finding that SpIEL is often superior to popular parameter-efficient fine-tuning methods like LoRA (low-rank adaptation) in terms of performance and comparable in terms of run time.

Quantization

Pheme: Efficient and Conversational Speech Generation

1 code implementation5 Jan 2024 Paweł Budzianowski, Taras Sereda, Tomasz Cichy, Ivan Vulić

However, certain applications, such as assistive conversational systems, require natural and conversational speech generation tools that also operate efficiently in real time.

DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models

2 code implementations4 Jan 2024 Songbo Hu, Xiaobin Wang, Zhangdie Yuan, Anna Korhonen, Ivan Vulić

We present DIALIGHT, a toolkit for developing and evaluating multilingual Task-Oriented Dialogue (ToD) systems which facilitates systematic evaluations and comparisons between ToD systems using fine-tuning of Pretrained Language Models (PLMs) and those utilising the zero-shot and in-context learning capabilities of Large Language Models (LLMs).

In-Context Learning Task-Oriented Dialogue Systems

On Task Performance and Model Calibration with Supervised and Self-Ensembled In-Context Learning

1 code implementation21 Dec 2023 Chengzu Li, Han Zhou, Goran Glavaš, Anna Korhonen, Ivan Vulić

Following the standard supervised fine-tuning (SFT) paradigm, in-context learning (ICL) has become an efficient approach propelled by the recent advancements in large language models (LLMs), yielding promising performance across various tasks in few-shot data setups.

In-Context Learning

$\textit{Dial BeInfo for Faithfulness}$: Improving Factuality of Information-Seeking Dialogue via Behavioural Fine-Tuning

no code implementations16 Nov 2023 Evgeniia Razumovskaia, Ivan Vulić, Pavle Marković, Tomasz Cichy, Qian Zheng, Tsung-Hsien Wen, Paweł Budzianowski

Factuality is a crucial requirement in information seeking dialogue: the system should respond to the user's queries so that the responses are meaningful and aligned with the knowledge provided to the system.

SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU

no code implementations16 Nov 2023 Evgeniia Razumovskaia, Goran Glavaš, Anna Korhonen, Ivan Vulić

Task-oriented dialogue (ToD) systems help users execute well-defined tasks across a variety of domains (e. g., $\textit{flight booking}$ or $\textit{food ordering}$), with their Natural Language Understanding (NLU) components being dedicated to the analysis of user utterances, predicting users' intents ($\textit{Intent Detection}$, ID) and extracting values for informational slots ($\textit{Value Extraction}$, VE).

Intent Detection Natural Language Understanding +1

Quantifying the Dialect Gap and its Correlates Across Languages

no code implementations23 Oct 2023 Anjali Kantharuban, Ivan Vulić, Anna Korhonen

Historically, researchers and consumers have noticed a decrease in quality when applying NLP tools to minority variants of languages (i. e. Puerto Rican Spanish or Swiss German), but studies exploring this have been limited to a select few languages.

Automatic Speech Recognition Machine Translation +2

On Bilingual Lexicon Induction with Large Language Models

1 code implementation21 Oct 2023 Yaoyiran Li, Anna Korhonen, Ivan Vulić

Bilingual Lexicon Induction (BLI) is a core task in multilingual NLP that still, to a large extent, relies on calculating cross-lingual word representations.

Bilingual Lexicon Induction Cross-Lingual Word Embeddings +8

Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning

1 code implementation19 Oct 2023 Han Zhou, Xingchen Wan, Ivan Vulić, Anna Korhonen

Prompt-based learning has been an effective paradigm for large pretrained language models (LLM), enabling few-shot or even zero-shot learning.

Combinatorial Optimization Zero-Shot Learning

One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging For Cross-Lingual Transfer

1 code implementation16 Oct 2023 Fabian David Schmidt, Ivan Vulić, Goran Glavaš

Because of this, model selection based on source-language validation is unreliable: it picks model snapshots with suboptimal target-language performance.

Model Selection NER +3

Multi3WOZ: A Multilingual, Multi-Domain, Multi-Parallel Dataset for Training and Evaluating Culturally Adapted Task-Oriented Dialog Systems

1 code implementation26 Jul 2023 Songbo Hu, Han Zhou, Mete Hergul, Milan Gritta, Guchun Zhang, Ignacio Iacobacci, Ivan Vulić, Anna Korhonen

Creating high-quality annotated data for task-oriented dialog (ToD) is known to be notoriously difficult, and the challenges are amplified when the goal is to create equitable, culturally adapted, and large-scale ToD datasets for multiple languages.

Translation

Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Data

no code implementations4 Jul 2023 Guangzhi Sun, Chao Zhang, Ivan Vulić, Paweł Budzianowski, Philip C. Woodland

In this work, we propose a Knowledge-Aware Audio-Grounded generative slot-filling framework, termed KA2G, that focuses on few-shot and zero-shot slot filling for ToD with speech input.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +6

Cross-Lingual Transfer with Target Language-Ready Task Adapters

no code implementations5 Jun 2023 Marinela Parović, Alan Ansell, Ivan Vulić, Anna Korhonen

We address this mismatch by exposing the task adapter to the target language adapter during training, and empirically validate several variants of the idea: in the simplest form, we alternate between using the source and target language adapters during task adapter training, which can be generalized to cycling over any set of language adapters.

Zero-Shot Cross-Lingual Transfer

Distilling Efficient Language-Specific Models for Cross-Lingual Transfer

1 code implementation2 Jun 2023 Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, Ivan Vulić

Specifically, we use a two-phase distillation approach, termed BiStil: (i) the first phase distils a general bilingual model from the MMT, while (ii) the second, task-specific phase sparsely fine-tunes the bilingual "student" model using a task-tuned variant of the original MMT as its "teacher".

Transfer Learning XLM-R +1

Translation-Enhanced Multilingual Text-to-Image Generation

no code implementations30 May 2023 Yaoyiran Li, Ching-Yun Chang, Stephen Rawls, Ivan Vulić, Anna Korhonen

Research on text-to-image generation (TTI) still predominantly focuses on the English language due to the lack of annotated image-caption data in other languages; in the long run, this might widen inequitable access to TTI technology.

Cross-lingual Text-to-Image Generation Crosslingual Text-to-Image Generation +6

Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging

1 code implementation26 May 2023 Fabian David Schmidt, Ivan Vulić, Goran Glavaš

The results indicate that averaging model checkpoints yields systematic and consistent performance gains across diverse target languages in all tasks.

Cross-Lingual Transfer Model Selection +4

CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models

1 code implementation23 May 2023 Benjamin Minixhofer, Jonas Pfeiffer, Ivan Vulić

We first address the data gap by introducing a dataset of 255k compound and non-compound words across 56 diverse languages obtained from Wiktionary.

Transfer-Free Data-Efficient Multilingual Slot Labeling

no code implementations22 May 2023 Evgeniia Razumovskaia, Ivan Vulić, Anna Korhonen

It is especially effective for the most challenging transfer-free few-shot setups, paving the way for quick and data-efficient bootstrapping of multilingual slot labelers for ToD.

Contrastive Learning Cross-Lingual Transfer +3

Romanization-based Large-scale Adaptation of Multilingual Language Models

no code implementations18 Apr 2023 Sukannya Purkayastha, Sebastian Ruder, Jonas Pfeiffer, Iryna Gurevych, Ivan Vulić

In order to boost the capacity of mPLMs to deal with low-resource and unseen languages, we explore the potential of leveraging transliteration on a massive scale.

Cross-Lingual Transfer Transliteration

Modular Deep Learning

no code implementations22 Feb 2023 Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, Edoardo Maria Ponti

Modular deep learning has emerged as a promising solution to these challenges.

Causal Inference Transfer Learning

AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning

1 code implementation28 Jan 2023 Han Zhou, Xingchen Wan, Ivan Vulić, Anna Korhonen

Large pretrained language models are widely used in downstream NLP tasks via task-specific fine-tuning, but such procedures can be costly.

Bayesian Optimisation Neural Architecture Search

Improving Generalization of Adapter-Based Cross-lingual Transfer with Scheduled Unfreezing

no code implementations13 Jan 2023 Chen Cecilia Liu, Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych

Our in-depth experiments reveal that scheduled unfreezing induces different learning dynamics compared to standard fine-tuning, and provide evidence that the dynamics of Fisher Information during training correlate with cross-lingual generalization performance.

Cross-Lingual Transfer Transfer Learning

MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialogue

no code implementations20 Dec 2022 Nikita Moghe, Evgeniia Razumovskaia, Liane Guillou, Ivan Vulić, Anna Korhonen, Alexandra Birch

We use MULTI3NLU++ to benchmark state-of-the-art multilingual models for the NLU tasks of intent detection and slot labelling for TOD systems in the multilingual setting.

Intent Detection Machine Translation +2

Reranking Overgenerated Responses for End-to-End Task-Oriented Dialogue Systems

1 code implementation7 Nov 2022 Songbo Hu, Ivan Vulić, Fangyu Liu, Anna Korhonen

At training, the high-scoring partition comprises all generated responses whose similarity to the gold response is higher than the similarity of the greedy response to the gold response.

Task-Oriented Dialogue Systems

Improving Bilingual Lexicon Induction with Cross-Encoder Reranking

1 code implementation30 Oct 2022 Yaoyiran Li, Fangyu Liu, Ivan Vulić, Anna Korhonen

This crucial step is done via 1) creating a word similarity dataset, comprising positive word pairs (i. e., true translations) and hard negative pairs induced from the original CLWE space, and then 2) fine-tuning an mPLM (e. g., mBERT or XLM-R) in a cross-encoder manner to predict the similarity scores.

Bilingual Lexicon Induction Cross-Lingual Word Embeddings +7

Can Pretrained Language Models (Yet) Reason Deductively?

1 code implementation12 Oct 2022 Zhangdie Yuan, Songbo Hu, Ivan Vulić, Anna Korhonen, Zaiqiao Meng

Acquiring factual knowledge with Pretrained Language Models (PLMs) has attracted increasing attention, showing promising performance in many knowledge-intensive tasks.

SLICER: Sliced Fine-Tuning for Low-Resource Cross-Lingual Transfer for Named Entity Recognition

1 code implementation Proceedings of the Conference on Empirical Methods in Natural Language Processing 2022 Fabian David Schmidt, Ivan Vulić, Goran Glavaš

Large multilingual language models generally demonstrate impressive results in zero-shot cross-lingual transfer, yet often fail to successfully transfer to low-resource languages, even for token-level prediction tasks like named entity recognition (NER).

Multilingual text classification named-entity-recognition +3

Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold

1 code implementation Findings (ACL) 2022 Sebastian Ruder, Ivan Vulić, Anders Søgaard

Most work targeting multilinguality, for example, considers only accuracy; most work on fairness or interpretability considers only English; and so on.

Computational Efficiency Fairness

Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog

1 code implementation NAACL 2022 Chia-Chien Hung, Anne Lauscher, Ivan Vulić, Simone Paolo Ponzetto, Goran Glavaš

We then introduce a new framework for multilingual conversational specialization of pretrained language models (PrLMs) that aims to facilitate cross-lingual transfer for arbitrary downstream TOD tasks.

Cross-Lingual Transfer dialog state tracking +1

Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders

no code implementations30 Apr 2022 Ivan Vulić, Goran Glavaš, Fangyu Liu, Nigel Collier, Edoardo Maria Ponti, Anna Korhonen

In this work, we probe SEs for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs.

Contrastive Learning Cross-Lingual Entity Linking +6

NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue

1 code implementation Findings (NAACL) 2022 Iñigo Casanueva, Ivan Vulić, Georgios P. Spithourakis, Paweł Budzianowski

2) The ontology is divided into domain-specific and generic (i. e., domain-universal) intent modules that overlap across domains, promoting cross-domain reusability of annotated examples.

Natural Language Understanding

Improved and Efficient Conversational Slot Labeling through Question Answering

no code implementations5 Apr 2022 Gabor Fuisz, Ivan Vulić, Samuel Gibbons, Inigo Casanueva, Paweł Budzianowski

In particular, we focus on modeling and studying \textit{slot labeling} (SL), a crucial component of NLU for dialog, through the QA optics, aiming to improve both its performance and efficiency, and make it more effective and resilient to working with limited task data.

Natural Language Understanding Question Answering

Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval

1 code implementation COLING 2022 Robert Litschko, Ivan Vulić, Goran Glavaš

Current approaches therefore commonly transfer rankers trained on English data to other languages and cross-lingual setups by means of multilingual encoders: they fine-tune all parameters of pretrained massively multilingual Transformers (MMTs, e. g., multilingual BERT) on English relevance judgments, and then deploy them in the target language(s).

Cross-Lingual Transfer Language Modelling +3

Improving Word Translation via Two-Stage Contrastive Learning

1 code implementation ACL 2022 Yaoyiran Li, Fangyu Liu, Nigel Collier, Anna Korhonen, Ivan Vulić

At Stage C1, we propose to refine standard cross-lingual linear maps between static word embeddings (WEs) via a contrastive learning objective; we also show how to integrate it into the self-learning procedure for even more refined cross-lingual maps.

Bilingual Lexicon Induction Contrastive Learning +7

Delving Deeper into Cross-lingual Visual Question Answering

1 code implementation15 Feb 2022 Chen Liu, Jonas Pfeiffer, Anna Korhonen, Ivan Vulić, Iryna Gurevych

2) We analyze cross-lingual VQA across different question types of varying complexity for different multilingual multimodal Transformers, and identify question types that are the most difficult to improve on.

Inductive Bias Question Answering +1

IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages

3 code implementations27 Jan 2022 Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, Ivan Vulić

Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups.

Cross-Modal Retrieval Few-Shot Learning +5

On Cross-Lingual Retrieval with Multilingual Text Encoders

1 code implementation21 Dec 2021 Robert Litschko, Ivan Vulić, Simone Paolo Ponzetto, Goran Glavaš

In this work we present a systematic empirical study focused on the suitability of the state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks across a number of diverse language pairs.

Re-Ranking Retrieval +2

MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models

1 code implementation CoNLL (EMNLP) 2021 Qianchu Liu, Fangyu Liu, Nigel Collier, Anna Korhonen, Ivan Vulić

Recent work indicated that pretrained language models (PLMs) such as BERT and RoBERTa can be transformed into effective sentence and word encoders even via simple self-supervised techniques.

Contextualised Word Representations Contrastive Learning +1

Towards Zero-shot Language Modeling

no code implementations IJCNLP 2019 Edoardo Maria Ponti, Ivan Vulić, Ryan Cotterell, Roi Reichart, Anna Korhonen

Motivated by this question, we aim at constructing an informative prior over neural weights, in order to adapt quickly to held-out languages in the task of character-level language modeling.

Language Modelling

Modelling Latent Translations for Cross-Lingual Transfer

1 code implementation23 Jul 2021 Edoardo Maria Ponti, Julia Kreutzer, Ivan Vulić, Siva Reddy

To remedy this, we propose a new technique that integrates both steps of the traditional pipeline (translation and classification) into a single model, by treating the intermediate translations as a latent random variable.

Cross-Lingual Transfer Few-Shot Learning +5

Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking

1 code implementation ACL 2021 Fangyu Liu, Ivan Vulić, Anna Korhonen, Nigel Collier

To this end, we propose and evaluate a series of cross-lingual transfer methods for the XL-BEL task, and demonstrate that general-domain bitext helps propagate the available English knowledge to languages with little to no in-domain data.

Cross-Lingual Transfer Entity Linking

Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems

no code implementations17 Apr 2021 Evgeniia Razumovskaia, Goran Glavaš, Olga Majewska, Edoardo M. Ponti, Anna Korhonen, Ivan Vulić

We find that the most critical factor preventing the creation of truly multilingual ToD systems is the lack of datasets in most languages for both training and evaluation.

Cross-Lingual Transfer Machine Translation +2

AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples

1 code implementation EMNLP 2021 Qianchu Liu, Edoardo M. Ponti, Diana McCarthy, Ivan Vulić, Anna Korhonen

In order to address these gaps, we present AM2iCo (Adversarial and Multilingual Meaning in Context), a wide-coverage cross-lingual and multilingual evaluation set; it aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts for 14 language pairs.

Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval

1 code implementation22 Mar 2021 Gregor Geigle, Jonas Pfeiffer, Nils Reimers, Ivan Vulić, Iryna Gurevych

Current state-of-the-art approaches to cross-modal retrieval process text and visual input jointly, relying on Transformer-based architectures with cross-attention mechanisms that attend over all words and objects in an image.

Cross-Modal Retrieval Retrieval

Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrieval

1 code implementation21 Jan 2021 Robert Litschko, Ivan Vulić, Simone Paolo Ponzetto, Goran Glavaš

Therefore, in this work we present a systematic empirical study focused on the suitability of the state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks across a large number of language pairs.

Cross-Lingual Word Embeddings Representation Learning +3

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

no code implementations ACL 2021 Mengjie Zhao, Yi Zhu, Ehsan Shareghi, Ivan Vulić, Roi Reichart, Anna Korhonen, Hinrich Schütze

Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT.

Few-Shot Learning

Verb Knowledge Injection for Multilingual Event Processing

no code implementations ACL 2021 Olga Majewska, Ivan Vulić, Goran Glavaš, Edoardo M. Ponti, Anna Korhonen

We investigate whether injecting explicit information on verbs' semantic-syntactic behaviour improves the performance of LM-pretrained Transformers in event extraction tasks -- downstream tasks for which accurate verb processing is paramount.

Event Extraction Language Modelling

UNKs Everywhere: Adapting Multilingual Language Models to New Scripts

2 code implementations EMNLP 2021 Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, Sebastian Ruder

The ultimate challenge is dealing with under-resourced languages not covered at all by the models and written in scripts unseen during pretraining.

Cross-Lingual Transfer

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

1 code implementation ACL 2021 Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, Iryna Gurevych

In this work, we provide a systematic and comprehensive empirical comparison of pretrained multilingual language models versus their monolingual counterparts with regard to their monolingual task performance.

Pretrained Multilingual Language Models

Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer

no code implementations11 Dec 2020 Marko Vidoni, Ivan Vulić, Goran Glavaš

Adapter modules, additional trainable parameters that enable efficient fine-tuning of pretrained transformers, have recently been used for language specialization of multilingual transformers, improving downstream zero-shot cross-lingual transfer.

NER POS +2

Emergent Communication Pretraining for Few-Shot Machine Translation

1 code implementation COLING 2020 Yaoyiran Li, Edoardo M. Ponti, Ivan Vulić, Anna Korhonen

On the other hand, this also provides an extrinsic evaluation protocol to probe the properties of emergent languages ex vitro.

Machine Translation NMT +2

ConVEx: Data-Efficient and Few-Shot Slot Labeling

no code implementations NAACL 2021 Matthew Henderson, Ivan Vulić

We propose ConVEx (Conversational Value Extractor), an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks.

Language Modelling

Probing Pretrained Language Models for Lexical Semantics

no code implementations EMNLP 2020 Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, Anna Korhonen

The success of large pretrained language models (LMs) such as BERT and RoBERTa has sparked interest in probing their representations, in order to unveil what types of knowledge they implicitly capture.

World Knowledge

Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation

3 code implementations15 Aug 2020 Goran Glavaš, Ivan Vulić

Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU).

Language Modelling Natural Language Understanding

AdapterHub: A Framework for Adapting Transformers

7 code implementations EMNLP 2020 Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych

We propose AdapterHub, a framework that allows dynamic "stitching-in" of pre-trained adapters for different tasks and languages.

XLM-R

Multidirectional Associative Optimization of Function-Specific Word Representations

1 code implementation ACL 2020 Daniela Gerz, Ivan Vulić, Marek Rei, Roi Reichart, Anna Korhonen

We present a neural framework for learning associations between interrelated groups of words such as the ones found in Subject-Verb-Object (SVO) structures.

From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers

no code implementations1 May 2020 Anne Lauscher, Vinit Ravishankar, Ivan Vulić, Goran Glavaš

Massively multilingual transformers pretrained with language modeling objectives (e. g., mBERT, XLM-R) have become a de facto default transfer paradigm for zero-shot cross-lingual transfer in NLP, offering unmatched transfer performance.

Cross-Lingual Word Embeddings Dependency Parsing +6

XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning

1 code implementation EMNLP 2020 Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, Anna Korhonen

In order to simulate human language capacity, natural language processing systems must be able to reason about the dynamics of everyday situations, including their possible causes and effects.

Ranked #3 on Cross-Lingual Transfer on XCOPA (using extra training data)

Cross-Lingual Transfer Translation +1

MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer

3 code implementations EMNLP 2020 Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, Sebastian Ruder

The main goal behind state-of-the-art pre-trained multilingual models such as multilingual BERT and XLM-R is enabling and bootstrapping NLP applications in low-resource languages through zero-shot or few-shot cross-lingual transfer.

Ranked #5 on Cross-Lingual Transfer on XCOPA (using extra training data)

Cross-Lingual Transfer named-entity-recognition +4

Towards Instance-Level Parser Selection for Cross-Lingual Transfer of Dependency Parsers

no code implementations COLING 2020 Robert Litschko, Ivan Vulić, Željko Agić, Goran Glavaš

Current methods of cross-lingual parser transfer focus on predicting the best parser for a low-resource target language globally, that is, "at treebank level".

Cross-Lingual Transfer POS

Are All Good Word Vector Spaces Isomorphic?

1 code implementation EMNLP 2020 Ivan Vulić, Sebastian Ruder, Anders Søgaard

Existing algorithms for aligning cross-lingual word vector spaces assume that vector spaces are approximately isomorphic.

Efficient Intent Detection with Dual Sentence Encoders

5 code implementations WS 2020 Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, Ivan Vulić

Building conversational systems in new domains and with added functionality requires resource-efficient models that work under low-data regimes (i. e., in few-shot setups).

Intent Detection Sentence

Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity

no code implementations10 Mar 2020 Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, Roi Reichart, Anna Korhonen

We introduce Multi-SimLex, a large-scale lexical resource and evaluation benchmark covering datasets for 12 typologically diverse languages, including major languages (e. g., Mandarin Chinese, Spanish, Russian) as well as less-resourced ones (e. g., Welsh, Kiswahili).

Cross-Lingual Word Embeddings Representation Learning +3

The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures

no code implementations EMNLP 2020 Haim Dubossarsky, Ivan Vulić, Roi Reichart, Anna Korhonen

Performance in cross-lingual NLP tasks is impacted by the (dis)similarity of languages at hand: e. g., previous work has suggested there is a connection between the expected success of bilingual lexicon induction (BLI) and the assumption of (approximate) isomorphism between monolingual embedding spaces.

Bilingual Lexicon Induction POS +1

A General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spaces

4 code implementations13 Sep 2019 Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, Ivan Vulić

Moreover, we successfully transfer debiasing models, by means of cross-lingual embedding spaces, and remove or attenuate biases in distributional word vector spaces of languages that lack readily available bias specifications.

Word Embeddings

Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity

1 code implementation COLING 2020 Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, Goran Glavaš

In this work, we complement such distributional knowledge with external lexical knowledge, that is, we integrate the discrete knowledge on word-level semantic similarity into pretraining.

Language Modelling Lexical Simplification +7

Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?

1 code implementation IJCNLP 2019 Ivan Vulić, Goran Glavaš, Roi Reichart, Anna Korhonen

A series of bilingual lexicon induction (BLI) experiments with 15 diverse languages (210 language pairs) show that fully unsupervised CLWE methods still fail for a large number of language pairs (e. g., they yield zero BLI performance for 87/210 pairs).

Bilingual Lexicon Induction Self-Learning

Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems

1 code implementation12 Jul 2019 Paweł Budzianowski, Ivan Vulić

Data scarcity is a long-standing and crucial challenge that hinders quick development of task-oriented dialogue systems across multiple domains: task-oriented dialogue models are expected to learn grammar, syntax, dialogue reasoning, decision making, and language generation from absurdly small amounts of task-specific data.

Decision Making Language Modelling +3

Training Neural Response Selection for Task-Oriented Dialogue Systems

1 code implementation ACL 2019 Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, Pei-Hao Su

Despite their popularity in the chatbot literature, retrieval-based models have had modest impact on task-oriented dialogue systems, with the main obstacle to their application being the low-data regime of most task-oriented dialogue tasks.

Chatbot Language Modelling +2

A Systematic Study of Leveraging Subword Information for Learning Word Representations

1 code implementation NAACL 2019 Yi Zhu, Ivan Vulić, Anna Korhonen

The use of subword-level information (e. g., characters, character n-grams, morphemes) has become ubiquitous in modern word representation learning.

Dependency Parsing Entity Typing +3

Fully Statistical Neural Belief Tracking

1 code implementation29 May 2018 Nikola Mrkšić, Ivan Vulić

This paper proposes an improvement to the existing data-driven Neural Belief Tracking (NBT) framework for Dialogue State Tracking (DST).

Dialogue State Tracking

Scoring Lexical Entailment with a Supervised Directional Similarity Network

1 code implementation ACL 2018 Marek Rei, Daniela Gerz, Ivan Vulić

Experiments show excellent performance on scoring graded lexical entailment, raising the state-of-the-art on the HyperLex dataset by approximately 25%.

Lexical Entailment Word Embeddings

On the Limitations of Unsupervised Bilingual Dictionary Induction

no code implementations ACL 2018 Anders Søgaard, Sebastian Ruder, Ivan Vulić

Unsupervised machine translation---i. e., not assuming any cross-lingual supervision signal, whether a dictionary, translations, or comparable corpora---seems impossible, but nevertheless, Lample et al. (2018) recently proposed a fully unsupervised machine translation (MT) model.

Graph Similarity Translation +1

Post-Specialisation: Retrofitting Vectors of Words Unseen in Lexical Resources

1 code implementation NAACL 2018 Ivan Vulić, Goran Glavaš, Nikola Mrkšić, Anna Korhonen

Word vector specialisation (also known as retrofitting) is a portable, light-weight approach to fine-tuning arbitrary distributional word vector spaces by injecting external knowledge from rich lexical resources such as WordNet.

Dialogue State Tracking Text Simplification +1

Unsupervised Cross-Lingual Information Retrieval using Monolingual Data Only

1 code implementation2 May 2018 Robert Litschko, Goran Glavaš, Simone Paolo Ponzetto, Ivan Vulić

We propose a fully unsupervised framework for ad-hoc cross-lingual information retrieval (CLIR) which requires no bilingual data at all.

Cross-Lingual Information Retrieval Retrieval

Specialising Word Vectors for Lexical Entailment

1 code implementation17 Oct 2017 Ivan Vulić, Nikola Mrkšić

We present LEAR (Lexical Entailment Attract-Repel), a novel post-processing method that transforms any input word vector space to emphasise the asymmetric relation of lexical entailment (LE), also known as the IS-A or hyponymy-hypernymy relation.

Lexical Entailment Relation +2

Cross-Lingual Induction and Transfer of Verb Classes Based on Word Vector Space Specialisation

no code implementations EMNLP 2017 Ivan Vulić, Nikola Mrkšić, Anna Korhonen

Existing approaches to automatic VerbNet-style verb classification are heavily dependent on feature engineering and therefore limited to languages with mature NLP pipelines.

Clustering Cross-Lingual Transfer +4

A Survey Of Cross-lingual Word Embedding Models

no code implementations15 Jun 2017 Sebastian Ruder, Ivan Vulić, Anders Søgaard

Cross-lingual representations of words enable us to reason about word meaning in multilingual contexts and are a key facilitator of cross-lingual transfer when developing natural language processing models for low-resource languages.

Cross-Lingual Transfer Cross-Lingual Word Embeddings +1

Morph-fitting: Fine-Tuning Word Vector Spaces with Simple Language-Specific Rules

no code implementations ACL 2017 Ivan Vulić, Nikola Mrkšić, Roi Reichart, Diarmuid Ó Séaghdha, Steve Young, Anna Korhonen

Morphologically rich languages accentuate two properties of distributional vector space models: 1) the difficulty of inducing accurate representations for low-frequency word forms; and 2) insensitivity to distinct lexical relations that have similar distributional signatures.

Dialogue State Tracking MORPH

Decoding Sentiment from Distributed Representations of Sentences

no code implementations SEMEVAL 2017 Edoardo Maria Ponti, Ivan Vulić, Anna Korhonen

Distributed representations of sentences have been developed recently to represent their meaning as real-valued vectors.

Negation Sentence

Survey on the Use of Typological Information in Natural Language Processing

no code implementations COLING 2016 Helen O'Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Anna Korhonen

In recent years linguistic typology, which classifies the world's languages according to their functional and structural properties, has been widely used to support multilingual NLP.

Multilingual NLP

Automatic Selection of Context Configurations for Improved Class-Specific Word Representations

no code implementations CONLL 2017 Ivan Vulić, Roy Schwartz, Ari Rappoport, Roi Reichart, Anna Korhonen

With our selected context configurations, we train on only 14% (A), 26. 2% (V), and 33. 6% (N) of all dependency-based contexts, resulting in a reduced training time.

Word Similarity

HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment

no code implementations CL 2017 Ivan Vulić, Daniela Gerz, Douwe Kiela, Felix Hill, Anna Korhonen

We introduce HyperLex - a dataset and evaluation resource that quantifies the extent of of the semantic category membership, that is, type-of relation also known as hyponymy-hypernymy or lexical entailment (LE) relation between 2, 616 concept pairs.

Lexical Entailment Relation +1

SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity

1 code implementation EMNLP 2016 Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, Anna Korhonen

Verbs play a critical role in the meaning of sentences, but these ubiquitous words have received little attention in recent distributional semantics research.

Representation Learning

Bilingual Distributed Word Representations from Document-Aligned Comparable Data

no code implementations24 Sep 2015 Ivan Vulić, Marie-Francine Moens

We propose a new model for learning bilingual word representations from non-parallel document-aligned data.

Representation Learning Sentence +2

Cannot find the paper you are looking for? You can Submit a new open access paper.