Search Results for author:
Found 81 papers, 44 papers with code
Semantic Data Set Construction from Human Clustering and Spatial Arrangement
We demonstrate how the resultant data set can be used for fine-grained analyses and evaluation of representation learning models on the intrinsic tasks of semantic clustering and semantic similarity.
Natural Language Processing for Multilingual Task-Oriented Dialogue
In this tutorial, we will thus discuss and demonstrate the importance of (building) multilingual ToD systems, and then provide a systematic overview of current research gaps, challenges and initiatives related to multilingual ToD systems, with a particular focus on their connections to current research and challenges in multilingual and low-resource NLP.
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold
Most work targeting multilinguality, for example, considers only accuracy; most work on fairness or interpretability considers only English; and so on.
Data Augmentation and Learned Layer Aggregation for Improved Multilingual Language Understanding in Dialogue
Scaling dialogue systems to a multitude of domains, tasks and languages relies on costly and time-consuming data annotation for different domain-task-language configurations.
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer
While offering (1) improved fine-tuning efficiency (by a factor of around 50 in our experiments), (2) a smaller parameter budget, and (3) increased language coverage, MAD-G remains competitive with more expensive methods for language-specific adapter training across the board.
Multi-SimLex: A Large-Scale Evaluation of Multilingual and Crosslingual Lexical Semantic Similarity
We introduce Multi-SimLex, a large-scale lexical resource and evaluation benchmark covering data sets for 12 typologically diverse languages, including major languages (e. g., Mandarin Chinese, Spanish, Russian) as well as less-resourced ones (e. g., Welsh, Kiswahili).
Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog
We then introduce a new framework for multilingual conversational specialization of pretrained language models (PrLMs) that aims to facilitate cross-lingual transfer for arbitrary downstream TOD tasks.
Exposing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders
Pretrained multilingual language models (LMs) can be successfully transformed into multilingual sentence encoders (SEs; e. g., LaBSE, xMPNET) via additional fine-tuning or model distillation on parallel data.
EVI: Multilingual Spoken Dialogue Tasks and Dataset for Knowledge-Based Enrolment, Verification, and Identification
Knowledge-based authentication is crucial for task-oriented spoken dialogue systems that offer personalised and privacy-focused services.
NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue
2) The ontology is divided into domain-specific and generic (i. e., domain-universal) intent modules that overlap across domains, promoting cross-domain reusability of annotated examples.
Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval
Current approaches therefore typically transfer rankers trained on English data to other languages and cross-lingual setups by means of multilingual encoders: they fine-tune all the parameters of a pretrained massively multilingual Transformer (MMT, e. g., multilingual BERT) on English relevance judgments and then deploy it in the target language.
Improved and Efficient Conversational Slot Labeling through Question Answering
In particular, we focus on modeling and studying \textit{slot labeling} (SL), a crucial component of NLU for dialog, through the QA optics, aiming to improve both its performance and efficiency, and make it more effective and resilient to working with limited task data.
Natural Language Understanding
Pretrained Language Models
+1
Improving Word Translation via Two-Stage Contrastive Learning
At Stage C1, we propose to refine standard cross-lingual linear maps between static word embeddings (WEs) via a contrastive learning objective; we also show how to integrate it into the self-learning procedure for even more refined cross-lingual maps.
Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation
Through this process we annotate a new large-scale dataset for training and evaluation of multilingual and cross-lingual ToD systems.
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups.
Cross-Lingual Visual Natural Language Inference
Cross-Modal Retrieval
+15
On Cross-Lingual Retrieval with Multilingual Text Encoders
In this work we present a systematic empirical study focused on the suitability of the state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks across a number of diverse language pairs.
Prix-LM: Pretraining for Multilingual Knowledge Base Construction
To achieve this, it is crucial to represent multilingual knowledge in a shared/unified space.
Composable Sparse Fine-Tuning for Cross-Lingual Transfer
Based on an in-depth analysis, we additionally find that sparsity is crucial to prevent both 1) interference between the fine-tunings to be composed and 2) overfitting.
ConvFiT: Conversational Fine-Tuning of Pretrained Language Models
Transformer-based language models (LMs) pretrained on large text collections are proven to store a wealth of semantic knowledge.
MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models
Recent work indicated that pretrained language models (PLMs) such as BERT and RoBERTa can be transformed into effective sentence and word encoders even via simple self-supervised techniques.
xGQA: Cross-Lingual Visual Question Answering
In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task.
Towards Zero-shot Language Modeling
Motivated by this question, we aim at constructing an informative prior over neural weights, in order to adapt quickly to held-out languages in the task of character-level language modeling.
Modelling Latent Translations for Cross-Lingual Transfer
To remedy this, we propose a new technique that integrates both steps of the traditional pipeline (translation and classification) into a single model, by treating the intermediate translations as a latent random variable.
RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models
We use the evaluation framework to benchmark the widely used conversational DialoGPT model along with the adaptations of four debiasing methods.
Conversational Response Generation
Pretrained Language Models
+1
Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking
To this end, we propose and evaluate a series of cross-lingual transfer methods for the XL-BEL task, and demonstrate that general-domain bitext helps propagate the available English knowledge to languages with little to no in-domain data.
Multilingual and Cross-Lingual Intent Detection from Spoken Data
We present a systematic study on multilingual and cross-lingual intent detection from spoken data.
Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems
We find that the most critical factor preventing the creation of truly multilingual ToD systems is the lack of datasets in most languages for both training and evaluation.
AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples
In order to address these gaps, we present AM2iCo (Adversarial and Multilingual Meaning in Context), a wide-coverage cross-lingual and multilingual evaluation set; it aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts for 14 language pairs.
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders
In this work, we demonstrate that it is possible to turn MLMs into effective universal lexical and sentence encoders even without any additional data and without any supervision.
Ranked #9 on
Semantic Textual Similarity
on STS16
Contrastive Learning
Cross-Lingual Semantic Textual Similarity
+3
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval
Current state-of-the-art approaches to cross-modal retrieval process text and visual input jointly, relying on Transformer-based architectures with cross-attention mechanisms that attend over all words and objects in an image.
Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrieval
Therefore, in this work we present a systematic empirical study focused on the suitability of the state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks across a large number of language pairs.
A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters
Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT.
Verb Knowledge Injection for Multilingual Event Processing
We investigate whether injecting explicit information on verbs' semantic-syntactic behaviour improves the performance of LM-pretrained Transformers in event extraction tasks -- downstream tasks for which accurate verb processing is paramount.
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts
The ultimate challenge is dealing with under-resourced languages not covered at all by the models and written in scripts unseen during pretraining.
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models
In this work, we provide a systematic and comprehensive empirical comparison of pretrained multilingual language models versus their monolingual counterparts with regard to their monolingual task performance.
Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer
Adapter modules, additional trainable parameters that enable efficient fine-tuning of pretrained transformers, have recently been used for language specialization of multilingual transformers, improving downstream zero-shot cross-lingual transfer.
Emergent Communication Pretraining for Few-Shot Machine Translation
On the other hand, this also provides an extrinsic evaluation protocol to probe the properties of emergent languages ex vitro.
ConVEx: Data-Efficient and Few-Shot Slot Labeling
We propose ConVEx (Conversational Value Extractor), an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks.
Probing Pretrained Language Models for Lexical Semantics
The success of large pretrained language models (LMs) such as BERT and RoBERTa has sparked interest in probing their representations, in order to unveil what types of knowledge they implicitly capture.
Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation
Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU).
AdapterHub: A Framework for Adapting Transformers
We propose AdapterHub, a framework that allows dynamic "stitching-in" of pre-trained adapters for different tasks and languages.
Span-ConveRT: Few-shot Span Extraction for Dialog with Pretrained Conversational Representations
We introduce Span-ConveRT, a light-weight model for dialog slot-filling which frames the task as a turn-based span extraction task.
Multidirectional Associative Optimization of Function-Specific Word Representations
We present a neural framework for learning associations between interrelated groups of words such as the ones found in Subject-Verb-Object (SVO) structures.
From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
Massively multilingual transformers pretrained with language modeling objectives (e. g., mBERT, XLM-R) have become a de facto default transfer paradigm for zero-shot cross-lingual transfer in NLP, offering unmatched transfer performance.
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning
In order to simulate human language capacity, natural language processing systems must be able to reason about the dynamics of everyday situations, including their possible causes and effects.
Ranked #1 on
Cross-Lingual Transfer
on XCOPA
(using extra training data)
MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
The main goal behind state-of-the-art pre-trained multilingual models such as multilingual BERT and XLM-R is enabling and bootstrapping NLP applications in low-resource languages through zero-shot or few-shot cross-lingual transfer.
Ranked #2 on
Cross-Lingual Transfer
on XCOPA
(using extra training data)
Towards Instance-Level Parser Selection for Cross-Lingual Transfer of Dependency Parsers
Current methods of cross-lingual parser transfer focus on predicting the best parser for a low-resource target language globally, that is, "at treebank level".
Are All Good Word Vector Spaces Isomorphic?
Existing algorithms for aligning cross-lingual word vector spaces assume that vector spaces are approximately isomorphic.
Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity
We introduce Multi-SimLex, a large-scale lexical resource and evaluation benchmark covering datasets for 12 typologically diverse languages, including major languages (e. g., Mandarin Chinese, Spanish, Russian) as well as less-resourced ones (e. g., Welsh, Kiswahili).
Efficient Intent Detection with Dual Sentence Encoders
Building conversational systems in new domains and with added functionality requires resource-efficient models that work under low-data regimes (i. e., in few-shot setups).
The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures
Performance in cross-lingual NLP tasks is impacted by the (dis)similarity of languages at hand: e. g., previous work has suggested there is a connection between the expected success of bilingual lexicon induction (BLI) and the assumption of (approximate) isomorphism between monolingual embedding spaces.
Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages
In this work, we propose a Bayesian generative model for the space of neural parameters.
ConveRT: Efficient and Accurate Conversational Representations from Transformers
General-purpose pretrained sentence encoders such as BERT are not ideal for real-world conversational AI applications; they are computationally heavy, slow, and expensive to train.
Ranked #1 on
Conversational Response Selection
on PolyAI AmazonQA
On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages
Recent work has validated the importance of subword information for word representation learning.
A General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spaces
Moreover, we successfully transfer debiasing models, by means of cross-lingual embedding spaces, and remove or attenuate biases in distributional word vector spaces of languages that lack readily available bias specifications.
Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity
In this work, we complement such distributional knowledge with external lexical knowledge, that is, we integrate the discrete knowledge on word-level semantic similarity into pretraining.
Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?
A series of bilingual lexicon induction (BLI) experiments with 15 diverse languages (210 language pairs) show that fully unsupervised CLWE methods still fail for a large number of language pairs (e. g., they yield zero BLI performance for 87/210 pairs).
PolyResponse: A Rank-based Approach to Task-Oriented Dialogue with Application in Restaurant Search and Booking
We present PolyResponse, a conversational search engine that supports task-oriented dialogue.
Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems
Data scarcity is a long-standing and crucial challenge that hinders quick development of task-oriented dialogue systems across multiple domains: task-oriented dialogue models are expected to learn grammar, syntax, dialogue reasoning, decision making, and language generation from absurdly small amounts of task-specific data.
Training Neural Response Selection for Task-Oriented Dialogue Systems
Despite their popularity in the chatbot literature, retrieval-based models have had modest impact on task-oriented dialogue systems, with the main obstacle to their application being the low-data regime of most task-oriented dialogue tasks.
A Systematic Study of Leveraging Subword Information for Learning Word Representations
The use of subword-level information (e. g., characters, character n-grams, morphemes) has become ubiquitous in modern word representation learning.
A Repository of Conversational Datasets
Progress in Machine Learning is often driven by the availability of large datasets, and consistent evaluation metrics for comparing modeling approaches.
Adversarial Propagation and Zero-Shot Cross-Lingual Transfer of Word Vector Specialization
Our adversarial post-specialization method propagates the external lexical knowledge to the full distributional space.
Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing
Linguistic typology aims to capture structural and semantic variation across the world's languages.
Fully Statistical Neural Belief Tracking
This paper proposes an improvement to the existing data-driven Neural Belief Tracking (NBT) framework for Dialogue State Tracking (DST).
Scoring Lexical Entailment with a Supervised Directional Similarity Network
Experiments show excellent performance on scoring graded lexical entailment, raising the state-of-the-art on the HyperLex dataset by approximately 25%.
On the Limitations of Unsupervised Bilingual Dictionary Induction
Unsupervised machine translation---i. e., not assuming any cross-lingual supervision signal, whether a dictionary, translations, or comparable corpora---seems impossible, but nevertheless, Lample et al. (2018) recently proposed a fully unsupervised machine translation (MT) model.
Post-Specialisation: Retrofitting Vectors of Words Unseen in Lexical Resources
Word vector specialisation (also known as retrofitting) is a portable, light-weight approach to fine-tuning arbitrary distributional word vector spaces by injecting external knowledge from rich lexical resources such as WordNet.
Unsupervised Cross-Lingual Information Retrieval using Monolingual Data Only
We propose a fully unsupervised framework for ad-hoc cross-lingual information retrieval (CLIR) which requires no bilingual data at all.
Specialising Word Vectors for Lexical Entailment
We present LEAR (Lexical Entailment Attract-Repel), a novel post-processing method that transforms any input word vector space to emphasise the asymmetric relation of lexical entailment (LE), also known as the IS-A or hyponymy-hypernymy relation.
Cross-Lingual Induction and Transfer of Verb Classes Based on Word Vector Space Specialisation
Existing approaches to automatic VerbNet-style verb classification are heavily dependent on feature engineering and therefore limited to languages with mature NLP pipelines.
A Survey Of Cross-lingual Word Embedding Models
Cross-lingual representations of words enable us to reason about word meaning in multilingual contexts and are a key facilitator of cross-lingual transfer when developing natural language processing models for low-resource languages.
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints
We present Attract-Repel, an algorithm for improving the semantic quality of word vectors by injecting constraints extracted from lexical resources.
Morph-fitting: Fine-Tuning Word Vector Spaces with Simple Language-Specific Rules
Morphologically rich languages accentuate two properties of distributional vector space models: 1) the difficulty of inducing accurate representations for low-frequency word forms; and 2) insensitivity to distinct lexical relations that have similar distributional signatures.
Decoding Sentiment from Distributed Representations of Sentences
Distributed representations of sentences have been developed recently to represent their meaning as real-valued vectors.
Survey on the Use of Typological Information in Natural Language Processing
In recent years linguistic typology, which classifies the world's languages according to their functional and structural properties, has been widely used to support multilingual NLP.
Automatic Selection of Context Configurations for Improved Class-Specific Word Representations
With our selected context configurations, we train on only 14% (A), 26. 2% (V), and 33. 6% (N) of all dependency-based contexts, resulting in a reduced training time.
HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment
We introduce HyperLex - a dataset and evaluation resource that quantifies the extent of of the semantic category membership, that is, type-of relation also known as hyponymy-hypernymy or lexical entailment (LE) relation between 2, 616 concept pairs.
SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity
Verbs play a critical role in the meaning of sentences, but these ubiquitous words have received little attention in recent distributional semantics research.
Bilingual Distributed Word Representations from Document-Aligned Comparable Data
We propose a new model for learning bilingual word representations from non-parallel document-aligned data.
