Search Results for author:
Found 66 papers, 11 papers with code
Language Technologies for Low Resource Languages: Sociolinguistic and Multilingual Insights
There is a growing interest in building language technologies (LTs) for low resource languages (LRLs).
A Case Study of Efficacy and Challenges in Practical Human-in-Loop Evaluation of NLP Systems Using Checklist
We conduct experiments for evaluating an offensive content detection system and use a data augmentation technique for improving the model using insights from Checklist.
Fluent but Culturally Distant: Can Regional Training Teach Cultural Understanding?
Large language models (LLMs) are used around the world but exhibit Western cultural tendencies.
A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applications
Misgendering is the act of referring to someone by a gender that does not match their chosen identity.
Uncovering inequalities in new knowledge learning by large language models across different languages
In this paper, we explore inequalities in new knowledge learning by LLMs across different languages and four key dimensions: effectiveness, transferability, prioritization, and robustness.
Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Model
A common challenge towards the adaptability of Large Language Models (LLMs) is their ability to learn new languages over time without hampering the model's performance on languages in which the model is already proficient (usually English).
Exploring Pretraining via Active Forgetting for Improving Cross Lingual Transfer for Decoder Language Models
Prior works have shown that encoder-only models such as BERT or XLM-RoBERTa show impressive cross lingual transfer of their capabilities from English to other languages.
Contamination Report for Multilingual Benchmarks
Benchmark contamination refers to the presence of test datasets in Large Language Model (LLM) pre-training or post-training data.
HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-World Multilingual Settings
Assessing the capabilities and limitations of large language models (LLMs) has garnered significant interest, yet the evaluation of multiple models in real-world scenarios remains rare.
JOOCI: a Framework for Learning Comprehensive Speech Representations
Given the orthogonal nature of other and content information, attempting to optimize both within a single embedding results in suboptimal solutions.
Speech Representation Learning Revisited: The Necessity of Separate Learnable Parameters and Robust Data Augmentation
Current self-supervised learning (SSL) methods such as HuBERT are very good at modeling the content information present in speech.
sPhinX: Sample Efficient Multilingual Instruction Fine-Tuning Through N-shot Guided Prompting
In order to address this, we introduce a novel recipe for creating a multilingual synthetic instruction tuning dataset, sPhinX, which is created by selectively translating instruction response pairs from English into 50 languages.
Improving Self Consistency in LLMs through Probabilistic Tokenization
Prior research has demonstrated noticeable performance gains through the use of probabilistic tokenizations, an approach that involves employing multiple tokenizations of the same input string during the training phase of a language model.
M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks
Since the release of ChatGPT, the field of Natural Language Processing has experienced rapid advancements, particularly in Large Language Models (LLMs) and their multimodal counterparts, Large Multimodal Models (LMMs).
Teaching LLMs to Abstain across Languages via Multilingual Feedback
Multilingual LLMs often have knowledge disparities across languages, with larger gaps in under-resourced languages.
PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data
We find that humans and LLMs agree fairly well in the pairwise setting but the agreement drops for direct assessment evaluation especially for languages such as Bengali and Odia.
Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Prompting
We observe that all models except GPT-4 show significant variations in their responses on both kinds of datasets for both kinds of prompts, casting doubt on the robustness of the culturally-conditioned prompting as a method for eliciting cultural bias in models or as an alignment strategy.
MS-HuBERT: Mitigating Pre-training and Inference Mismatch in Masked Language Modelling methods for learning Speech Representations
In recent years, self-supervised pre-training methods have gained significant traction in learning high-level information from raw speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios
The deployment of Large Language Models (LLMs) in real-world applications presents both opportunities and challenges, particularly in multilingual and code-mixed communication settings.
Bridging the Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs
Second, we introduce a new hybrid approach that synergizes LLM Retrieval Augmented Generation (RAG) with multilingual embeddings and achieves improved multilingual task performance.
METAL: Towards Multilingual Meta-Evaluation
This dataset is created specifically to evaluate LLM-based evaluators, which we refer to as meta-evaluation (METAL).
TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs
For scenarios where the model weights need to be kept private, we describe solutions from confidential computing and cryptography that can aid in private benchmarking.
DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures
Generative models are increasingly being used in various applications, such as text generation, commonsense reasoning, and question-answering.
A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
In this study, we propose a unified framework for systematically evaluating gender, race, and age biases in VLMs with respect to professions.
MAFIA: Multi-Adapter Fused Inclusive LanguAge Models
In this work, we aim to modularly debias a pretrained language model across multiple dimensions.
CultureLLM: Incorporating Cultural Differences into Large Language Models
Since multilingual cultural data are often expensive to collect, existing efforts handle this by prompt engineering or culture-specific pre-training.
MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models
Prior work on multilingual evaluation has shown that there is a large gap between the performance of LLMs on English and other languages.
MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks
We also perform a study on data contamination and find that several models are likely to be contaminated with multilingual evaluation benchmarks, necessitating approaches to detect and handle contamination while assessing the multilingual performance of LLMs.
Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation
Multilingualism is widespread around the world and code-switching (CSW) is a common practice among different language pairs/tuples across locations and regions.
Partial Rank Similarity Minimization Method for Quality MOS Prediction of Unseen Speech Synthesis Systems in Zero-Shot and Semi-supervised setting
That is the partial rank similarity is measured (PRS) rather than the individual MOS values as with the L1 loss.
Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?
Large Language Models (LLMs) excel in various Natural Language Processing (NLP) tasks, yet their evaluation, particularly in languages beyond the top $20$, remains inadequate due to existing benchmarks and metrics limitations.
On Evaluating and Mitigating Gender Biases in Multilingual Settings
While understanding and removing gender biases in language models has been a long-standing problem in Natural Language Processing, prior research work has primarily been limited to English.
Bridging the Language Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs
Large language models (LLMs) have revolutionized various domains but still struggle with non-Latin scripts and low-resource languages.
MEGA: Multilingual Evaluation of Generative AI
Most studies on generative LLMs have been restricted to English and it is unclear how capable these models are at understanding and generating text in other languages.
DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer
Zero-shot cross-lingual transfer is promising, however has been shown to be sub-optimal, with inferior transfer performance across low-resource languages.
Fairness in Language Models Beyond English: Gaps and Challenges
With language models becoming increasingly ubiquitous, it has become essential to address their inequitable treatment of diverse demographic groups and factors.
A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies
To fill this gap, we offer a survey of code-switching (C-S) covering the literature in linguistics with a reflection on the key issues in language technologies.
Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition
Code-switching poses a number of challenges and opportunities for multilingual automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
On the Calibration of Massively Multilingual Language Models
Massively Multilingual Language Models (MMLMs) have recently gained popularity due to their surprising effectiveness in cross-lingual transfer.
Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languages
Although recent Massively Multilingual Language Models (MMLMs) like mBERT and XLMR support around 100 languages, most existing multilingual NLP benchmarks provide evaluation data in only a handful of these languages with little linguistic diversity.
Multilingual CheckList: Generation and Evaluation
We compare the TEA CheckLists with CheckLists created with different levels of human intervention.
A Survey of Multilingual Models for Automatic Speech Recognition
Although Automatic Speech Recognition (ASR) systems have achieved human-like performance for a few languages, the majority of the world's languages do not have usable systems due to the lack of large speech datasets to train these models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Predicting the Performance of Multilingual NLP Models
Recent advancements in NLP have given us models like mBERT and XLMR that can serve over 100 languages.
On the Universality of Deep Contextual Language Models
Deep Contextual Language Models (LMs) like ELMO, BERT, and their successors dominate the landscape of Natural Language Processing due to their ability to scale across multiple tasks rapidly by pre-training a single model, followed by task-specific fine-tuning.
GCM: A Toolkit for Generating Synthetic Code-mixed Text
Code-mixing is common in multilingual communities around the world, and processing it is challenging due to the lack of labeled and unlabeled data.
Multilingual and code-switching ASR challenges for low resource Indian languages
For this purpose, we provide a total of ~600 hours of transcribed speech data, comprising train and test sets, in these languages including two code-switched language pairs, Hindi-English and Bengali-English.
mask-Net: Learning Context Aware Invariant Features using Adversarial Forgetting (Student Abstract)
Training a robust system, e. g., Speech to Text (STT), requires large datasets.
Cross-lingual and Multilingual Spoken Term Detection for Low-Resource Indian Languages
Since no publicly available dataset exists for Spoken Term Detection in these languages, we create a new dataset using a publicly available TTS dataset.
GLUECoS: An Evaluation Benchmark for Code-Switched NLP
We present results on all these tasks using cross-lingual word embedding models and multilingual models.
Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition
Recently, we showed that monolingual ASR systems fine-tuned on code-switched data deteriorate in performance on monolingual speech recognition, which is not desirable as ASR systems deployed in multilingual scenarios should recognize both monolingual and code-switched speech with high accuracy.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Learning to Recognize Code-switched Speech Without Forgetting Monolingual Speech Recognition
In this work, we show that fine-tuning ASR models on code-switched speech harms performance on monolingual speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Crowdsourcing Speech Data for Low-Resource Languages from Low-Income Workers
Unfortunately, collecting labelled speech data in any language is an expensive and resource-intensive task.
GLUECoS : An Evaluation Benchmark for Code-Switched NLP
We present results on all these tasks using cross-lingual word embedding models and multilingual models.
A New Dataset for Natural Language Inference from Code-mixed Conversations
Code-mixing is the use of more than one language in the same conversation or utterance, and is prevalent in multilingual communities all over the world.
Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities
In this paper, we examine and analyze the challenges associated with developing and introducing language technologies to low-resource language communities.
CoSSAT: Code-Switched Speech Annotation Tool
Code-switching refers to the alternation of two or more languages in a conversation or utterance and is common in multilingual communities across the world.
End-to-End ASR for Code-switched Hindi-English Speech
While the lack of data adversely affects the performance of end-to-end models, we see promising improvements with MTL and balancing the corpus.
A Survey of Code-switched Speech and Language Processing
Code-switching, the alternation of languages within a conversation or utterance, is a common communicative phenomenon that occurs in multilingual communities across the world.
Word Embeddings for Code-Mixed Language Processing
We compare three existing bilingual word embedding approaches, and a novel approach of training skip-grams on synthetic code-mixed text generated through linguistic models of code-mixing, on two tasks - sentiment analysis and POS tagging for code-mixed text.
Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data
Training language models for Code-mixed (CM) language is known to be a difficult problem because of lack of data compounded by the increased confusability due to the presence of more than one language.
Automatic Speech Recognition (ASR)
Language Identification
+4
Automatic Detection of Code-switching Style from Acoustics
We hypothesize that it may be useful for an ASR system to be able to first detect the switching style of a particular utterance from acoustics, and then use specialized language models or other adaptation techniques for decoding the speech.
Automatic Speech Recognition (ASR)
Language Identification
+1
Phone Merging For Code-Switched Speech Recognition
Though the best performance gain of 1. 2{\%} WER was observed with manually merged phones, we show experimentally that the manual phone merge is not optimal.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning
We introduce polyglot language models, recurrent neural network models trained to predict symbol sequences in many different languages using shared representations of symbols and conditioning on typological information about the language to be predicted.
Speech Synthesis of Code-Mixed Text
Most Text to Speech (TTS) systems today assume that the input text is in a single language and is written in the same language that the text needs to be synthesized in.
