Search Results for author:
Found 110 papers, 31 papers with code
We Need to Consider Disagreement in Evaluation
Instead, we suggest that we need to better capture the sources of disagreement to improve today’s evaluation practice.
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design
Over the last several years, end-to-end neural conversational agents have vastly improved their ability to carry unrestricted, open-domain conversations with humans.
Hard and Soft Evaluation of NLP models with BOOtSTrap SAmpling - BooStSa
Natural Language Processing (NLP) ‘s applied nature makes it necessary to select the most effective and robust models.
A Report on the VarDial Evaluation Campaign 2020
This paper presents the results of the VarDial Evaluation Campaign 2020 organized as part of the seventh workshop on Natural Language Processing (NLP) for Similar Languages, Varieties and Dialects (VarDial), co-located with COLING 2020.
Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals
Current language technology is ubiquitous and directly influences individuals’ lives worldwide.
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems
We then empirically assess the extent to which current tools can measure these effects and current systems display them.
Benchmarking Post-Hoc Interpretability Approaches for Transformer-based Misogyny Detection
In this paper, we provide the first benchmark study of interpretability approaches for hate speech detection.
XLM-EMO: Multilingual Emotion Prediction in Social Media Text
Detecting emotion in text allows social and computational scientists to study how people behave and react to online events.
MilaNLP @ WASSA: Does BERT Feel Sad When You Cry?
The paper describes the MilaNLP team’s submission (Bocconi University, Milan) in the WASSA 2021 Shared Task on Empathy Detection and Emotion Classification.
FEEL-IT: Emotion and Sentiment Classification for the Italian Language
While sentiment analysis is a popular task to understand people’s reactions online, we often need more nuanced information: is the post negative because the user is angry or sad?
Universal Joy A Data Set and Results for Classifying Emotions Across Languages
While emotions are universal aspects of human psychology, they are expressed differently across different languages and cultures.
Pipelines for Social Bias Testing of Large Language Models
We hope to open a discussion on the best methodologies to handle social bias testing in language models.
Conversations as a Source for Teaching Scientific Concepts at Different Education Levels
While language models hold great promise for educational applications, there are substantial challenges in training them to engage in meaningful and effective conversational teaching, especially when considering the diverse needs of various audiences.
SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety
Researchers and practitioners have met these concerns by introducing an abundance of new datasets for evaluating and improving LLM safety.
DADIT: A Dataset for Demographic Classification of Italian Twitter Users and a Comparison of Prediction Methods
DADIT enables us to train and compare the performance of various state-of-the-art models for the prediction of the gender and age of social media users.
Classist Tools: Social Class Correlates with Performance in NLP
We argue for the inclusion of socioeconomic class in future language technologies.
Impoverished Language Technology: The Lack of (Social) Class in NLP
Since Labov's (1964) foundational work on the social stratification of language, linguistics has dedicated concerted efforts towards understanding the relationships between socio-demographic factors and language production and perception.
Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution
We then analyze the emotions generated by the models in relation to the gender-event pairs.
Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions
We then discuss four lacunae: (1) the absence of demographic and cultural aspects does not account for the variation in how emotions are perceived, but instead assumes they are universally experienced in the same manner; (2) the poor fit of emotion categories from the two main emotion theories to the task; (3) the lack of standardized EA terminology hinders gap identification, comparison, and future goals; and (4) the absence of interdisciplinary research isolates EA from insights in other fields.
Multilingual Speech Models for Automatic Speech Recognition Exhibit Gender Performance Gaps
However, the advantaged group varies between languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models
Motivated by this discrepancy, we challenge the prevailing constrained evaluation paradigm for values and opinions in LLMs and explore more realistic unconstrained evaluations.
"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models
The open-ended nature of language generation makes the evaluation of autoregressive large language models (LLMs) challenging.
Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?
We compare pre-training models with human context via 1) group attributes, 2) individual users, and 3) a combined approach on 5 user- and document-level tasks.
Know Your Audience: Do LLMs Adapt to Different Age and Education Levels?
We find large variations in the readability of the answers by different LLMs.
How to Use Large Language Models for Text Coding: The Case of Fatherhood Roles in Public Policy Documents
Recent advances in large language models (LLMs) like GPT-3 and GPT-4 have opened up new opportunities for text analysis in political science.
Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Features
Existing work focuses on a few spoken language understanding (SLU) tasks, and explanations are difficult to interpret for most users.
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
In this paper, we introduce a new test suite called XSTest to identify such eXaggerated Safety behaviours in a systematic way.
Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variation
Recent studies emphasize the importance of considering human label variation in data annotation.
The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics
To account for sociodemographics in models of individual annotator behaviour, we introduce group-specific layers to multi-annotator models.
What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns
As 3rd-person pronoun usage shifts to include novel forms, e. g., neopronouns, we need more research on identity-inclusive NLP.
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible.
Leveraging Social Interactions to Detect Misinformation on Social Media
Keeping track of the sequence of the interactions during the time, we improve over previous state-of-the-art models.
Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement
We commonly use agreement measures to assess the utility of judgements made by human annotators in Natural Language Processing (NLP) tasks.
Beyond Digital "Echo Chambers": The Role of Viewpoint Diversity in Political Discussion
We find that the diversity scores for both Fragmentation and Representation are lower for immigration than for DST.
Bridging Fairness and Environmental Sustainability in Natural Language Processing
Fairness and environmental impact are important research directions for the sustainable development of artificial intelligence.
SocioProbe: What, When, and Where Language Models Learn about Sociodemographics
Our results show that PLMs do encode these sociodemographics, and that this knowledge is sometimes spread across the layers of some of the tested PLMs.
"It's Not Just Hate'': A Multi-Dimensional Perspective on Detecting Harmful Speech Online
Well-annotated data is a prerequisite for good Natural Language Processing models.
ProSiT! Latent Variable Discovery with PROgressive SImilarity Thresholds
The most common ways to explore latent document dimensions are topic models and clustering methods.
Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages
More data is needed, but annotating hateful content is expensive, time-consuming and potentially harmful to annotators.
The State of Profanity Obfuscation in Natural Language Processing
Work on hate speech has made the consideration of rude and harmful examples in scientific publications inevitable.
Is It Worth the (Environmental) Cost? Limited Evidence for Temporal Adaptation via Continuous Training
Consequently, we should continuously update our models with new data to expose them to new events and facts.
Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers
Previous work showed that incorporating demographic factors can consistently improve performance for various NLP tasks with traditional NLP models.
On the Limitations of Sociodemographic Adaptation with Transformers
We adapt the language representations for the sociodemographic dimensions of gender and age, using continuous language modeling and dynamic multi-task learning for adaptation, where we couple language modeling with the prediction of a sociodemographic class.
Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists
EAR also reveals overfitting terms, i. e., terms most likely to induce bias, to help identify their effect on the model, task, and predictions.
Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gender
Based on our observations and ethical considerations, we define a series of desiderata for modeling pronouns in language technology.
Twitter-Demographer: A Flow-based Tool to Enrich Twitter Data
Twitter data have become essential to Natural Language Processing (NLP) and social science research, driving various scientific discoveries in recent years.
Top-Down Influence? Predicting CEO Personality and Risk Impact from Speech Transcripts
How much does a CEO's personality impact the performance of their company?
Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks
To address this issue, we propose two contrasting paradigms for data annotation.
Language Invariant Properties in Natural Language Processing
We introduce language invariant properties: i. e., properties that should not change when we transform text, and how they can be used to quantitatively evaluate the robustness of transformation algorithms.
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
Over the last several years, end-to-end neural conversational agents have vastly improved in their ability to carry a chit-chat conversation with humans.
Beyond Black \& White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning
Supervised learning assumes that a ground truth label exists.
HONEST: Measuring Hurtful Sentence Completion in Language Models
Our results show that 4. 3{\%} of the time, language models complete a sentence with a hurtful word.
Ranked #1 on
Hurtful Sentence Completion
on HONEST
The Importance of Modeling Social Factors of Language: Theory and Practice
We show that current NLP systems systematically break down when faced with interpreting the social factors of language.
BERTective: Language Models and Contextual Information for Deception Detection
In most cases, however, the target texts{'} preceding context is not considered.
Helpful or Hierarchical? Predicting the Communicative Strategies of Chat Participants, and their Impact on Success
When interacting with each other, we motivate, advise, inform, show love or power towards our peers.
``You Sound Just Like Your Father'' Commercial Machine Translation Systems Include Stylistic Biases
The main goal of machine translation has been to convey the correct content.
Integrating Ethics into the NLP Curriculum
To raise awareness among future NLP practitioners and prevent inertia in the field, we need to place ethics in the curriculum for all NLP students{---}not as an elective, but as a core part of their education.
Cross-lingual Contextualized Topic Models with Zero-shot Learning
They all cover the same content, but the linguistic differences make it impossible to use traditional, bag-of-word-based topic models.
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence
Topic models extract groups of words from documents, whose interpretation as a topic hopefully allows for a better understanding of the data.
What the [MASK]? Making Sense of Language-Specific BERT Models
Driven by the potential of BERT models, the NLP community has started to investigate and generate an abundant number of BERT models that are trained on a particular language, and tested on a specific data domain and task.
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview
In this paper, we propose a unifying conceptualization: the predictive bias framework for NLP.
Hey Siri. Ok Google. Alexa: A topic modeling of user reviews for smart speakers
User reviews provide a significant source of information for companies to understand their market and audience.
Identifying Linguistic Areas for Geolocation
We create three sets of labels at different levels of granularity, and compare performance of a state-of-the-art geolocation model trained and tested with P2C labels to one with regular k-d tree labels.
Geolocation with Attention-Based Multitask Learning Models
Geolocation, predicting the location of a post based on text and other information, has a huge potential for several social media applications.
Dense Node Representation for Geolocation
Prior research has shown that geolocation can be substantially improved by including user network information.
Women's Syntactic Resilience and Men's Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing
Several linguistic studies have shown the prevalence of various lexical and grammatical patterns in texts authored by a person of a particular gender, but models for part-of-speech tagging and dependency parsing have still not adapted to account for these differences.
Increasing In-Class Similarity by Retrofitting Embeddings with Demographic Information
We use homophily cues to retrofit text-based author representations with non-linguistic information, and introduce a trade-off parameter.
Capturing Regional Variation with Distributed Place Representations and Geographic Retrofitting
Dialects are one of the main drivers of language variation, a major challenge for natural language processing tools.
Predicting News Headline Popularity with Syntactic and Semantic Knowledge Using Multi-Task Learning
However, we also find that performance is very similar to that of a simple Logistic Regression model over character n-grams.
The Social and the Neural Network: How to Make Natural Language Processing about People again
Over the years, natural language processing has increasingly focused on tasks that can be solved by statistical models, but ignored the social aspects of language.
Comparing Bayesian Models of Annotation
We evaluate these models along four aspects: comparison to gold labels, predictive accuracy for new annotations, annotator characterization, and item difficulty, using four datasets with varying degrees of noise in the form of random (spammy) annotators.
Multi-Task Learning for Mental Health using Social Media Text
We introduce initial groundwork for estimating suicide risk and mental health in a deep learning framework.
Huntsville, hospitals, and hockey teams: Names can reveal your location
Geolocation is the task of identifying a social media user{'}s primary location, and in natural language processing, there is a growing literature on to what extent automated analysis of social media posts can help.
End-to-End Information Extraction without Token-Level Supervision
End-to-end (E2E) models, which take raw text as input and produce the desired output directly, need not depend on token-level labels.
Exploring Language Variation Across Europe - A Web-based Tool for Computational Sociolinguistics
Language varies not only between countries, but also along regional and socio-demographic lines.
When POS data sets don't add up: Combatting sample bias
We present a systematic study of several Twitter POS data sets, the problems of label and data bias, discuss their effects on model performance, and show how to overcome them to learn models that perform well on various test sets, achieving relative error reduction of up to 21{\%}.
Augmenting English Adjective Senses with Supersenses
We develop a supersense taxonomy for adjectives, based on that of GermaNet, and apply it to English adjectives in WordNet using human annotation and supervised classification.
Crowdsourcing and annotating NER for Twitter \#drift
We present two new NER datasets for Twitter; a manually annotated set of 1, 467 tweets (kappa=0. 942) and a set of 2, 975 expert-corrected, crowdsourced NER annotated tweets from the dataset described in Finin et al. (2010).
