Search Results for author:
Found 54 papers, 21 papers with code
Clustering Examples in Multi-Dataset Benchmarks with Item Response Theory
In natural language processing, multi-dataset benchmarks for common tasks (e. g., SuperGLUE for natural language inference and MRQA for question answering) have risen in importance.
Gendered Language in Resumes and its Implications for Algorithmic Bias in Hiring
Despite growing concerns around gender bias in NLP models used in algorithmic hiring, there is little empirical work studying the extent and nature of gendered language in resumes. Using a corpus of 709k resumes from IT firms, we train a series of models to classify the gender of the applicant, thereby measuring the extent of gendered information encoded in resumes. We also investigate whether it is possible to obfuscate gender from resumes by removing gender identifiers, hobbies, gender sub-space in embedding models, etc. We find that there is a significant amount of gendered information in resumes even after obfuscation. A simple Tf-Idf model can learn to classify gender with AUROC=0. 75, and more sophisticated transformer-based models achieve AUROC=0. 8. We further find that gender predictive values have low correlation with gender direction of embeddings – meaning that, what is predictive of gender is much more than what is “gendered” in the masculine/feminine sense. We discuss the algorithmic bias and fairness implications of these findings in the hiring context.
Measuring the Language of Self-Disclosure across Corpora
Being able to reliably estimate self-disclosure – a key component of friendship and intimacy – from language is important for many psychology studies.
Item Response Theory for Efficient Human Evaluation of Chatbots
Conversational agent quality is currently assessed using human evaluation, and often requires an exorbitant number of comparisons to achieve statistical significance.
WASSA 2022 Shared Task: Predicting Empathy, Emotion and Personality in Reaction to News Stories
This paper presents the results that were obtained from WASSA 2022 shared task on predicting empathy, emotion, and personality in reaction to news stories.
Multi-Emotion Classification for Song Lyrics
Song lyrics convey a multitude of emotions to the listener and powerfully portray the emotional state of the writer or singer.
WASSA 2021 Shared Task: Predicting Empathy and Emotion in Reaction to News Stories
This paper presents the results that were obtained from the WASSA 2021 shared task on predicting empathy and emotions.
Topic Modeling for Maternal Health Using Reddit
This paper applies topic modeling to understand maternal health topics, concerns, and questions expressed in online communities on social networking sites.
Collecting Verified COVID-19 Question Answer Pairs
We release a dataset of over 2, 100 COVID19 related Frequently asked Question-Answer pairs scraped from over 40 trusted websites.
Using the Poly-encoder for a COVID-19 Question Answering System
To combat misinformation regarding COVID- 19 during this unprecedented pandemic, we propose a conversational agent that answers questions related to COVID-19.
DBOT: Artificial Intelligence for Systematic Long-Term Investing
We compare DBOT to its analytic parent, Damodaran, and highlight the research challenges involved in raising its current capability to that of Damodaran's.
Reasoning and the Trusting Behavior of DeepSeek and GPT: An Experiment Revealing Hidden Fault Lines in Large Language Models
When encountering increasingly frequent performance improvements or cost reductions from a new large language model (LLM), developers of applications leveraging LLMs must decide whether to take advantage of these improvements or stay with older tried-and-tested models.
From Human Annotation to LLMs: SILICON Annotation Workflow for Management Research
Unstructured text data annotation and analysis are fundamental to management research, often relying on human annotators through crowdsourcing platforms.
The Illusion of Empathy: How AI Chatbots Shape Conversation Perception
As AI chatbots increasingly incorporate empathy, understanding user-centered perceptions of chatbot empathy and its impact on conversation quality remains essential yet under-explored.
Socially Responsible Data for Large Multilingual Language Models
There is growing interest in multilingual LLMs, and various efforts are striving for models to accommodate languages of communities outside of the Global North, which include many languages that have been historically underrepresented in digital realms.
Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas
Thus, it may be the case that employing LLMs (which do not have such human factors) in these tasks results in a lack of variation in data, failing to reflect the diversity of human experiences.
Large Language Models Show Human-like Social Desirability Biases in Survey Responses
As Large Language Models (LLMs) become widely used to model and simulate human behavior, understanding their biases becomes critical.
On the Role of Summary Content Units in Text Summarization Evaluation
At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs).
Large Human Language Models: A Need and the Challenges
As research in human-centered NLP advances, there is a growing recognition of the importance of incorporating human and social factors into NLP models.
An Integrative Survey on Mental Health Conversational Agents to Bridge Computer Science and Medical Perspectives
Mental health conversational agents (a. k. a.
Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4
The advent and fast development of neural networks have revolutionized the research on dialogue systems and subsequently have triggered various challenges regarding their automatic evaluation.
How to Choose How to Choose Your Chatbot: A Massively Multi-System MultiReference Data Set for Dialog Metric Evaluation
We release MMSMR, a Massively Multi-System MultiReference dataset to enable future work on metrics and evaluation for dialog.
Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization
To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization.
Conceptor-Aided Debiasing of Large Language Models
We propose two methods of applying conceptors (1) bias subspace projection by post-processing by the conceptor NOT operation; and (2) a new architecture, conceptor-intervened BERT (CI-BERT), which explicitly incorporates the conceptor projection into all layers during training.
Automatic Document Selection for Efficient Encoder Pretraining
Building pretrained language models is considered expensive and data-intensive, but must we increase dataset size to achieve better performance?
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims.
Empathic Conversations: A Multi-level Dataset of Contextualized Conversations
Hence, we collected detailed characterization of the participants' traits, their self-reported empathetic response to news articles, their conversational partner other-report, and turn-by-turn third-party assessments of the level of self-disclosure, emotion, and empathy expressed.
Benchmark Data and Evaluation Framework for Intent Discovery Around COVID-19 Vaccine Hesitancy
We use this framework to report baseline intent discovery results over VIRADialogs, that highlight the difficulty of this task.
Linear Connectivity Reveals Generalization Strategies
It is widely accepted in the mode connectivity literature that when two neural networks are trained similarly on the same data, they are connected by a path through parameter space over which test set accuracy is maintained.
VIRATrustData: A Trust-Annotated Corpus of Human-Chatbot Conversations About COVID-19 Vaccines
Public trust in medical information is crucial for successful application of public health policies such as vaccine uptake.
Degendering Resumes for Fair Algorithmic Resume Screening
We investigate whether it is feasible to remove gendered information from resumes to mitigate potential bias in algorithmic resume screening.
Trees in transformers: a theoretical analysis of the Transformer's ability to represent trees
To date, there are no theoretical analyses of the Transformer's ability to capture tree structures.
Automatic Evaluation and Moderation of Open-domain Dialogue Systems
The development of Open-Domain Dialogue Systems (ODS)is a trending topic due to the large number of research challenges, large societal and business impact, and advances in the underlying technology.
Who speaks like a style of Vitamin: Towards Syntax-Aware DialogueSummarization using Multi-task Learning
To the best of our knowledge, our approach is the first method to apply multi-task learning to the dialogue summarization task.
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics.
Ranked #1 on
Extreme Summarization
on GEM-XSum
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+5
SMRT Chatbots: Improving Non-Task-Oriented Dialog with Simulated Multiple Reference Training
Non-task-oriented dialog models suffer from poor quality and non-diverse responses.
An Evaluation Protocol for Generative Conversational Systems
These findings demonstrate the feasibility of our protocol to evaluate conversational agents and evaluation sets.
Measuring the `I don't know' Problem through the Lens of Gricean Quantity
We consider the intrinsic evaluation of neural generative dialog models through the lens of Grice's Maxims of Conversation (1975).
Decoding Methods for Neural Narrative Generation
Narrative generation is an open-ended NLP task in which a model generates a story given a prompt.
COD3S: Diverse Generation with Discrete Semantic Signatures
We present COD3S, a novel method for generating semantically diverse sentences using neural sequence-to-sequence (seq2seq) models.
Incremental Neural Coreference Resolution in Constant Memory
We investigate modeling coreference resolution under a fixed memory constraint by extending an incremental clustering algorithm to utilize contextualized encoders and neural components.
Learning Word Ratings for Empathy and Distress from Document-Level User Responses
The underlying problem of learning word ratings from higher-level supervision has to date only been addressed in an ad hoc fashion and has not used deep learning methods.
Comparison of Diverse Decoding Methods from Conditional Language Models
While conditional language models have greatly improved in their ability to output high-quality natural language, many NLP applications benefit from being able to generate a diverse set of candidate sequences.
Conceptor Debiasing of Word Representations Evaluated on WEAT
Bias in word embeddings such as Word2Vec has been widely investigated, and many efforts made to remove such bias.
Continual Learning for Sentence Representations Using Conceptors
Distributed representations of sentences have become ubiquitous in natural language processing tasks.
Complexity-Weighted Loss and Diverse Reranking for Sentence Simplification
Sentence simplification is the task of rewriting texts so they are easier to understand.
Ranked #4 on
Text Simplification
on Newsela
Unsupervised Post-processing of Word Vectors via Conceptor Negation
Word vectors are at the core of many natural language processing tasks.
Learning Emotion from 100 Observations: Unexpected Robustness of Deep Learning under Strong Data Limitations
One of the major downsides of Deep Learning is its supposed need for vast amounts of training data.
Modeling Empathy and Distress in Reaction to News Stories
Computational detection and understanding of empathy is an important factor in advancing human-computer interaction.
Multiscale Hidden Markov Models For Covariance Prediction
This paper presents a novel variant of hierarchical hidden Markov models (HMMs), the multiscale hidden Markov model (MSHMM), and an associated spectral estimation and prediction scheme that is consistent, finds global optima, and is computationally efficient.
Neural Tree Transducers for Tree to Tree Learning
We introduce a novel approach to tree-to-tree learning, the neural tree transducer (NTT), a top-down depth first context-sensitive tree decoder, which is paired with recursive neural encoders.
Enterprise to Computer: Star Trek chatbot
In this work, we propose a design for a chatbot that captures the "style" of Star Trek by incorporating references from the show along with peculiar tones of the fictional characters therein.
Domain Aware Neural Dialog System
We investigate the task of building a domain aware chat system which generates intelligent responses in a conversation comprising of different domains.
Semantic Word Clusters Using Signed Normalized Graph Cuts
Vector space representations of words capture many aspects of word similarity, but such methods tend to make vector spaces in which antonyms (as well as synonyms) are close to each other.
