Search Results for author:
Found 63 papers, 8 papers with code
Online Abuse and Human Rights: WOAH Satellite Session at RightsCon 2020
In 2020 The Workshop on Online Abuse and Harms (WOAH) held a satellite panel at RightsCons 2020, an international human rights conference.
BeSt: The Belief and Sentiment Corpus
We present the BeSt corpus, which records cognitive state: who believes what (i. e., factuality), and who has what sentiment towards what.
Findings of the WOAH 5 Shared Task on Fine Grained Hateful Memes Detection
We present the results and main findings of the shared task at WOAH 5 on hateful memes detection.
Towards Geo-Culturally Grounded LLM Generations
Specifically, we compare the performance of standard LLMs, LLMs augmented with retrievals from a bespoke knowledge base (i. e., KB grounding), and LLMs augmented with retrievals from a web search (i. e., search grounding) on a series of cultural familiarity benchmarks.
Risks of Cultural Erasure in Large Language Models
As a result, language models will shape how people learn about, perceive and interact with global cultures making it important to consider whose knowledge systems and perspectives are represented in models.
Insights on Disagreement Patterns in Multimodal Safety Perception across Diverse Rater Groups
Our study shows that (1) there are significant differences across demographic groups (including intersectional groups) on how severe they assess the harm to be, and that these differences vary across different types of safety violations, (2) the diverse rater pool captures annotation patterns that are substantially different from expert raters trained on specific set of safety policies, and (3) the differences we observe in T2I safety are distinct from previously documented group level differences in text-based safety tasks.
Beyond Aesthetics: Cultural Competence in Text-to-Image Models
CUBE consists of 1) CUBE-1K, a set of high-quality prompts that enable the evaluation of cultural awareness, and 2) CUBE-CSpace, a larger dataset of cultural artifacts that serves as grounding to evaluate cultural diversity.
D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation
While human annotations play a crucial role in language technologies, annotator subjectivity has long been overlooked in data collection.
GeniL: A Multilingual Dataset on Generalizing Language
We argue that understanding the sentential context is crucial for detecting instances of generalization.
SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes
While generative multilingual models are rapidly being deployed, their safety and fairness evaluations are largely limited to resources collected in English.
ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation
First, we show that stereotypical attributes in ViSAGe are thrice as likely to be present in generated images of corresponding identities as compared to other attributes, and that the offensiveness of these depictions is especially higher for identities from Africa, South America, and South East Asia.
Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates
More importantly, we find that individual moral values play a crucial role in shaping these variations: moral concerns about Care and Purity are significant mediating factors driving cross-cultural differences.
SoUnD Framework: Analyzing (So)cial Representation in (Un)structured (D)ata
The unstructured nature of data used in foundation model development is a challenge to systematic analyses for making data use and documentation decisions.
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives
Human annotation plays a core role in machine learning -- annotations for supervised models, safety guardrails for generative models, and human feedback for reinforcement learning, to cite a few avenues.
A Taxonomy of Rater Disagreements: Surveying Challenges & Opportunities from the Perspective of Annotating Online Toxicity
Toxicity is an increasingly common and severe issue in online spaces.
SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models
Stereotype benchmark datasets are crucial to detect and mitigate social stereotypes about groups of people in NLP models.
MD3: The Multi-Dialect Dataset of Dialogues
We introduce a new dataset of conversational speech representing English from India, Nigeria, and the United States.
Cultural Re-contextualization of Fairness Research in Language Technologies in India
Recent research has revealed undesirable biases in NLP data and models.
Cultural Incongruencies in Artificial Intelligence
Artificial intelligence (AI) systems attempt to imitate human behavior.
Underspecification in Scene Description-to-Depiction Tasks
Questions regarding implicitness, ambiguity and underspecification are crucial for understanding the task validity and ethical concerns of multimodal image+text systems, yet have received little attention to date.
A Human Rights-Based Approach to Responsible AI
Research on fairness, accountability, transparency and ethics of AI-based interventions in society has gained much-needed momentum in recent years.
Re-contextualizing Fairness in NLP: The Case of India
In this paper, we focus on NLP fair-ness in the context of India.
CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation
Human annotated data plays a crucial role in machine learning (ML) research and development.
Evaluation Gaps in Machine Learning Practice
Forming a reliable judgement of a machine learning (ML) model's appropriateness for an application ecosystem is critical for its responsible use, and requires considering a broad range of factors including harms, benefits, and responsibilities.
PaLM: Scaling Language Modeling with Pathways
To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
Ranked #1 on
Coreference Resolution
on Winograd Schema Challenge
LaMDA: Language Models for Dialog Applications
We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding.
Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation
Human annotations play a crucial role in machine learning (ML) research and development.
Thinking Beyond Distributions in Testing Machine Learned Models
Testing practices within the machine learning (ML) community have centered around assessing a learned model's predictive performance measured against a test dataset, often drawn from the same distribution as the training dataset.
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations
Majority voting and averaging are common approaches employed to resolve annotator disagreements and derive single ground truth labels from multiple annotations.
On Releasing Annotator-Level Labels and Information in Datasets
A common practice in building NLP datasets, especially using crowd-sourced annotations, involves obtaining multiple annotator judgements on the same data instances, which are then flattened to produce a single "ground truth" label or score, through majority voting, averaging, or adjudication.
Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media
Online social media platforms increasingly rely on Natural Language Processing (NLP) techniques to detect abusive content at scale in order to mitigate the harms it causes to their users.
How Metaphors Impact Political Discourse: A Large-Scale Topic-Agnostic Study Using Neural Metaphor Detection
Metaphors are widely used in political rhetoric as an effective framing device.
Re-imagining Algorithmic Fairness in India and Beyond
Instead, we re-imagine algorithmic fairness in India and provide a roadmap to re-contextualise data and models, empower oppressed communities, and enable Fair-ML ecosystems.
Non-portability of Algorithmic Fairness in India
Conventional algorithmic fairness is Western in its sub-groups, values, and optimizations.
Learning to Recognize Dialect Features
Evaluation on a test set of 22 dialect features of Indian English demonstrates that these models learn to recognize many features with high accuracy, and that a few minimal pairs can be as effective for training as thousands of labeled examples.
Extending the Machine Learning Abstraction Boundary: A Complex Systems Approach to Incorporate Societal Context
Machine learning (ML) fairness research tends to focus primarily on mathematically-based interventions on often opaque algorithms or models and/or their immediate inputs and outputs.
Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics
Recent research on algorithmic fairness has highlighted that the problem formulation phase of ML system development can be a key source of bias that has significant downstream impacts on ML system fairness outcomes.
Social Biases in NLP Models as Barriers for Persons with Disabilities
Building equitable and inclusive NLP technologies demands consideration of whether and how social attitudes are represented in ML models.
Perturbation Sensitivity Analysis to Detect Unintended Model Biases
Data-driven statistical Natural Language Processing (NLP) techniques leverage large amounts of language data to build models that can understand language.
Power Networks: A Novel Neural Architecture to Predict Power Relations
Can language analysis reveal the underlying social power relations that exist between participants of an interaction?
Socially Responsible NLP
As language technologies have become increasingly prevalent, there is a growing awareness that decisions we make about our data, methods, and tools are often tied up with their impact on people and societies.
Author Commitment and Social Power: Automatic Belief Tagging to Infer the Social Context of Interactions
Understanding how social power structures affect the way we interact with one another is of great interest to social scientists who want to answer fundamental questions about human behavior, as well as to computer scientists who want to build automatic methods to infer the social contexts of interactions.
Detecting Institutional Dialog Acts in Police Traffic Stops
We apply computational dialog methods to police body-worn camera footage to model conversations between police officers and community members in traffic stops.
Dialog Structure Through the Lens of Gender, Gender Environment, and Power
In this paper, we study the interaction of power, gender, and dialog behavior in organizational interactions.
Computational Argumentation Quality Assessment in Natural Language
Research on computational argumentation faces the problem of how to automatically assess the quality of an argument or argumentation.
A Corpus of Wikipedia Discussions: Over the Years, with Topic, Power and Gender Labels
In order to gain a deep understanding of how social context manifests in interactions, we need data that represents interactions from a large community of people over a long period of time, capturing different aspects of social context.
Statistical modality tagging from rule-based annotations and crowdsourcing
We explore training an automatic modality tagger.
Annotations for Power Relations on Email Threads
In this paper, we describe a multi-layer annotation scheme for social power relations that are recognizable from online written interactions.
