Search Results for author:
Found 33 papers, 15 papers with code
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Language modeling via stochastic processes
Recent work in self-supervised learning suggests that models can learn good latent representations via contrastive learning, which can be effective for discriminative tasks.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
We tackle the problem of evaluating faithfulness of a generated summary given its source document.
Whose Opinions Do Language Models Reflect?
Language models (LMs) are increasingly being used in open-ended contexts, where the opinions reflected by LMs in response to subjective queries can have a profound impact, both on user satisfaction, as well as shaping the views of society at large.
WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization
As a set of baselines for further studies, we evaluate the performance of existing cross-lingual abstractive summarization methods on our dataset.
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+2
Studying Large Language Model Generalization with Influence Functions
When trying to gain better visibility into a machine learning model in order to understand and mitigate the associated risks, a potentially valuable source of evidence is: which training examples most contribute to a given behavior?
Towards Understanding Sycophancy in Language Models
Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses.
Benchmarking Large Language Models for News Summarization
Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood.
Question Decomposition Improves the Faithfulness of Model-Generated Reasoning
As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior.
Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization
Despite recent progress in abstractive summarization, systems still suffer from faithfulness errors.
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
To recognize and mitigate harms from large language models (LLMs), we need to understand the prevalence and nuances of stereotypes in LLM outputs.
Evaluating Human-Language Model Interaction
To evaluate human-LM interaction, we develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and dimensions to consider when designing evaluation metrics.
Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale
For example, we find cases of prompting for basic traits or social roles resulting in images reinforcing whiteness as ideal, prompting for occupations resulting in amplification of racial and gender disparities, and prompting for objects resulting in reification of American norms.
Contrastive Error Attribution for Finetuned Language Models
We show that existing approaches for error tracing, such as gradient-based influence measures, do not perform reliably for detecting faithfulness errors in NLG datasets.
Understanding the Effect of Gender and Stance in Opinion Expression in Debates on ``Abortion''
We use the gender and stance information to identify significant linguistic differences across individuals with different gender and stance.
Determining Relative Argument Specificity and Stance for Complex Argumentative Structures
Systems for automatic argument generation and debate require the ability to (1) determine the stance of any claims employed in the argument and (2) assess the specificity of each claim relative to the argument context.
A Corpus for Modeling User and Language Effects in Argumentation on Online Debating
Existing argumentation datasets have succeeded in allowing researchers to develop computational methods for analyzing the content, structure and linguistic features of argumentative text.
Exploring the Role of Prior Beliefs for Argument Persuasion
Public debate forums provide a common platform for exchanging opinions on a topic of interest.
Persuasion of the Undecided: Language vs. the Listener
In a study of users of a popular debate platform, we find first that different combinations of linguistic features are critical for predicting persuasion outcomes for UNDECIDED versus DECIDED members of the audience.
The Role of Pragmatic and Discourse Context in Determining Argument Impact
Research in the social sciences and psychology has shown that the persuasiveness of an argument depends not only the language employed, but also on attributes of the source/communicator, the audience, and the appropriateness and strength of the argument's claims given the pragmatic and discourse context of the argument.
Exploring the Role of Argument Structure in Online Debate Persuasion
Online debate forums provide users a platform to express their opinions on controversial topics while being exposed to opinions from diverse set of viewpoints.
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics.
Ranked #1 on
Extreme Summarization
on GEM-XSum
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+5
Towards Understanding Persuasion in Computational Argumentation
Even though studies in Social Sciences and Psychology have shown that source and audience effects are essential components of the persuasion process, most research in computational persuasion has focused solely on understanding the characteristics of persuasive language.
Spurious Correlations in Reference-Free Evaluation of Text Generation
Model-based, reference-free evaluation metrics have been proposed as a fast and cost-effective approach to evaluate Natural Language Generation (NLG) systems.
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims.
Opportunities and Risks of LLMs for Scalable Deliberation with Polis
In this paper, we explore the opportunities and risks associated with applying Large Language Models (LLMs) towards challenges with facilitating, moderating and summarizing the results of Polis engagements.
Towards Measuring the Representation of Subjective Global Opinions in Language Models
We first build a dataset, GlobalOpinionQA, comprised of questions and answers from cross-national surveys designed to capture diverse opinions on global issues across different countries.
Measuring Faithfulness in Chain-of-Thought Reasoning
Large language models (LLMs) perform better when they produce step-by-step, "Chain-of-Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i. e., its process for answering the question).
Specific versus General Principles for Constitutional AI
Constitutional AI offers an alternative, replacing human feedback with feedback from AI models conditioned only on a list of written principles.
Evaluating and Mitigating Discrimination in Language Model Decisions
We present a method for proactively evaluating the potential discriminatory impact of LMs in a wide range of use cases, including hypothetical use cases where they have not yet been deployed.
NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps
The use of words to convey speaker's intent is traditionally distinguished from the `mention' of words for quoting what someone said, or pointing out properties of a word.
