Search Results for author:
Found 128 papers, 67 papers with code
Threat Scenarios and Best Practices to Detect Neural Fake News
In this work, we discuss different threat scenarios from neural fake news generated by state-of-the-art language models.
CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge
Our study reveals that CulturalTeaming's various modes of AI assistance support annotators in creating cultural questions, that modern LLMs fail at, in a gamified manner.
DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages
This allows for a comprehensive evaluation of NLP system performance on different language varieties.
Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs
In this paper, we introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent, compared to what is revealed by prompting the target model with the training data directly, which is the dominant approach of quantifying memorization in LLMs.
Extracting Lexical Features from Dialects via Interpretable Dialect Classifiers
Identifying linguistic differences between dialects of a language often requires expert knowledge and meticulous human analysis.
Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks
Our experiments reveal that almost none of the existing detectors remain robust under all the attacks, and all detectors exhibit different loopholes.
DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detection
Large language models are limited by challenges in factuality and hallucinations to be directly employed off-the-shelf for judging the veracity of news articles, where factual accuracy is paramount.
Do Membership Inference Attacks Work on Large Language Models?
Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data.
Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration
Despite efforts to expand the knowledge of large language models (LLMs), knowledge gaps -- missing or outdated information in LLMs -- might always persist given the evolving nature of knowledge.
What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detection
Social media bot detection has always been an arms race between advancements in machine learning bot detectors and adversarial bot strategies to evade detection.
Tuning Language Models by Proxy
Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors.
Fine-grained Hallucination Detection and Editing for Language Models
On our benchmark, our automatic and human evaluations show that FAVA significantly outperforms ChatGPT and GPT-4 on fine-grained hallucination detection, and edits suggested by FAVA improve the factuality of LM-generated text.
P^3SUM: Preserving Author's Perspective in News Summarization with Diffusion Language Models
In this work, we take a first step towards designing summarization systems that are faithful to the author's intent, not only the semantic content of the article.
Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions
GEN-Z is generative, as it measures the LM likelihood of input text, conditioned on natural language descriptions of labels.
Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
The interactive use of large language models (LLMs) in AI assistants (at work, home, etc.)
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
In this work, we focus on LLM sensitivity to a quintessential class of meaning-preserving design choices: prompt formatting.
KGQuiz: Evaluating the Generalization of Encoded Knowledge in Large Language Models
To gain a better understanding of LLMs' knowledge abilities and their generalization, we evaluate 10 open-source and black-box LLMs on the KGQuiz benchmark across the five knowledge-intensive tasks and knowledge domains.
MatFormer: Nested Transformer for Elastic Inference
Furthermore, we observe that smaller encoders extracted from a universal MatFormer-based ViT (MatViT) encoder preserve the metric-space structure for adaptive large-scale retrieval.
On the Zero-Shot Generalization of Machine-Generated Text Detectors
The rampant proliferation of large language models, fluent enough to generate text indistinguishable from human-written language, gives unprecedented importance to the detection of machine-generated text.
SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation
Existing watermarking algorithms are vulnerable to paraphrase attacks because of their token-level design.
Knowledge Crosswords: Geometric Reasoning over Structured Knowledge with Large Language Models
We additionally propose two new approaches, Staged Prompting and Verify-All, to augment LLMs' ability to backtrack and verify structured constraints.
Resolving Knowledge Conflicts in Large Language Models
To this end, we introduce KNOWLEDGE CONFLICT, an evaluation framework for simulating contextual knowledge conflicts and quantitatively evaluating to what extent LLMs achieve these goals.
LatticeGen: A Cooperative Framework which Hides Generated Text in a Lattice for Privacy-Aware Generation on Cloud
In the current user-server interaction paradigm of prompted generation with large language models (LLM) on cloud, the server fully controls the generation process, which leaves zero options for users who want to keep the generated text to themselves.
Understanding In-Context Learning via Supportive Pretraining Data
We observe that a continued pretraining on this small subset significantly improves the model's ICL ability, by up to 18%.
Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker
We present SymbolicToM, a plug-and-play approach to reason about the belief states of multiple characters in reading comprehension tasks via explicit symbolic representation.
Examining risks of racial biases in NLP tools for child protective services
Given well-established racial bias in this setting, we investigate possible ways deployed NLP is liable to increase racial disparities.
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
Language models (LMs) often struggle to pay enough attention to the input context, and generate texts that are unfaithful or contain hallucinations.
GlobalBench: A Benchmark for Global Progress in Natural Language Processing
Despite the major advances in NLP, significant disparities in NLP system performance across languages still exist.
David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs
Diffusion-based language models are emerging as a promising alternative to autoregressive LMs: they approach the competence of autoregressive LMs while offering nuanced controllability at inference time.
BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer
Despite remarkable advancements in few-shot generalization in natural language processing, most models are developed and evaluated primarily in English.
TalkUp: Paving the Way for Understanding Empowering Language
Empowering language is important in many real-world contexts, from education to workplace dynamics to healthcare.
Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
Language models have graduated from being research prototypes to commercialized products offered as web APIs, and recent works have highlighted the multilingual capabilities of these products.
Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models
Ultimately, Knowledge Card framework enables dynamic synthesis and updates of knowledge from diverse domains.
Can Language Models Solve Graph Problems in Natural Language?
We then propose Build-a-Graph Prompting and Algorithmic Prompting, two instruction-based approaches to enhance LLMs in solving natural language graph problems.
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
We focus on hate speech and misinformation detection, aiming to empirically quantify the effects of political (social, economic) biases in pretraining data on the fairness of high-stakes social-oriented tasks.
FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge
We propose FactKB, a simple new approach to factuality evaluation that is generalizable across domains, in particular with respect to entities and relations.
Assessing Language Model Deployment with Risk Cards
However, there is no risk-centric framework for documenting the complexity of a landscape in which some risks are shared across models and contexts, while others are specific, and where certain conditions may be required for risks to manifest as harms.
On the Blind Spots of Model-Based Evaluation Metrics for Text Generation
In this work, we explore a useful but often neglected methodology for robustness analysis of text generation evaluation metrics: stress tests with synthetic data.
Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?
Large language models can perform new tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior.
SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control
Despite the growing success of diffusion models in continuous-valued domains (e. g., images), similar efforts for discrete domains such as text have yet to match the performance of autoregressive language models.
Gendered Mental Health Stigma in Masked Language Models
Mental health stigma prevents many individuals from receiving the appropriate care, and social psychology studies have shown that mental health tends to be overlooked in men.
Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation
Moreover, we uniquely propose iterative distillation of knowledge, where student models from the previous iteration of distillation serve as teacher models in the next iteration.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Abstractive summarization models often generate inconsistent summaries containing factual errors or hallucinated content.
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Recent advances in the capacity of large language models to generate human-like text have resulted in their increased adoption in user-facing settings.
KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding
With the advent of pretrained language models (LMs), increasing research efforts have been focusing on infusing commonsense and domain-specific knowledge to prepare LMs for downstream tasks.
Gradient-Based Constrained Sampling from Language Models
Large pretrained language models generate fluent text but are notoriously hard to controllably sample from.
ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Data
Large pretrained language models have been performing increasingly well in a variety of downstream tasks via prompting.
Challenges and Opportunities in Information Manipulation Detection: An Examination of Wartime Russian Media
NLP research on public opinion manipulation campaigns has primarily focused on detecting overt strategies such as fake news and disinformation.
Speaker Information Can Guide Models to Better Inductive Biases: A Case Study On Predicting Code-Switching
Natural language processing (NLP) models trained on people-generated data can be unreliable because, without any constraints, they can learn from spurious correlations that are not relevant to the task.
Unsupervised Keyphrase Extraction via Interpretable Neural Networks
Keyphrase extraction aims at automatically extracting a list of "important" phrases representing the key concepts in a document.
Improving the Diversity of Unsupervised Paraphrasing with Embedding Outputs
We present a novel technique for zero-shot paraphrase generation.
Can Machines Learn Morality? The Delphi Experiment
As AI systems become increasingly powerful and pervasive, there are growing concerns about machines' morality or a lack thereof.
Detecting Community Sensitive Norm Violations in Online Conversations
Online platforms and communities establish their own norms that govern what behavior is acceptable within the community.
Influence Tuning: Demoting Spurious Correlations via Instance Attribution and Instance-Driven Updates
Among the most critical limitations of deep learning NLP models are their lack of interpretability, and their reliance on spurious correlations.
Improving Span Representation for Domain-adapted Coreference Resolution
Recent work has shown fine-tuning neural coreference models can produce strong performance when adapting to different domains.
Efficient Test Time Adapter Ensembling for Low-resource Language Varieties
Adapters are light-weight modules that allow parameter-efficient fine-tuning of pretrained models.
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks.
Ranked #4 on
Visual Entailment
on SNLI-VE val
Controlled Text Generation as Continuous Optimization with Multiple Constraints
As large-scale language model pretraining pushes the state-of-the-art in text generation, recent work has turned to controlling attributes of the text such models generate.
A Survey of Race, Racism, and Anti-Racism in NLP
Despite inextricable ties between race and language, little work has considered race in NLP research and development.
Machine Translation into Low-resource Language Varieties
State-of-the-art machine translation (MT) systems are typically trained to generate the "standard" target language; however, many languages have multiple varieties (regional varieties, dialects, sociolects, non-native varieties) that are different from the standard language.
Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation
Experiments on classification, ranking and evaluation tasks across multiple datasets demonstrate that our approaches outperform strong baselines in providing informative negative examples for training dialogue systems.
DialoGraph: Incorporating Interpretable Strategy-Graph Networks into Negotiation Dialogues
To successfully negotiate a deal, it is not enough to communicate fluently: pragmatic planning of persuasive negotiation strategies is essential.
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Modern summarization models generate highly fluent but often factually unreliable outputs.
Simple and Efficient ways to Improve REALM
Dense retrieval has been shown to be effective for retrieving relevant documents for Open Domain QA, surpassing popular sparse retrieval methods like BM25.
An Exploration of Data Augmentation Techniques for Improving English to Tigrinya Translation
It has been shown that the performance of neural machine translation (NMT) drops starkly in low-resource conditions, often requiring large amounts of auxiliary data to achieve competitive results.
Evaluating the Morphosyntactic Well-formedness of Generated Texts
Text generation systems are ubiquitous in natural language processing applications.
SelfExplain: A Self-Explaining Architecture for Neural Text Classifiers
We introduce SelfExplain, a novel self-explaining model that explains a text classifier's predictions using phrase-based concepts.
Controlled Analyses of Social Biases in Wikipedia Bios
In this work, we present a methodology for analyzing Wikipedia pages about people that isolates dimensions of interest (e. g., gender), from other attributes (e. g., occupation).
Understanding Linguistic Accommodation in Code-Switched Human-Machine Dialogues
Code-switching is a ubiquitous phenomenon in multilingual communities.
Multilingual Contextual Affective Analysis of LGBT People Portrayals in Wikipedia
Specific lexical choices in narrative text reflect both the writer's attitudes towards people in the narrative and influence the audience's reactions.
End-to-End Differentiable GANs for Text Generation
We posit that this gap is due to autoregressive nature and architectural requirements for text generation as well as a fundamental difference between the definition of Wasserstein distance in image and text domains.
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
Massively multilingual models subsuming tens or even hundreds of languages pose great challenges to multi-task optimization.
Fortifying Toxic Speech Detectors Against Veiled Toxicity
Modern toxic speech detectors are incompetent in recognizing disguised offensive language, such as adversarial attacks that deliberately avoid known toxic lexicons, or manifestations of implicit bias.
On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages.
Automatic Extraction of Rules Governing Morphological Agreement
Using cross-lingual transfer, even with no expert annotations in the language of interest, our framework extracts a grammatical specification which is nearly equivalent to those created with large amounts of gold-standard annotated data.
Controlling Dialogue Generation with Semantic Exemplars
Dialogue systems pretrained with large language models generate locally coherent responses, but lack the fine-grained control over responses necessary to achieve specific goals.
LTIatCMU at SemEval-2020 Task 11: Incorporating Multi-Level Features for Multi-Granular Propaganda Span Identification
In this paper we describe our submission for the task of Propaganda Span Identification in news articles.
A Deep Reinforced Model for Zero-Shot Cross-Lingual Summarization with Bilingual Semantic Similarity Rewards
The model uses reinforcement learning to directly optimize a bilingual semantic similarity metric between the summaries generated in a target language and gold summaries in a source language.
Cross-Cultural Similarity Features for Cross-Lingual Transfer Learning of Pragmatically Motivated Tasks
Much work in cross-lingual transfer learning explored how to select better transfer languages for multilingual tasks, primarily focusing on typological and genealogical similarities between languages.
Demoting Racial Bias in Hate Speech Detection
In current hate speech datasets, there exists a high correlation between annotators' perceptions of toxicity and signals of African American English (AAE).
A Computational Analysis of Polarization on Indian and Pakistani Social Media
Between February 14, 2019 and March 4, 2019, a terrorist attack in Pulwama, Kashmir followed by retaliatory airstrikes led to rising tensions between India and Pakistan, two nuclear-armed countries.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
In this work, we investigate the use of influence functions for NLP, providing an alternative approach to interpreting neural text classifiers.
Unsupervised Discovery of Implicit Gender Bias
Despite their prevalence in society, social biases are difficult to identify, primarily because human judgements in this domain can be unreliable.
Balancing Training for Multilingual Neural Machine Translation
When training multilingual machine translation (MT) models that can translate to/from multiple languages, we are faced with imbalanced training sets: some languages have much more training data than others.
A Framework for the Computational Linguistic Analysis of Dehumanization
Dehumanization is a pernicious psychological process that often leads to extreme intergroup bias, hate speech, and violence aimed at targeted social groups.
StructSum: Summarization via Structured Representations
To this end, we propose incorporating latent and explicit dependencies across sentences in the source document into end-to-end single-document summarization models.
Where New Words Are Born: Distributional Semantic Analysis of Neologisms and Their Semantic Neighborhoods
Besides presenting a new linguistic application of distributional semantics, this study tackles the linguistic question of the role of language-internal factors (in our case, sparsity) in language change motivated by language-external factors (reflected in frequency growth).
Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts
Microaggressions are subtle, often veiled, manifestations of human biases.
A Margin-based Loss with Synthetic Negative Samples for Continuous-output Machine Translation
Neural models that eliminate the softmax bottleneck by generating word embeddings (rather than multinomial distributions over a vocabulary) attain faster training with fewer learnable parameters.
Learning to Generate Word- and Phrase-Embeddings for Efficient Phrase-Based Neural Machine Translation
In this paper, we introduce a phrase-based NMT model built upon continuous-output NMT, in which the decoder generates embeddings of words or phrases.
Augmenting Non-Collaborative Dialog Systems with Explicit Semantic and Strategic Dialog History
We train FSTs on a set of strategies and tactics used in negotiation dialogs.
A Dynamic Strategy Coach for Effective Negotiation
We consider a bargaining scenario where a seller and a buyer negotiate the price of an item for sale through a text-based dialog.
Topics to Avoid: Demoting Latent Confounds in Text Classification
Despite impressive performance on many text classification tasks, deep neural networks tend to learn frequent superficial patterns that are specific to the training data and do not always generalize well.
CMU-01 at the SIGMORPHON 2019 Shared Task on Crosslinguality and Context in Morphology
This paper presents the submission by the CMU-01 team to the SIGMORPHON 2019 task 2 of Morphological Analysis and Lemmatization in Context.
Measuring Bias in Contextualized Word Representations
Contextual word embeddings such as BERT have achieved state of the art performance in numerous NLP tasks.
Entity-Centric Contextual Affective Analysis
While contextualized word representations have improved state-of-the-art benchmarks in many NLP tasks, their potential usefulness for social-oriented tasks remains largely unexplored.
Contextual Affective Analysis: A Case Study of People Portrayals in Online #MeToo Stories
We show that while these articles are sympathetic towards women who have experienced sexual harassment, they consistently present men as most powerful, even after sexual assault allegations.
Social and Information Networks
Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings
Online texts -- across genres, registers, domains, and styles -- are riddled with human stereotypes, expressed in overt or subtle ways.
Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs
The Softmax function is used in the final layer of nearly all existing sequence-to-sequence models for language generation.
Framing and Agenda-setting in Russian News: a Computational Analysis of Intricate Political Strategies
Amidst growing concern over media manipulation, NLP attention has focused on overt strategies like censorship and "fake news'".
Socially Responsible NLP
As language technologies have become increasingly prevalent, there is a growing awareness that decisions we make about our data, methods, and tools are often tied up with their impact on people and societies.
Native Language Cognate Effects on Second Language Lexical Choice
We present a computational analysis of cognate effects on the spontaneous linguistic productions of advanced non-native speakers.
Style Transfer Through Back-Translation
We first learn a latent representation of the input sentence which is grounded in a language translation model in order to better preserve the meaning of the sentence while reducing stylistic properties.
Ranked #10 on
Unsupervised Text Style Transfer
on Yelp
Incorporating Dialectal Variability for Socially Equitable Language Identification
Language identification (LID) is a critical first step for processing multilingual text.
Correlation-based Intrinsic Evaluation of Word Vector Representations
We introduce QVEC-CCA--an intrinsic evaluation metric for word vector representations based on correlations of learned vectors with features extracted from linguistic resources.
Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning
We introduce polyglot language models, recurrent neural network models trained to predict symbol sequences in many different languages using shared representations of symbols and conditioning on typological information about the language to be predicted.
Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning
We use Bayesian optimization to learn curricula for word representation learning, optimizing performance on downstream tasks that depend on the learned representations as features.
Problems With Evaluation of Word Embeddings Using Word Similarity Tasks
Our study suggests that the use of word similarity tasks for evaluation of word vectors is not sustainable and calls for further research on evaluation methods.
Massively Multilingual Word Embeddings
We introduce new methods for estimating and evaluating embeddings of words in more than fifty languages in a single shared embedding space.
Morphological Inflection Generation Using Character Sequence to Sequence Learning
Morphological inflection generation is the task of generating the inflected form of a given lemma corresponding to a particular linguistic transformation.
Sparse Overcomplete Word Vector Representations
Current distributed representations of words show little resemblance to theories of lexical semantics.
A Unified Annotation Scheme for the Semantic/Pragmatic Components of Definiteness
We present a definiteness annotation scheme that captures the semantic, pragmatic, and discourse information, which we call communicative functions, associated with linguistic descriptions such as {``}a story about my speech{''}, {``}the story{''}, {``}every time I give it{''}, {``}this slideshow{''}.
Augmenting English Adjective Senses with Supersenses
We develop a supersense taxonomy for adjectives, based on that of GermaNet, and apply it to English adjectives in WordNet using human annotation and supervised classification.
