Search Results for author:
Found 28 papers, 20 papers with code
STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasings
In this paper, we present STAMP, a framework for detecting dataset membership-i. e., determining the inclusion of a dataset in the pretraining corpora of LLMs.
All That Glitters is Not Novel: Plagiarism in AI Generated Research
Unlike past efforts where experts evaluate the novelty and feasibility of research ideas, we request $13$ experts to operate under a different situational logic: to identify similarities between LLM-generated research documents and existing work.
Knowledge Graph Guided Evaluation of Abstention Techniques
Using SELECT, we benchmark different abstention techniques over six open-weight and closed-source models.
Richer Output for Richer Countries: Uncovering Geographical Disparities in Generated Stories and Travel Recommendations
While a large body of work inspects language models for biases concerning gender, race, occupation and religion, biases of geographical nature are relatively less explored.
Revisiting the Robustness of Watermarking to Paraphrasing Attacks
Amidst rising concerns about the internet being proliferated with content generated from language models (LMs), watermarking is seen as a principled way to certify whether text was generated from a model.
What Can Natural Language Processing Do for Peer Review?
We hope that our work will help set the agenda for research in machine-assisted scientific quality control in the age of AI, within the NLP community and beyond.
Evaluating Large Language Models for Health-related Queries with Presuppositions
As corporations rush to integrate large language models (LLMs) to their search offerings, it is critical that they provide factually accurate information that is robust to any presuppositions that a user may express.
Downstream Trade-offs of a Family of Text Watermarks
In this work, we evaluate the performance of LLMs watermarked using three different strategies over a diverse suite of tasks including those cast as k-class classification (CLS), multiple choice question answering (MCQ), short-form generation (e. g., open-ended question answering) and long-form generation (e. g., translation) tasks.
Geographical Erasure in Language Generation
Large language models (LLMs) encode vast amounts of world knowledge.
Goodhart's Law Applies to NLP's Explanation Benchmarks
First, we show that we can inflate a model's comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution test inputs.
Inspecting the Geographical Representativeness of Images from Text-to-Image Models
Recent progress in generative models has resulted in models that produce both realistic as well as relevant images for most textual inputs.
Model-tuning Via Prompts Makes NLP Models Adversarially Robust
Across 5 NLP datasets, 4 adversarial attacks, and 3 different models, MVP improves performance against adversarial substitutions by an average of 8% over standard methods and even outperforms adversarial training-based state-of-art defenses by 3. 5%.
Learning the Legibility of Visual Text Perturbations
In this work, we address this gap by learning models that predict the legibility of a perturbed string, and rank candidate perturbations based on their legibility.
Assisting Human Decisions in Document Matching
Many practical applications, ranging from paper-reviewer assignment in peer review to job-applicant matching for hiring, require human decision makers to identify relevant matches by combining their expertise with predictions from machine learning models.
Measures of Information Reflect Memorization Patterns
In this work, we hypothesize -- and subsequently show -- that the diversity in the activation patterns of different neurons is reflective of model generalization and memorization.
Learning to Scaffold: Optimizing Model Explanations for Teaching
In this work, leveraging meta-learning techniques, we extend this idea to improve the quality of the explanations themselves, specifically by optimizing explanations such that student models more effectively learn to simulate the original model.
Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations
Through our evaluation, we observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
Do Context-Aware Translation Models Pay the Right Attention?
Are models paying large amounts of attention to the same context?
Evaluating Explanations: How much do explanations from the teacher aid students?
While many methods purport to explain predictions by highlighting salient features, what aims these explanations serve and how they ought to be evaluated often go unstated.
Weakly- and Semi-supervised Evidence Extraction
For many prediction tasks, stakeholders desire not only predictions but also supporting evidence that a human can use to verify its correctness.
NeuSpell: A Neural Spelling Correction Toolkit
We introduce NeuSpell, an open-source toolkit for spelling correction in English.
Why and when should you pool? Analyzing Pooling in Recurrent Architectures
We find that pooling-based architectures substantially differ from their non-pooling equivalents in their learning ability and positional biases--which elucidate their performance benefits.
Learning to Deceive with Attention-Based Explanations
Attention mechanisms are ubiquitous components in neural architectures applied to natural language processing.
Combating Adversarial Misspellings with Robust Word Recognition
To combat adversarial spelling mistakes, we propose placing a word recognition model in front of the downstream classifier.
compare-mt: A Tool for Holistic Comparison of Language Generation Systems
In this paper, we describe compare-mt, a tool for holistic analysis and comparison of the results of systems for language generation tasks such as machine translation.
Measuring Density and Similarity of Task Relevant Information in Neural Representations
Neural models achieve state-of-the-art performance due to their ability to extract salient features useful to downstream tasks.
Simple and Effective Semi-Supervised Question Answering
Recent success of deep learning models for the task of extractive Question Answering (QA) is hinged on the availability of large annotated corpora.
SPINE: SParse Interpretable Neural Embeddings
We propose a novel variant of denoising k-sparse autoencoders that generates highly efficient and interpretable distributed word representations (word embeddings), beginning with existing word representations from state-of-the-art methods like GloVe and word2vec.
