no code implementations • 18 Apr 2025 • Saksham Rastogi, Pratyush Maini, Danish Pruthi
In this paper, we present STAMP, a framework for detecting dataset membership-i. e., determining the inclusion of a dataset in the pretraining corpora of LLMs.
no code implementations • 23 Feb 2025 • Tarun Gupta, Danish Pruthi
Unlike past efforts where experts evaluate the novelty and feasibility of research ideas, we request $13$ experts to operate under a different situational logic: to identify similarities between LLM-generated research documents and existing work.
no code implementations • 10 Dec 2024 • Kinshuk Vasisht, Navreet Kaur, Danish Pruthi
Using SELECT, we benchmark different abstention techniques over six open-weight and closed-source models.
1 code implementation • 11 Nov 2024 • Kirti Bhagat, Kinshuk Vasisht, Danish Pruthi
While a large body of work inspects language models for biases concerning gender, race, occupation and religion, biases of geographical nature are relatively less explored.
1 code implementation • 8 Nov 2024 • Saksham Rastogi, Danish Pruthi
Amidst rising concerns about the internet being proliferated with content generated from language models (LMs), watermarking is seen as a principled way to certify whether text was generated from a model.
1 code implementation • 10 May 2024 • Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wang, Xiaodan Zhu, Anna Rogers, Nihar B. Shah, Iryna Gurevych
We hope that our work will help set the agenda for research in machine-assisted scientific quality control in the age of AI, within the NLP community and beyond.
1 code implementation • 14 Dec 2023 • Navreet Kaur, Monojit Choudhury, Danish Pruthi
As corporations rush to integrate large language models (LLMs) to their search offerings, it is critical that they provide factually accurate information that is robust to any presuppositions that a user may express.
1 code implementation • 16 Nov 2023 • Anirudh Ajith, Sameer Singh, Danish Pruthi
In this work, we evaluate the performance of LLMs watermarked using three different strategies over a diverse suite of tasks including those cast as k-class classification (CLS), multiple choice question answering (MCQ), short-form generation (e. g., open-ended question answering) and long-form generation (e. g., translation) tasks.
1 code implementation • 23 Oct 2023 • Pola Schwöbel, Jacek Golebiowski, Michele Donini, Cédric Archambeau, Danish Pruthi
Large language models (LLMs) encode vast amounts of world knowledge.
no code implementations • 28 Aug 2023 • Jennifer Hsia, Danish Pruthi, Aarti Singh, Zachary C. Lipton
First, we show that we can inflate a model's comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution test inputs.
no code implementations • ICCV 2023 • Abhipsa Basu, R. Venkatesh Babu, Danish Pruthi
Recent progress in generative models has resulted in models that produce both realistic as well as relevant images for most textual inputs.
1 code implementation • 13 Mar 2023 • Mrigank Raman, Pratyush Maini, J. Zico Kolter, Zachary C. Lipton, Danish Pruthi
Across 5 NLP datasets, 4 adversarial attacks, and 3 different models, MVP improves performance against adversarial substitutions by an average of 8% over standard methods and even outperforms adversarial training-based state-of-art defenses by 3. 5%.
1 code implementation • 9 Mar 2023 • Dev Seth, Rickard Stureborg, Danish Pruthi, Bhuwan Dhingra
In this work, we address this gap by learning models that predict the legibility of a perturbed string, and rank candidate perturbations based on their legibility.
1 code implementation • 16 Feb 2023 • Joon Sik Kim, Valerie Chen, Danish Pruthi, Nihar B. Shah, Ameet Talwalkar
Many practical applications, ranging from paper-reviewer assignment in peer review to job-applicant matching for hiring, require human decision makers to identify relevant matches by combining their expertise with predictions from machine learning models.
no code implementations • 17 Oct 2022 • Rachit Bansal, Danish Pruthi, Yonatan Belinkov
In this work, we hypothesize -- and subsequently show -- that the diversity in the activation patterns of different neurons is reflective of model generalization and memorization.
1 code implementation • 22 Apr 2022 • Patrick Fernandes, Marcos Treviso, Danish Pruthi, André F. T. Martins, Graham Neubig
In this work, leveraging meta-learning techniques, we extend this idea to improve the quality of the explanations themselves, specifically by optimizing explanations such that student models more effectively learn to simulate the original model.
1 code implementation • 17 Dec 2021 • Siddhant Arora, Danish Pruthi, Norman Sadeh, William W. Cohen, Zachary C. Lipton, Graham Neubig
Through our evaluation, we observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
1 code implementation • ACL 2021 • Kayo Yin, Patrick Fernandes, Danish Pruthi, Aditi Chaudhary, André F. T. Martins, Graham Neubig
Are models paying large amounts of attention to the same context?
1 code implementation • 1 Dec 2020 • Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, William W. Cohen
While many methods purport to explain predictions by highlighting salient features, what aims these explanations serve and how they ought to be evaluated often go unstated.
1 code implementation • Findings of the Association for Computational Linguistics 2020 • Danish Pruthi, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton
For many prediction tasks, stakeholders desire not only predictions but also supporting evidence that a human can use to verify its correctness.
1 code implementation • EMNLP 2020 • Sai Muralidhar Jayanthi, Danish Pruthi, Graham Neubig
We introduce NeuSpell, an open-source toolkit for spelling correction in English.
1 code implementation • Findings of the Association for Computational Linguistics 2020 • Pratyush Maini, Keshav Kolluru, Danish Pruthi, Mausam
We find that pooling-based architectures substantially differ from their non-pooling equivalents in their learning ability and positional biases--which elucidate their performance benefits.
3 code implementations • ACL 2020 • Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton
Attention mechanisms are ubiquitous components in neural architectures applied to natural language processing.
3 code implementations • ACL 2019 • Danish Pruthi, Bhuwan Dhingra, Zachary C. Lipton
To combat adversarial spelling mistakes, we propose placing a word recognition model in front of the downstream classifier.
2 code implementations • NAACL 2019 • Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, Xinyi Wang, John Wieting
In this paper, we describe compare-mt, a tool for holistic analysis and comparison of the results of systems for language generation tasks such as machine translation.
no code implementations • 27 Sep 2018 • Danish Pruthi, Mansi Gupta, Nitish Kumar Kulkarni, Graham Neubig, Eduard Hovy
Neural models achieve state-of-the-art performance due to their ability to extract salient features useful to downstream tasks.
no code implementations • NAACL 2018 • Bhuwan Dhingra, Danish Pruthi, Dheeraj Rajagopal
Recent success of deep learning models for the task of extractive Question Answering (QA) is hinged on the availability of large annotated corpora.
2 code implementations • 23 Nov 2017 • Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, Eduard Hovy
We propose a novel variant of denoising k-sparse autoencoders that generates highly efficient and interpretable distributed word representations (word embeddings), beginning with existing word representations from state-of-the-art methods like GloVe and word2vec.