Search Results for author: Sameer Singh

Found 124 papers, 58 papers with code

Robustness and Adversarial Examples in Natural Language Processing

no code implementations EMNLP (ACL) 2021 Kai-Wei Chang, He He, Robin Jia, Sameer Singh

In particular, we will review recent studies on analyzing the weakness of NLP systems when facing adversarial inputs and data with a distribution shift.

Tweeki: Linking Named Entities on Twitter to a Knowledge Graph

no code implementations EMNLP (WNUT) 2020 Bahareh Harandizadeh, Sameer Singh

Further, there is a lack of a large, linked corpus of tweets to aid researchers, along with lack of gold dataset to evaluate the accuracy of entity linking.

Entity Linking

Zero- and Few-Shot NLP with Pretrained Language Models

no code implementations ACL 2022 Iz Beltagy, Arman Cohan, Robert Logan IV, Sewon Min, Sameer Singh

The ability to efficiently learn from little-to-no data is critical to applying NLP to tasks where data collection is costly or otherwise difficult.

Few-Shot Learning

COVIDLies: Detecting COVID-19 Misinformation on Social Media

no code implementations EMNLP (NLP-COVID19) 2020 Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, Sameer Singh

The ongoing pandemic has heightened the need for developing tools to flag COVID-19-related misinformation on the internet, specifically on social media such as Twitter.

Misconceptions Misinformation +2

Citations Beyond Self Citations: Identifying Authors, Affiliations, and Nationalities in Scientific Papers

1 code implementation WOSP 2020 Yoshitomo Matsubara, Sameer Singh

Our models are accurate; we identify at least one of authors, affiliations, and nationalities of held-out papers with 40. 3%, 47. 9% and 86. 0% accuracy respectively, from the top-10 guesses of our models.

Are Models Biased on Text without Gender-related Language?

1 code implementation1 May 2024 Catarina G Belém, Preethi Seshadri, Yasaman Razeghi, Sameer Singh

A key observation in prior work is that models reinforce stereotypes as a consequence of the gendered correlations that are present in the training data.

Fairness Sentence

MisgenderMender: A Community-Informed Approach to Interventions for Misgendering

no code implementations23 Apr 2024 Tamanna Hossain, Sunipa Dev, Sameer Singh

We are the first to address this lack of research into interventions for misgendering by conducting a survey of gender-diverse individuals in the US to understand perspectives about automated interventions for text-based misgendering.

Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills

no code implementations5 Feb 2024 Kolby Nottingham, Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Sameer Singh, Peter Clark, Roy Fox

We evaluate our method in the classic videogame NetHack and the text environment ScienceWorld to demonstrate SSO's ability to optimize a set of skills and perform in-context policy improvement.

Decision Making Language Modelling +1

Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals

no code implementations16 Nov 2023 Yanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe, Khyathi Raghavi, Vivek Srikumar, Sameer Singh, Noah A. Smith

Previous work has found that datasets with paired inputs are prone to correlations between a specific part of the input (e. g., the hypothesis in NLI) and the label; consequently, models trained only on those outperform chance.

counterfactual In-Context Learning +2

Performance Trade-offs of Watermarking Large Language Models

no code implementations16 Nov 2023 Anirudh Ajith, Sameer Singh, Danish Pruthi

However, implanting such signals alters the model's output distribution and can have unintended effects when watermarked LLMs are used for downstream applications.

Language Modelling Misinformation +6

What's In My Big Data?

1 code implementation31 Oct 2023 Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, Jesse Dodge

We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them.


EchoPrompt: Instructing the Model to Rephrase Queries for Improved In-context Learning

1 code implementation16 Sep 2023 Rajasekhar Reddy Mekala, Yasaman Razeghi, Sameer Singh

On average, EchoPrompt improves the Zero-shot-CoT performance of code-davinci-002 by 5% in numerical tasks and 13% in reading comprehension tasks.

Date Understanding GSM8K +4

The Bias Amplification Paradox in Text-to-Image Generation

1 code implementation1 Aug 2023 Preethi Seshadri, Sameer Singh, Yanai Elazar

Bias amplification is a phenomenon in which models exacerbate biases or stereotypes present in the training data.

Text-to-Image Generation

Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actors

no code implementations21 Jul 2023 Kolby Nottingham, Yasaman Razeghi, KyungMin Kim, JB Lanier, Pierre Baldi, Roy Fox, Sameer Singh

Large language models (LLMs) are being applied as actors for sequential decision making tasks in domains such as robotics and games, utilizing their general world knowledge and planning abilities.

Decision Making Language Modelling +2

Maestro: A Gamified Platform for Teaching AI Robustness

no code implementations14 Jun 2023 Margarita Geleta, Jiacen Xu, Manikanta Loya, Junlin Wang, Sameer Singh, Zhou Li, Sergio Gago-Masague

We assessed Maestro's influence on students' engagement, motivation, and learning success in robust AI.

Active Learning

MISGENDERED: Limits of Large Language Models in Understanding Pronouns

no code implementations6 Jun 2023 Tamanna Hossain, Sunipa Dev, Sameer Singh

Content Warning: This paper contains examples of misgendering and erasure that could be offensive and potentially triggering.


PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptions

no code implementations24 May 2023 Anthony Chen, Panupong Pasupat, Sameer Singh, Hongrae Lee, Kelvin Guu

These bottlenecks motivate the training of compact editors, which is challenging due to the scarcity of training data for this purpose.

Denoising Language Modelling

Coverage-based Example Selection for In-Context Learning

1 code implementation24 May 2023 Shivanshu Gupta, Matt Gardner, Sameer Singh

In-context learning (ICL), the ability of large language models to perform novel tasks by conditioning on a prompt with a few task examples, requires these examples to be informative about the test instance.

In-Context Learning Informativeness

TABLET: Learning From Instructions For Tabular Data

1 code implementation25 Apr 2023 Dylan Slack, Sameer Singh

Acquiring high-quality data is often a significant challenge in training machine learning (ML) models for tabular prediction, particularly in privacy-sensitive and costly domains like medicine and finance.

ART: Automatic multi-step reasoning and tool-use for large language models

2 code implementations16 Mar 2023 Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, Marco Tulio Ribeiro

We introduce Automatic Reasoning and Tool-use (ART), a framework that uses frozen LLMs to automatically generate intermediate reasoning steps as a program.

Successive Prompting for Decomposing Complex Questions

no code implementations8 Dec 2022 Dheeru Dua, Shivanshu Gupta, Sameer Singh, Matt Gardner

The intermediate supervision is typically manually written, which can be expensive to collect.

Question Answering

TCAB: A Large-Scale Text Classification Attack Benchmark

1 code implementation21 Oct 2022 Kalyani Asthana, Zhouhang Xie, Wencong You, Adam Noack, Jonathan Brophy, Sameer Singh, Daniel Lowd

In addition to the primary tasks of detecting and labeling attacks, TCAB can also be used for attack localization, attack target labeling, and attack characterization.

Abuse Detection Sentiment Analysis +2

Continued Pretraining for Better Zero- and Few-Shot Promptability

1 code implementation19 Oct 2022 Zhaofeng Wu, Robert L. Logan IV, Pete Walsh, Akshita Bhagia, Dirk Groeneveld, Sameer Singh, Iz Beltagy

We demonstrate that a simple recipe, continued pretraining that incorporates a trainable prompt during multi-task learning, leads to improved promptability in both zero- and few-shot settings compared to existing methods, up to 31% relative.

Language Modelling Meta-Learning +1

Quantifying Social Biases Using Templates is Unreliable

no code implementations9 Oct 2022 Preethi Seshadri, Pouya Pezeshkpour, Sameer Singh

Recently, there has been an increase in efforts to understand how large language models (LLMs) propagate and amplify social biases.

Attribute Benchmarking +1

TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations

1 code implementation8 Jul 2022 Dylan Slack, Satyapriya Krishna, Himabindu Lakkaraju, Sameer Singh

In real-world evaluations with humans, 73% of healthcare workers (e. g., doctors and nurses) agreed they would use TalkToModel over baseline point-and-click systems for explainability in a disease prediction task, and 85% of ML professionals agreed TalkToModel was easier to use for computing explanations.

BIG-bench Machine Learning Disease Prediction +1

Learning to Query Internet Text for Informing Reinforcement Learning Agents

1 code implementation25 May 2022 Kolby Nottingham, Alekhya Pyla, Sameer Singh, Roy Fox

We show that our method correctly learns to execute queries to maximize reward in a reinforcement learning setting.

reinforcement-learning Reinforcement Learning (RL)

ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension

2 code implementations ACL 2022 Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, Anna Rohrbach

Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain.

Image Classification Referring Expression +1

Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation

1 code implementation16 Mar 2022 Shivanshu Gupta, Sameer Singh, Matt Gardner

A growing body of research has demonstrated the inability of NLP models to generalize compositionally and has tried to alleviate it through specialized architectures, training schemes, and data augmentation, among other approaches.

Data Augmentation Semantic Parsing

Impact of Pretraining Term Frequencies on Few-Shot Reasoning

no code implementations15 Feb 2022 Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, Sameer Singh

Pretrained Language Models (LMs) have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings.

Rethinking Explainability as a Dialogue: A Practitioner's Perspective

1 code implementation3 Feb 2022 Himabindu Lakkaraju, Dylan Slack, Yuxin Chen, Chenhao Tan, Sameer Singh

Overall, we hope our work serves as a starting place for researchers and engineers to design interactive explainability systems.

BIG-bench Machine Learning

Identifying Adversarial Attacks on Text Classifiers

no code implementations21 Jan 2022 Zhouhang Xie, Jonathan Brophy, Adam Noack, Wencong You, Kalyani Asthana, Carter Perkins, Sabrina Reis, Sameer Singh, Daniel Lowd

The landscape of adversarial attacks against text classifiers continues to grow, with new attacks developed every year and many of them available in standard toolkits, such as TextAttack and OpenAttack.

Abuse Detection Adversarial Text +2

FRUIT: Faithfully Reflecting Updated Information in Text

no code implementations NAACL 2022 Robert L. Logan IV, Alexandre Passos, Sameer Singh, Ming-Wei Chang

Textual knowledge bases such as Wikipedia require considerable effort to keep up to date and consistent.

Entity-Based Knowledge Conflicts in Question Answering

1 code implementation EMNLP 2021 Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, Sameer Singh

To understand how models use these sources together, we formalize the problem of knowledge conflicts, where the contextual information contradicts the learned information.

Hallucination Out-of-Distribution Generalization +1

Modular Framework for Visuomotor Language Grounding

no code implementations5 Sep 2021 Kolby Nottingham, Litian Liang, Daeyun Shin, Charless C. Fowlkes, Roy Fox, Sameer Singh

Natural language instruction following tasks serve as a valuable test-bed for grounded language and robotics research.

Instruction Following

Benchmarking Scalable Methods for Streaming Cross Document Entity Coreference

1 code implementation ACL 2021 Robert L Logan IV, Andrew McCallum, Sameer Singh, Dan Bikel

We investigate: how to best encode mentions, which clustering algorithms are most effective for grouping mentions, how models transfer to different domains, and how bounding the number of mentions tracked during inference impacts performance.

Benchmarking Clustering +2

Enforcing Consistency in Weakly Supervised Semantic Parsing

1 code implementation ACL 2021 Nitish Gupta, Sameer Singh, Matt Gardner

The predominant challenge in weakly supervised semantic parsing is that of spurious programs that evaluate to correct answers for the wrong reasons.

Semantic Parsing Visual Reasoning

Combining Feature and Instance Attribution to Detect Artifacts

no code implementations Findings (ACL) 2022 Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, Byron C. Wallace

In this paper we evaluate use of different attribution methods for aiding identification of training data artifacts.

Feature Attributions and Counterfactual Explanations Can Be Manipulated

no code implementations23 Jun 2021 Dylan Slack, Sophie Hilgard, Sameer Singh, Himabindu Lakkaraju

As machine learning models are increasingly used in critical decision-making settings (e. g., healthcare, finance), there has been a growing emphasis on developing methods to explain model predictions.

BIG-bench Machine Learning counterfactual +1

Counterfactual Explanations Can Be Manipulated

no code implementations NeurIPS 2021 Dylan Slack, Sophie Hilgard, Himabindu Lakkaraju, Sameer Singh

In this work, we introduce the first framework that describes the vulnerabilities of counterfactual explanations and shows how they can be manipulated.

counterfactual Counterfactual Explanation +1

Learning with Instance Bundles for Reading Comprehension

no code implementations EMNLP 2021 Dheeru Dua, Pradeep Dasigi, Sameer Singh, Matt Gardner

When training most modern reading comprehension models, all the questions associated with a context are treated as being independent from each other.

Reading Comprehension

Competency Problems: On Finding and Removing Artifacts in Language Data

no code implementations EMNLP 2021 Matt Gardner, William Merrill, Jesse Dodge, Matthew E. Peters, Alexis Ross, Sameer Singh, Noah A. Smith

In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems.


An Empirical Comparison of Instance Attribution Methods for NLP

1 code implementation NAACL 2021 Pouya Pezeshkpour, Sarthak Jain, Byron C. Wallace, Sameer Singh

Instance attribution methods constitute one means of accomplishing these goals by retrieving training instances that (may have) led to a particular prediction.


Paired Examples as Indirect Supervision in Latent Decision Models

no code implementations EMNLP 2021 Nitish Gupta, Sameer Singh, Matt Gardner, Dan Roth

Such an objective does not require external supervision for the values of the latent output, or even the end task, yet provides an additional training signal to that provided by individual training examples themselves.

Out-of-Distribution Generalization Question Answering +2

Calibrate Before Use: Improving Few-Shot Performance of Language Models

5 code implementations19 Feb 2021 Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, Sameer Singh

We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art.

Few-Shot Learning

Head Network Distillation: Splitting Distilled Deep Neural Networks for Resource-Constrained Edge Computing Systems

2 code implementations20 Nov 2020 Yoshitomo Matsubara, Davide Callegaro, Sabur Baidya, Marco Levorato, Sameer Singh

In this paper, we propose to modify the structure and training process of DNN models for complex image classification tasks to achieve in-network compression in the early network layers.

Edge-computing Image Classification +2

Interpreting Predictions of NLP Models

no code implementations EMNLP 2020 Eric Wallace, Matt Gardner, Sameer Singh

Although neural NLP models are highly expressive and empirically successful, they also systematically fail in counterintuitive ways and are opaque in their decision-making process.

Decision Making

Concealed Data Poisoning Attacks on NLP Models

no code implementations NAACL 2021 Eric Wallace, Tony Z. Zhao, Shi Feng, Sameer Singh

In this work, we develop a new data poisoning attack that allows an adversary to control model predictions whenever a desired trigger phrase is present in the input.

Data Poisoning Language Modelling +2

Gradient-based Analysis of NLP Models is Manipulable

no code implementations Findings of the Association for Computational Linguistics 2020 Junlin Wang, Jens Tuyls, Eric Wallace, Sameer Singh

Gradient-based analysis methods, such as saliency map visualizations and adversarial input perturbations, have found widespread use in interpreting neural NLP models due to their simplicity, flexibility, and most importantly, their faithfulness.

text-classification Text Classification

MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics

1 code implementation EMNLP 2020 Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner

Posing reading comprehension as a generation problem provides a great deal of flexibility, allowing for open-ended questions with few restrictions on possible answers.

Question Answering Reading Comprehension

Evaluating NLP Models via Contrast Sets

no code implementations1 Oct 2020 Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, A. Zhang, Ben Zhou

Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.

Reading Comprehension Sentiment Analysis

Reliable Post hoc Explanations: Modeling Uncertainty in Explainability

1 code implementation NeurIPS 2021 Dylan Slack, Sophie Hilgard, Sameer Singh, Himabindu Lakkaraju

In this paper, we address the aforementioned challenges by developing a novel Bayesian framework for generating local explanations along with their associated uncertainty.

Feature Importance

Benefits of Intermediate Annotations in Reading Comprehension

no code implementations ACL 2020 Dheeru Dua, Sameer Singh, Matt Gardner

Complex compositional reading comprehension datasets require performing latent sequential decisions that are learned via supervision from the final answer.

Reading Comprehension

On Importance Sampling-Based Evaluation of Latent Language Models

no code implementations ACL 2020 Robert L. Logan IV, Matt Gardner, Sameer Singh

In addition, we elucidate subtle differences in how importance sampling is applied in these works that can have substantial effects on the final estimates, as well as provide theoretical results which reinforce the validity of this technique.

Dynamic Sampling Strategies for Multi-Task Reading Comprehension

no code implementations ACL 2020 Ananth Gottumukkala, Dheeru Dua, Sameer Singh, Matt Gardner

Building general reading comprehension systems, capable of solving multiple datasets at the same time, is a recent aspirational goal in the research community.

Multi-Task Learning Reading Comprehension

Image Augmentations for GAN Training

no code implementations4 Jun 2020 Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, Han Zhang

We provide new state-of-the-art results for conditional generation on CIFAR-10 with both consistency loss and contrastive loss as additional regularizations.

Image Augmentation Image Generation

Beyond Accuracy: Behavioral Testing of NLP models with CheckList

4 code implementations ACL 2020 Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh

Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors.

Question Answering Sentiment Analysis

Obtaining Faithful Interpretations from Compositional Neural Networks

1 code implementation ACL 2020 Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, Matt Gardner

Neural module networks (NMNs) are a popular approach for modeling compositionality: they achieve high accuracy when applied to problems in language and vision, while reflecting the compositional structure of the problem in the network architecture.

Explain Your Move: Understanding Agent Actions Using Focused Feature Saliency

1 code implementation ICLR 2020 Piyush Gupta, Nikaash Puri, Sukriti Verma, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, Sameer Singh

We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that our approach generates saliency maps that are more interpretable for humans than existing approaches.

Atari Games Board Games +2

Revisiting Evaluation of Knowledge Base Completion Models

no code implementations AKBC 2020 Pouya Pezeshkpour, Yifan Tian, Sameer Singh

To address these issues, we gather a semi-complete KG referred as YAGO3-TC, using a random subgraph from the test and validation data of YAGO3-10, which enables us to compute accurate triple classification accuracy on this data.

Knowledge Base Completion Triple Classification

Improved Consistency Regularization for GANs

no code implementations11 Feb 2020 Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, Han Zhang

Recent work has increased the performance of Generative Adversarial Networks (GANs) by enforcing a consistency cost on the discriminator.

Image Generation

ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension

no code implementations29 Dec 2019 Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Sameer Singh, Matt Gardner

A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity typing to entity tracking and understanding the implications of the context.

Entity Typing Machine Reading Comprehension +2

Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution

2 code implementations23 Dec 2019 Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, Sameer Singh

We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that SARFA generates saliency maps that are more interpretable for humans than existing approaches.

Atari Games Board Games +2

Neural Module Networks for Reasoning over Text

2 code implementations ICLR 2020 Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, Matt Gardner

Answering compositional questions that require multiple steps of reasoning against text is challenging, especially when they involve discrete, symbolic operations.

Inductive Bias

Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods

2 code implementations6 Nov 2019 Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, Himabindu Lakkaraju

Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous.

Compositional Generalization with Tree Stack Memory Units

3 code implementations5 Nov 2019 Forough Arabshahi, Zhichu Lu, Pranay Mundra, Sameer Singh, Animashree Anandkumar

We study compositional generalization, viz., the problem of zero-shot generalization to novel compositions of concepts in a domain.

Mathematical Reasoning Zero-shot Generalization

Evaluating Question Answering Evaluation

no code implementations WS 2019 Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner

Our study suggests that while current metrics may be suitable for existing QA datasets, they limit the complexity of QA datasets that can be created.

Answer Generation Multiple-choice +1

Comprehensive Multi-Dataset Evaluation of Reading Comprehension

no code implementations WS 2019 Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Sameer Singh, Matt Gardner

A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity typing to entity tracking and understanding the implications of the context.

Entity Typing Natural Language Understanding +2

Improving Differentially Private Models with Active Learning

no code implementations2 Oct 2019 Zhengli Zhao, Nicolas Papernot, Sameer Singh, Neoklis Polyzotis, Augustus Odena

Broad adoption of machine learning techniques has increased privacy concerns for models trained on sensitive data such as medical records.

Active Learning

Distilled Split Deep Neural Networks for Edge-Assisted Real-Time Systems

2 code implementations1 Oct 2019 Yoshitomo Matsubara, Sabur Baidya, Davide Callegaro, Marco Levorato, Sameer Singh

Offloading the execution of complex Deep Neural Networks (DNNs) models to compute-capable devices at the network edge, that is, edge servers, can significantly reduce capture-to-output delay.

Edge-computing Image Classification +2

AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models

1 code implementation IJCNLP 2019 Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, Sameer Singh

Neural NLP models are increasingly accurate but are imperfect and opaque---they break in counterintuitive ways and leave end users puzzled at their behavior.

Language Modelling Masked Language Modeling +1

Do NLP Models Know Numbers? Probing Numeracy in Embeddings

1 code implementation IJCNLP 2019 Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, Matt Gardner

The ability to understand and work with numbers (numeracy) is critical for many complex reasoning tasks.

Question Answering

Knowledge Enhanced Contextual Word Representations

1 code implementation IJCNLP 2019 Matthew E. Peters, Mark Neumann, Robert L. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith

Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities.

Entity Linking Entity Typing +3

Universal Adversarial Triggers for Attacking and Analyzing NLP

1 code implementation IJCNLP 2019 Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh

We define universal adversarial triggers: input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset.

Language Modelling Reading Comprehension

Are Red Roses Red? Evaluating Consistency of Question-Answering Models

1 code implementation ACL 2019 Marco Tulio Ribeiro, Carlos Guestrin, Sameer Singh

Although current evaluation of question-answering systems treats predictions in isolation, we need to consider the relationship between predictions to measure true understanding.

Question Answering valid +1

GenderQuant: Quantifying Mention-Level Genderedness

no code implementations NAACL 2019 {Ananya}, Nitya Parthasarthi, Sameer Singh

Language is gendered if the context surrounding a mention is suggestive of a particular binary gender for that mention.


Deep Adversarial Learning for NLP

no code implementations NAACL 2019 William Yang Wang, Sameer Singh, Jiwei Li

Adversarial learning is a game-theoretic learning paradigm, which has achieved huge successes in the field of Computer Vision recently.

Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications

1 code implementation NAACL 2019 Pouya Pezeshkpour, Yifan Tian, Sameer Singh

We use these techniques to evaluate the robustness of link prediction models (by measuring sensitivity to additional facts), study interpretability through the facts most responsible for predictions (by identifying the most influential neighbors), and detect incorrect facts in the knowledge base.

Link Prediction

PoMo: Generating Entity-Specific Post-Modifiers in Context

no code implementations NAACL 2019 Jun Seok Kang, Robert L. Logan IV, Zewei Chu, Yang Chen, Dheeru Dua, Kevin Gimpel, Sameer Singh, Niranjan Balasubramanian

Given a sentence about a target entity, the task is to automatically generate a post-modifier phrase that provides contextually relevant information about the entity.


Embedding Multimodal Relational Data for Knowledge Base Completion

2 code implementations EMNLP 2018 Pouya Pezeshkpour, Liyan Chen, Sameer Singh

In this paper, we propose multimodal knowledge base embeddings (MKBE) that use different neural encoders for this variety of observed data, and combine them with existing relational models to learn embeddings of the entities and multimodal data.

Imputation Knowledge Base Completion +1

Semantically Equivalent Adversarial Rules for Debugging NLP models

1 code implementation ACL 2018 Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin

Complex machine learning models for NLP are often brittle, making different predictions for input instances that are extremely similar semantically.

Data Augmentation Question Answering +3

Compact Factorization of Matrices Using Generalized Round-Rank

1 code implementation1 May 2018 Pouya Pezeshkpour, Carlos Guestrin, Sameer Singh

Matrix factorization is a well-studied task in machine learning for compactly representing large, noisy data.

Combining Symbolic Expressions and Black-box Function Evaluations in Neural Programs

1 code implementation ICLR 2018 Forough Arabshahi, Sameer Singh, Animashree Anandkumar

This is because they mostly rely either on black-box function evaluations that do not capture the structure of the program, or on detailed execution traces that are expensive to obtain, and hence the training data has poor coverage of the domain under consideration.


Multimodal Attribute Extraction

1 code implementation29 Nov 2017 Robert L. Logan IV, Samuel Humeau, Sameer Singh

The broad goal of information extraction is to derive structured information from unstructured data.

Attribute Attribute Extraction

Generating Natural Adversarial Examples

1 code implementation ICLR 2018 Zhengli Zhao, Dheeru Dua, Sameer Singh

Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed.

Adversarial Attack Image Classification +3

Entity Linking via Joint Encoding of Types, Descriptions, and Context

no code implementations EMNLP 2017 Nitish Gupta, Sameer Singh, Dan Roth

For accurate entity linking, we need to capture various information aspects of an entity, such as its description in a KB, contexts in which it is mentioned, and structured knowledge.

Entity Linking

Relational Learning and Feature Extraction by Querying over Heterogeneous Information Networks

no code implementations25 Jul 2017 Parisa Kordjamshidi, Sameer Singh, Daniel Khashabi, Christos Christodoulopoulos, Mark Summons, Saurabh Sinha, Dan Roth

In particular, we provide an initial prototype for a relational and graph traversal query language where queries are directly used as relational features for structured machine learning models.

BIG-bench Machine Learning Knowledge Graphs +1

Better call Saul: Flexible Programming for Learning and Inference in NLP

1 code implementation COLING 2016 Parisa Kordjamshidi, Daniel Khashabi, Christos Christodoulopoulos, Bhargav Mangipudi, Sameer Singh, Dan Roth

We present a novel way for designing complex joint inference and learning models using Saul (Kordjamshidi et al., 2015), a recently-introduced declarative learning-based programming language (DeLBP).

Part-Of-Speech Tagging Probabilistic Programming +1

Programs as Black-Box Explanations

no code implementations22 Nov 2016 Sameer Singh, Marco Tulio Ribeiro, Carlos Guestrin

Recent work in model-agnostic explanations of black-box machine learning has demonstrated that interpretability of complex models does not have to come at the cost of accuracy or model flexibility.

Program induction

Nothing Else Matters: Model-Agnostic Explanations By Identifying Prediction Invariance

no code implementations17 Nov 2016 Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin

At the core of interpretable machine learning is the question of whether humans are able to make accurate predictions about a model's behavior.

Interpretable Machine Learning

Model-Agnostic Interpretability of Machine Learning

no code implementations16 Jun 2016 Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin

Understanding why machine learning models behave the way they do empowers both system designers and end-users in many ways: in model selection, feature engineering, in order to trust and act upon the predictions, and in more intuitive user interfaces.

BIG-bench Machine Learning Feature Engineering +1

Connotation Frames: A Data-Driven Investigation

no code implementations ACL 2016 Hannah Rashkin, Sameer Singh, Yejin Choi

Through a particular choice of a predicate (e. g., "x violated y"), a writer can subtly connote a range of implied sentiments and presupposed facts about the entities x and y: (1) writer's perspective: projecting x as an "antagonist"and y as a "victim", (2) entities' perspective: y probably dislikes x, (3) effect: something bad happened to y, (4) value: y is something valuable, and (5) mental state: y is distressed by the event.

Collectively Embedding Multi-Relational Data for Predicting User Preferences

no code implementations23 Apr 2015 Nitish Gupta, Sameer Singh

Matrix factorization has found incredible success and widespread application as a collaborative filtering based approach to recommendations.

Attribute Collaborative Filtering

Design Challenges for Entity Linking

no code implementations TACL 2015 Xiao Ling, Sameer Singh, Daniel S. Weld

Recent research on entity linking (EL) has introduced a plethora of promising techniques, ranging from deep neural networks to joint inference.

Entity Linking Relation Extraction +1

Anytime Belief Propagation Using Sparse Domains

no code implementations14 Nov 2013 Sameer Singh, Sebastian Riedel, Andrew McCallum

Belief Propagation has been widely used for marginal inference, however it is slow on problems with large-domain variables and high-order factors.


FACTORIE: Probabilistic Programming via Imperatively Defined Factor Graphs

no code implementations NeurIPS 2009 Andrew Mccallum, Karl Schultz, Sameer Singh

Discriminatively trained undirected graphical models have had wide empirical success, and there has been increasing interest in toolkits that ease their application to complex relational data.

Probabilistic Programming

Cannot find the paper you are looking for? You can Submit a new open access paper.