Search Results for author: Sameer Singh

Found 90 papers, 39 papers with code

Tweeki: Linking Named Entities on Twitter to a Knowledge Graph

no code implementations EMNLP (WNUT) 2020 Bahareh Harandizadeh, Sameer Singh

Further, there is a lack of a large, linked corpus of tweets to aid researchers, along with lack of gold dataset to evaluate the accuracy of entity linking.

Entity Linking

Robustness and Adversarial Examples in Natural Language Processing

no code implementations EMNLP (ACL) 2021 Kai-Wei Chang, He He, Robin Jia, Sameer Singh

In particular, we will review recent studies on analyzing the weakness of NLP systems when facing adversarial inputs and data with a distribution shift.

COVIDLies: Detecting COVID-19 Misinformation on Social Media

no code implementations EMNLP (NLP-COVID19) 2020 Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, Sameer Singh

The ongoing pandemic has heightened the need for developing tools to flag COVID-19-related misinformation on the internet, specifically on social media such as Twitter.

Misinformation Stance Detection

Citations Beyond Self Citations: Identifying Authors, Affiliations, and Nationalities in Scientific Papers

1 code implementation WOSP 2020 Yoshitomo Matsubara, Sameer Singh

Our models are accurate; we identify at least one of authors, affiliations, and nationalities of held-out papers with 40. 3%, 47. 9% and 86. 0% accuracy respectively, from the top-10 guesses of our models.

Entity-Based Knowledge Conflicts in Question Answering

1 code implementation EMNLP 2021 Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, Sameer Singh

To understand how models use these sources together, we formalize the problem of knowledge conflicts, where the contextual information contradicts the learned information.

Question Answering

Modular Framework for Visuomotor Language Grounding

no code implementations5 Sep 2021 Kolby Nottingham, Litian Liang, Daeyun Shin, Charless C. Fowlkes, Roy Fox, Sameer Singh

Natural language instruction following tasks serve as a valuable test-bed for grounded language and robotics research.

Benchmarking Scalable Methods for Streaming Cross Document Entity Coreference

1 code implementation ACL 2021 Robert L Logan IV, Andrew McCallum, Sameer Singh, Dan Bikel

We investigate: how to best encode mentions, which clustering algorithms are most effective for grouping mentions, how models transfer to different domains, and how bounding the number of mentions tracked during inference impacts performance.

Entity Disambiguation Entity Linking

Enforcing Consistency in Weakly Supervised Semantic Parsing

1 code implementation ACL 2021 Nitish Gupta, Sameer Singh, Matt Gardner

The predominant challenge in weakly supervised semantic parsing is that of spurious programs that evaluate to correct answers for the wrong reasons.

Semantic Parsing Visual Reasoning

Combining Feature and Instance Attribution to Detect Artifacts

no code implementations1 Jul 2021 Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, Byron C. Wallace

Training the large deep neural networks that dominate NLP requires large datasets.

Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

1 code implementation24 Jun 2021 Robert L. Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, Sebastian Riedel

Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning.

Few-Shot Learning

Feature Attributions and Counterfactual Explanations Can Be Manipulated

no code implementations23 Jun 2021 Dylan Slack, Sophie Hilgard, Sameer Singh, Himabindu Lakkaraju

As machine learning models are increasingly used in critical decision-making settings (e. g., healthcare, finance), there has been a growing emphasis on developing methods to explain model predictions.

Decision Making

Counterfactual Explanations Can Be Manipulated

no code implementations NeurIPS 2021 Dylan Slack, Sophie Hilgard, Himabindu Lakkaraju, Sameer Singh

In this work, we introduce the first framework that describes the vulnerabilities of counterfactual explanations and shows how they can be manipulated.

Counterfactual Explanation Crime Prediction

Learning with Instance Bundles for Reading Comprehension

no code implementations EMNLP 2021 Dheeru Dua, Pradeep Dasigi, Sameer Singh, Matt Gardner

When training most modern reading comprehension models, all the questions associated with a context are treated as being independent from each other.

Reading Comprehension

Competency Problems: On Finding and Removing Artifacts in Language Data

no code implementations EMNLP 2021 Matt Gardner, William Merrill, Jesse Dodge, Matthew E. Peters, Alexis Ross, Sameer Singh, Noah A. Smith

In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems.

Language understanding

An Empirical Comparison of Instance Attribution Methods for NLP

1 code implementation NAACL 2021 Pouya Pezeshkpour, Sarthak Jain, Byron C. Wallace, Sameer Singh

Instance attribution methods constitute one means of accomplishing these goals by retrieving training instances that (may have) led to a particular prediction.

Paired Examples as Indirect Supervision in Latent Decision Models

no code implementations EMNLP 2021 Nitish Gupta, Sameer Singh, Matt Gardner, Dan Roth

Such an objective does not require external supervision for the values of the latent output, or even the end task, yet provides an additional training signal to that provided by individual training examples themselves.

Question Answering Question Generation

Calibrate Before Use: Improving Few-Shot Performance of Language Models

2 code implementations19 Feb 2021 Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, Sameer Singh

We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art.

Few-Shot Learning

Head Network Distillation: Splitting Distilled Deep Neural Networks for Resource-Constrained Edge Computing Systems

2 code implementations20 Nov 2020 Yoshitomo Matsubara, Davide Callegaro, Sabur Baidya, Marco Levorato, Sameer Singh

In this paper, we propose to modify the structure and training process of DNN models for complex image classification tasks to achieve in-network compression in the early network layers.

Edge-computing Knowledge Distillation +1

Interpreting Predictions of NLP Models

no code implementations EMNLP 2020 Eric Wallace, Matt Gardner, Sameer Singh

Although neural NLP models are highly expressive and empirically successful, they also systematically fail in counterintuitive ways and are opaque in their decision-making process.

Decision Making

Concealed Data Poisoning Attacks on NLP Models

no code implementations NAACL 2021 Eric Wallace, Tony Z. Zhao, Shi Feng, Sameer Singh

In this work, we develop a new data poisoning attack that allows an adversary to control model predictions whenever a desired trigger phrase is present in the input.

Data Poisoning Language Modelling +2

Gradient-based Analysis of NLP Models is Manipulable

no code implementations Findings of the Association for Computational Linguistics 2020 Junlin Wang, Jens Tuyls, Eric Wallace, Sameer Singh

Gradient-based analysis methods, such as saliency map visualizations and adversarial input perturbations, have found widespread use in interpreting neural NLP models due to their simplicity, flexibility, and most importantly, their faithfulness.

Text Classification

MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics

1 code implementation EMNLP 2020 Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner

Posing reading comprehension as a generation problem provides a great deal of flexibility, allowing for open-ended questions with few restrictions on possible answers.

Question Answering Reading Comprehension

Evaluating NLP Models via Contrast Sets

no code implementations1 Oct 2020 Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, A. Zhang, Ben Zhou

Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.

Reading Comprehension Sentiment Analysis

Reliable Post hoc Explanations: Modeling Uncertainty in Explainability

no code implementations NeurIPS 2021 Dylan Slack, Sophie Hilgard, Sameer Singh, Himabindu Lakkaraju

In this paper, we address the aforementioned challenges by developing a novel Bayesian framework for generating local explanations along with their associated uncertainty.

Feature Importance

Dynamic Sampling Strategies for Multi-Task Reading Comprehension

no code implementations ACL 2020 Ananth Gottumukkala, Dheeru Dua, Sameer Singh, Matt Gardner

Building general reading comprehension systems, capable of solving multiple datasets at the same time, is a recent aspirational goal in the research community.

Multi-Task Learning Reading Comprehension

On Importance Sampling-Based Evaluation of Latent Language Models

no code implementations ACL 2020 Robert L. Logan IV, Matt Gardner, Sameer Singh

In addition, we elucidate subtle differences in how importance sampling is applied in these works that can have substantial effects on the final estimates, as well as provide theoretical results which reinforce the validity of this technique.

Benefits of Intermediate Annotations in Reading Comprehension

no code implementations ACL 2020 Dheeru Dua, Sameer Singh, Matt Gardner

Complex compositional reading comprehension datasets require performing latent sequential decisions that are learned via supervision from the final answer.

Reading Comprehension

Image Augmentations for GAN Training

no code implementations4 Jun 2020 Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, Han Zhang

We provide new state-of-the-art results for conditional generation on CIFAR-10 with both consistency loss and contrastive loss as additional regularizations.

Image Augmentation Image Generation

Beyond Accuracy: Behavioral Testing of NLP models with CheckList

4 code implementations ACL 2020 Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh

Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors.

Question Answering Sentiment Analysis

Obtaining Faithful Interpretations from Compositional Neural Networks

1 code implementation ACL 2020 Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, Matt Gardner

Neural module networks (NMNs) are a popular approach for modeling compositionality: they achieve high accuracy when applied to problems in language and vision, while reflecting the compositional structure of the problem in the network architecture.

Explain Your Move: Understanding Agent Actions Using Focused Feature Saliency

1 code implementation ICLR 2020 Piyush Gupta, Nikaash Puri, Sukriti Verma, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, Sameer Singh

We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that our approach generates saliency maps that are more interpretable for humans than existing approaches.

Atari Games Board Games

Revisiting Evaluation of Knowledge Base Completion Models

no code implementations AKBC 2020 Pouya Pezeshkpour, Yifan Tian, Sameer Singh

To address these issues, we gather a semi-complete KG referred as YAGO3-TC, using a random subgraph from the test and validation data of YAGO3-10, which enables us to compute accurate triple classification accuracy on this data.

Knowledge Base Completion Knowledge Graph Completion +1

Improved Consistency Regularization for GANs

no code implementations11 Feb 2020 Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, Han Zhang

Recent work has increased the performance of Generative Adversarial Networks (GANs) by enforcing a consistency cost on the discriminator.

Image Generation

ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension

no code implementations29 Dec 2019 Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Sameer Singh, Matt Gardner

A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity typing to entity tracking and understanding the implications of the context.

Entity Typing Language understanding +3

Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution

2 code implementations23 Dec 2019 Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, Sameer Singh

We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that SARFA generates saliency maps that are more interpretable for humans than existing approaches.

Atari Games Board Games

Neural Module Networks for Reasoning over Text

2 code implementations ICLR 2020 Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, Matt Gardner

Answering compositional questions that require multiple steps of reasoning against text is challenging, especially when they involve discrete, symbolic operations.

Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods

2 code implementations6 Nov 2019 Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, Himabindu Lakkaraju

Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous.

Compositional Generalization with Tree Stack Memory Units

3 code implementations5 Nov 2019 Forough Arabshahi, Zhichu Lu, Pranay Mundra, Sameer Singh, Animashree Anandkumar

We study compositional generalization, viz., the problem of zero-shot generalization to novel compositions of concepts in a domain.

Mathematical Reasoning

Evaluating Question Answering Evaluation

no code implementations WS 2019 Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner

Our study suggests that while current metrics may be suitable for existing QA datasets, they limit the complexity of QA datasets that can be created.

Question Answering

Comprehensive Multi-Dataset Evaluation of Reading Comprehension

no code implementations WS 2019 Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Sameer Singh, Matt Gardner

A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity typing to entity tracking and understanding the implications of the context.

Entity Typing Language understanding +3

Improving Differentially Private Models with Active Learning

no code implementations2 Oct 2019 Zhengli Zhao, Nicolas Papernot, Sameer Singh, Neoklis Polyzotis, Augustus Odena

Broad adoption of machine learning techniques has increased privacy concerns for models trained on sensitive data such as medical records.

Active Learning Fine-tuning

Distilled Split Deep Neural Networks for Edge-Assisted Real-Time Systems

2 code implementations1 Oct 2019 Yoshitomo Matsubara, Sabur Baidya, Davide Callegaro, Marco Levorato, Sameer Singh

Offloading the execution of complex Deep Neural Networks (DNNs) models to compute-capable devices at the network edge, that is, edge servers, can significantly reduce capture-to-output delay.

Edge-computing Knowledge Distillation +1

AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models

1 code implementation IJCNLP 2019 Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, Sameer Singh

Neural NLP models are increasingly accurate but are imperfect and opaque---they break in counterintuitive ways and leave end users puzzled at their behavior.

Language Modelling Reading Comprehension

Do NLP Models Know Numbers? Probing Numeracy in Embeddings

1 code implementation IJCNLP 2019 Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, Matt Gardner

The ability to understand and work with numbers (numeracy) is critical for many complex reasoning tasks.

Question Answering

Knowledge Enhanced Contextual Word Representations

1 code implementation IJCNLP 2019 Matthew E. Peters, Mark Neumann, Robert L. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith

Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities.

Entity Linking Entity Typing +3

Universal Adversarial Triggers for Attacking and Analyzing NLP

1 code implementation IJCNLP 2019 Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh

We define universal adversarial triggers: input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset.

Language Modelling Reading Comprehension

Are Red Roses Red? Evaluating Consistency of Question-Answering Models

1 code implementation ACL 2019 Marco Tulio Ribeiro, Carlos Guestrin, Sameer Singh

Although current evaluation of question-answering systems treats predictions in isolation, we need to consider the relationship between predictions to measure true understanding.

Question Answering Visual Question Answering

Deep Adversarial Learning for NLP

no code implementations NAACL 2019 William Yang Wang, Sameer Singh, Jiwei Li

Adversarial learning is a game-theoretic learning paradigm, which has achieved huge successes in the field of Computer Vision recently.

GenderQuant: Quantifying Mention-Level Genderedness

no code implementations NAACL 2019 {Ananya}, Nitya Parthasarthi, Sameer Singh

Language is gendered if the context surrounding a mention is suggestive of a particular binary gender for that mention.

Coreference Resolution

Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications

no code implementations NAACL 2019 Pouya Pezeshkpour, Yifan Tian, Sameer Singh

We use these techniques to evaluate the robustness of link prediction models (by measuring sensitivity to additional facts), study interpretability through the facts most responsible for predictions (by identifying the most influential neighbors), and detect incorrect facts in the knowledge base.

Link Prediction

PoMo: Generating Entity-Specific Post-Modifiers in Context

no code implementations NAACL 2019 Jun Seok Kang, Robert L. Logan IV, Zewei Chu, Yang Chen, Dheeru Dua, Kevin Gimpel, Sameer Singh, Niranjan Balasubramanian

Given a sentence about a target entity, the task is to automatically generate a post-modifier phrase that provides contextually relevant information about the entity.

Embedding Multimodal Relational Data for Knowledge Base Completion

2 code implementations EMNLP 2018 Pouya Pezeshkpour, Liyan Chen, Sameer Singh

In this paper, we propose multimodal knowledge base embeddings (MKBE) that use different neural encoders for this variety of observed data, and combine them with existing relational models to learn embeddings of the entities and multimodal data.

Imputation Knowledge Base Completion +1

Semantically Equivalent Adversarial Rules for Debugging NLP models

1 code implementation ACL 2018 Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin

Complex machine learning models for NLP are often brittle, making different predictions for input instances that are extremely similar semantically.

Data Augmentation Question Answering +3

Compact Factorization of Matrices Using Generalized Round-Rank

1 code implementation1 May 2018 Pouya Pezeshkpour, Carlos Guestrin, Sameer Singh

Matrix factorization is a well-studied task in machine learning for compactly representing large, noisy data.

Combining Symbolic Expressions and Black-box Function Evaluations in Neural Programs

1 code implementation ICLR 2018 Forough Arabshahi, Sameer Singh, Animashree Anandkumar

This is because they mostly rely either on black-box function evaluations that do not capture the structure of the program, or on detailed execution traces that are expensive to obtain, and hence the training data has poor coverage of the domain under consideration.

Multimodal Attribute Extraction

1 code implementation29 Nov 2017 Robert L. Logan IV, Samuel Humeau, Sameer Singh

The broad goal of information extraction is to derive structured information from unstructured data.

Generating Natural Adversarial Examples

1 code implementation ICLR 2018 Zhengli Zhao, Dheeru Dua, Sameer Singh

Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed.

Adversarial Attack Image Classification +3

Entity Linking via Joint Encoding of Types, Descriptions, and Context

no code implementations EMNLP 2017 Nitish Gupta, Sameer Singh, Dan Roth

For accurate entity linking, we need to capture various information aspects of an entity, such as its description in a KB, contexts in which it is mentioned, and structured knowledge.

Entity Linking

Relational Learning and Feature Extraction by Querying over Heterogeneous Information Networks

no code implementations25 Jul 2017 Parisa Kordjamshidi, Sameer Singh, Daniel Khashabi, Christos Christodoulopoulos, Mark Summons, Saurabh Sinha, Dan Roth

In particular, we provide an initial prototype for a relational and graph traversal query language where queries are directly used as relational features for structured machine learning models.

Knowledge Graphs Relational Reasoning

Better call Saul: Flexible Programming for Learning and Inference in NLP

1 code implementation COLING 2016 Parisa Kordjamshidi, Daniel Khashabi, Christos Christodoulopoulos, Bhargav Mangipudi, Sameer Singh, Dan Roth

We present a novel way for designing complex joint inference and learning models using Saul (Kordjamshidi et al., 2015), a recently-introduced declarative learning-based programming language (DeLBP).

Part-Of-Speech Tagging Probabilistic Programming +1

Programs as Black-Box Explanations

no code implementations22 Nov 2016 Sameer Singh, Marco Tulio Ribeiro, Carlos Guestrin

Recent work in model-agnostic explanations of black-box machine learning has demonstrated that interpretability of complex models does not have to come at the cost of accuracy or model flexibility.

Program induction

Nothing Else Matters: Model-Agnostic Explanations By Identifying Prediction Invariance

no code implementations17 Nov 2016 Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin

At the core of interpretable machine learning is the question of whether humans are able to make accurate predictions about a model's behavior.

Interpretable Machine Learning

Model-Agnostic Interpretability of Machine Learning

no code implementations16 Jun 2016 Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin

Understanding why machine learning models behave the way they do empowers both system designers and end-users in many ways: in model selection, feature engineering, in order to trust and act upon the predictions, and in more intuitive user interfaces.

Feature Engineering Model Selection

Connotation Frames: A Data-Driven Investigation

no code implementations ACL 2016 Hannah Rashkin, Sameer Singh, Yejin Choi

Through a particular choice of a predicate (e. g., "x violated y"), a writer can subtly connote a range of implied sentiments and presupposed facts about the entities x and y: (1) writer's perspective: projecting x as an "antagonist"and y as a "victim", (2) entities' perspective: y probably dislikes x, (3) effect: something bad happened to y, (4) value: y is something valuable, and (5) mental state: y is distressed by the event.

Collectively Embedding Multi-Relational Data for Predicting User Preferences

no code implementations23 Apr 2015 Nitish Gupta, Sameer Singh

Matrix factorization has found incredible success and widespread application as a collaborative filtering based approach to recommendations.

Collaborative Filtering

Design Challenges for Entity Linking

no code implementations TACL 2015 Xiao Ling, Sameer Singh, Daniel S. Weld

Recent research on entity linking (EL) has introduced a plethora of promising techniques, ranging from deep neural networks to joint inference.

Entity Linking Relation Extraction +1

Anytime Belief Propagation Using Sparse Domains

no code implementations14 Nov 2013 Sameer Singh, Sebastian Riedel, Andrew McCallum

Belief Propagation has been widely used for marginal inference, however it is slow on problems with large-domain variables and high-order factors.

FACTORIE: Probabilistic Programming via Imperatively Defined Factor Graphs

no code implementations NeurIPS 2009 Andrew Mccallum, Karl Schultz, Sameer Singh

Discriminatively trained undirected graphical models have had wide empirical success, and there has been increasing interest in toolkits that ease their application to complex relational data.

Probabilistic Programming

Training Factor Graphs with Reinforcement Learning for Efficient MAP Inference

no code implementations NeurIPS 2009 Khashayar Rohanimanesh, Sameer Singh, Andrew McCallum, Michael J. Black

Large, relational factor graphs with structure defined by first-order logic or other languages give rise to notoriously difficult inference problems.

Cannot find the paper you are looking for? You can Submit a new open access paper.