Search Results for author:
Found 109 papers, 51 papers with code
Zero- and Few-Shot NLP with Pretrained Language Models
The ability to efficiently learn from little-to-no data is critical to applying NLP to tasks where data collection is costly or otherwise difficult.
Robustness and Adversarial Examples in Natural Language Processing
In particular, we will review recent studies on analyzing the weakness of NLP systems when facing adversarial inputs and data with a distribution shift.
What Models Know About Their Attackers: Deriving Attacker Information From Latent Representations
Adversarial attacks curated against NLP models are increasingly becoming practical threats.
COVIDLies: Detecting COVID-19 Misinformation on Social Media
The ongoing pandemic has heightened the need for developing tools to flag COVID-19-related misinformation on the internet, specifically on social media such as Twitter.
Citations Beyond Self Citations: Identifying Authors, Affiliations, and Nationalities in Scientific Papers
Our models are accurate; we identify at least one of authors, affiliations, and nationalities of held-out papers with 40. 3%, 47. 9% and 86. 0% accuracy respectively, from the top-10 guesses of our models.
Tweeki: Linking Named Entities on Twitter to a Knowledge Graph
Further, there is a lack of a large, linked corpus of tweets to aid researchers, along with lack of gold dataset to evaluate the accuracy of entity linking.
ART: Automatic multi-step reasoning and tool-use for large language models
We introduce Automatic Reasoning and Tool-use (ART), a framework that uses frozen LLMs to automatically generate intermediate reasoning steps as a program.
Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling
Reinforcement learning (RL) agents typically learn tabula rasa, without prior knowledge of the world, which makes learning complex tasks with sparse rewards difficult.
To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering
Recent advances in open-domain question answering (ODQA) have demonstrated impressive accuracy on standard Wikipedia style benchmarks.
Successive Prompting for Decomposing Complex Questions
The intermediate supervision is typically manually written, which can be expensive to collect.
TCAB: A Large-Scale Text Classification Attack Benchmark
In addition to the primary tasks of detecting and labeling attacks, TCAB can also be used for attack localization, attack target labeling, and attack characterization.
Continued Pretraining for Better Zero- and Few-Shot Promptability
We demonstrate that a simple recipe, continued pretraining that incorporates a trainable prompt during multi-task learning, leads to improved promptability in both zero- and few-shot settings compared to existing methods, up to 31% relative.
Quantifying Social Biases Using Templates is Unreliable
Recently, there has been an increase in efforts to understand how large language models (LLMs) propagate and amplify social biases.
Factual and Informative Review Generation for Explainable Recommendation
Recent models can generate fluent and grammatical synthetic reviews while accurately predicting user ratings.
TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations
In real-world evaluations with humans, 73% of healthcare workers (e. g., doctors and nurses) agreed they would use TalkToModel over baseline point-and-click systems for explainability in a disease prediction task, and 85% of ML professionals agreed TalkToModel was easier to use for computing explanations.
Learning to Query Internet Text for Informing Reinforcement Learning Agents
We show that our method correctly learns to execute queries to maximize reward in a reinforcement learning setting.
ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension
Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain.
Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation
A growing body of research has demonstrated the inability of NLP models to generalize compositionally and has tried to alleviate it through specialized architectures, training schemes, and data augmentation, among other approaches.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Pretrained Language Models (LMs) have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings.
Rethinking Explainability as a Dialogue: A Practitioner's Perspective
Overall, we hope our work serves as a starting place for researchers and engineers to design interactive explainability systems.
Identifying Adversarial Attacks on Text Classifiers
The landscape of adversarial attacks against text classifiers continues to grow, with new attacks developed every year and many of them available in standard toolkits, such as TextAttack and OpenAttack.
BottleFit: Learning Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computing
We show that BottleFit decreases power consumption and latency respectively by up to 49% and 89% with respect to (w. r. t.)
FRUIT: Faithfully Reflecting Updated Information in Text
Textual knowledge bases such as Wikipedia require considerable effort to keep up to date and consistent.
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts
Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning.
Entity-Based Knowledge Conflicts in Question Answering
To understand how models use these sources together, we formalize the problem of knowledge conflicts, where the contextual information contradicts the learned information.
Modular Framework for Visuomotor Language Grounding
Natural language instruction following tasks serve as a valuable test-bed for grounded language and robotics research.
Benchmarking Scalable Methods for Streaming Cross Document Entity Coreference
We investigate: how to best encode mentions, which clustering algorithms are most effective for grouping mentions, how models transfer to different domains, and how bounding the number of mentions tracked during inference impacts performance.
Enforcing Consistency in Weakly Supervised Semantic Parsing
The predominant challenge in weakly supervised semantic parsing is that of spurious programs that evaluate to correct answers for the wrong reasons.
Combining Feature and Instance Attribution to Detect Artifacts
In this paper we evaluate use of different attribution methods for aiding identification of training data artifacts.
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning.
Feature Attributions and Counterfactual Explanations Can Be Manipulated
As machine learning models are increasingly used in critical decision-making settings (e. g., healthcare, finance), there has been a growing emphasis on developing methods to explain model predictions.
Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP
These experiments on AmbER sets show their utility as an evaluation tool and highlight the weaknesses of popular retrieval systems.
Counterfactual Explanations Can Be Manipulated
In this work, we introduce the first framework that describes the vulnerabilities of counterfactual explanations and shows how they can be manipulated.
Generative Context Pair Selection for Multi-hop Question Answering
Compositional reasoning tasks like multi-hop question answering, require making latent decisions to get the final answer, given a question.
Learning with Instance Bundles for Reading Comprehension
When training most modern reading comprehension models, all the questions associated with a context are treated as being independent from each other.
Competency Problems: On Finding and Removing Artifacts in Language Data
In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems.
An Empirical Comparison of Instance Attribution Methods for NLP
Instance attribution methods constitute one means of accomplishing these goals by retrieving training instances that (may have) led to a particular prediction.
Paired Examples as Indirect Supervision in Latent Decision Models
Such an objective does not require external supervision for the values of the latent output, or even the end task, yet provides an additional training signal to that provided by individual training examples themselves.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art.
Head Network Distillation: Splitting Distilled Deep Neural Networks for Resource-Constrained Edge Computing Systems
In this paper, we propose to modify the structure and training process of DNN models for complex image classification tasks to achieve in-network compression in the early network layers.
Interpreting Predictions of NLP Models
Although neural NLP models are highly expressive and empirically successful, they also systematically fail in counterintuitive ways and are opaque in their decision-making process.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining.
Concealed Data Poisoning Attacks on NLP Models
In this work, we develop a new data poisoning attack that allows an adversary to control model predictions whenever a desired trigger phrase is present in the input.
Gradient-based Analysis of NLP Models is Manipulable
Gradient-based analysis methods, such as saliency map visualizations and adversarial input perturbations, have found widespread use in interpreting neural NLP models due to their simplicity, flexibility, and most importantly, their faithfulness.
MedICaT: A Dataset of Medical Images, Captions, and Textual References
To address challenges in figure retrieval and figure-to-text alignment, we introduce MedICaT, a dataset of medical images in context.
MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics
Posing reading comprehension as a generation problem provides a great deal of flexibility, allowing for open-ended questions with few restrictions on possible answers.
Evaluating NLP Models via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Reliable Post hoc Explanations: Modeling Uncertainty in Explainability
In this paper, we address the aforementioned challenges by developing a novel Bayesian framework for generating local explanations along with their associated uncertainty.
Benefits of Intermediate Annotations in Reading Comprehension
Complex compositional reading comprehension datasets require performing latent sequential decisions that are learned via supervision from the final answer.
Dynamic Sampling Strategies for Multi-Task Reading Comprehension
Building general reading comprehension systems, capable of solving multiple datasets at the same time, is a recent aspirational goal in the research community.
On Importance Sampling-Based Evaluation of Latent Language Models
In addition, we elucidate subtle differences in how importance sampling is applied in these works that can have substantial effects on the final estimates, as well as provide theoretical results which reinforce the validity of this technique.
Image Augmentations for GAN Training
We provide new state-of-the-art results for conditional generation on CIFAR-10 with both consistency loss and contrastive loss as additional regularizations.
Beyond Accuracy: Behavioral Testing of NLP models with CheckList
Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors.
Obtaining Faithful Interpretations from Compositional Neural Networks
Neural module networks (NMNs) are a popular approach for modeling compositionality: they achieve high accuracy when applied to problems in language and vision, while reflecting the compositional structure of the problem in the network architecture.
Explain Your Move: Understanding Agent Actions Using Focused Feature Saliency
We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that our approach generates saliency maps that are more interpretable for humans than existing approaches.
Evaluating Models' Local Decision Boundaries via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Revisiting Evaluation of Knowledge Base Completion Models
To address these issues, we gather a semi-complete KG referred as YAGO3-TC, using a random subgraph from the test and validation data of YAGO3-10, which enables us to compute accurate triple classification accuracy on this data.
Improved Consistency Regularization for GANs
Recent work has increased the performance of Generative Adversarial Networks (GANs) by enforcing a consistency cost on the discriminator.
ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension
A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity typing to entity tracking and understanding the implications of the context.
Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution
We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that SARFA generates saliency maps that are more interpretable for humans than existing approaches.
Neural Module Networks for Reasoning over Text
Answering compositional questions that require multiple steps of reasoning against text is challenging, especially when they involve discrete, symbolic operations.
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous.
Compositional Generalization with Tree Stack Memory Units
We study compositional generalization, viz., the problem of zero-shot generalization to novel compositions of concepts in a domain.
Evaluating Question Answering Evaluation
Our study suggests that while current metrics may be suitable for existing QA datasets, they limit the complexity of QA datasets that can be created.
Comprehensive Multi-Dataset Evaluation of Reading Comprehension
A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity typing to entity tracking and understanding the implications of the context.
Improving Differentially Private Models with Active Learning
Broad adoption of machine learning techniques has increased privacy concerns for models trained on sensitive data such as medical records.
Distilled Split Deep Neural Networks for Edge-Assisted Real-Time Systems
Offloading the execution of complex Deep Neural Networks (DNNs) models to compute-capable devices at the network edge, that is, edge servers, can significantly reduce capture-to-output delay.
AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
Neural NLP models are increasingly accurate but are imperfect and opaque---they break in counterintuitive ways and leave end users puzzled at their behavior.
Do NLP Models Know Numbers? Probing Numeracy in Embeddings
The ability to understand and work with numbers (numeracy) is critical for many complex reasoning tasks.
Knowledge Enhanced Contextual Word Representations
Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities.
Ranked #11 on
Entity Linking
on AIDA-CoNLL
Universal Adversarial Triggers for Attacking and Analyzing NLP
We define universal adversarial triggers: input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset.
Barack's Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling
Modeling human language requires the ability to not only generate fluent text but also encode factual knowledge.
Are Red Roses Red? Evaluating Consistency of Question-Answering Models
Although current evaluation of question-answering systems treats predictions in isolation, we need to consider the relationship between predictions to measure true understanding.
Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling
Modeling human language requires the ability to not only generate fluent text but also encode factual knowledge.
Compositional Questions Do Not Necessitate Multi-hop Reasoning
Multi-hop reading comprehension (RC) questions are challenging because they require reading and reasoning over multiple paragraphs.
GenderQuant: Quantifying Mention-Level Genderedness
Language is gendered if the context surrounding a mention is suggestive of a particular binary gender for that mention.
Deep Adversarial Learning for NLP
Adversarial learning is a game-theoretic learning paradigm, which has achieved huge successes in the field of Computer Vision recently.
Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications
We use these techniques to evaluate the robustness of link prediction models (by measuring sensitivity to additional facts), study interpretability through the facts most responsible for predictions (by identifying the most influential neighbors), and detect incorrect facts in the knowledge base.
PoMo: Generating Entity-Specific Post-Modifiers in Context
Given a sentence about a target entity, the task is to automatically generate a post-modifier phrase that provides contextually relevant information about the entity.
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
We introduce a new English reading comprehension benchmark, DROP, which requires Discrete Reasoning Over the content of Paragraphs.
Ranked #11 on
Question Answering
on DROP Test
Embedding Multimodal Relational Data for Knowledge Base Completion
In this paper, we propose multimodal knowledge base embeddings (MKBE) that use different neural encoders for this variety of observed data, and combine them with existing relational models to learn embeddings of the entities and multimodal data.
Interpretation of Natural Language Rules in Conversational Machine Reading
This task requires both the interpretation of rules and the application of background knowledge.
Semantically Equivalent Adversarial Rules for Debugging NLP models
Complex machine learning models for NLP are often brittle, making different predictions for input instances that are extremely similar semantically.
Compact Factorization of Matrices Using Generalized Round-Rank
Matrix factorization is a well-studied task in machine learning for compactly representing large, noisy data.
Combining Symbolic Expressions and Black-box Function Evaluations in Neural Programs
This is because they mostly rely either on black-box function evaluations that do not capture the structure of the program, or on detailed execution traces that are expensive to obtain, and hence the training data has poor coverage of the domain under consideration.
Multimodal Attribute Extraction
The broad goal of information extraction is to derive structured information from unstructured data.
Generating Natural Adversarial Examples
Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed.
Entity Linking via Joint Encoding of Types, Descriptions, and Context
For accurate entity linking, we need to capture various information aspects of an entity, such as its description in a KB, contexts in which it is mentioned, and structured knowledge.
Relational Learning and Feature Extraction by Querying over Heterogeneous Information Networks
In particular, we provide an initial prototype for a relational and graph traversal query language where queries are directly used as relational features for structured machine learning models.
Better call Saul: Flexible Programming for Learning and Inference in NLP
We present a novel way for designing complex joint inference and learning models using Saul (Kordjamshidi et al., 2015), a recently-introduced declarative learning-based programming language (DeLBP).
Programs as Black-Box Explanations
Recent work in model-agnostic explanations of black-box machine learning has demonstrated that interpretability of complex models does not have to come at the cost of accuracy or model flexibility.
Nothing Else Matters: Model-Agnostic Explanations By Identifying Prediction Invariance
At the core of interpretable machine learning is the question of whether humans are able to make accurate predictions about a model's behavior.
Model-Agnostic Interpretability of Machine Learning
Understanding why machine learning models behave the way they do empowers both system designers and end-users in many ways: in model selection, feature engineering, in order to trust and act upon the predictions, and in more intuitive user interfaces.
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Despite widespread adoption, machine learning models remain mostly black boxes.
Connotation Frames: A Data-Driven Investigation
Through a particular choice of a predicate (e. g., "x violated y"), a writer can subtly connote a range of implied sentiments and presupposed facts about the entities x and y: (1) writer's perspective: projecting x as an "antagonist"and y as a "victim", (2) entities' perspective: y probably dislikes x, (3) effect: something bad happened to y, (4) value: y is something valuable, and (5) mental state: y is distressed by the event.
Collectively Embedding Multi-Relational Data for Predicting User Preferences
Matrix factorization has found incredible success and widespread application as a collaborative filtering based approach to recommendations.
Design Challenges for Entity Linking
Recent research on entity linking (EL) has introduced a plethora of promising techniques, ranging from deep neural networks to joint inference.
Anytime Belief Propagation Using Sparse Domains
Belief Propagation has been widely used for marginal inference, however it is slow on problems with large-domain variables and high-order factors.
FACTORIE: Probabilistic Programming via Imperatively Defined Factor Graphs
Discriminatively trained undirected graphical models have had wide empirical success, and there has been increasing interest in toolkits that ease their application to complex relational data.
Training Factor Graphs with Reinforcement Learning for Efficient MAP Inference
Large, relational factor graphs with structure defined by first-order logic or other languages give rise to notoriously difficult inference problems.
