Search Results for author:
Found 122 papers, 57 papers with code
Training Factor Graphs with Reinforcement Learning for Efficient MAP Inference
Large, relational factor graphs with structure defined by first-order logic or other languages give rise to notoriously difficult inference problems.
FACTORIE: Probabilistic Programming via Imperatively Defined Factor Graphs
Discriminatively trained undirected graphical models have had wide empirical success, and there has been increasing interest in toolkits that ease their application to complex relational data.
Anytime Belief Propagation Using Sparse Domains
Belief Propagation has been widely used for marginal inference, however it is slow on problems with large-domain variables and high-order factors.
Design Challenges for Entity Linking
Recent research on entity linking (EL) has introduced a plethora of promising techniques, ranging from deep neural networks to joint inference.
Collectively Embedding Multi-Relational Data for Predicting User Preferences
Matrix factorization has found incredible success and widespread application as a collaborative filtering based approach to recommendations.
Connotation Frames: A Data-Driven Investigation
Through a particular choice of a predicate (e. g., "x violated y"), a writer can subtly connote a range of implied sentiments and presupposed facts about the entities x and y: (1) writer's perspective: projecting x as an "antagonist"and y as a "victim", (2) entities' perspective: y probably dislikes x, (3) effect: something bad happened to y, (4) value: y is something valuable, and (5) mental state: y is distressed by the event.
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Despite widespread adoption, machine learning models remain mostly black boxes.
Model-Agnostic Interpretability of Machine Learning
Understanding why machine learning models behave the way they do empowers both system designers and end-users in many ways: in model selection, feature engineering, in order to trust and act upon the predictions, and in more intuitive user interfaces.
Nothing Else Matters: Model-Agnostic Explanations By Identifying Prediction Invariance
At the core of interpretable machine learning is the question of whether humans are able to make accurate predictions about a model's behavior.
Programs as Black-Box Explanations
Recent work in model-agnostic explanations of black-box machine learning has demonstrated that interpretability of complex models does not have to come at the cost of accuracy or model flexibility.
Better call Saul: Flexible Programming for Learning and Inference in NLP
We present a novel way for designing complex joint inference and learning models using Saul (Kordjamshidi et al., 2015), a recently-introduced declarative learning-based programming language (DeLBP).
Relational Learning and Feature Extraction by Querying over Heterogeneous Information Networks
In particular, we provide an initial prototype for a relational and graph traversal query language where queries are directly used as relational features for structured machine learning models.
Entity Linking via Joint Encoding of Types, Descriptions, and Context
For accurate entity linking, we need to capture various information aspects of an entity, such as its description in a KB, contexts in which it is mentioned, and structured knowledge.
Generating Natural Adversarial Examples
Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed.
Multimodal Attribute Extraction
The broad goal of information extraction is to derive structured information from unstructured data.
Combining Symbolic Expressions and Black-box Function Evaluations in Neural Programs
This is because they mostly rely either on black-box function evaluations that do not capture the structure of the program, or on detailed execution traces that are expensive to obtain, and hence the training data has poor coverage of the domain under consideration.
Compact Factorization of Matrices Using Generalized Round-Rank
Matrix factorization is a well-studied task in machine learning for compactly representing large, noisy data.
Semantically Equivalent Adversarial Rules for Debugging NLP models
Complex machine learning models for NLP are often brittle, making different predictions for input instances that are extremely similar semantically.
Interpretation of Natural Language Rules in Conversational Machine Reading
This task requires both the interpretation of rules and the application of background knowledge.
Embedding Multimodal Relational Data for Knowledge Base Completion
In this paper, we propose multimodal knowledge base embeddings (MKBE) that use different neural encoders for this variety of observed data, and combine them with existing relational models to learn embeddings of the entities and multimodal data.
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
We introduce a new English reading comprehension benchmark, DROP, which requires Discrete Reasoning Over the content of Paragraphs.
Ranked #14 on
Question Answering
on DROP Test
PoMo: Generating Entity-Specific Post-Modifiers in Context
Given a sentence about a target entity, the task is to automatically generate a post-modifier phrase that provides contextually relevant information about the entity.
Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications
We use these techniques to evaluate the robustness of link prediction models (by measuring sensitivity to additional facts), study interpretability through the facts most responsible for predictions (by identifying the most influential neighbors), and detect incorrect facts in the knowledge base.
Deep Adversarial Learning for NLP
Adversarial learning is a game-theoretic learning paradigm, which has achieved huge successes in the field of Computer Vision recently.
GenderQuant: Quantifying Mention-Level Genderedness
Language is gendered if the context surrounding a mention is suggestive of a particular binary gender for that mention.
Compositional Questions Do Not Necessitate Multi-hop Reasoning
Multi-hop reading comprehension (RC) questions are challenging because they require reading and reasoning over multiple paragraphs.
Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling
Modeling human language requires the ability to not only generate fluent text but also encode factual knowledge.
Barack's Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling
Modeling human language requires the ability to not only generate fluent text but also encode factual knowledge.
Are Red Roses Red? Evaluating Consistency of Question-Answering Models
Although current evaluation of question-answering systems treats predictions in isolation, we need to consider the relationship between predictions to measure true understanding.
Universal Adversarial Triggers for Attacking and Analyzing NLP
We define universal adversarial triggers: input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset.
Knowledge Enhanced Contextual Word Representations
Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities.
Ranked #9 on
Relation Classification
on TACRED
Do NLP Models Know Numbers? Probing Numeracy in Embeddings
The ability to understand and work with numbers (numeracy) is critical for many complex reasoning tasks.
AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
Neural NLP models are increasingly accurate but are imperfect and opaque---they break in counterintuitive ways and leave end users puzzled at their behavior.
Distilled Split Deep Neural Networks for Edge-Assisted Real-Time Systems
Offloading the execution of complex Deep Neural Networks (DNNs) models to compute-capable devices at the network edge, that is, edge servers, can significantly reduce capture-to-output delay.
Improving Differentially Private Models with Active Learning
Broad adoption of machine learning techniques has increased privacy concerns for models trained on sensitive data such as medical records.
Evaluating Question Answering Evaluation
Our study suggests that while current metrics may be suitable for existing QA datasets, they limit the complexity of QA datasets that can be created.
Comprehensive Multi-Dataset Evaluation of Reading Comprehension
A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity typing to entity tracking and understanding the implications of the context.
Compositional Generalization with Tree Stack Memory Units
We study compositional generalization, viz., the problem of zero-shot generalization to novel compositions of concepts in a domain.
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous.
Neural Module Networks for Reasoning over Text
Answering compositional questions that require multiple steps of reasoning against text is challenging, especially when they involve discrete, symbolic operations.
Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution
We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that SARFA generates saliency maps that are more interpretable for humans than existing approaches.
ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension
A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity typing to entity tracking and understanding the implications of the context.
Improved Consistency Regularization for GANs
Recent work has increased the performance of Generative Adversarial Networks (GANs) by enforcing a consistency cost on the discriminator.
Revisiting Evaluation of Knowledge Base Completion Models
To address these issues, we gather a semi-complete KG referred as YAGO3-TC, using a random subgraph from the test and validation data of YAGO3-10, which enables us to compute accurate triple classification accuracy on this data.
Evaluating Models' Local Decision Boundaries via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Explain Your Move: Understanding Agent Actions Using Focused Feature Saliency
We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that our approach generates saliency maps that are more interpretable for humans than existing approaches.
Obtaining Faithful Interpretations from Compositional Neural Networks
Neural module networks (NMNs) are a popular approach for modeling compositionality: they achieve high accuracy when applied to problems in language and vision, while reflecting the compositional structure of the problem in the network architecture.
Beyond Accuracy: Behavioral Testing of NLP models with CheckList
Although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models, while alternative approaches for evaluating models either focus on individual tasks or on specific behaviors.
Image Augmentations for GAN Training
We provide new state-of-the-art results for conditional generation on CIFAR-10 with both consistency loss and contrastive loss as additional regularizations.
Dynamic Sampling Strategies for Multi-Task Reading Comprehension
Building general reading comprehension systems, capable of solving multiple datasets at the same time, is a recent aspirational goal in the research community.
Benefits of Intermediate Annotations in Reading Comprehension
Complex compositional reading comprehension datasets require performing latent sequential decisions that are learned via supervision from the final answer.
On Importance Sampling-Based Evaluation of Latent Language Models
In addition, we elucidate subtle differences in how importance sampling is applied in these works that can have substantial effects on the final estimates, as well as provide theoretical results which reinforce the validity of this technique.
Reliable Post hoc Explanations: Modeling Uncertainty in Explainability
In this paper, we address the aforementioned challenges by developing a novel Bayesian framework for generating local explanations along with their associated uncertainty.
Evaluating NLP Models via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics
Posing reading comprehension as a generation problem provides a great deal of flexibility, allowing for open-ended questions with few restrictions on possible answers.
MedICaT: A Dataset of Medical Images, Captions, and Textual References
To address challenges in figure retrieval and figure-to-text alignment, we introduce MedICaT, a dataset of medical images in context.
Gradient-based Analysis of NLP Models is Manipulable
Gradient-based analysis methods, such as saliency map visualizations and adversarial input perturbations, have found widespread use in interpreting neural NLP models due to their simplicity, flexibility, and most importantly, their faithfulness.
Concealed Data Poisoning Attacks on NLP Models
In this work, we develop a new data poisoning attack that allows an adversary to control model predictions whenever a desired trigger phrase is present in the input.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining.
Interpreting Predictions of NLP Models
Although neural NLP models are highly expressive and empirically successful, they also systematically fail in counterintuitive ways and are opaque in their decision-making process.
Head Network Distillation: Splitting Distilled Deep Neural Networks for Resource-Constrained Edge Computing Systems
In this paper, we propose to modify the structure and training process of DNN models for complex image classification tasks to achieve in-network compression in the early network layers.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art.
Paired Examples as Indirect Supervision in Latent Decision Models
Such an objective does not require external supervision for the values of the latent output, or even the end task, yet provides an additional training signal to that provided by individual training examples themselves.
An Empirical Comparison of Instance Attribution Methods for NLP
Instance attribution methods constitute one means of accomplishing these goals by retrieving training instances that (may have) led to a particular prediction.
Competency Problems: On Finding and Removing Artifacts in Language Data
In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems.
Generative Context Pair Selection for Multi-hop Question Answering
Compositional reasoning tasks like multi-hop question answering, require making latent decisions to get the final answer, given a question.
Learning with Instance Bundles for Reading Comprehension
When training most modern reading comprehension models, all the questions associated with a context are treated as being independent from each other.
Counterfactual Explanations Can Be Manipulated
In this work, we introduce the first framework that describes the vulnerabilities of counterfactual explanations and shows how they can be manipulated.
Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP
These experiments on AmbER sets show their utility as an evaluation tool and highlight the weaknesses of popular retrieval systems.
Feature Attributions and Counterfactual Explanations Can Be Manipulated
As machine learning models are increasingly used in critical decision-making settings (e. g., healthcare, finance), there has been a growing emphasis on developing methods to explain model predictions.
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning.
Combining Feature and Instance Attribution to Detect Artifacts
In this paper we evaluate use of different attribution methods for aiding identification of training data artifacts.
Enforcing Consistency in Weakly Supervised Semantic Parsing
The predominant challenge in weakly supervised semantic parsing is that of spurious programs that evaluate to correct answers for the wrong reasons.
Benchmarking Scalable Methods for Streaming Cross Document Entity Coreference
We investigate: how to best encode mentions, which clustering algorithms are most effective for grouping mentions, how models transfer to different domains, and how bounding the number of mentions tracked during inference impacts performance.
Modular Framework for Visuomotor Language Grounding
Natural language instruction following tasks serve as a valuable test-bed for grounded language and robotics research.
Entity-Based Knowledge Conflicts in Question Answering
To understand how models use these sources together, we formalize the problem of knowledge conflicts, where the contextual information contradicts the learned information.
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts
Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning.
FRUIT: Faithfully Reflecting Updated Information in Text
Textual knowledge bases such as Wikipedia require considerable effort to keep up to date and consistent.
BottleFit: Learning Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computing
We show that BottleFit decreases power consumption and latency respectively by up to 49% and 89% with respect to (w. r. t.)
Identifying Adversarial Attacks on Text Classifiers
The landscape of adversarial attacks against text classifiers continues to grow, with new attacks developed every year and many of them available in standard toolkits, such as TextAttack and OpenAttack.
Rethinking Explainability as a Dialogue: A Practitioner's Perspective
Overall, we hope our work serves as a starting place for researchers and engineers to design interactive explainability systems.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Pretrained Language Models (LMs) have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings.
Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation
A growing body of research has demonstrated the inability of NLP models to generalize compositionally and has tried to alleviate it through specialized architectures, training schemes, and data augmentation, among other approaches.
ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension
Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain.
Learning to Query Internet Text for Informing Reinforcement Learning Agents
We show that our method correctly learns to execute queries to maximize reward in a reinforcement learning setting.
TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations
In real-world evaluations with humans, 73% of healthcare workers (e. g., doctors and nurses) agreed they would use TalkToModel over baseline point-and-click systems for explainability in a disease prediction task, and 85% of ML professionals agreed TalkToModel was easier to use for computing explanations.
Factual and Informative Review Generation for Explainable Recommendation
Recent models can generate fluent and grammatical synthetic reviews while accurately predicting user ratings.
Quantifying Social Biases Using Templates is Unreliable
Recently, there has been an increase in efforts to understand how large language models (LLMs) propagate and amplify social biases.
Continued Pretraining for Better Zero- and Few-Shot Promptability
We demonstrate that a simple recipe, continued pretraining that incorporates a trainable prompt during multi-task learning, leads to improved promptability in both zero- and few-shot settings compared to existing methods, up to 31% relative.
TCAB: A Large-Scale Text Classification Attack Benchmark
In addition to the primary tasks of detecting and labeling attacks, TCAB can also be used for attack localization, attack target labeling, and attack characterization.
Successive Prompting for Decomposing Complex Questions
The intermediate supervision is typically manually written, which can be expensive to collect.
To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering
Recent advances in open-domain question answering (ODQA) have demonstrated impressive accuracy on standard Wikipedia style benchmarks.
Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling
Reinforcement learning (RL) agents typically learn tabula rasa, without prior knowledge of the world.
ART: Automatic multi-step reasoning and tool-use for large language models
We introduce Automatic Reasoning and Tool-use (ART), a framework that uses frozen LLMs to automatically generate intermediate reasoning steps as a program.
TABLET: Learning From Instructions For Tabular Data
Acquiring high-quality data is often a significant challenge in training machine learning (ML) models for tabular prediction, particularly in privacy-sensitive and costly domains like medicine and finance.
Post Hoc Explanations of Language Models Can Improve Language Models
Large Language Models (LLMs) have demonstrated remarkable capabilities in performing complex tasks.
Coverage-based Example Selection for In-Context Learning
In-context learning (ICL), the ability of large language models to perform novel tasks by conditioning on a prompt with a few task examples, requires these examples to be informative about the test instance.
PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptions
These bottlenecks motivate the training of compact editors, which is challenging due to the scarcity of training data for this purpose.
MISGENDERED: Limits of Large Language Models in Understanding Pronouns
Content Warning: This paper contains examples of misgendering and erasure that could be offensive and potentially triggering.
Maestro: A Gamified Platform for Teaching AI Robustness
We assessed Maestro's influence on students' engagement, motivation, and learning success in robust AI.
Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actors
Large language models (LLMs) are being applied as actors for sequential decision making tasks in domains such as robotics and games, utilizing their general world knowledge and planning abilities.
The Bias Amplification Paradox in Text-to-Image Generation
Bias amplification is a phenomenon in which models exacerbate biases or stereotypes present in the training data.
EchoPrompt: Instructing the Model to Rephrase Queries for Improved In-context Learning
On average, EchoPrompt improves the Zero-shot-CoT performance of code-davinci-002 by 5% in numerical tasks and 13% in reading comprehension tasks.
What's In My Big Data?
We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them.
Performance Trade-offs of Watermarking Large Language Models
However, implanting such signals alters the model's output distribution and can have unintended effects when watermarked LLMs are used for downstream applications.
Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals
Previous work has found that datasets with paired inputs are prone to correlations between a specific part of the input (e. g., the hypothesis in NLI) and the label; consequently, models trained only on those outperform chance.
Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills
We evaluate our method in the classic videogame NetHack and the text environment ScienceWorld to demonstrate SSO's ability to optimize a set of skills and perform in-context policy improvement.
What Models Know About Their Attackers: Deriving Attacker Information From Latent Representations
Adversarial attacks curated against NLP models are increasingly becoming practical threats.
Robustness and Adversarial Examples in Natural Language Processing
In particular, we will review recent studies on analyzing the weakness of NLP systems when facing adversarial inputs and data with a distribution shift.
Zero- and Few-Shot NLP with Pretrained Language Models
The ability to efficiently learn from little-to-no data is critical to applying NLP to tasks where data collection is costly or otherwise difficult.
Tweeki: Linking Named Entities on Twitter to a Knowledge Graph
Further, there is a lack of a large, linked corpus of tweets to aid researchers, along with lack of gold dataset to evaluate the accuracy of entity linking.
COVIDLies: Detecting COVID-19 Misinformation on Social Media
The ongoing pandemic has heightened the need for developing tools to flag COVID-19-related misinformation on the internet, specifically on social media such as Twitter.
Citations Beyond Self Citations: Identifying Authors, Affiliations, and Nationalities in Scientific Papers
Our models are accurate; we identify at least one of authors, affiliations, and nationalities of held-out papers with 40. 3%, 47. 9% and 86. 0% accuracy respectively, from the top-10 guesses of our models.
