Search Results for author:
Found 107 papers, 35 papers with code
To Find Waldo You Need Contextual Cues: Debiasing Who’s Waldo
We find that the original Who’s Waldo dataset compiled for this task contains a large number of biased samples that are solvable simply by heuristic methods; for instance, in many cases the first name in the sentence corresponds to the largest bounding box, or the sequence of names in the sentence corresponds to an exact left-to-right order in the image.
Deeply Embedded Knowledge Representation & Reasoning For Natural Language Question Answering: A Practitioner’s Perspective
Successful application of Knowledge Representation and Reasoning (KR) in Natural Language Understanding (NLU) is largely limited by the availability of a robust and general purpose natural language parser.
Reframing Instructional Prompts to GPTk’s Language
Our experiments compare the zero-shot and few-shot performance of LMs prompted with reframed instructions on 12 NLP tasks across 6 categories.
Dr.ICL: Demonstration-Retrieved In-context Learning
In-context learning (ICL), teaching a large language model (LLM) to perform a task with few-shot demonstrations rather than adjusting the model parameters, has emerged as a strong paradigm for using LLMs.
Can NLP Models Correctly Reason Over Contexts that Break the Common Assumptions?
Can state-of-the-art NLP models correctly reason over the contexts of such scenarios?
A Unified Evaluation Framework for Novelty Detection and Accommodation in NLP with an Instantiation in Authorship Attribution
To initiate a systematic research in this important area of 'dealing with novelties', we introduce 'NoveltyTask', a multi-stage task to evaluate a system's performance on pipelined novelty 'detection' and 'accommodation' tasks.
Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA
Despite remarkable progress made in natural language processing, even the state-of-the-art models often make incorrect predictions.
Prompt-Based Learning for Thread Structure Prediction in Cybersecurity Forums
We propose a method called Next Paragraph Prediction with Instructional Prompting (NPP-IP) to predict thread structures while grounded on the context around posts.
Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environments
Learning to detect, characterize and accommodate novelties is a challenge that agents operating in open-world domains need to address to be able to guarantee satisfactory task performance.
Exploring the Limits of Transfer Learning with Unified Model in the Cybersecurity Domain
We also show that with a few examples, UTS can be adapted to novel unseen tasks and the nature of data
InstructABSA: Instruction Learning for Aspect Based Sentiment Analysis
In this paper, we present InstructABSA, Aspect Based Sentiment Analysis (ABSA) using the instruction learning paradigm for the ABSA subtasks: Aspect Term Extraction (ATE), Aspect Term Sentiment Classification (ATSC), and Joint Task modeling.
Ranked #1 on
Aspect Extraction
on SemEval 2014 Task 4 Sub Task 2
Real-Time Visual Feedback to Guide Benchmark Creation: A Human-and-Metric-in-the-Loop Workflow
In pursuit of creating better benchmarks, we propose VAIDA, a novel benchmark creation paradigm for NLP, that focuses on guiding crowdworkers, an under-explored facet of addressing benchmark idiosyncrasies.
Lexi: Self-Supervised Learning of the UI Language
Along with text, these resources include visual content such as UI screenshots and images of application icons referenced in the text.
Benchmarking Spatial Relationships in Text-to-Image Generation
We investigate the ability of T2I models to generate correct spatial relationships among objects and present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image.
Learning Action-Effect Dynamics for Hypothetical Vision-Language Reasoning Task
'Actions' play a vital role in how humans interact with the world.
Learning Action-Effect Dynamics from Pairs of Scene-graphs
'Actions' play a vital role in how humans interact with the world.
Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?
Comparing with the FiD reader, this approach matches its accuracy by utilizing just 18. 32% of its reader inference cost and also outperforms it by achieving up to 55. 10% accuracy on NQ Open.
CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering
Videos often capture objects, their visible properties, their motion, and the interactions between different objects.
Lila: A Unified Benchmark for Mathematical Reasoning
Mathematical reasoning skills are essential for general-purpose intelligent systems to perform tasks from grocery shopping to climate modeling.
Ranked #1 on
Mathematical Reasoning
on Lila (OOD)
Pretrained Transformers Do not Always Improve Robustness
Pretrained Transformers (PT) have been shown to improve Out of Distribution (OOD) robustness than traditional models such as Bag of Words (BOW), LSTMs, Convolutional Neural Networks (CNN) powered by Word2Vec and Glove embeddings.
"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibility
We show that even state-of-the-art models such as GPT-3, GPT-2, and T5 struggle to answer the feasibility questions correctly.
A Survey of Parameters Associated with the Quality of Benchmarks in NLP
Inspired by successful quality indices in several domains such as power, food, and water, we take the first step towards a metric by identifying certain language properties that can represent various possible interactions leading to biases in a benchmark.
Hardness of Samples Need to be Quantified for a Reliable Evaluation System: Exploring Potential Opportunities with a New Task
Evaluation of models on benchmarks is unreliable without knowing the degree of sample hardness; this subsequently overestimates the capability of AI systems and limits their adoption in real world applications.
Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems
Through comprehensive experiments in multiple task settings that differ in the number of models available for cascading (K value), we show that cascading improves both the computational efficiency and the prediction accuracy.
Investigating the Failure Modes of the AUC metric and Exploring Alternatives for Evaluating Systems in Safety Critical Applications
We find limitations in AUC; e. g., a model having higher AUC is not always better in performing selective answering.
A Study on the Efficiency and Generalization of Light Hybrid Retrievers
Driven by this question, we leverage an indexing-efficient dense retriever (i. e. DrBoost) and introduce a LITE retriever that further reduces the memory of DrBoost.
Reasoning about Actions over Visual and Linguistic Modalities: A Survey
'Actions' play a vital role in how humans interact with the world and enable them to achieve desired goals.
BioTABQA: Instruction Learning for Biomedical Table Question Answering
To the best of our knowledge, none of TQA datasets exist in the biomedical domain where tables are frequently used to present information.
Improving Diversity with Adversarially Learned Transformations for Domain Generalization
To be successful in single source domain generalization, maximizing diversity of synthesized domains has emerged as one of the most effective strategies.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Is a Question Decomposition Unit All We Need?
Large Language Models (LMs) have achieved state-of-the-art performance on many Natural Language Processing (NLP) benchmarks.
Let the Model Decide its Curriculum for Multitask Learning
Curriculum learning strategies in prior multi-task learning approaches arrange datasets in a difficulty hierarchy either based on human perception or by exhaustively searching the optimal arrangement.
Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions
In this work, we hypothesize that annotators pick up on patterns in the crowdsourcing instructions, which bias them to write many similar examples that are then over-represented in the collected data.
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning
Recently, instructional prompts have shown significant improvement towards multi-task generalization; however, the effect of instructional prompts and Multi-Task Learning (MTL) has not been systematically studied in the biomedical domain.
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems.
To Find Waldo You Need Contextual Cues: Debiasing Who's Waldo
We find that the original Who's Waldo dataset compiled for this task contains a large number of biased samples that are solvable simply by heuristic methods; for instance, in many cases the first name in the sentence corresponds to the largest bounding box, or the sequence of names in the sentence corresponds to an exact left-to-right order in the image.
How Many Data Samples is an Additional Instruction Worth?
However, they can write alternate instructions to represent an instruction task.
Less is More: Summary of Long Instructions is Better for Program Synthesis
We show that LMs benefit from the summarized version of complicated questions.
Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustness
However, the effect of data modification on adversarial robustness remains unclear.
Choose Your QA Model Wisely: A Systematic Study of Generative and Extractive Readers for Question Answering
Among several interesting findings, it is important to highlight that (1) the generative readers perform better in long context QA, (2) the extractive readers perform better in short context while also showing better out-of-domain generalization, and (3) the encoder of encoder-decoder PrLMs (e. g., T5) turns out to be a strong extractive reader and outperforms the standard choice of encoder-only PrLMs (e. g., RoBERTa).
ILDAE: Instance-Level Difficulty Analysis of Evaluation Data
Knowledge of questions' difficulty level helps a teacher in several ways, such as estimating students' potential quickly by asking carefully selected questions and improving quality of examination by modifying trivial and hard questions.
Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings
In order to equip NLP systems with selective prediction capability, several task-specific approaches have been proposed.
Improving Biomedical Information Retrieval with Neural Retrievers
We show that BM25 and our method can complement each other, and a simple hybrid model leads to further gains in the large corpus setting.
Unsupervised Natural Language Inference Using PHL Triplet Generation
Transformer-based models achieve impressive performance on numerous Natural Language Inference (NLI) benchmarks when trained on respective training datasets.
A Bayesian Approach for Medical Inquiry and Disease Inference in Automated Differential Diagnosis
Unlike previous work that simulates data from given probabilities and uses ML algorithms on them, we directly use the Quick Medical Reference (QMR) belief network, and apply Bayesian inference in the inference phase and Bayesian experimental design in the inquiry phase.
Semantically Distributed Robust Optimization for Vision-and-Language Inference
Analysis of vision-and-language models has revealed their brittleness under linguistic phenomena such as paraphrasing, negation, textual entailment, and word substitutions with synonyms or antonyms.
A Simple Approach to Jointly Rank Passages and Select Relevant Sentences in the OBQA Context
Furthermore, we propose consistency and similarity constraints to promote the correlation and interaction between passage ranking and sentence selection. The experiments demonstrate that our framework can achieve competitive results with previous systems and outperform the baseline by 28\% in terms of exact matching of relevant sentences on the HotpotQA dataset.
Reframing Instructional Prompts to GPTk's Language
Our experiments compare the zero-shot and few-shot performance of LMs prompted with reframed instructions on 12 NLP tasks across 6 categories.
Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model
Some possible reasons can be the tokenizers and pre-training objectives which are not specifically designed to learn and preserve numeracy.
Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering
The visual retriever aims to retrieve relevant knowledge, and the visual reader seeks to predict answers based on given knowledge.
Weakly Supervised Relative Spatial Reasoning for Visual Question Answering
In this work, we evaluate the faithfulness of V\&L models to such geometric understanding, by formulating the prediction of pair-wise relative locations of objects as a classification as well as a regression task.
Interviewer-Candidate Role Play: Towards Developing Real-World NLP Systems
However, our task leaves a significant challenge for NLP researchers to further improve OOD performance at each stage.
Commonsense Reasoning with Implicit Knowledge in Natural Language
Two common approaches to this are (i) Use of well-structured commonsense present in knowledge graphs, and (ii) Use of progressively larger transformer language models.
CLEVR\_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images
Most existing research on visual question answering (VQA) is limited to information explicitly present in an image or a video.
Constructing Flow Graphs from Procedural Cybersecurity Texts
We must read the whole text to identify the relevant information or identify the instruction flows to complete a task, which is prone to failures.
Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction
In this work, we propose Masked Noun-Phrase Prediction (MNPP), a pre-training strategy to tackle pronoun resolution in a fully unsupervised setting.
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones.
CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images
Most existing research on visual question answering (VQA) is limited to information explicitly present in an image or a video.
`Just because you are right, doesn't mean I am wrong': Overcoming a bottleneck in development and evaluation of Open-Ended VQA tasks
GQA (CITATION) is a dataset for real-world visual reasoning and compositional question answering.
'Just because you are right, doesn't mean I am wrong': Overcoming a Bottleneck in the Development and Evaluation of Open-Ended Visual Question Answering (VQA) Tasks
GQA~\citep{hudson2019gqa} is a dataset for real-world visual reasoning and compositional question answering.
Variable Name Recovery in Decompiled Binary Code using Constrained Masked Language Modeling
Inspired by recent advances in natural language processing, we propose a novel solution to infer variable names in decompiled code based on Masked Language Modeling, Byte-Pair Encoding, and neural architectures such as Transformers and BERT.
Self-Supervised Test-Time Learning for Reading Comprehension
Recent work on unsupervised question answering has shown that models can be trained with procedurally generated question-answer pairs and can achieve performance competitive with supervised methods.
Can Transformers Reason About Effects of Actions?
A recent work has shown that transformers are able to "reason" with facts and rules in a limited setting where the rules are natural language expressions of conjunctions of conditions implying a conclusion.
WeaQA: Weak Supervision via Captions for Visual Question Answering
Methodologies for training visual question answering (VQA) models assume the availability of datasets with human-annotated \textit{Image-Question-Answer} (I-Q-A) triplets.
Attribute-Guided Adversarial Training for Robustness to Natural Perturbations
While this deviation may not be exactly known, its broad characterization is specified a priori, in terms of attributes.
Language-Conditioned Imitation Learning for Robot Manipulation Tasks
Imitation learning is a popular approach for teaching motor skills to robots.
MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
In this paper, we present MUTANT, a training paradigm that exposes the model to perceptually similar, yet semantically distinct mutations of the input, to improve OOD generalization, such as the VQA-CP challenge.
Multi-Perspective Semantic Information Retrieval
Information Retrieval (IR) is the task of obtaining pieces of data (such as documents or snippets of text) that are relevant to a particular query or need from a large repository of information.
Towards Improving Selective Prediction Ability of NLP Systems
In (IID, OOD) settings, we show that the representations learned by our calibrator result in an improvement of (15. 81%, 5. 64%) and (6. 19%, 13. 9%) over 'MaxProb' -- a selective prediction baseline -- on NLI and DD tasks respectively.
DQI: A Guide to Benchmark Evaluation
A `state of the art' model A surpasses humans in a benchmark B, but fails on similar benchmarks C, D, and E. What does B have that the other benchmarks do not?
Our Evaluation Metric Needs an Update to Encourage Generalization
In order to stop the inflation in model performance -- and thus overestimation in AI systems' capabilities -- we propose a simple and novel evaluation metric, WOOD Score, that encourages generalization during evaluation.
Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks
However, there exists a strong need for a benchmark which can evaluate the abilities of models, in performing question format independent numerical reasoning, as (i) the numerical reasoning capabilities we want to teach are not controlled by question formats, (ii) for numerical reasoning technology to have the best possible application, it must be able to process language and reason in a way that is not exclusive to a single format, task, dataset or domain.
DQI: Measuring Data Quality in NLP
The data creation paradigm consists of several data visualizations to help data creators (i) understand the quality of data and (ii) visualize the impact of the created data instance on the overall quality.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
The aim of all Question Answering (QA) systems is to be able to generalize to unseen questions.
Visuo-Linguistic Question Answering (VLQA) Challenge
Understanding images and text together is an important aspect of cognition and building advanced Artificial Intelligence (AI) systems.
Knowledge Fusion and Semantic Knowledge Ranking for Open Domain Question Answering
Open Domain Question Answering requires systems to retrieve external knowledge and perform multi-hop reasoning by composing knowledge spread over multiple sentences.
Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning
In videos that involve active agents such as humans, the agent's actions can bring about myriad changes in the scene.
Natural Language QA Approaches using Reasoning with External Knowledge
The challenges inspired by Winograd's councilmen example, and recent developments such as the Rebooting AI book, various NLQA datasets, research on knowledge acquisition in the NLQA context, and their use in various NLQA models have brought the issue of NLQA using ``reasoning'' with external knowledge to the forefront.
VQA-LOL: Visual Question Answering under the Lens of Logic
We propose our {Lens of Logic (LOL)} model which uses question-attention and logic-attention to understand logical connectives in the question, and a novel Fr\'echet-Compatibility Loss, which ensures that the answers of the component questions and the composed question are consistent with the inferred logical operation.
Imitation Learning of Robot Policies by Combining Language, Vision and Demonstration
In this work we propose a novel end-to-end imitation learning approach which combines natural language, vision, and motion information to produce an abstract representation of a task, which in turn is used to synthesize specific motion controllers at run-time.
Knowledge Guided Named Entity Recognition for BioMedical Text
In this work, we formulate the NER task as a multi-answer knowledge guided QA task (KGQA) which helps to predict entities only by assigning B, I and O tags without associating entity types with the tags.
Imitation Learning of Robot Policies using Language, Vision and Motion
In this work we propose a novel end-to-end imitation learning approach which combines natural language, vision, and motion information to produce an abstract representation of a task, which in turn can be used to synthesize specific motion controllers at run-time.
How Additional Knowledge can Improve Natural Language Commonsense Question Answering?
Recently several datasets have been proposed to encourage research in Question Answering domains where commonsense knowledge is expected to play an important role.
A Generate-Validate Approach to Answering Questions about Qualitative Relationships
Qualitative relationships describe how increasing or decreasing one property (e. g. altitude) affects another (e. g. temperature).
Identification of Adverse Drug Reaction Mentions in Tweets -- SMM4H Shared Task 2019
Analyzing social media posts can offer insights into a wide range of topics that are commonly discussed online, providing valuable information for studying various health-related phenomena reported online.
Careful Selection of Knowledge to solve Open Book Question Answering
Open book question answering is a type of natural language based QA (NLQA) where questions are expected to be answered with respect to a given set of open book facts, and common knowledge about a topic.
Ranked #8 on
Question Answering
on OpenBookQA
Combining Knowledge Hunting and Neural Language Models to Solve the Winograd Schema Challenge
Our end-to-end system built in such a manner improves on the accuracy of two of the available language model based approaches by 5. 53{\%} and 7. 7{\%} respectively.
Integrating Knowledge and Reasoning in Image Understanding
Deep learning based data-driven approaches have been successfully applied in various image understanding applications ranging from object recognition, semantic segmentation to visual question answering.
Blocksworld Revisited: Learning and Reasoning to Generate Event-Sequences from Image Pairs
The process of identifying changes or transformations in a scene along with the ability of reasoning about their causes and effects, is a key aspect of intelligence.
Declarative Question Answering over Knowledge Bases containing Natural Language Text with Answer Set Programming
While in recent years machine learning (ML) based approaches have been the popular approach in developing end-to-end question answering systems, such systems often struggle when additional knowledge is needed to correctly answer the questions.
Understanding Roles and Entities: Datasets and Models for Natural Language Inference
We present two new datasets and a novel attention mechanism for Natural Language Inference (NLI).
Developing and Using Special-Purpose Lexicons for Cohort Selection from Clinical Notes
To help automate the process, National NLP Clinical Challenges (N2C2) conducted a shared challenge by defining 13 criteria for clinical trial cohort selection and by providing training and test datasets.
Spatial Knowledge Distillation to aid Visual Reasoning
We propose a framework that combines recent advances in knowledge distillation (teacher-student framework), relational reasoning and probabilistic logical languages to incorporate such knowledge in existing neural networks for the task of Visual Question Answering.
Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering
Here we adopt Visual Question Answering (VQA) as an example task, where a system is expected to answer a question in natural language about an image.
Incremental and Iterative Learning of Answer Set Programs from Mutually Distinct Examples
This paper is under consideration for acceptance in
Answering Image Riddles using Vision and Reasoning through Probabilistic Soft Logic
We compile a dataset of over 3k riddles where each riddle consists of 4 images and a groundtruth answer.
From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge
Specifically, commonsense reasoning is applied on (a) detections obtained from existing perception methods on given images, (b) a "commonsense" knowledge base constructed using natural language processing of image annotations and (c) lexical ontological knowledge from resources such as WordNet.
An Action Language for Multi-Agent Domains: Foundations
It also allows the specification of agents' dynamic awareness of action occurrences which has future implications on what agents' know about the world and other agents' knowledge about the world.
Event-Object Reasoning with Curated Knowledge Bases: Deriving Missing Information
The broader goal of our research is to formulate answers to why and how questions with respect to knowledge bases, such as AURA.
Encoding Higher Level Extensions of Petri Nets in Answer Set Programming
Answering realistic questions about biological systems and pathways similar to the ones used by text books to test understanding of students about biological systems is one of our long term research goals.
Encoding Petri Nets in Answer Set Programming for Simulation Based Reasoning
However, we need to make extensions to the Petri Net model and also reason with multiple simulation runs and parallel state evolutions.
