Search Results for author:
Found 41 papers, 17 papers with code
Do We Need to Create Big Datasets to Learn a Task?
Since the language models have already been pre-trained with huge amount of data and have basic linguistic knowledge, there is no need to create big datasets to learn a task.
InstructABSA: Instruction Learning for Aspect Based Sentiment Analysis
In this paper, we present InstructABSA, Aspect Based Sentiment Analysis (ABSA) using the instruction learning paradigm for the ABSA subtasks: Aspect Term Extraction (ATE), Aspect Term Sentiment Classification (ATSC), and Joint Task modeling.
Ranked #1 on
Aspect Extraction
on SemEval 2014 Task 4 Sub Task 2
Real-Time Visual Feedback to Guide Benchmark Creation: A Human-and-Metric-in-the-Loop Workflow
In pursuit of creating better benchmarks, we propose VAIDA, a novel benchmark creation paradigm for NLP, that focuses on guiding crowdworkers, an under-explored facet of addressing benchmark idiosyncrasies.
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations.
Lila: A Unified Benchmark for Mathematical Reasoning
Mathematical reasoning skills are essential for general-purpose intelligent systems to perform tasks from grocery shopping to climate modeling.
Ranked #1 on
Mathematical Reasoning
on Lila (OOD)
"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibility
We show that even state-of-the-art models such as GPT-3, GPT-2, and T5 struggle to answer the feasibility questions correctly.
A Survey of Parameters Associated with the Quality of Benchmarks in NLP
Inspired by successful quality indices in several domains such as power, food, and water, we take the first step towards a metric by identifying certain language properties that can represent various possible interactions leading to biases in a benchmark.
Hardness of Samples Need to be Quantified for a Reliable Evaluation System: Exploring Potential Opportunities with a New Task
Evaluation of models on benchmarks is unreliable without knowing the degree of sample hardness; this subsequently overestimates the capability of AI systems and limits their adoption in real world applications.
Pretrained Transformers Do not Always Improve Robustness
Pretrained Transformers (PT) have been shown to improve Out of Distribution (OOD) robustness than traditional models such as Bag of Words (BOW), LSTMs, Convolutional Neural Networks (CNN) powered by Word2Vec and Glove embeddings.
Investigating the Failure Modes of the AUC metric and Exploring Alternatives for Evaluating Systems in Safety Critical Applications
We find limitations in AUC; e. g., a model having higher AUC is not always better in performing selective answering.
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions.
Ranked #3 on
Science Question Answering
on ScienceQA
HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models
We demonstrate the efficacy of our technique HELP ME THINK on a variety of tasks.
BioTABQA: Instruction Learning for Biomedical Table Question Answering
To the best of our knowledge, none of TQA datasets exist in the biomedical domain where tables are frequently used to present information.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Is a Question Decomposition Unit All We Need?
Large Language Models (LMs) have achieved state-of-the-art performance on many Natural Language Processing (NLP) benchmarks.
Let the Model Decide its Curriculum for Multitask Learning
Curriculum learning strategies in prior multi-task learning approaches arrange datasets in a difficulty hierarchy either based on human perception or by exhaustively searching the optimal arrangement.
Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions
In this work, we hypothesize that annotators pick up on patterns in the crowdsourcing instructions, which bias them to write many similar examples that are then over-represented in the collected data.
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning
Recently, instructional prompts have shown significant improvement towards multi-task generalization; however, the effect of instructional prompts and Multi-Task Learning (MTL) has not been systematically studied in the biomedical domain.
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems.
How Many Data Samples is an Additional Instruction Worth?
However, they can write alternate instructions to represent an instruction task.
Less is More: Summary of Long Instructions is Better for Program Synthesis
We show that LMs benefit from the summarized version of complicated questions.
Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustness
However, the effect of data modification on adversarial robustness remains unclear.
A Proposal to Study "Is High Quality Data All We Need?"
Our hypothesis is based on the fact that deep neural networks are data driven models, and data is what leads/misleads models.
ILDAE: Instance-Level Difficulty Analysis of Evaluation Data
Knowledge of questions' difficulty level helps a teacher in several ways, such as estimating students' potential quickly by asking carefully selected questions and improving quality of examination by modifying trivial and hard questions.
Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings
In order to equip NLP systems with selective prediction capability, several task-specific approaches have been proposed.
Reframing Instructional Prompts to GPTk's Language
Our experiments compare the zero-shot and few-shot performance of LMs prompted with reframed instructions on 12 NLP tasks across 6 categories.
Context-NER : Contextual Phrase Generation at Scale
We also introduce the EDGAR10-Q dataset for the same, which is a corpus of 1, 500 publicly traded companies.
Ranked #1 on
ContextNER
on EDGAR10-Q Dataset
Development of User-friendly Smart Grid Architecture
As systems like smart grid continue to become complex on a daily basis, emerging issues demand complex solutions that can deal with parameters in multiple domains of engineering.
Interviewer-Candidate Role Play: Towards Developing Real-World NLP Systems
However, our task leaves a significant challenge for NLP researchers to further improve OOD performance at each stage.
Commonsense Reasoning with Implicit Knowledge in Natural Language
Two common approaches to this are (i) Use of well-structured commonsense present in knowledge graphs, and (ii) Use of progressively larger transformer language models.
Front Contribution instead of Back Propagation
We show that our algorithm produces the exact same output as BP, in contrast to several recently proposed algorithms approximating BP.
How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation
Models that top leaderboards often perform unsatisfactorily when deployed in real world applications; this has necessitated rigorous and expensive pre-deployment model testing.
Constructing Flow Graphs from Procedural Cybersecurity Texts
We must read the whole text to identify the relevant information or identify the instruction flows to complete a task, which is prone to failures.
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones.
Towards Improving Selective Prediction Ability of NLP Systems
In (IID, OOD) settings, we show that the representations learned by our calibrator result in an improvement of (15. 81%, 5. 64%) and (6. 19%, 13. 9%) over 'MaxProb' -- a selective prediction baseline -- on NLI and DD tasks respectively.
DQI: A Guide to Benchmark Evaluation
A `state of the art' model A surpasses humans in a benchmark B, but fails on similar benchmarks C, D, and E. What does B have that the other benchmarks do not?
Our Evaluation Metric Needs an Update to Encourage Generalization
In order to stop the inflation in model performance -- and thus overestimation in AI systems' capabilities -- we propose a simple and novel evaluation metric, WOOD Score, that encourages generalization during evaluation.
Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks
However, there exists a strong need for a benchmark which can evaluate the abilities of models, in performing question format independent numerical reasoning, as (i) the numerical reasoning capabilities we want to teach are not controlled by question formats, (ii) for numerical reasoning technology to have the best possible application, it must be able to process language and reason in a way that is not exclusive to a single format, task, dataset or domain.
DQI: Measuring Data Quality in NLP
The data creation paradigm consists of several data visualizations to help data creators (i) understand the quality of data and (ii) visualize the impact of the created data instance on the overall quality.
How Additional Knowledge can Improve Natural Language Commonsense Question Answering?
Recently several datasets have been proposed to encourage research in Question Answering domains where commonsense knowledge is expected to play an important role.
