Search Results for author:
Found 31 papers, 6 papers with code
Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models
Experimental results show that there is a significant drop in the performance of LLMs as the reasoning steps/depth increases (average accuracy of ~68% at depth-1 to ~43% at depth-5).
Efficient Transmission Scheme for LEO Satellite-Based NB-IoT: A Data-Driven Perspective
This study analyses the medium access control (MAC) layer aspects of a low-Earth-orbit (LEO) satellite-based Internet of Things (IoT) network.
Investigating and Addressing Hallucinations of LLMs in Tasks Involving Negation
Recent research has focused on investigating and addressing this problem for a variety of tasks such as biography generation, question answering, abstractive summarization, and dialogue generation.
Chaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategies
This study explores the sycophantic tendencies of Large Language Models (LLMs), where these models tend to provide answers that match what users want to hear, even if they are not entirely correct.
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
Existing work investigating this reasoning ability of LLMs has focused only on a couple of inference rules (such as modus ponens and modus tollens) of propositional and first-order logic.
The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness
As Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications, their safety concerns become critical areas of NLP research.
Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE
Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks; however, their large size makes their inference slow and computationally expensive.
Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models
This work strives to understand the proficiency of LLMs in logical reasoning by offering a brief review of the latest progress in this area; with a focus on the logical reasoning datasets, tasks, and the methods adopted to utilize LLMs for reasoning.
Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?
Though state-of-the-art (SOTA) NLP systems have achieved remarkable performance on a variety of language understanding tasks, they primarily focus on questions that have a correct and a definitive answer.
A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation
Specifically, the detection technique achieves a recall of ~88% and the mitigation technique successfully mitigates 57. 6% of the correctly detected hallucinations.
Can NLP Models Correctly Reason Over Contexts that Break the Common Assumptions?
Can state-of-the-art NLP models correctly reason over the contexts of such scenarios?
A Unified Evaluation Framework for Novelty Detection and Accommodation in NLP with an Instantiation in Authorship Attribution
To initiate a systematic research in this important area of 'dealing with novelties', we introduce 'NoveltyTask', a multi-stage task to evaluate a system's performance on pipelined novelty 'detection' and 'accommodation' tasks.
Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA
Despite remarkable progress made in natural language processing, even the state-of-the-art models often make incorrect predictions.
Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environments
Learning to detect, characterize and accommodate novelties is a challenge that agents operating in open-world domains need to address to be able to guarantee satisfactory task performance.
Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?
Comparing with the FiD reader, this approach matches its accuracy by utilizing just 18. 32% of its reader inference cost and also outperforms it by achieving up to 55. 10% accuracy on NQ Open.
Performance Analysis of LEO Satellite-Based IoT Networks in the Presence of Interference
The paper also presents simplified expressions for the OP under a high signal-to-noise ratio (SNR) assumption, which are utilized to optimize the system parameters for achieving a target OP.
"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibility
We show that even state-of-the-art models such as GPT-3, GPT-2, and T5 struggle to answer the feasibility questions correctly.
Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems
Through comprehensive experiments in multiple task settings that differ in the number of models available for cascading (K value), we show that cascading improves both the computational efficiency and the prediction accuracy.
Let the Model Decide its Curriculum for Multitask Learning
Curriculum learning strategies in prior multi-task learning approaches arrange datasets in a difficulty hierarchy either based on human perception or by exhaustively searching the optimal arrangement.
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems.
ILDAE: Instance-Level Difficulty Analysis of Evaluation Data
Knowledge of questions' difficulty level helps a teacher in several ways, such as estimating students' potential quickly by asking carefully selected questions and improving quality of examination by modifying trivial and hard questions.
Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings
In order to equip NLP systems with selective prediction capability, several task-specific approaches have been proposed.
Unsupervised Natural Language Inference Using PHL Triplet Generation
Transformer-based models achieve impressive performance on numerous Natural Language Inference (NLI) benchmarks when trained on respective training datasets.
Hybrid Transceiver Design for Tera-Hertz MIMO Systems Relying on Bayesian Learning Aided Sparse Channel Estimation
Hybrid transceiver design in multiple-input multiple-output (MIMO) Tera-Hertz (THz) systems relying on sparse channel state information (CSI) estimation techniques is conceived.
Interviewer-Candidate Role Play: Towards Developing Real-World NLP Systems
However, our task leaves a significant challenge for NLP researchers to further improve OOD performance at each stage.
On the Performance of the Primary and Secondary Links in a 3-D Underlay Cognitive Molecular Communication
In this work, we consider a system in three-dimensional (3-D) space with two coexisting communication links, each between a point transmitter and fully-absorbing spherical receiver (FAR), where the one link (termed primary) has priority over the second link (termed secondary).
Information Theory Information Theory
Can Transformers Reason About Effects of Actions?
A recent work has shown that transformers are able to "reason" with facts and rules in a limited setting where the rules are natural language expressions of conjunctions of conditions implying a conclusion.
Towards Improving Selective Prediction Ability of NLP Systems
In (IID, OOD) settings, we show that the representations learned by our calibrator result in an improvement of (15. 81%, 5. 64%) and (6. 19%, 13. 9%) over 'MaxProb' -- a selective prediction baseline -- on NLI and DD tasks respectively.
Beamformed Energy Detection in the Presence of an Interferer for Cognitive mmWave Network
To demonstrate the bound on the system performance, the proposed sensing scheme is designed under the knowledge of full channel state information (CSI) at the SU for the PU-SU and Interferer-SU channels.
Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks
However, there exists a strong need for a benchmark which can evaluate the abilities of models, in performing question format independent numerical reasoning, as (i) the numerical reasoning capabilities we want to teach are not controlled by question formats, (ii) for numerical reasoning technology to have the best possible application, it must be able to process language and reason in a way that is not exclusive to a single format, task, dataset or domain.
