Search Results for author:
Found 23 papers, 15 papers with code
A Survey of Small Language Models
We propose a novel taxonomy for categorizing the methods used to optimize SLMs, including model compression, pruning, and quantization techniques.
Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark
The best scores achieved on PolyMATH are ~41%, ~36%, and ~27%, obtained by Claude-3. 5 Sonnet, GPT-4o and Gemini-1. 5 Pro respectively - highlighting the logical and visual complexity of these questions.
Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter?
Then, we develop an LLM-based framework for large-scale subjective evaluation (i. e., identifying errors) and an objective metric, PuzzleEval, to evaluate the correctness of reasoning chains.
Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs
Motivated by this, we propose a systematically created human-annotated dataset consisting of coherent summaries for five publicly available datasets and natural language user feedback, offering valuable insights into how to improve coherence in extractive summaries.
Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models
Experimental results show that there is a significant drop in the performance of LLMs as the reasoning steps/depth increases (average accuracy of ~68% at depth-1 to ~43% at depth-5).
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
Existing work investigating this reasoning ability of LLMs has focused only on a couple of inference rules (such as modus ponens and modus tollens) of propositional and first-order logic.
LongBoX: Evaluating Transformers on Long-Sequence Clinical Tasks
Assessing these models on long sequences is crucial since prior work in the general domain has demonstrated performance degradation of LLMs on longer texts.
Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE
Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks; however, their large size makes their inference slow and computationally expensive.
TarGEN: Targeted Data Generation with Large Language Models
Finally, when pre-finetuned on our synthetic SuperGLUE dataset, T5-3B yields impressive results on the OpenLLM leaderboard, surpassing the model trained on the Self-Instruct dataset by 4. 14% points.
Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models
This work strives to understand the proficiency of LLMs in logical reasoning by offering a brief review of the latest progress in this area; with a focus on the logical reasoning datasets, tasks, and the methods adopted to utilize LLMs for reasoning.
Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?
Though state-of-the-art (SOTA) NLP systems have achieved remarkable performance on a variety of language understanding tasks, they primarily focus on questions that have a correct and a definitive answer.
MDDial: A Multi-turn Differential Diagnosis Dialogue Dataset with Reliability Evaluation
We introduce a unified score for the ADD system that takes into account the interplay between symptoms and diagnosis.
EDM3: Event Detection as Multi-task Text Generation
We show that EDM3 helps to learn transferable knowledge that can be leveraged to perform Event Detection and its subtasks concurrently, mitigating the error propagation inherent in pipelined approaches.
Can NLP Models Correctly Reason Over Contexts that Break the Common Assumptions?
Can state-of-the-art NLP models correctly reason over the contexts of such scenarios?
BioTABQA: Instruction Learning for Biomedical Table Question Answering
To the best of our knowledge, none of TQA datasets exist in the biomedical domain where tables are frequently used to present information.
Is a Question Decomposition Unit All We Need?
Large Language Models (LMs) have achieved state-of-the-art performance on many Natural Language Processing (NLP) benchmarks.
Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions
In this work, we hypothesize that annotators pick up on patterns in the crowdsourcing instructions, which bias them to write many similar examples that are then over-represented in the collected data.
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning
Recently, instructional prompts have shown significant improvement towards multi-task generalization; however, the effect of instructional prompts and Multi-Task Learning (MTL) has not been systematically studied in the biomedical domain.
How Many Data Samples is an Additional Instruction Worth?
However, they can write alternate instructions to represent an instruction task.
Less is More: Summary of Long Instructions is Better for Program Synthesis
We show that LMs benefit from the summarized version of complicated questions.
COVID-19: Comparative Analysis of Methods for Identifying Articles Related to Therapeutics and Vaccines without Using Labeled Data
Here we proposed an approach to analyze text classification methods based on the presence or absence of task-specific terms (and their synonyms) in the text.
AdaGAN: Adaptive GAN for Many-to-Many Non-Parallel Voice Conversion
In this paper, we propose a novel style transfer architecture, which can also be extended to generate voices even for target speakers whose data were not used in the training (i. e., case of zero-shot learning).
