Search Results for author:
Found 28 papers, 18 papers with code
FinMath: Injecting a Tree-structured Solver for Question Answering over Financial Reports
This paper proposes a new framework named FinMath, which improves the model’s numerical reasoning capacity by injecting a tree-structured neural model to perform multi-step numerical reasoning.
Evaluating LLMs at Detecting Errors in LLM Responses
This work introduces ReaLMistake, the first error detection benchmark consisting of objective, realistic, and diverse errors made by LLMs.
MIMIR: A Streamlined Platform for Personalized Agent Tuning in Domain Expertise
Recently, large language models (LLMs) have evolved into interactive agents, proficient in planning, tool use, and task execution across a wide variety of tasks.
Prioritizing Safeguarding Over Autonomy: Risks of LLM Agents for Science
Intelligent agents powered by large language models (LLMs) have demonstrated substantial promise in autonomously conducting experiments and facilitating scientific discoveries across various disciplines.
Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models
In the face of uncertainty, the ability to seek information is of fundamental importance.
ML-Bench: Evaluating Large Language Models for Code Generation in Repository-Level Machine Learning Tasks
While Large Language Models (LLMs) have demonstrated proficiency in code generation benchmarks, translating these results into practical development scenarios - where leveraging existing repository-level libraries is the norm - remains challenging.
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Large language models (LLMs), despite their remarkable progress across various general domains, encounter significant barriers in medicine and healthcare.
DocMath-Eval: Evaluating Numerical Reasoning Capabilities of LLMs in Understanding Long Documents with Tabular Data
This paper introduces DocMath-Eval, a comprehensive benchmark specifically designed to evaluate the numerical reasoning and problem-solving capabilities of LLMs in the context of understanding and analyzing financial documents containing both text and tables.
On Evaluating the Integration of Reasoning and Action in LLM Agents with Database Question Answering
A key discovery is the identification of two primary bottlenecks hindering effective interaction: the capacity for planning and the ability to generate multiple SQL queries.
KnowledgeMath: Knowledge-Intensive Math Word Problem Solving in Finance Domains
We introduce KnowledgeMath, a novel benchmark designed to evaluate LLMs' capabilities in applying financial knowledge to solve complex math word problems.
Investigating Data Contamination in Modern Benchmarks for Large Language Models
Recent observations have underscored a disparity between the inflated benchmark scores and the actual performance of LLMs, raising concerns about potential contamination of evaluation benchmarks.
Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization
Our study reveals that instruction controllable text summarization remains a challenging task for LLMs, since (1) all LLMs evaluated still make factual and other types of errors in their summaries; (2) all LLM-based evaluation methods cannot achieve a strong alignment with human annotators when judging the quality of candidate summaries; (3) different LLMs show large performance gaps in summary generation and evaluation.
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models
Recently, large language models (LLMs), especially those that are pretrained on code, have demonstrated strong capabilities in generating programs from natural language inputs in a few-shot or even zero-shot manner.
Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?
Despite the remarkable capabilities of Large Language Models (LLMs) like GPT-4, producing complex, structured tabular data remains challenging.
ODSum: New Benchmarks for Open Domain Multi-Document Summarization
Open-domain Multi-Document Summarization (ODMDS) is a critical tool for condensing vast arrays of documents into coherent, concise summaries.
RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations
Despite significant progress having been made in question answering on tabular data (Table QA), it's unclear whether, and to what extent existing Table QA models are robust to task-specific perturbations, e. g., replacing key question entities or shuffling table columns.
Investigating Table-to-Text Generation Capabilities of LLMs in Real-World Information Seeking Scenarios
These include the LogicNLG and our newly-constructed LoTNLG datasets for data insight generation, along with the FeTaQA and our newly-constructed F2WTQ datasets for query-based generation.
QTSumm: Query-Focused Summarization over Tabular Data
Motivated by this, we define a new query-focused table summarization task, where text generation models have to perform human-like reasoning and analysis over the given table to generate a tailored summary.
Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies
In-context learning (ICL) has emerged as a new approach to various natural language processing tasks, utilizing large language models (LLMs) to make predictions based on context that has been supplemented with a few examples or task-specific instructions.
Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation
Interpretability and efficiency are two important considerations for the adoption of neural automatic metrics.
LoFT: Enhancing Faithfulness and Diversity for Table-to-Text Generation via Logic Form Control
Logical Table-to-Text (LT2T) generation is tasked with generating logically faithful sentences from tables.
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation
Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
Reasoning over tabular data requires both table structure understanding and a broad set of table reasoning skills.
Ranked #3 on
Semantic Parsing
on WikiSQL
(Denotation accuracy (test) metric)
FOLIO: Natural Language Reasoning with First-Order Logic
We present FOLIO, a human-annotated, open-domain, and logically complex and diverse dataset for reasoning in natural language (NL), equipped with first order logic (FOL) annotations.
MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data
Numerical reasoning over hybrid data containing both textual and tabular content (e. g., financial reports) has recently attracted much attention in the NLP community.
R2D2: Robust Data-to-Text with Replacement Detection
Unfaithful text generation is a common problem for text generation systems.
Apparel-invariant Feature Learning for Apparel-changed Person Re-identification
Therefore, it is critical to learn an apparel-invariant person representation under cases like cloth changing or several persons wearing similar clothes.
MusiCoder: A Universal Music-Acoustic Encoder Based on Transformers
The performance of MusiCoder is evaluated in two downstream music annotation tasks.
