Search Results for author:
Found 108 papers, 74 papers with code
Overview of the Second Workshop on Scholarly Document Processing
With the ever-increasing pace of research and high volume of scholarly communication, scholars face a daunting task.
GUIR @ LongSumm 2020: Learning to Generate Long Summaries from Scientific Documents
A two stage model that additionally includes an abstraction step using BART; and 3.
Zero- and Few-Shot NLP with Pretrained Language Models
The ability to efficiently learn from little-to-no data is critical to applying NLP to tasks where data collection is costly or otherwise difficult.
Overview of the Third Workshop on Scholarly Document Processing
With the ever-increasing pace of research and high volume of scholarly communication, scholars face a daunting task.
Overview of the First Shared Task on Multi Perspective Scientific Document Summarization (MuP)
We present the main findings of MuP 2022 shared task, the first shared task on multi-perspective scientific document summarization.
MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
We introduce MMVU, a comprehensive expert-level, multi-discipline benchmark for evaluating foundation models in video understanding.
ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning
To address these challenges, we present ChemAgent, a novel framework designed to improve the performance of LLMs through a dynamic, self-updating library.
Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference
Due to the high cost and time-consuming nature of human evaluations, an automatic LLM bencher (i. e., an automatic evaluation framework that aims to rank LLMs based on their alignment with human preferences) is indispensable.
HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation
First, we propose a general recipe for generating more challenging versions of existing benchmarks, resulting in three new benchmarks: HumanEval Pro, MBPP Pro, and BigCodeBench-Lite Pro, specifically designed to assess LLMs on self-invoking code generation.
ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain
To address this issue in the field of chemistry, we introduce ChemSafetyBench, a benchmark designed to evaluate the accuracy and safety of LLM responses.
FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents
We introduce FinDVer, a comprehensive benchmark specifically designed to evaluate the explainable claim verification capabilities of LLMs in the context of understanding and analyzing long, hybrid-content financial documents.
SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers
We introduce SciDQA, a new dataset for reading comprehension that challenges LLMs for a deep understanding of scientific articles, consisting of 2, 937 QA pairs.
Bayesian Calibration of Win Rate Estimation with LLM Evaluators
Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for assessing the quality of text generations from LLMs.
M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models
Existing benchmarks for evaluating foundation models mainly focus on single-document, text-only tasks.
MDCure: A Scalable Pipeline for Multi-Document Instruction-Following
Multi-document (MD) processing is crucial for LLMs to handle real-world tasks such as summarization and question-answering across large sets of documents.
COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
To achieve robust alignment with general preferences, we model the alignment problem as a two-player zero-sum game, where the Nash equilibrium policy guarantees a 50% win rate against any competing policy.
TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models
Our study of existing benchmarks shows that this capability of MFMs is likely overestimated as many questions can be solved by using a single, few, or out-of-order frames.
P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains
We show that human-written reasoning chains significantly boost the logical reasoning capabilities of LLMs via many-shot prompting and fine-tuning.
ReIFE: Re-evaluating Instruction-Following Evaluation
The automatic evaluation of instruction following typically involves using large language models (LLMs) to assess response quality.
INC-Math: Integrating Natural Language and Code for Enhanced Mathematical Reasoning in Large Language Models
Large Language Models (LLMs) are commonly used to generate solutions for mathematical reasoning problems in the following formats: natural language, code, or a combination of both.
RouterRetriever: Exploring the Benefits of Routing over Multiple Expert Embedding Models
To our knowledge, RouterRetriever is the first work to demonstrate the advantages of using multiple domain-specific expert embedding models with effective routing over a single, general-purpose embedding model in retrieval tasks.
Understanding Reference Policies in Direct Preference Optimization
In this work, we explore an under-investigated aspect of DPO - its dependency on the reference model or policy.
Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation
This survey serves as a succinct overview of the most recent advancements in data contamination research, providing a straightforward guide for the benefit of future research endeavors.
Step-Back Profiling: Distilling User History for Personalized Scientific Writing
To conduct the experiments, we construct a Personalized Scientific Writing (PSW) dataset to study multi-user personalization.
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
We present SciRIFF (Scientific Resource for Instruction-Following and Finetuning), a dataset of 137K instruction-following demonstrations for 54 tasks covering five essential scientific literature understanding capabilities: information extraction, summarization, question answering, claim verification, and classification.
NExT: Teaching Large Language Models to Reason about Code Execution
A fundamental skill among human developers is the ability to understand and reason about program execution.
Evaluating LLMs at Detecting Errors in LLM Responses
This work introduces ReaLMistake, the first error detection benchmark consisting of objective, realistic, and diverse errors made by LLMs.
MIMIR: A Streamlined Platform for Personalized Agent Tuning in Domain Expertise
Recently, large language models (LLMs) have evolved into interactive agents, proficient in planning, tool use, and task execution across a wide variety of tasks.
FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions
First, we introduce our dataset FollowIR, which contains a rigorous instruction evaluation benchmark as well as a training set for helping IR models learn to better follow real-world instructions.
On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization
We study the behavior of the underlying losses between factual and non-factual examples, to understand and refine the performance of LT. We demonstrate that LT's performance is limited when the underlying assumption that noisy targets have higher NLL loss is not satisfied, and find that word-level NLL among entities provides better signal for distinguishing factuality.
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models
While large language models have achieved remarkable performance on various code generation benchmarks, there have been growing concerns regarding potential contamination of these benchmarks as they may be leaked into pretraining and finetuning data.
Calibrating Long-form Generations from Large Language Models
Addressing this gap, we introduce a unified calibration framework, in which both the correctness of the LLMs' responses and their associated confidence levels are treated as distributions across a range of scores.
Prioritizing Safeguarding Over Autonomy: Risks of LLM Agents for Science
Intelligent agents powered by large language models (LLMs) have demonstrated substantial promise in autonomously conducting experiments and facilitating scientific discoveries across various disciplines.
OLMo: Accelerating the Science of Language Models
Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs.
Observable Propagation: Uncovering Feature Vectors in Transformers
A key goal of current mechanistic interpretability research in NLP is to find linear features (also called "feature vectors") for transformers: directions in activation space corresponding to concepts that are used by a given model in its computation.
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Large language models (LLMs), despite their remarkable progress across various general domains, encounter significant barriers in medicine and healthcare.
DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents
Recent LLMs have demonstrated remarkable performance in solving exam-like math word problems.
ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code
Despite Large Language Models (LLMs) like GPT-4 achieving impressive results in function-level code generation, they struggle with repository-scale code understanding (e. g., coming up with the right arguments for calling routines), requiring a deeper comprehension of complex file interactions.
On Evaluating the Integration of Reasoning and Action in LLM Agents with Database Question Answering
A key discovery is the identification of two primary bottlenecks hindering effective interaction: the capacity for planning and the ability to generate multiple SQL queries.
FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains
Finally, we evaluate a wide spectrum of 44 LLMs with both Chain-of-Thought and Program-of-Thought prompting methods.
Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders
Prevailing research practice today often relies on training dense retrievers on existing large datasets such as MSMARCO and then experimenting with ways to improve zero-shot generalization capabilities to unseen domains.
Investigating Data Contamination in Modern Benchmarks for Large Language Models
Recent observations have underscored a disparity between the inflated benchmark scores and the actual performance of LLMs, raising concerns about potential contamination of evaluation benchmarks.
Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization
Our study reveals that instruction controllable text summarization remains a challenging task for LLMs, since (1) all LLMs evaluated still make factual and other types of errors in their summaries; (2) no LLM-based evaluation methods can achieve a strong alignment with human annotators when judging the quality of candidate summaries; (3) different LLMs show large performance gaps in summary generation and evaluation capabilities.
Medical Text Simplification: Optimizing for Readability with Unlikelihood Training and Reranked Beam Search Decoding
Text simplification has emerged as an increasingly useful application of AI for bridging the communication gap in specialized fields such as medicine, where the lexicon is often dominated by technical jargon and complex constructs.
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models
Recently, large language models (LLMs), especially those that are pretrained on code, have demonstrated strong capabilities in generating programs from natural language inputs in a few-shot or even zero-shot manner.
ODSum: New Benchmarks for Open Domain Multi-Document Summarization
Open-domain Multi-Document Summarization (ODMDS) is a critical tool for condensing vast arrays of documents into coherent, concise summaries.
Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?
Despite the remarkable capabilities of Large Language Models (LLMs) like GPT-4, producing complex, structured tabular data remains challenging.
When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets
Using large language models (LMs) for query or document expansion can improve generalization in information retrieval.
Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering
The integration of multi-document pre-training objectives into language models has resulted in remarkable improvements in multi-document downstream tasks.
Investigating Table-to-Text Generation Capabilities of LLMs in Real-World Information Seeking Scenarios
These include the LogicNLG and our newly-constructed LoTNLG datasets for data insight generation, along with the FeTaQA and our newly-constructed F2WTQ datasets for query-based generation.
A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents
Many real-world applications (e. g., note taking, search) require extracting a sentence or paragraph from a document and showing that snippet to a human outside of the source document.
On Learning to Summarize with Large Language Models as References
Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets.
QTSumm: Query-Focused Summarization over Tabular Data
Motivated by this, we define a new query-focused table summarization task, where text generation models have to perform human-like reasoning and analysis over the given table to generate a tailored summary.
Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies
In-context learning (ICL) has emerged as a new approach to various natural language processing tasks, utilizing large language models (LLMs) to make predictions based on context that has been supplemented with a few examples or task-specific instructions.
ReFIT: Relevance Feedback from a Reranker during Inference
Retrieve-and-rerank is a prevalent framework in neural information retrieval, wherein a bi-encoder network initially retrieves a pre-defined number of candidates (e. g., K=100), which are then reranked by a more powerful cross-encoder model.
TESS: Text-to-Text Self-Conditioned Simplex Diffusion
Diffusion models have emerged as a powerful paradigm for generation, obtaining strong performance in various continuous domains.
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Motivated by our survey, we present LongEval, a set of guidelines for human evaluation of faithfulness in long-form summaries that addresses the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores?
The Semantic Scholar Open Data Platform
The volume of scientific output is creating an urgent need for automated tools to help scientists keep up with developments in their field.
Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval
Via extensive automatic and human evaluation, we determine: (1) state-of-the-art summarizers suffer large reductions in performance when applied to open-domain MDS, (2) additional training in the open-domain setting can reduce this sensitivity to imperfect retrieval, and (3) summarizers are insensitive to the retrieval of duplicate documents and the order of retrieved documents, but highly sensitive to other errors, like the retrieval of irrelevant documents.
Ranked #1 on
Multi-Document Summarization
on MS^2
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
In response, we introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations.
SciFact-Open: Towards open-domain scientific claim verification
While research on scientific claim verification has led to the development of powerful systems that appear to approach human performance, these approaches have yet to be tested in a realistic setting against large corpora of scientific literature.
FOLIO: Natural Language Reasoning with First-Order Logic
We present FOLIO, a human-annotated, logically complex and diverse dataset for reasoning in natural language (NL), equipped with first-order logic (FOL) annotations.
Embedding Recycling for Language Models
Real-world applications of neural language models often involve running many different models over the same corpus.
Improving the Generalizability of Depression Detection by Leveraging Clinical Questionnaires
In dataset-transfer experiments on three social media datasets, we find that grounding the model in PHQ9's symptoms substantially improves its ability to generalize to out-of-distribution data compared to a standard BERT-based approach.
Generating Scientific Claims for Zero-Shot Scientific Fact Checking
To address this challenge, we propose scientific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences, and demonstrate its usefulness in zero-shot fact checking for biomedical claims.
Long Context Question Answering via Supervised Contrastive Learning
Long-context question answering (QA) tasks require reasoning over a long document or multiple documents.
MultiVerS: Improving scientific claim verification with weak supervision and full-document context
Our approach outperforms two competitive baselines on three scientific claim verification datasets, with particularly strong performance in zero / few-shot domain adaptation experiments.
Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity
We present a new scientific document similarity model based on matching fine-grained aspects of texts.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We introduce PRIMERA, a pre-trained model for multi-document representation with a focus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data.
Ranked #1 on
Multi-Document Summarization
on Multi-News
FLEX: Unifying Evaluation for Few-Shot NLP
Few-shot NLP research is highly active, yet conducted in disjoint research threads with evaluation suites that lack challenging-yet-realistic testing setups and fail to employ careful experimental design.
MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting
In our work, we address this research gap by proposing a novel framework for CCA as a document-level context extraction and labeling task.
Beyond Paragraphs: NLP for Long Sequences
In this tutorial, we aim at bringing interested NLP researchers up to speed about the recent and ongoing techniques for document-level representation learning.
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Readers of academic research papers often read with the goal of answering specific questions.
Ranked #1 on
Evidence Selection
on QASPER
Simplified Data Wrangling with ir_datasets
Managing the data for Information Retrieval (IR) experiments can be challenging.
CDLM: Cross-Document Language Modeling
We introduce a new pretraining approach geared for multi-document language modeling, incorporating two key ideas into the masked language modeling self-supervised objective.
Ranked #1 on
Citation Recommendation
on AAN test
On Generating Extended Summaries of Long Documents
We then present our results on three long summarization datasets, arXiv-Long, PubMed-Long, and Longsumm.
Ranked #1 on
Extended Summarization
on arXiv-Long Test
ParsiNLU: A Suite of Language Understanding Challenges for Persian
Despite the progress made in recent years in addressing natural language understanding (NLU) challenges, the majority of this progress remains to be concentrated on resource-rich languages like English.
ABNIRML: Analyzing the Behavior of Neural IR Models
Pretrained contextualized language models such as BERT and T5 have established a new state-of-the-art for ad-hoc search.
SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search
With worldwide concerns surrounding the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), there is a rapidly growing body of scientific literature on the virus.
SLEDGE: A Simple Yet Effective Baseline for COVID-19 Scientific Knowledge Search
In this work, we present a search system called SLEDGE, which utilizes SciBERT to effectively re-rank articles.
TLDR: Extreme Summarization of Scientific Documents
We introduce TLDR generation, a new form of extreme summarization, for scientific papers.
Fact or Fiction: Verifying Scientific Claims
We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that SUPPORTS or REFUTES a given scientific claim, and to identify rationales justifying each decision.
SPECTER: Document-level Representation Learning using Citation-informed Transformers
We propose SPECTER, a new method to generate document-level embedding of scientific documents based on pretraining a Transformer language model on a powerful signal of document-level relatedness: the citation graph.
Ranked #1 on
Document Classification
on SciDocs (MAG)
Longformer: The Long-Document Transformer
To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer.
Ranked #2 on
Question Answering
on WikiHop
Ranking Significant Discrepancies in Clinical Reports
This allows medical practitioners to easily identify and learn from the reports in which their interpretation most substantially differed from that of the attending physician (who finalized the report).
SUPP.AI: Finding Evidence for Supplement-Drug Interactions
We fine-tune the contextualized word representations of the RoBERTa language model using labeled DDI data, and apply the fine-tuned model to identify supplement interactions.
Pretrained Language Models for Sequential Sentence Classification
As a step toward better document-level understanding, we explore classification of a sequence of sentences into their corresponding categories, a task that requires understanding sentences in context of the document.
Ontology-Aware Clinical Abstractive Summarization
Automatically generating accurate summaries from clinical reports could save a clinician's time, improve summary coverage, and reduce errors.
CEDR: Contextualized Embeddings for Document Ranking
We call this joint approach CEDR (Contextualized Embeddings for Document Ranking).
Ranked #3 on
Ad-Hoc Information Retrieval
on TREC Robust04
Structural Scaffolds for Citation Intent Classification in Scientific Publications
Identifying the intent of a citation in scientific papers (e. g., background information, use of methods, comparing results) is critical for machine reading of individual publications and automated analysis of the scientific literature.
Ranked #2 on
Sentence Classification
on SciCite
SciBERT: A Pretrained Language Model for Scientific Text
Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive.
Ranked #1 on
Sentence Classification
on Paper Field
(using extra training data)
RSDD-Time: Temporal Annotation of Self-Reported Mental Health Diagnoses
Self-reported diagnosis statements have been widely employed in studying language related to mental health in social media.
SMHD: A Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions
Mental health is a significant and growing public health concern.
Helping or Hurting? Predicting Changes in Users' Risk of Self-Harm Through Online Community Interactions
In recent years, online communities have formed around suicide and self-harm prevention.
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
Neural abstractive summarization models have led to promising results in summarizing relatively short documents.
Ranked #4 on
Unsupervised Extractive Summarization
on Pubmed
GU IRLAB at SemEval-2018 Task 7: Tree-LSTMs for Scientific Relation Classification
SemEval 2018 Task 7 focuses on relation ex- traction and classification in scientific literature.
Depression and Self-Harm Risk Assessment in Online Forums
We propose methods for identifying posts in support communities that may indicate a risk of self-harm, and demonstrate that our approach outperforms strong previously proposed methods for identifying such posts.
Identifying Harm Events in Clinical Care through Medical Narratives
Preventable medical errors are estimated to be among the leading causes of injury and death in the United States.
GUIR at SemEval-2017 Task 12: A Framework for Cross-Domain Clinical Temporal Information Extraction
Clinical TempEval 2017 (SemEval 2017 Task 12) addresses the task of cross-domain temporal extraction from clinical text.
Scientific document summarization via citation contextualization and scientific discourse
We present a framework for scientific summarization which takes advantage of the citations and the scientific discourse structure.
Contextualizing Citations for Scientific Summarization using Word Embeddings and Domain Knowledge
Citation texts are sometimes not very informative or in some cases inaccurate by themselves; they need the appropriate context from the referenced paper to reflect its exact contributions.
Scientific Article Summarization Using Citation-Context and Article's Discourse Structure
We propose a summarization approach for scientific articles which takes advantage of citation-context and the document discourse model.
A Neural Attention Model for Categorizing Patient Safety Events
Medical errors are leading causes of death in the US and as such, prevention of these errors is paramount to promoting health care.
Triaging Content Severity in Online Mental Health Forums
Our analysis on the interaction of the moderators with the users further indicates that without an automatic way to identify critical content, it is indeed challenging for the moderators to provide timely response to the users in need.
Revisiting Summarization Evaluation for Scientific Articles
Finally, we propose an alternative metric for summarization evaluation which is based on the content relevance between a system generated summary and the corresponding human written summaries.
