Search Results for author:
Found 54 papers, 35 papers with code
Overview of the Third Workshop on Scholarly Document Processing
With the ever-increasing pace of research and high volume of scholarly communication, scholars face a daunting task.
Overview of the Second Workshop on Scholarly Document Processing
With the ever-increasing pace of research and high volume of scholarly communication, scholars face a daunting task.
FABLES: Evaluating faithfulness and content selection in book-length summarization
While LLM-based auto-raters have proven reliable for factuality and coherence in other settings, we implement several LLM raters of faithfulness and find that none correlates strongly with human annotations, especially with regard to detecting unfaithful claims.
FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions
We introduce our dataset FollowIR, which contains a rigorous instruction evaluation benchmark as well as a training set for helping IR models learn to better follow real-world instructions.
KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions
To evaluate the capabilities of current LLMs on this task, we construct KIWI, a dataset of knowledge-intensive writing instructions in the scientific domain.
OLMo: Accelerating the Science of Language Models
Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs.
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Language models have become a critical technology to tackling a wide range of natural language processing tasks, yet many details about how the best-performing language models were developed are not reported.
InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification
Text simplification aims to make technical texts more accessible to laypeople but often results in deletion of information and vagueness.
Paloma: A Benchmark for Evaluating Language Model Fit
We invite submissions to our benchmark and organize results by comparability based on compliance with guidelines such as removal of benchmark contamination from pretraining.
Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders
Prevailing research practice today often relies on training dense retrievers on existing large datasets such as MSMARCO and then experimenting with ways to improve zero-shot generalization capabilities to unseen domains.
The Rise of Open Science: Tracking the Evolution and Perceived Value of Data and Methods Link-Sharing Practices
In recent years, funding agencies and journals increasingly advocate for open science practices (e. g. data and method sharing) to improve the transparency, access, and reproducibility of science.
BooookScore: A systematic exploration of book-length summarization in the era of LLMs
We find that closed-source LLMs such as GPT-4 and Claude 2 produce summaries with higher BooookScore than those generated by open-source models.
When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets
Using large language models (LMs) for query or document expansion can improve generalization in information retrieval.
Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation
In response, we introduce Pentathlon, a benchmark for holistic and realistic evaluation of model efficiency.
Are Layout-Infused Language Models Robust to Layout Distribution Shifts? A Case Study with Scientific Documents
Recent work has shown that infusing layout features into language models (LMs) improves processing of visually-rich documents such as scientific papers.
A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents
Many real-world applications (e. g., note taking, search) require extracting a sentence or paragraph from a document and showing that snippet to a human outside of the source document.
Decomposing Complex Queries for Tip-of-the-tongue Retrieval
When re-finding items, users who forget or are uncertain about identifying details often rely on creative strategies for expressing their information needs -- complex queries that describe content elements (e. g., book characters or events), information beyond the document text (e. g., descriptions of book covers), or personal context (e. g., when they read a book).
Complex Mathematical Symbol Definition Structures: A Dataset and Model for Coordination Resolution in Definition Extraction
We also introduce a new definition extraction method that masks mathematical symbols, creates a copy of each sentence for each symbol, specifies a target symbol, and predicts its corresponding definition spans using slot filling.
Beyond Summarization: Designing AI Support for Real-World Expository Writing Tasks
In this position paper, we argue that developing AI supports for expository writing has unique and exciting research challenges and can lead to high real-world impacts.
The Semantic Reader Project: Augmenting Scholarly Documents through AI-Powered Interactive Reading Interfaces
Scholarly publications are key to the transfer of knowledge from scholars to others.
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings.
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Motivated by our survey, we present LongEval, a set of guidelines for human evaluation of faithfulness in long-form summaries that addresses the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores?
The Semantic Scholar Open Data Platform
The volume of scientific output is creating an urgent need for automated tools to help scientists keep up with developments in their field.
Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval
Via extensive automatic and human evaluation, we determine: (1) state-of-the-art summarizers suffer large reductions in performance when applied to open-domain MDS, (2) additional training in the open-domain setting can reduce this sensitivity to imperfect retrieval, and (3) summarizers are insensitive to the retrieval of duplicate documents and the order of retrieved documents, but highly sensitive to other errors, like the retrieval of irrelevant documents.
Ranked #1 on
Multi-Document Summarization
on MS^2
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
SciFact-Open: Towards open-domain scientific claim verification
While research on scientific claim verification has led to the development of powerful systems that appear to approach human performance, these approaches have yet to be tested in a realistic setting against large corpora of scientific literature.
Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities
With the advent of large language models, methods for abstractive summarization have made great strides, creating potential for use in applications to aid knowledge workers processing unwieldy document collections.
ACCoRD: A Multi-Document Approach to Generating Diverse Descriptions of Scientific Concepts
We present ACCoRD, an end-to-end system tackling the novel task of generating sets of descriptions of scientific concepts.
Data Governance in the Age of Large-Scale Data-Driven Language Technology
The recent emergence and adoption of Machine Learning technology, and specifically of Large Language Models, has drawn attention to the need for systematic and transparent management of language data.
Generating Scientific Claims for Zero-Shot Scientific Fact Checking
To address this challenge, we propose scientific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences, and demonstrate its usefulness in zero-shot fact checking for biomedical claims.
Paper Plain: Making Medical Research Papers Approachable to Healthcare Consumers with Natural Language Processing
When seeking information not covered in patient-friendly documents, like medical pamphlets, healthcare consumers may turn to the research literature.
MultiVerS: Improving scientific claim verification with weak supervision and full-document context
Our approach outperforms two competitive baselines on three scientific claim verification datasets, with particularly strong performance in zero / few-shot domain adaptation experiments.
Exploring The Role of Local and Global Explanations in Recommender Systems
However, we note that the two explanation approaches may be better compared in the context of a higher-stakes or more opaque domain.
Overview and Insights from the SciVer Shared Task on Scientific Claim Verification
We present an overview of the SciVer shared task, presented at the 2nd Scholarly Document Processing (SDP) workshop at NAACL 2021.
FLEX: Unifying Evaluation for Few-Shot NLP
Few-shot NLP research is highly active, yet conducted in disjoint research threads with evaluation suites that lack challenging-yet-realistic testing setups and fail to employ careful experimental design.
MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting
In our work, we address this research gap by proposing a novel framework for CCA as a document-level context extraction and labeling task.
VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups
Experiments are conducted on a newly curated evaluation suite, S2-VLUE, that unifies existing automatically-labeled datasets and includes a new dataset of manual annotations covering diverse papers from 19 scientific disciplines.
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Readers of academic research papers often read with the goal of answering specific questions.
Ranked #1 on
Evidence Selection
on QASPER
Searching for Scientific Evidence in a Pandemic: An Overview of TREC-COVID
We present an overview of the TREC-COVID Challenge, an information retrieval (IR) shared task to evaluate search on scientific literature related to COVID-19.
Document-Level Definition Detection in Scholarly Documents: Existing Models, Error Analyses, and Future Directions
Based on this analysis, we develop a new definition detection system, HEDDEx, that utilizes syntactic features, transformer encoders, and heuristic filters, and evaluate it on a standard sentence-level benchmark.
Augmenting Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols
We introduce ScholarPhi, an augmented reading interface with four novel features: (1) tooltips that surface position-sensitive definitions from elsewhere in a paper, (2) a filter over the paper that "declutters" it to reveal how the term or symbol is used across the paper, (3) automatic equation diagrams that expose multiple definitions in parallel, and (4) an automatically generated glossary of important terms and symbols.
TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection
TREC-COVID is a community evaluation designed to build a test collection that captures the information needs of biomedical researchers using the scientific literature during a pandemic.
TLDR: Extreme Summarization of Scientific Documents
We introduce TLDR generation, a new form of extreme summarization, for scientific papers.
Fact or Fiction: Verifying Scientific Claims
We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that SUPPORTS or REFUTES a given scientific claim, and to identify rationales justifying each decision.
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
Language models pretrained on text from a wide variety of sources form the foundation of today's NLP.
CORD-19: The COVID-19 Open Research Dataset
The COVID-19 Open Research Dataset (CORD-19) is a growing resource of scientific papers on COVID-19 and related historical coronavirus research.
LIMEADE: From AI Explanations to Advice Taking
We show our method improves accuracy compared to a rigorous baseline on the image classification domains.
Explaining Relationships Between Scientific Documents
We address the task of explaining relationships between two scientific documents using natural language text.
S2ORC: The Semantic Scholar Open Research Corpus
We introduce S2ORC, a large corpus of 81. 1M English-language academic papers spanning many academic disciplines.
Discourse Understanding and Factual Consistency in Abstractive Summarization
We introduce a general framework for abstractive summarization with factual consistency and distinct modeling of the narrative flow in an output summary.
SciBERT: A Pretrained Language Model for Scientific Text
Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive.
Ranked #1 on
Sentence Classification
on Paper Field
(using extra training data)
Combining Distant and Direct Supervision for Neural Relation Extraction
In relation extraction with distant supervision, noisy labels make it difficult to train quality models.
Ontology Alignment in the Biomedical Domain Using Entity Definitions and Context
Ontology alignment is the task of identifying semantically equivalent entities from two given ontologies.
Construction of the Literature Graph in Semantic Scholar
We describe a deployed scalable system for organizing published scientific literature into a heterogeneous graph to facilitate algorithmic manipulation and discovery.
