Search Results for author: Luca Soldaini

Found 39 papers, 14 papers with code

SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature

no code implementations10 Jun 2024 David Wadden, Kejian Shi, Jacob Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, Arman Cohan

We present SciRIFF (Scientific Resource for Instruction-Following and Finetuning), a dataset of 137K instruction-following demonstrations for 54 tasks covering five essential scientific literature understanding capabilities: information extraction, summarization, question answering, claim verification, and classification.

Claim Verification Instruction Following +3

Overview of the TREC 2023 NeuCLIR Track

no code implementations11 Apr 2024 Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, Eugene Yang

The principal tasks are ranked retrieval of news in one of the three languages, using English topics.

Information Retrieval Retrieval

FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions

1 code implementation22 Mar 2024 Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, Luca Soldaini

First, we introduce our dataset FollowIR, which contains a rigorous instruction evaluation benchmark as well as a training set for helping IR models learn to better follow real-world instructions.

Information Retrieval Retrieval +1

KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions

no code implementations6 Mar 2024 Fangyuan Xu, Kyle Lo, Luca Soldaini, Bailey Kuehl, Eunsol Choi, David Wadden

To evaluate the capabilities of current LLMs on this task, we construct KIWI, a dataset of knowledge-intensive writing instructions in the scientific domain.

Instruction Following

Paloma: A Benchmark for Evaluating Language Model Fit

no code implementations16 Dec 2023 Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hannaneh Hajishirzi, Noah A. Smith, Kyle Richardson, Jesse Dodge

We invite submissions to our benchmark and organize results by comparability based on compliance with guidelines such as removal of benchmark contamination from pretraining.

Language Modelling

Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders

1 code implementation16 Nov 2023 Hyunji Lee, Luca Soldaini, Arman Cohan, Minjoon Seo, Kyle Lo

Prevailing research practice today often relies on training dense retrievers on existing large datasets such as MSMARCO and then experimenting with ways to improve zero-shot generalization capabilities to unseen domains.

Data Augmentation Domain Generalization +2

What's In My Big Data?

1 code implementation31 Oct 2023 Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, Jesse Dodge

We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them.

Benchmarking

The Surveillance AI Pipeline

no code implementations26 Sep 2023 Pratyusha Ria Kalluri, William Agnew, Myra Cheng, Kentrell Owens, Luca Soldaini, Abeba Birhane

Moreover, the majority of these technologies specifically enable extracting data about human bodies and body parts.

Bound by the Bounty: Collaboratively Shaping Evaluation Processes for Queer AI Harms

no code implementations15 Jul 2023 Organizers Of QueerInAI, Nathan Dennler, Anaelia Ovalle, Ashwin Singh, Luca Soldaini, Arjun Subramonian, Huy Tu, William Agnew, Avijit Ghosh, Kyra Yee, Irene Font Peradejordi, Zeerak Talat, Mayra Russo, Jess de Jesus de Pinho Pinhal

However, these auditing processes have been criticized for their failure to integrate the knowledge of marginalized communities and consider the power dynamics between auditors and the communities.

A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents

no code implementations24 May 2023 Benjamin Newman, Luca Soldaini, Raymond Fok, Arman Cohan, Kyle Lo

Many real-world applications (e. g., note taking, search) require extracting a sentence or paragraph from a document and showing that snippet to a human outside of the source document.

Question Answering Question Generation +2

Overview of the TREC 2022 NeuCLIR Track

no code implementations24 Apr 2023 Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, Eugene Yang

This is the first year of the TREC Neural CLIR (NeuCLIR) track, which aims to study the impact of neural approaches to cross-language information retrieval.

Information Retrieval Retrieval

One-Shot Labeling for Automatic Relevance Estimation

1 code implementation22 Feb 2023 Sean MacAvaney, Luca Soldaini

We then explore various approaches for predicting the relevance of unjudged documents with respect to a query and the known relevant document, including nearest neighbor, supervised, and prompting techniques.

Retrieval

Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval

no code implementations20 Dec 2022 John Giorgi, Luca Soldaini, Bo wang, Gary Bader, Kyle Lo, Lucy Lu Wang, Arman Cohan

Via extensive automatic and human evaluation, we determine: (1) state-of-the-art summarizers suffer large reductions in performance when applied to open-domain MDS, (2) additional training in the open-domain setting can reduce this sensitivity to imperfect retrieval, and (3) summarizers are insensitive to the retrieval of duplicate documents and the order of retrieved documents, but highly sensitive to other errors, like the retrieval of irrelevant documents.

Document Summarization Multi-Document Summarization +1

Knowledge Transfer from Answer Ranking to Answer Generation

no code implementations23 Oct 2022 Matteo Gabburo, Rik Koncel-Kedziorski, Siddhant Garg, Luca Soldaini, Alessandro Moschitti

In this paper, we propose to train a GenQA model by transferring knowledge from a trained AS2 model, to overcome the aforementioned issue.

Answer Generation Question Answering +2

Embedding Recycling for Language Models

1 code implementation11 Jul 2022 Jon Saad-Falcon, Amanpreet Singh, Luca Soldaini, Mike D'Arcy, Arman Cohan, Doug Downey

Real-world applications of neural language models often involve running many different models over the same corpus.

Question Answering Text Classification

Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection

no code implementations20 May 2022 Luca Di Liello, Siddhant Garg, Luca Soldaini, Alessandro Moschitti

An important task for designing QA systems is answer sentence selection (AS2): selecting the sentence containing (or constituting) the answer to a question from a set of retrieved relevant documents.

Answer Selection Sentence

Paragraph-based Transformer Pre-training for Multi-Sentence Inference

1 code implementation NAACL 2022 Luca Di Liello, Siddhant Garg, Luca Soldaini, Alessandro Moschitti

Our evaluation on three AS2 and one fact verification datasets demonstrates the superiority of our pre-training technique over the traditional ones for transformers used as joint models for multi-candidate inference tasks, as well as when used as cross-encoders for sentence-pair formulations of these tasks.

Answer Selection Fact Verification +1

Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems

1 code implementation15 Jan 2022 Yoshitomo Matsubara, Luca Soldaini, Eric Lind, Alessandro Moschitti

CERBERUS consists of two components: a stack of transformer layers that is used to encode inputs, and a set of ranking heads; unlike traditional distillation technique, each of them is trained by distilling a different large transformer architecture in a way that preserves the diversity of the ensemble members.

Efficient Neural Network Question Answering +1

Cross-Lingual Open-Domain Question Answering with Answer Sentence Generation

no code implementations14 Oct 2021 Benjamin Muller, Luca Soldaini, Rik Koncel-Kedziorski, Eric Lind, Alessandro Moschitti

Our cross-lingual generative system outperforms answer sentence selection baselines for all 5 languages and monolingual generative pipelines for three out of five languages studied.

Answer Generation Generative Question Answering +3

Answer Generation for Retrieval-based Question Answering Systems

no code implementations Findings (ACL) 2021 Chao-Chun Hsu, Eric Lind, Luca Soldaini, Alessandro Moschitti

Recent advancements in transformer-based models have greatly improved the ability of Question Answering (QA) systems to provide correct answers; in particular, answer sentence selection (AS2) models, core components of retrieval-based systems, have achieved impressive results.

Answer Generation Question Answering +2

Multi-task Learning of Spoken Language Understanding by Integrating N-Best Hypotheses with Hierarchical Attention

no code implementations COLING 2020 Mingda Li, Xinyue Liu, Weitong Ruan, Luca Soldaini, Wael Hamza, Chengwei Su

The comparison shows that our model could recover the transcription by integrating the fragmented information among hypotheses and identifying the frequent error patterns of the ASR module, and even rewrite the query for a better understanding, which reveals the characteristic of multi-task learning of broadcasting knowledge.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +6

Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing

no code implementations30 Jan 2020 Subendhu Rongali, Luca Soldaini, Emilio Monti, Wael Hamza

Virtual assistants such as Amazon Alexa, Apple Siri, and Google Assistant often rely on a semantic parsing component to understand which action(s) to execute for an utterance spoken by its users.

Semantic Parsing slot-filling +1

Improving Spoken Language Understanding By Exploiting ASR N-best Hypotheses

no code implementations11 Jan 2020 Mingda Li, Weitong Ruan, Xinyue Liu, Luca Soldaini, Wael Hamza, Chengwei Su

The NLU module usually uses the first best interpretation of a given speech in downstream tasks such as domain and intent classification.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +5

Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learning

1 code implementation30 Dec 2019 Sean MacAvaney, Luca Soldaini, Nazli Goharian

While billions of non-English speaking users rely on search engines every day, the problem of ad-hoc information retrieval is rarely studied for non-English languages.

Ad-Hoc Information Retrieval Information Retrieval +2

Cannot find the paper you are looking for? You can Submit a new open access paper.