Search Results for author:
Found 47 papers, 21 papers with code
The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
In the context of the TREC 2024 RAG Track, we calibrate a fully automatic approach against strategies where nuggets are created manually or semi-manually by human assessors and then assigned manually to system answers.
Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges
Retrieval-augmented generation (RAG) enables large language models (LLMs) to generate answers with citations from source documents containing "ground truth", thereby reducing system hallucinations.
Judging the Judges: A Collection of LLM-Generated Relevance Judgements
Using Large Language Models (LLMs) for relevance assessments offers promising opportunities to improve Information Retrieval (IR), Natural Language Processing (NLP), and related fields.
JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment
The effective training and evaluation of retrieval systems require a substantial amount of relevance judgments, which are traditionally collected from human assessors -- a process that is both costly and time-consuming.
Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework
Within the TREC setup, we are able to calibrate our fully automatic process against a manual process whereby nuggets are created by human assessors semi-manually and then assigned manually to system answers.
A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look
This paper reports on the results of a large-scale evaluation (the TREC 2024 RAG Track) where four different relevance assessment approaches were deployed in situ: the "standard" fully manual process that NIST has implemented for decades and three different alternatives that take advantage of LLMs to different extents using the open-source UMBRELA tool.
SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval
Large-scale test collections play a crucial role in Information Retrieval (IR) research.
Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024
The first edition of the workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) took place in July 2024, co-located with the ACM SIGIR Conference 2024 in the USA (SIGIR 2024).
LLMJudge: LLMs for Relevance Judgments
The evaluation and tuning of a search system is largely based on relevance labels, which indicate whether a document is useful for a specific search and user.
Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
In our work, we lay out the steps we've made towards making this track a reality -- we describe the details of our reusable framework, Ragnar\"ok, explain the curation of the new MS MARCO V2. 1 collection choice, release the development topics for the track, and standardize the I/O definitions which assist the end user.
UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor
Copious amounts of relevance judgments are necessary for the effective training and accurate evaluation of retrieval systems.
Synthetic Test Collections for Retrieval Evaluation
Previous studies demonstrate that LLMs have the potential to generate synthetic relevance judgments for use in the evaluation of IR systems.
MS MARCO Web Search: a Large-scale Information-rich Web Dataset with Millions of Real Click Labels
Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals.
Towards Group-aware Search Success
Traditional measures of search success often overlook the varying information needs of different demographic groups.
Large language models can accurately predict searcher preferences
It takes careful feedback from real users, which by definition is the highest-quality first-party gold data that can be derived, and develops an large language model prompt that agrees with that data.
Patterns of gender-specializing query reformulation
There are many ways a query, the search results, and a demographic attribute such as gender may relate, leading us to hypothesize different causes for these reformulation patterns, such as under-representation on the original result page or based on the linguistic theory of markedness.
Zero-shot Clarifying Question Generation for Conversational Search
In this work, we innovatively explore generating clarifying questions in a zero-shot setting to overcome the cold start problem and we propose a constrained clarifying question generation system which uses both question templates and query facets to guide the effective and precise question generation.
Are We There Yet? A Decision Framework for Replacing Term Based Retrieval with Dense Retrieval Systems
Recently, several dense retrieval (DR) models have demonstrated competitive performance to term-based retrieval that are ubiquitous in search systems.
Human Preferences as Dueling Bandits
Based on these simulations, one algorithm stands out for its potential.
Less is Less: When Are Snippets Insufficient for Human vs Machine Relevance Estimation?
This raises questions about when such summaries are sufficient for relevance estimation by the ranking model or the human assessor, and whether humans and machines benefit from the document's full text in similar ways.
Neural Approaches to Conversational Information Retrieval
A conversational information retrieval (CIR) system is an information retrieval (IR) system with a conversational interface which allows users to interact with the system to seek information via multi-turn conversations of natural language, in spoken or written form.
Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking
An emerging recipe for achieving state-of-the-art effectiveness in neural document re-ranking involves utilizing large pre-trained language models - e. g., BERT - to evaluate all individual passages in the document and then aggregating the outputs by pooling or additional Transformer layers.
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field.
TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime
The TREC Deep Learning (DL) Track studies ad hoc search in the large data regime, meaning that a large set of human-labeled training data is available.
Improving Transformer-Kernel Ranking Model Using Conformer and Query Term Independence
The Transformer-Kernel (TK) model has demonstrated strong reranking performance on the TREC Deep Learning benchmark -- and can be considered to be an efficient (but slightly less effective) alternative to other Transformer-based architectures that employ (i) large-scale pretraining (high training cost), (ii) joint encoding of query and document (high inference cost), and (iii) larger number of Transformer layers (both high training and high inference costs).
Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard
Leaderboards are a ubiquitous part of modern research in applied machine learning.
Overview of the TREC 2020 deep learning track
This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime.
Conformer-Kernel with Query Term Independence at TREC 2020 Deep Learning Track
We benchmark Conformer-Kernel models under the strict blind evaluation setting of the TREC 2020 Deep Learning track.
Conformer-Kernel with Query Term Independence for Document Retrieval
In this work, we extend the TK architecture to the full retrieval setting by incorporating the query term independence assumption.
Scalable Methods for Calculating Term Co-Occurrence Frequencies
Search techniques make use of elementary information such as term frequencies and document lengths in computation of similarity weighting.
MIMICS: A Large-Scale Data Collection for Search Clarification
In this paper, we introduce MIMICS, a collection of search clarification datasets for real web search queries sampled from the Bing query logs.
ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search
Users of Web search engines reveal their information needs through queries and clicks, making click logs a useful asset for information retrieval.
Analyzing and Learning from User Interactions for Search Clarification
We also propose a model for learning representation for clarifying questions based on the user interaction data as implicit feedback.
Local Self-Attention over Long Text for Efficient Document Retrieval
In this work, we propose a local self-attention which considers a moving window over the document terms and for each term attends only to other terms in the same window.
On the Reliability of Test Collections for Evaluating Systems of Different Types
As deep learning based models are increasingly being used for information retrieval (IR), a major challenge is to ensure the availability of test collections for measuring their quality.
Overview of the TREC 2019 deep learning track
The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime.
Macaw: An Extensible Conversational Information Seeking Platform
Such research will require data and tools, to allow the implementation and study of conversational systems.
Duet at TREC 2019 Deep Learning Track
This report discusses three submissions based on the Duet architecture to the Deep Learning track at TREC 2019.
Generic Intent Representation in Web Search
This paper presents GEneric iNtent Encoder (GEN Encoder) which learns a distributed representation space for user intent in search.
Incorporating Query Term Independence Assumption for Efficient Retrieval and Ranking using Deep Neural Networks
Deep neural IR models, in contrast, compare the whole query to the document and are, therefore, typically employed only for late stage re-ranking.
An Axiomatic Approach to Regularizing Neural Ranking Models
The training of these models involve a search for appropriate parameter values based on large quantities of labeled examples.
An Updated Duet Model for Passage Re-ranking
We propose several small modifications to Duet---a deep neural ranking model---and evaluate the updated model on the MS MARCO passage ranking task.
Ranked #4 on
Passage Re-Ranking
on MS MARCO
Neural Models for Information Retrieval
Neural ranking models for information retrieval (IR) use shallow or deep neural networks to rank search results in response to a query.
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering.
Learning to Match Using Local and Distributed Representations of Text for Web Search
Models such as latent semantic analysis and those based on neural embeddings learn distributed representations of text, and match the query against the document in the latent semantic space.
Query Expansion with Locally-Trained Word Embeddings
Continuous space word embeddings have received a great deal of attention in the natural language processing and machine learning communities for their ability to model term similarity and other relationships.
A Dual Embedding Space Model for Document Ranking
A fundamental goal of search engines is to identify, given a query, documents that have relevant text.
