Search Results for author:
Found 35 papers, 10 papers with code
XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation
In this paper, we introduce XGLUE, a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora, and evaluate their performance across a diverse set of cross-lingual tasks.
ColBERT-serve: Efficient Multi-Stage Memory-Mapped Scoring
We study serving retrieval models, specifically late interaction models like ColBERT, to many concurrent users at once and under a small budget, in which the index may not fit in memory.
Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges
Retrieval-augmented generation (RAG) enables large language models (LLMs) to generate answers with citations from source documents containing "ground truth", thereby reducing system hallucinations.
The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
In the context of the TREC 2024 RAG Track, we calibrate a fully automatic approach against strategies where nuggets are created manually or semi-manually by human assessors and then assigned manually to system answers.
Inference Scaling for Bridging Retrieval and Augmented Generation
Retrieval-augmented generation (RAG) has emerged as a popular approach to steering the output of a large language model (LLM) by incorporating retrieved contexts as inputs.
Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
This paper presents the training methodology of Arctic-Embed 2. 0, a set of open-source text embedding models built for accurate and efficient multilingual retrieval.
Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework
Within the TREC setup, we are able to calibrate our fully automatic process against a manual process whereby nuggets are created by human assessors semi-manually and then assigned manually to system answers.
A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look
This paper reports on the results of a large-scale evaluation (the TREC 2024 RAG Track) where four different relevance assessment approaches were deployed in situ: the "standard" fully manual process that NIST has implemented for decades and three different alternatives that take advantage of LLMs to different extents using the open-source UMBRELA tool.
SuffixDecoding: A Model-Free Approach to Speeding Up Large Language Model Inference
For open-ended chat and code generation tasks, SuffixDecoding achieves up to $1. 4\times$ higher output throughput than SpecInfer and up to $1. 1\times$ lower time-per-token (TPOT) latency.
Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
In our work, we lay out the steps we've made towards making this track a reality -- we describe the details of our reusable framework, Ragnar\"ok, explain the curation of the new MS MARCO V2. 1 collection choice, release the development topics for the track, and standardize the I/O definitions which assist the end user.
Synthetic Test Collections for Retrieval Evaluation
Previous studies demonstrate that LLMs have the potential to generate synthetic relevance judgments for use in the evaluation of IR systems.
Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models
This report describes the training dataset creation and recipe behind the family of \texttt{arctic-embed} text embedding models (a set of five models ranging from 22 to 334 million parameters with weights open-sourced under an Apache-2 license).
Overview of the TREC 2023 Product Product Search Track
The focus this year was the creation of a reusable collection and evaluation of the impact of the use of metadata and multi-modal data on retrieval accuracy.
Noise-Robust Dense Retrieval via Contrastive Alignment Post Training
The success of contextual word representations and advances in neural information retrieval have made dense vector-based retrieval a standard approach for passage and document ranking.
To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiency
Sequence-to-sequence language models can be used to produce abstractive summaries which are coherent, relevant, and concise.
Dense Sparse Retrieval: Using Sparse Language Models for Inference Efficient Dense Retrieval
Vector-based retrieval systems have become a common staple for academic and industrial search applications because they provide a simple and scalable way of extending the search to leverage contextual representations for documents and queries.
Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoders
In this paper, we consider the problem of improving the inference latency of language model-based dense retrieval systems by introducing structural compression and model size asymmetry between the context and query encoders.
oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes
In this paper, we introduce the range of oBERTa language models, an easy-to-use set of language models which allows Natural Language Processing (NLP) practitioners to obtain between 3. 8 and 24. 3 times faster models without expertise in model compression.
Compressing Cross-Lingual Multi-Task Models at Qualtrics
Experience management is an emerging business area where organizations focus on understanding the feedback of customers and employees in order to improve their end-to-end experiences.
Sparse*BERT: Sparse Models Generalize To New tasks and Domains
Our experimentation shows that models that are pruned during pretraining using general domain masked language models can transfer to novel domains and tasks without extensive hyperparameter exploration or specialized approaches.
The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models
We perform an in-depth study of the accuracy-compression trade-off for unstructured weight pruning of BERT models.
IMG2SMI: Translating Molecular Structure Images to Simplified Molecular-input Line-entry System
A large portion of chemistry literature focuses on new molecules and reactions between molecules.
Curriculum learning for language modeling
Language Models like ELMo and BERT have provided robust representations of natural language, which serve as the language understanding component for a diverse range of downstream tasks. Curriculum learning is a method that employs a structured training regime instead, which has been leveraged in computer vision and machine translation to improve model training speed and model performance.
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field.
TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime
The TREC Deep Learning (DL) Track studies ad hoc search in the large data regime, meaning that a large set of human-labeled training data is available.
Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard
Leaderboards are a ubiquitous part of modern research in applied machine learning.
Informational entropy thresholds as a physical mechanism to explain power-law time distributions in sequential decision-making
While frameworks based on physical grounds (like the Drift-Diffusion Model) have been exhaustively used in psychology and neuroscience to describe perceptual decision-making in humans, analogous approaches for more complex situations like sequential (tree-like) decision making are still absent.
Decision Making
Physics and Society
Disordered Systems and Neural Networks
Adaptation and Self-Organizing Systems
Overview of the TREC 2020 deep learning track
This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime.
ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search
Users of Web search engines reveal their information needs through queries and clicks, making click logs a useful asset for information retrieval.
On the Reliability of Test Collections for Evaluating Systems of Different Types
As deep learning based models are increasingly being used for information retrieval (IR), a major challenge is to ensure the availability of test collections for measuring their quality.
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks.
Overview of the TREC 2019 deep learning track
The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime.
Open Domain Web Keyphrase Extraction Beyond Language Modeling
This paper studies keyphrase extraction in real-world scenarios where documents are from diverse domains and have variant content quality.
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering.
