Search Results for author:
Found 94 papers, 65 papers with code
Distantly-Supervised Dense Retrieval Enables Open-Domain Question Answering without Evidence Annotation
This paper investigates whether models can learn to find evidence from a large corpus, with only distant supervision from answer labels for model training, thereby generating no additional annotation cost.
Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder
Dense retrieval requires high-quality text sequence embeddings to support effective search in the representation space.
Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviews
Which large language model (LLM) is better?
PIP-KAG: Mitigating Knowledge Conflicts in Knowledge-Augmented Generation via Parametric Pruning
Knowledge-Augmented Generation (KAG) has shown great promise in updating the internal memory of Large Language Models (LLMs) by integrating external knowledge.
Data-Efficient Pretraining with Group-Level Data Influence Modeling
Data-efficient pretraining has shown tremendous potential to elevate scaling laws.
Craw4LLM: Efficient Web Crawling for LLM Pretraining
Web crawl is a main source of large language models' (LLMs) pretraining data, but the majority of crawled web pages are discarded in pretraining due to low data quality.
Harness Local Rewards for Global Benefits: Effective Text-to-Video Generation Alignment with Patch-level Reward Models
To address this issue, we propose a post-training strategy for VGMs, HALO, which explicitly incorporates local feedback from a patch reward model, providing detailed and comprehensive training signals with the video reward model for advanced VGM optimization.
Efficient Multi-Agent System Training with Data Influence-Oriented Tree Search
Monte Carlo Tree Search (MCTS) based methods provide promising approaches for generating synthetic data to enhance the self-training of Large Language Model (LLM) based multi-agent systems (MAS).
Fairshare Data Pricing for Large Language Models
Training data is a pivotal resource for building large language models (LLMs), but unfair pricing in data markets poses a serious challenge for both data buyers (e. g., LLM builders) and sellers (e. g., human annotators), which discourages market participation, reducing data quantity and quality.
RAGViz: Diagnose and Visualize Retrieval-Augmented Generation
With a built-in user interface, retrieval index, and Large Language Model (LLM) backbone, RAGViz provides two main functionalities: (1) token and document-level attention visualization, and (2) generation comparison upon context document addition and removal.
Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning
In this paper, we propose Montessori-Instruct, a novel data synthesis framework that tailors the data synthesis ability of the teacher language model toward the student language model's learning process.
Interpret and Control Dense Retrieval with Sparse Latent Features
Dense embeddings deliver strong retrieval performance but often lack interpretability and controllability.
RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards
Our experiments on various knowledge-intensive tasks demonstrate that DDR significantly outperforms the SFT method, particularly for LLMs with smaller-scale parameters that depend more on the retrieved knowledge.
Harnessing Webpage UIs for Text-Rich Visual Understanding
Text-rich visual understanding-the ability to process environments where dense textual content is integrated with visuals-is crucial for multimodal large language models (MLLMs) to interact effectively with structured environments.
Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generation
Further analysis indicates that employing our factually-informed training strategy imposes an effective supervision signal, without relying on explicit diagnostic label guidance, and successfully propagates fact-aware capabilities from the multimodal retriever to the multimodal foundation model in radiology report generation.
In-Context Probing Approximates Influence Function for Data Valuation
Data valuation quantifies the value of training data, and is used for data attribution (i. e., determining the contribution of training data towards model predictions), and data selection; both of which are important for curating high-quality datasets to train large language models.
ResearchArena: Benchmarking LLMs' Ability to Collect and Organize Information as Research Agents
In particular, we establish an offline environment comprising 12. 0M full-text academic papers and 7. 9K survey papers, which evaluates agents' ability to locate supporting materials for composing the survey on a topic, rank the located papers based on their impact, and organize these into a hierarchical knowledge mind-map.
MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models
In this paper, we introduce model-aware data selection with data influence models (MATES), where a data influence model continuously adapts to the evolving data preferences of the pretraining model and then selects the data most effective for the current pretraining progress.
MS MARCO Web Search: a Large-scale Information-rich Web Dataset with Millions of Real Click Labels
Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals.
Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval
This study investigates the existence of positional biases in Transformer-based models for text representation learning, particularly in the context of web document retrieval.
Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression
It finetunes the compression plugin module and uses the representations of gist tokens to emulate the raw prompts in the vanilla language model.
Cleaner Pretraining Corpus Curation with Neural Web Scraping
The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans.
ActiveRAG: Autonomously Knowledge Assimilation and Accommodation through Retrieval-Augmented Agents
Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to leverage external knowledge, enhancing their performance on knowledge-intensive tasks.
ED-Copilot: Reduce Emergency Department Wait Time with Language Model Diagnostic Assistance
In collaboration with ED clinicians, we use public patient data to curate MIMIC-ED-Assist, a benchmark for AI systems to suggest laboratory tests that minimize wait time while accurately predicting critical outcomes such as death.
An In-depth Look at Gemini's Language Abilities
The recently released Google Gemini class of models are the first to comprehensively report results that rival the OpenAI GPT series across a wide variety of tasks.
Enhancing Dense Retrievers' Robustness with Group-level Reweighting
In this paper, we introduce WebDRO, an efficient approach for clustering the web graph data and optimizing group weights to enhance the robustness of dense retrieval models.
MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin
To facilitate the multi-modal retrieval tasks, we build the ClueWeb22-MM dataset based on the ClueWeb22 dataset, which regards anchor texts as queries, and extracts the related text and image documents from anchor-linked web pages.
Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model
We first validate the efficacy of Toolink in harnessing the model's creativity and CoS ability on ChatGPT.
Text Matching Improves Sequential Recommendation by Reducing Popularity Biases
TASTE alleviates the cold start problem by representing long-tail items using full-text modeling and bringing the benefits of pretrained language models to recommendation systems.
Improving Multitask Retrieval by Promoting Task Specialization
In multitask retrieval, a single retriever is trained to retrieve relevant contexts for multiple tasks.
Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data
SANTA proposes two pretraining methods to make language models structure-aware and learn effective representations for structured data: 1) Structured Data Alignment, which utilizes the natural alignment relations between structured data and unstructured data for structure-aware pretraining.
Fusion-in-T5: Unifying Document Ranking Signals for Improved Information Retrieval
Common document ranking pipelines in search systems are cascade systems that involve multiple ranking layers to integrate different information step-by-step.
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5.
Unsupervised Dense Retrieval Training with Web Anchors
Based on their commonalities, we train an unsupervised dense retriever, Anchor-DR, with a contrastive learning task that matches the anchor text and the linked document.
Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories
In this paper we improve the zero-shot generalization ability of language models via Mixture-Of-Memory Augmentation (MoMA), a mechanism that retrieves augmentation documents from multiple information corpora ("external memories"), with the option to "plug in" new memory at inference time.
ClueWeb22: 10 Billion Web Documents with Visual and Semantic Information
ClueWeb22, the newest iteration of the ClueWeb line of datasets, provides 10 billion web pages affiliated with rich information.
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
In this paper, we investigate the instability in the standard dense retrieval training, which iterates between model training and hard negative selection using the being-trained model.
COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning
We present a new zero-shot dense retrieval (ZeroDR) method, COCO-DR, to improve the generalization ability of dense retrieval by combating the distribution shifts between source training tasks and target scenarios.
Ranked #1 on
Zero-shot Text Search
on CQADupStack
Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval
To learn a unified embedding space for multi-modal retrieval, UniVL-DR proposes two techniques: 1) Universal embedding optimization strategy, which contrastively optimizes the embedding space using the modality-balanced hard negatives; 2) Image verbalization method, which bridges the modality gap between images and texts in the raw data space.
Dimension Reduction for Efficient Dense Retrieval via Conditional Autoencoder
These embeddings need to be high-dimensional to fit training signals and guarantee the retrieval effectiveness of dense retrievers.
Ranked #1 on
Information Retrieval
on MS MARCO
P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning
In this paper, we identify and study the two mismatches between pre-training and ranking fine-tuning: the training schema gap regarding the differences in training objectives and model architectures, and the task knowledge gap considering the discrepancy between the knowledge needed in ranking and that learned during pre-training.
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model.
Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
We present a new framework AMOS that pretrains text encoders with an Adversarial learning curriculum via a Mixture Of Signals from multiple auxiliary generators.
Neural Approaches to Conversational Information Retrieval
A conversational information retrieval (CIR) system is an information retrieval (IR) system with a conversational interface which allows users to interact with the system to seek information via multi-turn conversations of natural language, in spoken or written form.
Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations
Dense retrieval (DR) methods conduct text retrieval by first encoding texts in the embedding space and then matching them by nearest neighbor search.
Distantly-Supervised Evidence Retrieval Enables Question Answering without Evidence Annotation
Open-domain question answering answers a question based on evidence retrieved from a large corpus.
Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representation
Dense retrieval (DR) methods conduct text retrieval by first encoding texts in the embedding space and then matching them by nearest neighbor search.
TIAGE: A Benchmark for Topic-Shift Aware Dialog Modeling
In research on dialog systems, the ability to actively and smoothly transition to new topics is often ignored.
Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback
This paper proposes ANCE-PRF, a new query encoder that uses pseudo relevance feedback (PRF) to improve query representations for dense retrieval.
More Robust Dense Retrieval with Contrastive Dual Learning
With contrastive learning, the dual training object of DANCE learns more tailored representations for queries and documents to keep the embedding space smooth and uniform, thriving on the ranking performance of DANCE on the MS MARCO document retrieval task.
Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature
A prominent case in point is the explosion of the biomedical literature on COVID-19, which swelled to hundreds of thousands of papers in a matter of months.
Few-Shot Conversational Dense Retrieval
In this paper, we present a Conversational Dense Retrieval system, ConvDR, that learns contextualized embeddings for multi-turn conversational queries and retrieves documents solely using embedding dot products.
Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval
Complex question answering often requires finding a reasoning chain that consists of multiple evidence pieces.
Complex Factoid Question Answering with a Free-Text Knowledge Graph
DELFT's advantage comes from both the high coverage of its free-text knowledge graph-more than double that of dbpedia relations-and the novel graph neural network which reasons on the rich but noisy free-text evidence.
Data Augmentation for Abstractive Query-Focused Multi-Document Summarization
The progress in Query-focused Multi-Document Summarization (QMDS) has been limited by the lack of sufficient largescale high-quality training datasets.
Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder
Dense retrieval requires high-quality text sequence embeddings to support effective search in the representation space.
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics.
OpenMatch: An Open Source Library for Neu-IR Research
OpenMatch is a Python-based library that serves for Neural Information Retrieval (Neu-IR) research.
Pretrain Knowledge-Aware Language Models
Rather, we simply signal the existence of entities to the input of the transformer in pretraining, with an entity-extended tokenizer; and at the output, with an additional entity prediction task.
Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision
To democratize the benefits of Neu-IR, this paper presents MetaAdaptRank, a domain adaptive learning method that generalizes Neu-IR models from label-rich source domains to few-shot target domains.
CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search
Neural rankers based on deep pretrained language models (LMs) have been shown to improve many information retrieval benchmarks.
Adapting Open Domain Fact Extraction and Verification to COVID-FACT through In-Domain Language Modeling
With the epidemic of COVID-19, verifying the scientifically false online information, such as fake news and maliciously fabricated statements, has become crucial.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
In this paper, we develop a general framework for interpretable natural language understanding that requires only a small set of human annotated explanations for training.
Long Document Ranking with Query-Directed Sparse Transformer
In this paper, we design Query-Directed Sparse attention that induces IR-axiomatic structures in transformer self-attention.
Text Classification Using Label Names Only: A Language Model Self-Training Approach
In this paper, we explore the potential of only using the label name of each class to train classification models on unlabeled data, without using any labeled documents.
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
In this paper, we identify that the main bottleneck is in the training mechanisms, where the negative instances used in training are not representative of the irrelevant documents in testing.
Ranked #7 on
Passage Retrieval
on Natural Questions
Proceedings of the KG-BIAS Workshop 2020 at AKBC 2020
The KG-BIAS 2020 workshop touches on biases and how they surface in knowledge graphs (KGs), biases in the source data that is used to create KGs, methods for measuring or remediating bias in KGs, but also identifying other biases such as how and which languages are represented in automatically constructed KGs or how personal KGs might incur inherent biases.
Few-Shot Generative Conversational Query Rewriting
Conversational query rewriting aims to reformulate a concise conversational query to a fully specified, context-independent query that can be effectively handled by existing information retrieval systems.
Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention
Transformers have achieved new heights modeling natural language as a sequence of text tokens.
Ranked #42 on
Question Answering
on HotpotQA
Capturing Global Informativeness in Open Domain Keyphrase Extraction
Open-domain KeyPhrase Extraction (KPE) aims to extract keyphrases from documents without domain or quality restrictions, e. g., web pages with variant domains and qualities.
TREC CAsT 2019: The Conversational Assistance Track Overview
A common theme through the runs is the use of BERT-based neural reranking methods.
Selective Weak Supervision for Neural Information Retrieval
This paper democratizes neural information retrieval to scenarios where large scale relevance training signals are not available.
TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network
Taxonomies consist of machine-interpretable semantics and provide valuable knowledge for many web applications.
Contrastive Multi-document Question Generation
In the SL stage, a single-document question generator is trained.
Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs
Human conversations naturally evolve around related concepts and scatter to multi-hop concepts.
Open Domain Web Keyphrase Extraction Beyond Language Modeling
This paper studies keyphrase extraction in real-world scenarios where documents are from diverse domains and have variant content quality.
Fine-grained Fact Verification with Kernel Graph Attention Network
Fact Verification requires fine-grained natural language inference capability that finds subtle clues to identify the syntactical and semantically correct but not well-supported claims.
Ranked #5 on
Fact Verification
on FEVER
Explore Entity Embedding Effectiveness in Entity Retrieval
Entity embedding learns lots of semantic information from the knowledge graph and represents entities with a low-dimensional representation, which provides an opportunity to establish interactions between query related entities and candidate entities for entity retrieval.
Latent Relation Language Models
In this paper, we propose Latent Relation Language Models (LRLMs), a class of language models that parameterizes the joint distribution over the words in a document and the entities that occur therein via knowledge graph relations.
Neural Document Expansion with User Feedback
This paper presents a neural document expansion approach (NeuDEF) that enriches document representations for neural ranking models.
Generic Intent Representation in Web Search
This paper presents GEneric iNtent Encoder (GEN Encoder) which learns a distributed representation space for user intent in search.
Target-Guided Open-Domain Conversation
We study the problem of imposing conversational goals on open-domain chat agents.
Understanding the Behaviors of BERT in Ranking
This paper studies the performances and behaviors of BERT in ranking tasks.
An Axiomatic Approach to Regularizing Neural Ranking Models
The training of these models involve a search for appropriate parameter values based on large quantities of labeled examples.
Consistency and Variation in Kernel Neural Ranking Model
This paper studies the consistency of the kernel-based neural ranking model K-NRM, a recent state-of-the-art neural IR model, which is important for reproducible research and deployment in the industry.
Automatic Event Salience Identification
Our analyses demonstrate that our neural model captures interesting connections between salience and discourse unit relations (e. g., scripts and frame structures).
Entity-Duet Neural Ranking: Understanding the Role of Knowledge Graph Semantics in Neural Information Retrieval
This paper presents the Entity-Duet Neural Ranking Model (EDRM), which introduces knowledge graphs to neural search systems.
Towards Better Text Understanding and Retrieval through Kernel Entity Salience Modeling
The salience model also improves ad hoc search accuracy, providing effective ranking features by modeling the salience of query entities in candidate documents.
Convolutional Neural Networks for Soft Matching N-Grams in Ad-hoc Search
This paper presents Conv-KNRM, a Convolutional Kernel-based Neural Ranking Model that models n-gram soft matches for ad-hoc search.
End-to-End Neural Ad-hoc Ranking with Kernel Pooling
Given a query and a set of documents, K-NRM uses a translation matrix that models word-level similarities via word embeddings, a new kernel-pooling technique that uses kernels to extract multi-level soft match features, and a learning-to-rank layer that combines those features into the final ranking score.
Word-Entity Duet Representations for Document Ranking
This paper presents a word-entity duet framework for utilizing knowledge bases in ad-hoc retrieval.
