Search Results for author:
Found 27 papers, 16 papers with code
Benchmarking LLM-based Relevance Judgment Methods
In addition to a traditional comparison based on system rankings using Kendall correlations, we also examine how well LLM judgments align with human preferences, as inferred from relevance grades.
A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment
In addition to investigating the impact of prompt variations on agreement with human labels, we compare human- and LLM-generated prompts and analyze differences among different LLMs as judges.
exHarmony: Authorship and Citations for Benchmarking the Reviewer Assignment Problem
The peer review process is crucial for ensuring the quality and reliability of scholarly work, yet assigning suitable reviewers remains a significant challenge.
EMPRA: Embedding Perturbation Rank Attack against Neural Ranking Models
Adversarial attacks seek to manipulate the ranking of documents, with the intention of exposing users to targeted content.
Offline Evaluation of Set-Based Text-to-Image Generation
Our proposed offline evaluation metrics for TTI not only capture how relevant generated images are with respect to the user's ideation need but also take into consideration the diversity and arrangement of the set of generated images.
IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents
Seamless interaction between AI agents and humans using natural language remains a key goal in AI research.
Assessing and Verifying Task Utility in LLM-Powered Applications
The rapid development of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents, assisting humans in their daily tasks.
Ranked List Truncation for Large Language Model-based Re-Ranking
RLT is crucial for re-ranking as it can improve re-ranking efficiency by sending variable-length candidate lists to a re-ranker on a per-query basis.
Generative Information Retrieval Evaluation
We resolve this apparent circularity in two ways: 1) by viewing LLM-based assessment as a form of "slow search", where a slower IR system is used for evaluation and training of a faster production IR system; and 2) by recognizing a continuing need to ground evaluation in human assessment, even if the characteristics of that human assessment must change.
A Comparison of Methods for Evaluating Generative IR
Given that Gen-IR systems do not generate responses from a fixed set, we assume that methods for Gen-IR evaluation must largely depend on LLM-generated labels.
Query Performance Prediction using Relevance Judgments Generated by Large Language Models
To solve the challenges, we devise an approximation strategy to predict an IR measure considering recall and propose to fine-tune open-source LLMs using human-labeled relevance judgments.
Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applications
The rapid development in the field of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents to assist humans in their daily tasks.
Fréchet Distance for Offline Evaluation of Information Retrieval Systems with Sparse Labels
The rapid advancement of natural language processing, information retrieval (IR), computer vision, and other technologies has presented significant challenges in evaluating the performance of these systems.
Adapting Standard Retrieval Benchmarks to Evaluate Generated Answers
In the second, we compare generated answers to the top results retrieved by a diverse set of retrieval models, ranging from traditional approaches to advanced methods, allowing us to measure improvements without human judgments.
Retrieving Supporting Evidence for Generative Question Answering
After presenting a question to an LLM and receiving a generated answer, we query the corpus with the combination of the question + generated answer.
Generative Question Answering
Open-Domain Question Answering
+1
Retrieving Supporting Evidence for LLMs Generated Answers
After presenting a question to an LLM and receiving a generated answer, we query the corpus with the combination of the question + generated answer.
Query Performance Prediction: From Ad-hoc to Conversational Search
The QPP task is to predict the retrieval quality of a search system for a query without relevance judgments.
Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions
Human intelligence's adaptability is remarkable, allowing us to adjust to new tasks and multi-modal environments swiftly.
Collecting Interactive Multi-modal Datasets for Grounded Language Understanding
Human intelligence can remarkably adapt quickly to new tasks and environments.
Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language Instructions
The adoption of pre-trained language models to generate action plans for embodied agents is a promising research strategy.
Early Stage Sparse Retrieval with Entity Linking
In this work, we propose boosting the performance of sparse retrievers by expanding both the queries and the documents with linked entities in two formats for the entity names: 1) explicit and 2) hashed.
Unsupervised Question Clarity Prediction Through Retrieved Item Coherency
While the research community has paid substantial attention to the problem of predicting query ambiguity in traditional search contexts, researchers have paid relatively little attention to predicting when this ambiguity is sufficient to warrant clarification in the context of conversational systems.
IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022
Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions.
PREME: Preference-based Meeting Exploration through an Interactive Questionnaire
The recent increase in the volume of online meetings necessitates automated tools for managing and organizing the material, especially when an attendee has missed the discussion and needs assistance in quickly exploring it.
Making Large Language Models Interactive: A Pioneer Study on Supporting Complex Information-Seeking Tasks with Implicit Constraints
Current interactive systems with natural language interfaces lack the ability to understand a complex information-seeking request which expresses several implicit constraints at once, and there is no prior information about user preferences e. g.,"find hiking trails around San Francisco which are accessible with toddlers and have beautiful scenery in summer", where output is a list of possible suggestions for users to start their exploration.
Predicting Efficiency/Effectiveness Trade-offs for Dense vs. Sparse Retrieval Strategy Selection
These hybrid retrievers leverage low-cost, exact-matching based sparse retrievers along with dense retrievers to bridge the semantic gaps between query and documents.
Shallow pooling for sparse labels
To test this observation, we employed crowdsourced workers to make preference judgments between the top item returned by a modern neural ranking stack and a judged relevant item.
