Search Results for author:
Found 73 papers, 39 papers with code
Improving the Reusability of Conversational Search Test Collections
We show that filling the holes using few-shot training the Llama 3. 1 model enables a fairer comparison between the new system and the systems contributed to the pool.
Conversational Gold: Evaluating Personalized Conversational Search System using Gold Nuggets
Built on this resource, we provide a framework for long-form answer generation evaluation, involving nuggets extraction and nuggets matching, linked to retrieval.
PSCon: Toward Conversational Product Search
Conversational Product Search (CPS) is confined to simulated conversations due to the lack of real-world CPS datasets that reflect human-like language.
Judging the Judges: A Collection of LLM-Generated Relevance Judgements
Using Large Language Models (LLMs) for relevance assessments offers promising opportunities to improve Information Retrieval (IR), Natural Language Processing (NLP), and related fields.
Multi-Turn Multi-Modal Question Clarification for Enhanced Conversational Understanding
Motivated by this, we introduce the Multi-turn Multi-modal Clarifying Questions (MMCQ) task, which combines text and visual modalities to refine user queries in a multi-turn conversation.
IRLab@iKAT24: Learned Sparse Retrieval with Multi-aspect LLM Query Generation for Conversational Search
The Interactive Knowledge Assistant Track (iKAT) 2024 focuses on advancing conversational assistants, able to adapt their interaction and responses from personalized user knowledge.
AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs
Human evaluation and CrowdLLM show that the AGENT-CQ - generation stage, consistently outperforms baselines in various aspects of question and answer quality.
DiSCo Meets LLMs: A Unified Approach for Sparse Retrieval and Contextual Distillation in Conversational Search
Conversational Search (CS) is the task of retrieving relevant documents from a corpus within a conversational context, combining retrieval with conversational context modeling.
Unlocking Markets: A Multilingual Benchmark to Cross-Market Question Answering
We then conduct experiments to benchmark our dataset, using models ranging from traditional lexical models to LLMs in both single-market and cross-market scenarios across McMarket and the corresponding LLM subset.
A Comparative Analysis of Faithfulness Metrics and Humans in Citation Evaluation
To investigate the effectiveness of faithfulness metrics in fine-grained scenarios, we propose a comparative evaluation framework that assesses the metric effectiveness in distinguishing citations between three-category support levels: full, partial, and no support.
Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024
The first edition of the workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) took place in July 2024, co-located with the ACM SIGIR Conference 2024 in the USA (SIGIR 2024).
LLMJudge: LLMs for Relevance Judgments
The evaluation and tuning of a search system is largely based on relevance labels, which indicate whether a document is useful for a specific search and user.
Generative Retrieval with Few-shot Indexing
Existing generative retrieval (GR) approaches rely on training-based indexing, i. e., fine-tuning a model to memorise the associations between a query and the document identifier (docid) of a relevant document.
Interactions with Generative Information Retrieval Systems
We will also briefly describe multi-modal interactions in generative information retrieval.
Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics
To investigate the effectiveness of faithfulness metrics in fine-grained scenarios, we propose a comparative evaluation framework that assesses the metric effectiveness in distinguishing citations between three-category support levels: full, partial, and no support.
Can We Use Large Language Models to Fill Relevance Judgment Holes?
Incomplete relevance judgments limit the re-usability of test collections.
TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants
Conversational information seeking has evolved rapidly in the last few years with the development of Large Language Models (LLMs), providing the basis for interpreting and responding in a naturalistic manner to user requests.
Are We Really Achieving Better Beyond-Accuracy Performance in Next Basket Recommendation?
So far, most NBR studies have focused on optimizing the accuracy of the recommendation, whereas optimizing for beyond-accuracy metrics, e. g., item fairness and diversity remains largely unexplored.
Ranked List Truncation for Large Language Model-based Re-Ranking
RLT is crucial for re-ranking as it can improve re-ranking efficiency by sending variable-length candidate lists to a re-ranker on a per-query basis.
Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs
Workers are more susceptible to user feedback on usefulness and interestingness compared to LLMs on interestingness and relevance.
Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems
Using the first user utterance as context leads to consistent ratings, akin to those obtained using the entire dialogue, with significantly reduced annotation effort.
Query Performance Prediction using Relevance Judgments Generated by Large Language Models
To solve the challenges, we devise an approximation strategy to predict an IR measure considering recall and propose to fine-tune open-source LLMs using human-labeled relevance judgments.
Generating Multi-Aspect Queries for Conversational Search
To test this hypothesis, we conduct extensive experiments on five widely used CIS datasets where we leverage LLMs to generate multi-aspect queries to represent the information need for each utterance in multiple query rewrites.
CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems
In this work, we leverage large language models (LLMs) and unlock their ability to generate satisfaction-aware counterfactual dialogues to augment the set of original dialogues of a test collection.
Self-seeding and Multi-intent Self-instructing LLMs for Generating Intent-aware Information-Seeking dialogs
Furthermore, we propose SOLID-RL, which is further trained to generate a dialog in one step on the data generated by SOLID.
Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search
Therefore, we propose to add images to clarifying questions and formulate the novel task of asking multimodal clarifying questions in open-domain, mixed-initiative conversational search systems.
Clarifying the Path to User Satisfaction: An Investigation into Clarification Usefulness
Clarifying questions are an integral component of modern information retrieval systems, directly impacting user satisfaction and overall system performance.
Analyzing Coherency in Facet-based Clarification Prompt Generation for Search
Clarifying user's information needs is an essential component of modern search systems.
TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview
Conversational Information Seeking has evolved rapidly in the last few years with the development of Large Language Models providing the basis for interpreting and responding in a naturalistic manner to user requests.
Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions
Our framework involves two LLMs interacting on a specific topic, with the first LLM acting as a student, generating questions to explore a given search topic.
Query Performance Prediction: From Ad-hoc to Conversational Search
The QPP task is to predict the retrieval quality of a search system for a query without relevance judgments.
Generating Synthetic Documents for Cross-Encoder Re-Rankers: A Comparative Study of ChatGPT and Human Experts
We introduce a new dataset, ChatGPT-RetrievalQA, and compare the effectiveness of models fine-tuned on LLM-generated and human-generated data.
On the Impact of Outlier Bias on User Clicks
We therefore propose an outlier-aware click model that accounts for both outlier and position bias, called outlier-aware position-based model ( OPBM).
Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond
To this end, we propose a user simulator-based framework for multi-turn interactions with a variety of mixed-initiative CS systems.
Market-Aware Models for Efficient Cross-Market Recommendation
We consider the cross-market recommendation (CMR) task, which involves recommendation in a low-resource target market using data from a richer, auxiliary source market.
Experiments on Generalizability of BERTopic on Multi-Domain Short Text
We further analyze the performance of the HDBSCAN clustering algorithm utilized by BERTopic and find that it classifies a majority of the documents as outliers.
Towards Confidence-aware Calibrated Recommendation
Calibration ensures that the distribution of recommended item categories is consistent with the user's historical data.
IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022
Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions.
Experiments on Generalizability of User-Oriented Fairness in Recommender Systems
In this paper, we re-produce a user-oriented fairness study and provide extensive experiments to analyze the dependency of their proposed method on various fairness and recommendation aspects, including the recommendation domain, nature of the base ranking model, and user grouping method.
Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021
The primary goal of the competition is to approach the problem of how to build interactive agents that learn to solve a task while provided with grounded natural language instructions in a collaborative environment.
Understanding User Satisfaction with Task-oriented Dialogue Systems
What is the influence of user experience on the user satisfaction rating of TDS as opposed to, or in addition to, utility?
Conversational Recommendation
Task-Oriented Dialogue Systems
Evaluating Mixed-initiative Conversational Search Systems via User Simulation
Clarifying the underlying user information need by asking clarifying questions is an important feature of modern conversational search system.
Mental Disorders on Online Social Media Through the Lens of Language and Behaviour: Analysis and Visualisation
As for emotional expression, we observe that affected users tend to share emotions more regularly than control individuals on average.
Towards Building Economic Models of Conversational Search
Various conceptual and descriptive models of conversational search have been proposed in the literature -- while useful, they do not provide insights into how interaction between the agent and user would change in response to the costs and benefits of the different interactions.
A Systematic Analysis on the Impact of Contextual Information on Point-of-Interest Recommendation
The major contributions of this paper are: (i) providing an extensive survey of context-aware location recommendation (ii) quantifying and analyzing the impact of different contextual information (e. g., social, temporal, spatial, and categorical) in the POI recommendation on available baselines and two new linear and non-linear models, that can incorporate all the major contextual information into a single recommendation model, and (iii) evaluating the considered models using two well-known real-world datasets.
Leveraging Social Influence based on Users Activity Centers for Point-of-Interest Recommendation
To this end, we model social influence based on two factors: similarities between users in terms of common check-ins and the friendships between them.
Understanding and Mitigating the Effect of Outliers in Fair Ranking
We formalize outlierness in a ranking, show that outliers are present in realistic datasets, and present the results of an eye-tracking study, showing that users scanning order and the exposure of items are influenced by the presence of outliers.
NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment
Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions.
The Impact of User Demographics and Task Types on Cross-App Mobile Search
Recent developments in the mobile app industry have resulted in various types of mobile apps, each targeting a different need and a specific audience.
BERT for Target Apps Selection: Analyzing the Diversity and Performance of BERT in Unified Mobile Search
A unified mobile search framework aims to identify the mobile apps that can satisfy a user's information need and route the user's query to them.
Analysing Mixed Initiatives and Search Strategies during Conversational Search
In this paper, we present a model for conversational search -- from which we instantiate different observed conversational search strategies, where the agent elicits: (i) Feedback-First, or (ii) Feedback-After.
Keyword Extraction for Improved Document Retrieval in Conversational Search
Recent research has shown that mixed-initiative conversational search, based on the interaction between users and computers to clarify and improve a query, provides enormous advantages.
Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions
Enabling open-domain dialogue systems to ask clarifying questions when appropriate is an important direction for improving the quality of the system response.
Cross-Market Product Recommendation
We introduce and formalize the problem of cross-market product recommendation, i. e., market adaptation.
Ranking Clarifying Questions Based on Predicted User Engagement
This research demonstrates the potential for ranking clarification panes based on lexical information only and can serve as a first neural baseline for future research to improve on.
User Engagement Prediction for Clarification in Search
Prompting the user for clarification in a search session can be very beneficial to the system as the user's explicit feedback helps the system improve retrieval massively.
MICROS: Mixed-Initiative ConveRsatiOnal Systems Workshop
The 1st edition of the workshop on Mixed-Initiative ConveRsatiOnal Systems (MICROS@ECIR2021) aims at investigating and collecting novel ideas and contributions in the field of conversational systems.
Context-Aware Target Apps Selection and Recommendation for Enhancing Personal Mobile Assistants
Here we focus on context-aware models to leverage the rich contextual information available to mobile devices.
On Estimating the Training Cost of Conversational Recommendation Systems
Conversational recommendation systems have recently gain a lot of attention, as users can continuously interact with the system over multiple conversational turns.
ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)
The main aim of the conversational systems is to return an appropriate answer in response to the user requests.
Longformer for MS MARCO Document Re-ranking Task
Two step document ranking, where the initial retrieval is done by a classical information retrieval method, followed by neural re-ranking model, is the new standard.
Analysing the Effect of Clarifying Questions on Document Ranking in Conversational Search
Recent research on conversational search highlights the importance of mixed-initiative in conversations.
A Tool for Conducting User Studies on Mobile Devices
With the ever-growing interest in the area of mobile information retrieval and the ongoing fast development of mobile devices and, as a consequence, mobile apps, an active research area lies in studying users' behavior and search queries users submit on mobile devices.
Joint Geographical and Temporal Modeling based on Matrix Factorization for Point-of-Interest Recommendation
Previous studies show that incorporating contextual information such as geographical and temporal influences is necessary to improve POI recommendation by addressing the data sparsity problem.
Harnessing Evolution of Multi-Turn Conversations for Effective Answer Retrieval
With the improvements in speech recognition and voice generation technologies over the last years, a lot of companies have sought to develop conversation understanding systems that run on mobile phones or smart home devices through natural language interfaces.
A Joint Two-Phase Time-Sensitive Regularized Collaborative Ranking Model for Point of Interest Recommendation
In this article, we propose a two-phase CR algorithm that incorporates the geographical influence of POIs and is regularized based on the variance of POIs popularity and users' activities over time.
LGLMF: Local Geographical based Logistic Matrix Factorization Model for POI Recommendation
To address these problems, a POI recommendation method is proposed in this paper based on a Local Geographical Model, which considers both users' and locations' points of view.
Category-Aware Location Embedding for Point-of-Interest Recommendation
With the recent advances of neural models, much work has sought to leverage neural networks to learn neural embeddings in a pre-training phase that achieve an improved representation of POIs and consequently a better recommendation.
Asking Clarifying Questions in Open-Domain Information-Seeking Conversations
In this paper, we formulate the task of asking clarifying questions in open-domain information-seeking conversational systems.
ANTIQUE: A Non-Factoid Question Answering Benchmark
Despite the importance of the task, the community still feels the significant lack of large-scale non-factoid question answering collections with real questions and comprehensive relevance judgments.
Venue Suggestion Using Social-Centric Scores
These scores model each user by focusing on the different types of information extracted from venues that they have previously visited.
Graph-Based Semi-Supervised Conditional Random Fields For Spoken Language Understanding Using Unaligned Data
We experiment graph-based Semi-Supervised Learning (SSL) of Conditional Random Fields (CRF) for the application of Spoken Language Understanding (SLU) on unaligned data.
