Search Results for author:
Found 69 papers, 36 papers with code
Improving Dialogue State Tracking with Turn-based Loss Function and Sequential Data Augmentation
While state-of-the-art Dialogue State Tracking (DST) models show promising results, all of them rely on a traditional cross-entropy loss function during the training process, which may not be optimal for improving the joint goal accuracy.
Uncertainty and Traffic-Aware Active Learning for Semantic Parsing
Collecting training data for semantic parsing is a time-consuming and expensive task.
Towards Understanding Bias in Synthetic Data for Evaluation
Previous work~\cite{rahmani2024synthetic} showed that synthetic test collections have the potential to be used for system evaluation, however, more analysis is needed to validate this claim.
PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants
Large language models (LLMs) have advanced conversational AI assistants.
Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation
Retrieval-Augmented Generation (RAG) leverages large language models (LLMs) combined with external contexts to enhance the accuracy and reliability of generated responses.
Machine-generated text detection prevents language model collapse
As Large Language Models (LLMs) become increasingly prevalent, their generated outputs are proliferating across the web, risking a future where machine-generated content dilutes human-authored text.
Judging the Judges: A Collection of LLM-Generated Relevance Judgements
Using Large Language Models (LLMs) for relevance assessments offers promising opportunities to improve Information Retrieval (IR), Natural Language Processing (NLP), and related fields.
KEIR @ ECIR 2025: The Second Workshop on Knowledge-Enhanced Information Retrieval
Pretrained language models (PLMs) like BERT and GPT-4 have become the foundation for modern information retrieval (IR) systems.
TrustRAG: Enhancing Robustness and Trustworthiness in RAG
Retrieval-Augmented Generation (RAG) systems enhance large language models (LLMs) by integrating external knowledge sources, enabling more accurate and contextually relevant responses tailored to user queries.
JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment
The effective training and evaluation of retrieval systems require a substantial amount of relevance judgments, which are traditionally collected from human assessors -- a process that is both costly and time-consuming.
SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval
Large-scale test collections play a crucial role in Information Retrieval (IR) research.
LLMJudge: LLMs for Relevance Judgments
The evaluation and tuning of a search system is largely based on relevance labels, which indicate whether a document is useful for a specific search and user.
Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024
The first edition of the workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) took place in July 2024, co-located with the ACM SIGIR Conference 2024 in the USA (SIGIR 2024).
Adaptive Retrieval-Augmented Generation for Conversational Systems
Despite the success of integrating large language models into the development of conversational systems, many studies have shown the effectiveness of retrieving and augmenting external knowledge for informative responses.
Understanding the Role of User Profile in the Personalization of Large Language Models
Utilizing user profiles to personalize Large Language Models (LLMs) has been shown to enhance the performance on a wide range of tasks.
Instruction Tuning With Loss Over Instructions
We observe that IM is especially beneficial when trained on datasets with lengthy instructions paired with brief outputs, or under the Superficial Alignment Hypothesis (SAH) where a small amount of training examples are used for instruction tuning.
Synthetic Test Collections for Retrieval Evaluation
Previous studies demonstrate that LLMs have the potential to generate synthetic relevance judgments for use in the evaluation of IR systems.
Clarifying the Path to User Satisfaction: An Investigation into Clarification Usefulness
Clarifying questions are an integral component of modern information retrieval systems, directly impacting user satisfaction and overall system performance.
Benchmarking LLMs via Uncertainty Quantification
The proliferation of open-source Large Language Models (LLMs) from various institutions has highlighted the urgent need for comprehensive evaluation methods.
A Toolbox for Modelling Engagement with Educational Videos
With the advancement and utility of Artificial Intelligence (AI), personalising education to a global population could be a cornerstone of new educational systems in the future.
Improving Conversational Recommendation Systems via Bias Analysis and Language-Model-Enhanced Data Augmentation
Conversational Recommendation System (CRS) is a rapidly growing research area that has gained significant attention alongside advancements in language modelling techniques.
Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting
Furthermore, we propose distilling the rewriting capabilities of LLMs into smaller models to reduce rewriting latency.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Further, it employs a fulfillment representation layer for learning how many task attributes have been fulfilled in the dialogue, an importance predictor component for calculating the importance of task attributes.
A Survey on Asking Clarification Questions Datasets in Conversational Systems
The ability to understand a user's underlying needs is critical for conversational systems, especially with limited input from users in a conversation.
Towards Asking Clarification Questions for Information Seeking on Task-Oriented Dialogues
Task-oriented dialogue systems aim at providing users with task-specific services.
Rethinking Semi-supervised Learning with Language Models
Semi-supervised learning (SSL) is a popular setting aiming to effectively utilize unlabelled data to improve model performance in downstream natural language processing (NLP) tasks.
Modeling User Satisfaction Dynamics in Dialogue via Hawkes Process
It assumes that the performance of a dialogue system can be measured by user satisfaction and uses an estimator to simulate users.
Scalable Educational Question Generation with Pre-trained Language Models
We develop \textit{EduQG}, a novel educational question generation model built by adapting a large language model.
Query-specific Variable Depth Pooling via Query Performance Prediction towards Reducing Relevance Assessment Effort
However, in this paper we argue that the annotation effort can be substantially reduced if the depth of the pool is made a variable quantity for each query, the rationale being that the number of documents relevant to the information need can widely vary across queries.
Task2KB: A Public Task-Oriented Knowledge Base
Search engines and conversational assistants are commonly used to help users complete their every day tasks such as booking travel, cooking, etc.
Pre-Training With Scientific Text Improves Educational Question Generation
With the boom of digital educational materials and scalable e-learning systems, the potential for realising AI-assisted personalised learning has skyrocketed.
MetaASSIST: Robust Dialogue State Tracking with Meta Learning
Experimental results demonstrate that all three schemes can achieve competitive performance.
Just Mix Once: Worst-group Generalization by Group Interpolation
Advances in deep learning theory have revealed how average generalization relies on superficial patterns in data.
Evaluation Metrics for Measuring Bias in Search Engine Results
However, search engine results do not necessarily cover all the viewpoints of a search query topic, and they can be biased towards a specific view since search engine results are returned based on relevance, which is calculated using many features and sophisticated algorithms where search neutrality is not necessarily the focal point.
Can Population-based Engagement Improve Personalisation? A Novel Dataset and Experiments
This work explores how population-based engagement prediction can address cold-start at scale in large learning resource collections.
Integrated Weak Learning
We introduce a label model that can learn to aggregate weak supervision sources differently for different datapoints and takes into consideration the performance of the end-model during training.
ViralBERT: A User Focused BERT-Based Approach to Virality Prediction
We collect a dataset of 330k tweets to train ViralBERT and validate the efficacy of our model using baselines from current studies in this field.
Dynamic Schema Graph Fusion Network for Multi-Domain Dialogue State Tracking
Existing approaches that have considered such relations generally fall short in: (1) fusing prior slot-domain membership relations and dialogue-aware dynamic slot relations explicitly, and (2) generalizing to unseen domains.
ASSIST: Towards Label Noise-Robust Dialogue State Tracking
In this paper, instead of improving the annotation quality further, we propose a general framework, named ASSIST (lAbel noiSe-robuSt dIalogue State Tracking), to train DST models robustly from noisy labels.
Watch Less and Uncover More: Could Navigation Tools Help Users Search and Explore Videos?
The user questionnaire revealed that participants found the Content Flow Bar helpful and enjoyable for finding relevant information in videos.
Semantic TrueLearn: Using Semantic Knowledge Graphs in Recommendation Systems
In informational recommenders, many challenges arise from the need to handle the semantic and hierarchical structure between knowledge areas.
Variational Continual Bayesian Meta-Learning
Conventional meta-learning considers a set of tasks from a stationary distribution.
Towards More Accountable Search Engines: Online Evaluation of Representation Bias
Information availability affects people's behavior and perception of the world.
Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations
Predominantly, two formulations are used for sentence-pair tasks: bi-encoders and cross-encoders.
Ranked #6 on
Semantic Textual Similarity
on STS15
Sample Efficient Model Evaluation
Labelling data is a major practical bottleneck in training and testing classifiers.
PEEK: A Large Dataset of Learner Engagement with Educational Videos
One of the main challenges in advancing this research direction is the scarcity of large, publicly available datasets.
Estimation of Fair Ranking Metrics with Incomplete Judgments
There is increasing attention to evaluating the fairness of search system ranking decisions.
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field.
TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime
The TREC Deep Learning (DL) Track studies ad hoc search in the large data regime, meaning that a large set of human-labeled training data is available.
MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation
The annotations in the training set remain unchanged (same as MultiWOZ 2. 1) to elicit robust and noise-resilient model training.
Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard
Leaderboards are a ubiquitous part of modern research in applied machine learning.
Overview of the TREC 2020 deep learning track
This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime.
Slot Self-Attentive Dialogue State Tracking
Then a stacked slot self-attention is applied on these features to learn the correlations among slots.
VLEngagement: A Dataset of Scientific Video Lectures for Evaluating Population-based Engagement
This paper introduces VLEngagement, a novel dataset that consists of content-based and video-specific features extracted from publicly available scientific video lectures and several metrics related to user engagement.
Unsupervised Few-Bits Semantic Hashing with Implicit Topics Modeling
Semantic hashing is a powerful paradigm for representing texts as compact binary hash codes.
ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search
Users of Web search engines reveal their information needs through queries and clicks, making click logs a useful asset for information retrieval.
Predicting Engagement in Video Lectures
The explosion of Open Educational Resources (OERs) in the recent years creates the demand for scalable, automatic approaches to process and evaluate OERs, with the end goal of identifying and recommending the most suitable educational materials for learners.
On the Reliability of Test Collections for Evaluating Systems of Different Types
As deep learning based models are increasingly being used for information retrieval (IR), a major challenge is to ensure the availability of test collections for measuring their quality.
Overview of the TREC 2019 deep learning track
The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime.
Towards an Integrative Educational Recommender for Lifelong Learners
One of the most ambitious use cases of computer-assisted learning is to build a recommendation system for lifelong learning.
TrueLearn: A Family of Bayesian Algorithms to Match Lifelong Learners to Open Educational Resources
The recent advances in computer-assisted learning systems and the availability of open educational resources today promise a pathway to providing cost-efficient, high-quality education to large masses of learners.
Disentangling Interpretable Generative Parameters of Random and Real-World Graphs
While a wide range of interpretable generative procedures for graphs exist, matching observed graph topologies with such procedures and choices for its parameters remains an open problem.
Self-Attentive Hawkes Processes
The proposed method adapts self-attention to fit the intensity function of Hawkes processes.
Incorporating Query Term Independence Assumption for Efficient Retrieval and Ranking using Deep Neural Networks
Deep neural IR models, in contrast, compare the whole query to the document and are, therefore, typically employed only for late stage re-ranking.
Towards Task Understanding in Visual Settings
We consider the problem of understanding real world tasks depicted in visual images.
Extracting Hierarchies of Search Tasks & Subtasks via a Bayesian Nonparametric Approach
As a result, significant amount of research has been devoted to extracting proper representations of tasks in order to enable search systems to help users complete their tasks, as well as providing the end user with better query suggestions, for better recommendations, for satisfaction prediction, and for improved personalization in terms of tasks.
