Search Results for author: Maarten de Rijke

Found 218 papers, 139 papers with code

AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs

no code implementations25 Oct 2024 Clemencia Siro, Yifei Yuan, Mohammad Aliannejadi, Maarten de Rijke

Human evaluation and CrowdLLM show that the AGENT-CQ - generation stage, consistently outperforms baselines in various aspects of question and answer quality.

Conversational Search Retrieval

Evaluation of Attribution Bias in Retrieval-Augmented Large Language Models

no code implementations16 Oct 2024 Amin Abolghasemi, Leif Azzopardi, Seyyed Hadi Hashemi, Maarten de Rijke, Suzan Verberne

Our results show that adding authorship information to source documents can significantly change the attribution quality of LLMs by 3% to 18%.

Attribute counterfactual +2

MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization

no code implementations10 Oct 2024 Yougang Lyu, Lingyong Yan, Zihan Wang, Dawei Yin, Pengjie Ren, Maarten de Rijke, Zhaochun Ren

As large language models (LLMs) are rapidly advancing and achieving near-human capabilities, aligning them with human values is becoming more urgent.

Information Discovery in e-Commerce

no code implementations8 Oct 2024 Zhaochun Ren, Xiangnan He, Dawei Yin, Maarten de Rijke

Methods for information discovery in e-commerce largely focus on improving the effectiveness of e-commerce search and recommender systems, on enriching and using knowledge graphs to support e-commerce, and on developing innovative question answering and bot-based solutions that help to connect people to goods and services.

Information Retrieval Knowledge Graphs +2

Cognitive Biases in Large Language Models for News Recommendation

no code implementations3 Oct 2024 Yougang Lyu, XiaoYu Zhang, Zhaochun Ren, Maarten de Rijke

Despite large language models (LLMs) increasingly becoming important components of news recommender systems, employing LLMs in such systems introduces new risks, such as the influence of cognitive biases in LLMs.

Data Augmentation Misinformation +3

Generative Retrieval Meets Multi-Graded Relevance

no code implementations27 Sep 2024 Yubao Tang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Xueqi Cheng

GR$^2$ focuses on two key components: ensuring relevant and distinct identifiers, and implementing multi-graded constrained contrastive training.

Decoder Information Retrieval +1

Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method

1 code implementation23 Sep 2024 Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng

We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text.

Proximal Ranking Policy Optimization for Practical Safety in Counterfactual Learning to Rank

no code implementations15 Sep 2024 Shashank Gupta, Harrie Oosterhuis, Maarten de Rijke

We propose a novel approach, proximal ranking policy optimization (PRPO), that provides safety in deployment without assumptions about user behavior.

counterfactual Learning-To-Rank

Real World Conversational Entity Linking Requires More Than Zeroshots

1 code implementation2 Sep 2024 Mohanna Hoveyda, Arjen P. de Vries, Maarten de Rijke, Faegheh Hasibi

Entity linking (EL) in conversations faces notable challenges in practical applications, primarily due to the scarcity of entity-annotated conversational datasets and sparse knowledge bases (KB) containing domain-specific, long-tail entities.

Entity Linking

Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank

no code implementations29 Jul 2024 Shashank Gupta, Harrie Oosterhuis, Maarten de Rijke

Our experiments show that both our novel safe doubly robust method and PRPO provide higher performance than the existing safe inverse propensity scoring approach.

counterfactual Learning-To-Rank

Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval

no code implementations16 Jul 2024 Yubao Tang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng

Generative retrieval uses differentiable search indexes to directly generate relevant document identifiers in response to a query.

Memorization Retrieval

Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective

1 code implementation9 Jul 2024 Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng

Recent advances in neural information retrieval (IR) models have significantly enhanced their effectiveness over various IR tasks.

Information Retrieval Retrieval

Robust Information Retrieval

no code implementations13 Jun 2024 Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke

Beyond effectiveness, the robustness of an information retrieval (IR) system is increasingly attracting attention.

Adversarial Robustness Information Retrieval +1

Let Me Do It For You: Towards LLM Empowered Recommendation via Tool Learning

no code implementations24 May 2024 Yuyue Zhao, Jiancan Wu, Xiang Wang, Wei Tang, Dingxian Wang, Maarten de Rijke

Through the integration of LLMs, ToolRec enables conventional recommender systems to become external tools with a natural language interface.

Attribute Recommendation Systems

Optimal Baseline Corrections for Off-Policy Contextual Bandits

1 code implementation9 May 2024 Shashank Gupta, Olivier Jeunen, Harrie Oosterhuis, Maarten de Rijke

The foundation of our framework is the derivation of an equivalent baseline correction for all of the existing control variates.

Decision Making Multi-Armed Bandits +1

QFMTS: Generating Query-Focused Summaries over Multi-Table Inputs

no code implementations8 May 2024 Weijia Zhang, Vaishali Pal, Jia-Hong Huang, Evangelos Kanoulas, Maarten de Rijke

Our approach, which comprises a table serialization module, a summarization controller, and a large language model (LLM), utilizes textual queries and multiple tables to generate query-dependent table summaries tailored to users' information needs.

Language Modelling Large Language Model

Are We Really Achieving Better Beyond-Accuracy Performance in Next Basket Recommendation?

no code implementations2 May 2024 Ming Li, Yuanna Liu, Sami Jullien, Mozhdeh Ariannezhad, Mohammad Aliannejadi, Andrew Yates, Maarten de Rijke

So far, most NBR studies have focused on optimizing the accuracy of the recommendation, whereas optimizing for beyond-accuracy metrics, e. g., item fairness and diversity remains largely unexplored.

Diversity Fairness +3

A First Look at Selection Bias in Preference Elicitation for Recommendation

1 code implementation1 May 2024 Shashank Gupta, Harrie Oosterhuis, Maarten de Rijke

Despite the fact that the extreme sparsity of preference elicitation interactions make them severely more prone to selection bias than natural interactions, the effect of selection bias in preference elicitation on the resulting recommendations has not been studied yet.

Recommendation Systems Selection bias

Ranked List Truncation for Large Language Model-based Re-Ranking

1 code implementation28 Apr 2024 Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, Maarten de Rijke

RLT is crucial for re-ranking as it can improve re-ranking efficiency by sending variable-length candidate lists to a re-ranker on a per-query basis.

Language Modelling Large Language Model +2

ExcluIR: Exclusionary Neural Information Retrieval

1 code implementation26 Apr 2024 WenHao Zhang, Mengqi Zhang, Shiguang Wu, Jiahuan Pei, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, Pengjie Ren

However, in information retrieval community, there is little research on exclusionary retrieval, where users express what they do not want in their queries.

Information Retrieval Retrieval

Large Language Models for Next Point-of-Interest Recommendation

1 code implementation19 Apr 2024 Peibo Li, Maarten de Rijke, Hao Xue, Shuang Ao, Yang song, Flora D. Salim

Our results show that the proposed framework outperforms the state-of-the-art models in all three datasets.

Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs

1 code implementation19 Apr 2024 Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke

Workers are more susceptible to user feedback on usefulness and interestingness compared to LLMs on interestingness and relevance.

Task-Oriented Dialogue Systems

Estimating the Hessian Matrix of Ranking Objectives for Stochastic Learning to Rank with Gradient Boosted Trees

1 code implementation18 Apr 2024 Jingwei Kang, Maarten de Rijke, Harrie Oosterhuis

Stochastic learning to rank (LTR) is a recent branch in the LTR field that concerns the optimization of probabilistic ranking models.

Fairness Learning-To-Rank

Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems

1 code implementation15 Apr 2024 Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke

Using the first user utterance as context leads to consistent ratings, akin to those obtained using the entire dialogue, with significantly reduced annotation effort.

Task-Oriented Dialogue Systems

Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset

2 code implementations3 Apr 2024 Philipp Hager, Romain Deffayet, Jean-Michel Renders, Onno Zoeter, Maarten de Rijke

Our experiments reveal that gains in click prediction do not necessarily translate to enhanced ranking performance on expert relevance annotations, implying that conclusions strongly depend on how success is measured in this benchmark.

Learning-To-Rank

Multi-granular Adversarial Attacks against Black-box Neural Ranking Models

no code implementations2 Apr 2024 Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng

However, limiting perturbations to a single level of granularity may reduce the flexibility of adversarial examples, thereby diminishing the potential threat of the attack.

Adversarial Attack Decision Making +4

Query Performance Prediction using Relevance Judgments Generated by Large Language Models

1 code implementation1 Apr 2024 Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, Maarten de Rijke

To solve the challenges, we devise an approximation strategy to predict an IR measure considering recall and propose to fine-tune open-source LLMs using human-labeled relevance judgments.

Information Retrieval Language Modelling +2

Generative Retrieval as Multi-Vector Dense Retrieval

1 code implementation31 Mar 2024 Shiguang Wu, Wenda Wei, Mengqi Zhang, Zhumin Chen, Jun Ma, Zhaochun Ren, Maarten de Rijke, Pengjie Ren

Both methods compute relevance as a sum of products of query and document vectors and an alignment matrix.

Decoder Retrieval

Are Large Language Models Good at Utility Judgments?

1 code implementation28 Mar 2024 Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng

Retrieval-augmented generation (RAG) is considered to be a promising approach to alleviate the hallucination issue of large language models (LLMs), and it has received widespread attention from researchers recently.

Answer Generation Benchmarking +5

CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems

no code implementations27 Mar 2024 Amin Abolghasemi, Zhaochun Ren, Arian Askari, Mohammad Aliannejadi, Maarten de Rijke, Suzan Verberne

In this work, we leverage large language models (LLMs) and unlock their ability to generate satisfaction-aware counterfactual dialogues to augment the set of original dialogues of a test collection.

counterfactual Data Augmentation +1

RankingSHAP -- Listwise Feature Attribution Explanations for Ranking Models

1 code implementation24 Mar 2024 Maria Heuss, Maarten de Rijke, Avishek Anand

We evaluate RankingSHAP for commonly used learning-to-rank datasets to showcase the more nuanced use of an attribution method while highlighting the limitations of selection-based explanations.

Learning-To-Rank valid

Listwise Generative Retrieval Models via a Sequential Learning Process

1 code implementation19 Mar 2024 Yubao Tang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Xueqi Cheng

Specifically, we view the generation of a ranked docid list as a sequence learning process: at each step we learn a subset of parameters that maximizes the corresponding generation likelihood of the $i$-th docid given the (preceding) top $i-1$ docids.

Retrieval

Measuring Bias in a Ranked List using Term-based Representations

no code implementations9 Mar 2024 Amin Abolghasemi, Leif Azzopardi, Arian Askari, Maarten de Rijke, Suzan Verberne

With TExFAIR, we extend the AWRF framework to allow for the evaluation of fairness in settings with term-based representations of groups in documents in a ranked list.

Document Ranking Fairness +1

MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning

1 code implementation27 Feb 2024 Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, Jiahuan Pei

Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring pre-trained large language models (LLMs), especially as the models' scale and the diversity of tasks increase.

Diversity Instruction Following +2

Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning

1 code implementation27 Feb 2024 Maurits Bleeker, Mariya Hendriksen, Andrew Yates, Maarten de Rijke

Hence, contrastive losses are not sufficient to learn task-optimal representations, i. e., representations that contain all task-relevant information shared between the image and associated captions.

Contrastive Learning Representation Learning

Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control

1 code implementation27 Feb 2024 Thong Nguyen, Mariya Hendriksen, Andrew Yates, Maarten de Rijke

Our proposed approach efficiently transforms dense vectors from a frozen dense model into sparse lexical vectors.

Image Retrieval Text Retrieval

CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks

1 code implementation26 Feb 2024 Jiafeng Guo, Changjiang Zhou, Ruqing Zhang, Jiangui Chen, Maarten de Rijke, Yixing Fan, Xueqi Cheng

Very recently, a pre-trained generative retrieval model for KILTs, named CorpusBrain, was proposed and reached new state-of-the-art retrieval performance.

Retrieval

Query Augmentation by Decoding Semantics from Brain Signals

1 code implementation24 Feb 2024 Ziyi Ye, Jingtao Zhan, Qingyao Ai, Yiqun Liu, Maarten de Rijke, Christina Lioma, Tuukka Ruotsalo

If the quality of the initially retrieved documents is low, then the effectiveness of query augmentation would be limited as well.

Document Ranking

KnowTuning: Knowledge-aware Fine-tuning for Large Language Models

2 code implementations17 Feb 2024 Yougang Lyu, Lingyong Yan, Shuaiqiang Wang, Haibo Shi, Dawei Yin, Pengjie Ren, Zhumin Chen, Maarten de Rijke, Zhaochun Ren

To address these problems, we propose a knowledge-aware fine-tuning (KnowTuning) method to improve fine-grained and coarse-grained knowledge awareness of LLMs.

Question Answering

Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search

1 code implementation12 Feb 2024 Yifei Yuan, Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke, Wai Lam

Therefore, we propose to add images to clarifying questions and formulate the novel task of asking multimodal clarifying questions in open-domain, mixed-initiative conversational search systems.

4k Conversational Search

Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Off

no code implementations16 Dec 2023 Yu-An Liu, Ruqing Zhang, Mingkun Zhang, Wei Chen, Maarten de Rijke, Jiafeng Guo, Xueqi Cheng

We decompose the robust ranking error into two components, i. e., a natural ranking error for effectiveness evaluation and a boundary ranking error for assessing adversarial robustness.

Adversarial Robustness Information Retrieval

SARDINE: A Simulator for Automated Recommendation in Dynamic and Interactive Environments

1 code implementation28 Nov 2023 Romain Deffayet, Thibaut Thonet, Dongyoon Hwang, Vassilissa Lehoux, Jean-Michel Renders, Maarten de Rijke

Simulators can provide valuable insights for researchers and practitioners who wish to improve recommender systems, because they allow one to easily tweak the experimental setup in which recommender systems operate, and as a result lower the cost of identifying general trends and uncovering novel findings about the candidate methods.

counterfactual Learning-To-Rank +1

Language Generation from Brain Recordings

1 code implementation16 Nov 2023 Ziyi Ye, Qingyao Ai, Yiqun Liu, Maarten de Rijke, Min Zhang, Christina Lioma, Tuukka Ruotsalo

Inspired by recent research that revealed associations between the brain and the large computational language models, we propose a generative language BCI that utilizes the capacity of a large language model (LLM) jointly with a semantic brain decoder to directly generate language from functional magnetic resonance imaging (fMRI) input.

Decoder Language Modelling +3

Learning Robust Sequential Recommenders through Confident Soft Labels

1 code implementation4 Nov 2023 Shiguang Wu, Xin Xin, Pengjie Ren, Zhumin Chen, Jun Ma, Maarten de Rijke, Zhaochun Ren

CSRec contains a teacher module that generates high-quality and confident soft labels and a student module that acts as the target recommender and is trained on the combination of dense, soft labels and sparse, one-hot labels.

Multi-class Classification Sequential Recommendation

Hierarchical Forecasting at Scale

1 code implementation19 Oct 2023 Olivier Sprangers, Wander Wadman, Sebastian Schelter, Maarten de Rijke

We implement our sparse hierarchical loss function within an existing forecasting model at bol, a large European e-commerce platform, resulting in an improved forecasting performance of 2% at the product level.

Production Forecasting Time Series

From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification

1 code implementation18 Oct 2023 Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng

We argue that, rather than relevance, for FV we need to focus on the utility that a claim verifier derives from the retrieved evidence.

Fact Verification Retrieval

Generalizing Few-Shot Named Entity Recognizers to Unseen Domains with Type-Related Features

1 code implementation15 Oct 2023 Zihan Wang, Ziqi Zhao, Zhumin Chen, Pengjie Ren, Maarten de Rijke, Zhaochun Ren

To address this limitation, recent studies enable generalization to an unseen target domain with only a few labeled examples using data augmentation techniques.

Data Augmentation few-shot-ner +5

Predictive Uncertainty-based Bias Mitigation in Ranking

1 code implementation18 Sep 2023 Maria Heuss, Daniel Cohen, Masoud Mansoury, Maarten de Rijke, Carsten Eickhoff

Prior work on bias mitigation often assumes that ranking scores, which correspond to the utility that a document holds for a user, can be accurately determined.

Fairness

Continual Learning for Generative Retrieval over Dynamic Corpora

no code implementations29 Aug 2023 Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, Xueqi Cheng

We put forward a novel Continual-LEarner for generatiVE Retrieval (CLEVER) model and make two major contributions to continual learning for GR: (i) To encode new documents into docids with low computational cost, we present Incremental Product Quantization, which updates a partial quantization codebook according to two adaptive thresholds; and (ii) To memorize new documents for querying without forgetting previous knowledge, we propose a memory-augmented learning mechanism, to form meaningful connections between old and new documents.

Continual Learning Quantization +1

Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Method

no code implementations19 Aug 2023 Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, Xueqi Cheng

The AREA task is meant to trick DR models into retrieving a target document that is outside the initial set of candidate documents retrieved by the DR model in response to a query.

Adversarial Attack Attribute +2

The Impact of Group Membership Bias on the Quality and Fairness of Exposure in Ranking

no code implementations5 Aug 2023 Ali Vardasbi, Maarten de Rijke, Fernando Diaz, Mostafa Dehghani

With group membership bias, the utility of the sensitive groups is under-estimated, hence, without correcting for this bias, a supposedly fair ranking is not truly fair.

Fairness Learning-To-Rank +1

Masked and Swapped Sequence Modeling for Next Novel Basket Recommendation in Grocery Shopping

1 code implementation2 Aug 2023 Ming Li, Mozhdeh Ariannezhad, Andrew Yates, Maarten de Rijke

In next basket recommendation (NBR), it is useful to distinguish between repeat items, i. e., items that a user has consumed before, and explore items, i. e., items that a user has not consumed before.

Next-basket recommendation

Answering Ambiguous Questions via Iterative Prompting

1 code implementation8 Jul 2023 Weiwei Sun, Hengyi Cai, Hongshen Chen, Pengjie Ren, Zhumin Chen, Maarten de Rijke, Zhaochun Ren

To provide feasible answers to an ambiguous question, one approach is to directly predict all valid answers, but this can struggle with balancing relevance and diversity.

Diversity Open-Domain Question Answering +1

Distributional Reinforcement Learning with Dual Expectile-Quantile Regression

no code implementations26 May 2023 Sami Jullien, Romain Deffayet, Jean-Michel Renders, Paul Groth, Maarten de Rijke

Motivated by the efficiency of $L_2$-based learning, we propose to jointly learn expectiles and quantiles of the return distribution in a way that allows efficient learning while keeping an estimate of the full distribution of returns.

Continuous Control Distributional Reinforcement Learning +4

MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering

1 code implementation22 May 2023 Vaishali Pal, Andrew Yates, Evangelos Kanoulas, Maarten de Rijke

Recent advances in tabular question answering (QA) with large language models are constrained in their coverage and only answer questions over a single table.

Question Answering

Improving Implicit Feedback-Based Recommendation through Multi-Behavior Alignment

1 code implementation9 May 2023 Xin Xin, Xiangyuan Liu, Hanbing Wang, Pengjie Ren, Zhumin Chen, Jiahuan Lei, Xinlei Shi, Hengliang Luo, Joemon Jose, Maarten de Rijke, Zhaochun Ren

Recommender systems that learn from implicit feedback often use large volumes of a single type of implicit user feedback, such as clicks, to enhance the prediction of sparse target behavior such as purchases.

Denoising Open-Ended Question Answering +2

On the Impact of Outlier Bias on User Clicks

1 code implementation1 May 2023 Fatemeh Sarvi, Ali Vardasbi, Mohammad Aliannejadi, Sebastian Schelter, Maarten de Rijke

We therefore propose an outlier-aware click model that accounts for both outlier and position bias, called outlier-aware position-based model ( OPBM).

counterfactual Learning-To-Rank +1

A Unified Generative Retriever for Knowledge-Intensive Language Tasks via Prompt Learning

1 code implementation28 Apr 2023 Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yiqun Liu, Yixing Fan, Xueqi Cheng

Learning task-specific retrievers that return relevant contexts at an appropriate level of semantic granularity, such as a document retriever, passage retriever, sentence retriever, and entity retriever, may help to achieve better performance on the end-to-end task.

Retrieval Sentence

Topic-oriented Adversarial Attacks against Black-box Neural Ranking Models

1 code implementation28 Apr 2023 Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, Xueqi Cheng

In this paper, we focus on a more general type of perturbation and introduce the topic-oriented adversarial ranking attack task against NRMs, which aims to find an imperceptible perturbation that can promote a target document in ranking for a group of queries with the same topic.

Information Retrieval Retrieval

Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization

1 code implementation26 Apr 2023 Shashank Gupta, Harrie Oosterhuis, Maarten de Rijke

For the CLTR field, our novel exposure-based risk minimization method enables practitioners to adopt CLTR methods in a safer manner that mitigates many of the risks attached to previous methods.

counterfactual Learning-To-Rank

An Offline Metric for the Debiasedness of Click Models

2 code implementations19 Apr 2023 Romain Deffayet, Philipp Hager, Jean-Michel Renders, Maarten de Rijke

We prove that debiasedness is a necessary condition for recovering unbiased and consistent relevance scores and for the invariance of click prediction under covariate shift.

counterfactual Learning-To-Rank +1

Generative Slate Recommendation with Reinforcement Learning

no code implementations20 Jan 2023 Romain Deffayet, Thibaut Thonet, Jean-Michel Renders, Maarten de Rijke

Our findings suggest that representation learning using generative models is a promising direction towards generalizable RL-based slate recommendation.

Recommendation Systems reinforcement-learning +3

Scene-centric vs. Object-centric Image-Text Cross-modal Retrieval: A Reproducibility Study

1 code implementation12 Jan 2023 Mariya Hendriksen, Svitlana Vakulenko, Ernst Kuiper, Maarten de Rijke

Additionally, we select two scene-centric datasets, and three object-centric datasets, and determine the relative performance of the selected models on these datasets.

Cross-Modal Retrieval Object +1

Modeling Sequential Recommendation as Missing Information Imputation

1 code implementation4 Jan 2023 Yujie Lin, Zhumin Chen, Zhaochun Ren, Chenyang Wang, Qiang Yan, Maarten de Rijke, Xiuzhen Cheng, Pengjie Ren

To address the limitation of sequential recommenders with side information, we define a way to fuse side information and alleviate the problem of missing side information by proposing a unified task, namely the missing information imputation (MII), which randomly masks some feature fields in a given sequence of items, including item IDs, and then forces a predictive model to recover them.

Imputation Sequential Recommendation

Offline Evaluation for Reinforcement Learning-based Recommendation: A Critical Issue and Some Alternatives

no code implementations3 Jan 2023 Romain Deffayet, Thibaut Thonet, Jean-Michel Renders, Maarten de Rijke

In this paper, we argue that the paradigm commonly adopted for offline evaluation of sequential recommender systems is unsuitable for evaluating reinforcement learning-based recommenders.

Offline RL Recommendation Systems +3

Contrastive Learning Reduces Hallucination in Conversations

1 code implementation20 Dec 2022 Weiwei Sun, Zhengliang Shi, Shen Gao, Pengjie Ren, Maarten de Rijke, Zhaochun Ren

MixCL effectively reduces the hallucination of LMs in conversations and achieves the highest performance among LM-based dialogue agents in terms of relevancy and factuality.

Contrastive Learning Hallucination

Feature-Level Debiased Natural Language Understanding

1 code implementation11 Dec 2022 Yougang Lyu, Piji Li, Yechang Yang, Maarten de Rijke, Pengjie Ren, Yukun Zhao, Dawei Yin, Zhaochun Ren

We also propose a dynamic negative sampling strategy to capture the dynamic influence of biases by employing a bias-only model to dynamically select the most similar biased negative samples.

Contrastive Learning Natural Language Understanding

RADio -- Rank-Aware Divergence Metrics to Measure Normative Diversity in News Recommendations

no code implementations17 Sep 2022 Sanne Vrijenhoek, Gabriel Bénédict, Mateo Gutierrez Granada, Daan Odijk, Maarten de Rijke

In traditional recommender system literature, diversity is often seen as the opposite of similarity, and typically defined as the distance between identified topics, categories or word models.

Diversity Recommendation Systems

Certified Robustness to Word Substitution Ranking Attack for Neural Ranking Models

1 code implementation14 Sep 2022 Chen Wu, Ruqing Zhang, Jiafeng Guo, Wei Chen, Yixing Fan, Maarten de Rijke, Xueqi Cheng

A ranking model is said to be Certified Top-$K$ Robust on a ranked list when it is guaranteed to keep documents that are out of the top $K$ away from the top $K$ under any attack.

Information Retrieval Retrieval

Intersection of Parallels as an Early Stopping Criterion

1 code implementation19 Aug 2022 Ali Vardasbi, Maarten de Rijke, Mostafa Dehghani

Using this result, we propose to train two parallel instances of a linear model, initialized with different random seeds, and use their intersection as a signal to detect overfitting.

counterfactual Learning-To-Rank

Towards the Use of Saliency Maps for Explaining Low-Quality Electrocardiograms to End Users

no code implementations6 Jul 2022 Ana Lucic, Sheeraz Ahmad, Amanda Furtado Brinhosa, Vera Liao, Himani Agrawal, Umang Bhatt, Krishnaram Kenthapadi, Alice Xiang, Maarten de Rijke, Nicholas Drabowski

In this paper, we report on ongoing work regarding (i) the development of an AI system for flagging and explaining low-quality medical images in real-time, (ii) an interview study to understand the explanation needs of stakeholders using the AI system at OurCompany, and, (iii) a longitudinal user study design to examine the effect of including explanations on the workflow of the technicians in our clinics.

Explainable Artificial Intelligence (XAI)

Debiasing Learning for Membership Inference Attacks Against Recommender Systems

1 code implementation24 Jun 2022 Zihan Wang, Na Huang, Fei Sun, Pengjie Ren, Zhumin Chen, Hengliang Luo, Maarten de Rijke, Zhaochun Ren

To address the above limitations, we propose a Debiasing Learning for Membership Inference Attacks against recommender systems (DL-MIA) framework that has four main components: (1) a difference vector generator, (2) a disentangled encoder, (3) a weight estimator, and (4) an attack model.

Recommendation Systems

A Simulation Environment and Reinforcement Learning Method for Waste Reduction

no code implementations30 May 2022 Sami Jullien, Mozhdeh Ariannezhad, Paul Groth, Maarten de Rijke

We frame inventory restocking as a new reinforcement learning task that exhibits stochastic behavior conditioned on the agent's actions, making the environment partially observable.

Distributional Reinforcement Learning reinforcement-learning +2

Fairness of Exposure in Light of Incomplete Exposure Estimation

1 code implementation25 May 2022 Maria Heuss, Fatemeh Sarvi, Maarten de Rijke

In this work, we discuss how to approach fairness of exposure in cases where the policy contains rankings of which, due to inter-item dependencies, we cannot reliably estimate the exposure distribution.

Fairness

State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study

1 code implementation10 May 2022 Jin Huang, Harrie Oosterhuis, Bunyamin Cetinkaya, Thijs Rood, Maarten de Rijke

In response to these shortcomings, we reproduce and expand on the existing comparison of attention-based state encoders (1) in the publicly available debiased RL4Rec SOFA simulator with (2) a different RL method, (3) more state encoders, and (4) a different dataset.

reinforcement-learning Reinforcement Learning (RL)

Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrieval

1 code implementation28 Apr 2022 Maurits Bleeker, Andrew Yates, Maarten de Rijke

We add an additional decoder to the contrastive ICR framework, to reconstruct the input caption in a latent space of a general-purpose sentence encoder, which prevents the image and caption encoder from suppressing predictive features.

Contrastive Learning Retrieval +1

Probabilistic Permutation Graph Search: Black-Box Optimization for Fairness in Ranking

1 code implementation28 Apr 2022 Ali Vardasbi, Fatemeh Sarvi, Maarten de Rijke

Different from PL, where pointwise logits are used as the distribution parameters, in PPG pairwise inversion probabilities together with a reference permutation construct the distribution.

Fairness Learning-To-Rank

Understanding User Satisfaction with Task-oriented Dialogue Systems

no code implementations26 Apr 2022 Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke

What is the influence of user experience on the user satisfaction rating of TDS as opposed to, or in addition to, utility?

Conversational Recommendation Task-Oriented Dialogue Systems

Summarization with Graphical Elements

1 code implementation15 Apr 2022 Maartje ter Hoeve, Julia Kiseleva, Maarten de Rijke

Motivated from these two angles, we propose a new task: summarization with graphical elements, and we verify that these summaries are helpful for a critical mass of people.

Text Summarization

Parameter-Efficient Abstractive Question Answering over Tables or Text

1 code implementation dialdoc (ACL) 2022 Vaishali Pal, Evangelos Kanoulas, Maarten de Rijke

In this work, we study parameter-efficient abstractive QA in encoder-decoder models over structured tabular data and unstructured textual data using only 1. 5% additional parameters for each modality.

abstractive question answering Decoder +1

PRADA: Practical Black-Box Adversarial Attacks against Neural Ranking Models

no code implementations4 Apr 2022 Chen Wu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng

We focus on the decision-based black-box attack setting, where the attackers cannot directly get access to the model information, but can only query the target model to obtain the rank positions of the partial retrieved list.

Document Ranking Information Retrieval +1

Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems

2 code implementations2 Apr 2022 Weiwei Sun, Shuyu Guo, Shuo Zhang, Pengjie Ren, Zhumin Chen, Maarten de Rijke, Zhaochun Ren

Employing existing user simulators to evaluate TDSs is challenging as user simulators are primarily designed to optimize dialogue policies for TDSs and have limited evaluation capabilities.

Conversational Recommendation Task-Oriented Dialogue Systems

Do Lessons from Metric Learning Generalize to Image-Caption Retrieval?

1 code implementation14 Feb 2022 Maurits Bleeker, Maarten de Rijke

Recent progress in metric learning has given rise to new loss functions that outperform the triplet loss on tasks such as image retrieval and representation learning.

Image Retrieval Metric Learning +3

Understanding and Mitigating the Effect of Outliers in Fair Ranking

1 code implementation21 Dec 2021 Fatemeh Sarvi, Maria Heuss, Mohammad Aliannejadi, Sebastian Schelter, Maarten de Rijke

We formalize outlierness in a ranking, show that outliers are present in realistic datasets, and present the results of an eye-tracking study, showing that users scanning order and the exposure of items are influenced by the presence of outliers.

Fairness Outlier Detection +1

Parameter Efficient Deep Probabilistic Forecasting

1 code implementation6 Dec 2021 Olivier Sprangers, Sebastian Schelter, Maarten de Rijke

However, these methods require a large number of parameters to be learned, which imposes high memory requirements on the computational resources for training such models.

Probabilistic Time Series Forecasting Time Series

It Is Different When Items Are Older: Debiasing Recommendations When Selection Bias and User Preferences Are Dynamic

1 code implementation24 Nov 2021 Jin Huang, Harrie Oosterhuis, Maarten de Rijke

We theoretically show that in a dynamic scenario in which both the selection bias and user preferences are dynamic, existing debiasing methods are no longer unbiased.

Recommendation Systems Selection bias

Reproducibility as a Mechanism for Teaching Fairness, Accountability, Confidentiality, and Transparency in Artificial Intelligence

no code implementations1 Nov 2021 Ana Lucic, Maurits Bleeker, Sami Jullien, Samarth Bhargav, Maarten de Rijke

In this work, we explain the setup for a technical, graduate-level course on Fairness, Accountability, Confidentiality, and Transparency in Artificial Intelligence (FACT-AI) at the University of Amsterdam, which teaches FACT-AI concepts through the lens of reproducibility.

Fairness

A Next Basket Recommendation Reality Check

1 code implementation29 Sep 2021 Ming Li, Sami Jullien, Mozhdeh Ariannezhad, Maarten de Rijke

We propose a set of metrics that measure the repeat/explore ratio and performance of NBR models.

Next-basket recommendation

ReMeDi: Resources for Multi-domain, Multi-service, Medical Dialogues

1 code implementation1 Sep 2021 Guojun Yan, Jiahuan Pei, Pengjie Ren, Zhaochun Ren, Xin Xin, Huasheng Liang, Maarten de Rijke, Zhumin Chen

(1) there is no dataset with large-scale medical dialogues that covers multiple medical services and contains fine-grained medical labels (i. e., intents, actions, slots, values), and (2) there is no set of established benchmarks for MDSs for multi-domain, multi-service medical dialogues.

Benchmarking Contrastive Learning +2

sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification

1 code implementation24 Aug 2021 Gabriel Bénédict, Vincent Koops, Daan Odijk, Maarten de Rijke

We propose a loss function, sigmoidF1, which is an approximation of the F1 score that (1) is smooth and tractable for stochastic gradient descent, (2) naturally approximates a multilabel metric, and (3) estimates label propensities and label counts.

Classification

A Human-machine Collaborative Framework for Evaluating Malevolence in Dialogues

1 code implementation ACL 2021 Yangjun Zhang, Pengjie Ren, Maarten de Rijke

HMCEval casts dialogue evaluation as a sample assignment problem, where we need to decide to assign a sample to a human or a machine for evaluation.

Dialogue Evaluation

Learning to Ask Conversational Questions by Optimizing Levenshtein Distance

1 code implementation ACL 2021 Zhongkun Liu, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Maarten de Rijke, Ming Zhou

Conversational Question Simplification (CQS) aims to simplify self-contained questions into conversational ones by incorporating some conversational characteristics, e. g., anaphora and ellipsis.

News Article Retrieval in Context for Event-centric Narrative Creation

1 code implementation In2Writing (ACL) 2022 Nikos Voskarides, Edgar Meij, Sabrina Sauer, Maarten de Rijke

Given an incomplete narrative that specifies a main event and a context, we aim to retrieve news articles that discuss relevant events that would enable the continuation of the narrative.

Retrieval

Few-Shot Electronic Health Record Coding through Graph Contrastive Learning

1 code implementation29 Jun 2021 Shanshan Wang, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Huasheng Liang, Qiang Yan, Evangelos Kanoulas, Maarten de Rijke

We seek to improve the performance for both frequent and rare ICD codes by using a contrastive graph-based EHR coding framework, CoGraph, which re-casts EHR coding as a few-shot learning task.

Contrastive Learning Few-Shot Learning

Improving Transformer-based Sequential Recommenders through Preference Editing

no code implementations23 Jun 2021 Muyang Ma, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Huasheng Liang, Jun Ma, Maarten de Rijke

By doing so, the SR model is able to learn how to identify common and unique user preferences, and thereby do better user preference extraction and representation.

Self-Supervised Learning Sequential Recommendation

Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression

1 code implementation3 Jun 2021 Olivier Sprangers, Sebastian Schelter, Maarten de Rijke

We propose Probabilistic Gradient Boosting Machines (PGBM), a method to create probabilistic predictions with a single ensemble of decision trees in a computationally efficient manner.

regression Time Series Analysis

Wizard of Search Engine: Access to Information Through Conversations with Search Engines

1 code implementation18 May 2021 Pengjie Ren, Zhongkun Liu, Xiaomeng Song, Hongtao Tian, Zhumin Chen, Zhaochun Ren, Maarten de Rijke

(2) We release a benchmark dataset, called wizard of search engine (WISE), which allows for comprehensive and in-depth research on all aspects of CIS.

Intent Detection Keyphrase Extraction +1

Improving Response Quality with Backward Reasoning in Open-domain Dialogue Systems

1 code implementation30 Apr 2021 Ziming Li, Julia Kiseleva, Maarten de Rijke

The proposed backward reasoning step pushes the model to produce more informative and coherent content because the forward generation step's output is used to infer the dialogue context in the backward direction.

Decoder Response Generation

A Cooperative Memory Network for Personalized Task-oriented Dialogue Systems with Incomplete User Profiles

1 code implementation16 Feb 2021 Jiahuan Pei, Pengjie Ren, Maarten de Rijke

We find that CoMemNN is able to enrich user profiles effectively, which results in an improvement of 3. 06% in terms of response selection accuracy compared to state-of-the-art methods.

Attribute Task-Oriented Dialogue Systems

Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank

1 code implementation11 Feb 2021 Harrie Oosterhuis, Maarten de Rijke

We introduce the Generalization and Specialization (GENSPEC) algorithm, a robust feature-based counterfactual LTR method that pursues per-query memorization when it is safe to do so.

counterfactual Learning-To-Rank +1

CF-GNNExplainer: Counterfactual Explanations for Graph Neural Networks

1 code implementation5 Feb 2021 Ana Lucic, Maartje ter Hoeve, Gabriele Tolomei, Maarten de Rijke, Fabrizio Silvestri

In this work, we propose a method for generating CF explanations for GNNs: the minimal perturbation to the input (graph) data such that the prediction changes.

counterfactual

Abstractive Opinion Tagging

1 code implementation18 Jan 2021 Qintong Li, Piji Li, Xinyi Li, Zhaochun Ren, Zhumin Chen, Maarten de Rijke

In this paper, we propose the abstractive opinion tagging task, where systems have to automatically generate a ranked list of opinion tags that are based on, but need not occur in, a given set of user-generated reviews.

Sentence

Analyzing and Predicting Purchase Intent in E-commerce: Anonymous vs. Identified Customers

no code implementations16 Dec 2020 Mariya Hendriksen, Ernst Kuiper, Pim Nauts, Sebastian Schelter, Maarten de Rijke

In this paper, we focus on purchase prediction for both anonymous and identified sessions on an e-commerce platform.

Descriptive

What Makes a Good and Useful Summary? Incorporating Users in Automatic Summarization Research

no code implementations NAACL 2022 Maartje ter Hoeve, Julia Kiseleva, Maarten de Rijke

Motivated by our findings, we present ways to mitigate this mismatch in future research on automatic summarization: we propose research directions that impact the design, the development and the evaluation of automatically generated summaries.

Survey Text Summarization

Unifying Online and Counterfactual Learning to Rank

1 code implementation8 Dec 2020 Harrie Oosterhuis, Maarten de Rijke

With the introduction of the intervention-aware estimator, we aim to bridge the online/counterfactual LTR division as it is shown to be highly effective in both online and counterfactual scenarios.

counterfactual Learning-To-Rank +1

Conversational Browsing

1 code implementation7 Dec 2020 Svitlana Vakulenko, Vadim Savenkov, Maarten de Rijke

How can we better understand the mechanisms behind multi-turn information seeking dialogues?

Information Retrieval Question Answering +1

Mixed Information Flow for Cross-domain Sequential Recommendations

1 code implementation1 Dec 2020 Muyang Ma, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Lifan Zhao, Jun Ma, Maarten de Rijke

One of the key challenges in cross-domain sequential recommendation is to grasp and transfer the flow of information from multiple domains so as to promote recommendations in all domains.

Sequential Recommendation Transfer Learning

When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank

1 code implementation24 Aug 2020 Ali Vardasbi, Harrie Oosterhuis, Maarten de Rijke

Our main contribution is a new estimator based on affine corrections: it both reweights clicks and penalizes items displayed on ranks with high trust bias.

counterfactual Learning-To-Rank +1

Diversifying Task-oriented Dialogue Response Generation with Prototype Guided Paraphrasing

1 code implementation7 Aug 2020 Phillip Lippe, Pengjie Ren, Hinda Haned, Bart Voorn, Maarten de Rijke

Instead of generating a response from scratch, P2-Net generates system responses by paraphrasing template-based responses.

Diversity Response Generation +1

Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking

1 code implementation24 Jul 2020 Harrie Oosterhuis, Maarten de Rijke

LogOpt turns the counterfactual approach - which is indifferent to the logging policy - into an online approach, where the algorithm decides what rankings to display.

counterfactual Position +1

A Comparison of Supervised Learning to Match Methods for Product Search

1 code implementation20 Jul 2020 Fatemeh Sarvi, Nikos Voskarides, Lois Mooiman, Sebastian Schelter, Maarten de Rijke

As recent learning to match methods have made important advances in bridging the vocabulary gap for these traditional IR areas, we investigate their potential in the context of product search.

ARC Attribute +3

Pre-Trained Models for Heterogeneous Information Networks

no code implementations7 Jul 2020 Yang Fang, Xiang Zhao, Yifan Chen, Weidong Xiao, Maarten de Rijke

We propose a self-supervised pre-training and fine-tuning framework, PF-HIN, to capture the features of a heterogeneous information network.

Clustering Link Prediction +3

Optimizing Interactive Systems via Data-Driven Objectives

no code implementations19 Jun 2020 Ziming Li, Julia Kiseleva, Alekh Agarwal, Maarten de Rijke, Ryen W. White

Effective optimization is essential for real-world interactive systems to provide a satisfactory user experience in response to changing user behavior.

Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank

no code implementations25 May 2020 Ali Vardasbi, Maarten de Rijke, Ilya Markov

Unbiased CLTR requires click propensities to compensate for the difference between user clicks and true relevance of search results via IPS.

counterfactual Learning-To-Rank

An Analysis of Mixed Initiative and Collaboration in Information-Seeking Dialogues

no code implementations25 May 2020 Svitlana Vakulenko, Evangelos Kanoulas, Maarten de Rijke

The ability to engage in mixed-initiative interaction is one of the core requirements for a conversational search system.

Conversational Search

Query Resolution for Conversational Search with Limited Supervision

1 code implementation24 May 2020 Nikos Voskarides, Dan Li, Pengjie Ren, Evangelos Kanoulas, Maarten de Rijke

Context from the conversational history can be used to arrive at a better expression of the current turn query, defined as the task of query resolution.

Conversational Search Passage Retrieval +1

Accelerated Convergence for Counterfactual Learning to Rank

1 code implementation21 May 2020 Rolf Jagerman, Maarten de Rijke

Counterfactual Learning to Rank (LTR) algorithms learn a ranking model from logged user interactions, often collected using a production system.

counterfactual Learning-To-Rank

Policy-Aware Unbiased Learning to Rank for Top-k Rankings

1 code implementation18 May 2020 Harrie Oosterhuis, Maarten de Rijke

We prove that the policy-aware estimator is unbiased if every relevant item has a non-zero probability to appear in the top-k ranking.

counterfactual Learning-To-Rank +1

Rethinking Item Importance in Session-based Recommendation

no code implementations9 May 2020 Zhiqiang Pan, Fei Cai, Yanxiang Ling, Maarten de Rijke

We employ a modified self-attention mechanism to estimate item importance in a session, which is then used to predict user's long-term preference.

Session-Based Recommendations

WN-Salience: A Corpus of News Articles with Entity Salience Annotations

no code implementations LREC 2020 Chuan Wu, Evangelos Kanoulas, Maarten de Rijke, Wei Lu

To support research on entity salience, we present a new dataset, the WikiNews Salience dataset (WN-Salience), which can be used to benchmark tasks such as entity salience detection and salient entity linking.

Entity Linking

Conversations with Search Engines: SERP-based Conversational Response Generation

1 code implementation29 Apr 2020 Pengjie Ren, Zhumin Chen, Zhaochun Ren, Evangelos Kanoulas, Christof Monz, Maarten de Rijke

In this paper, we address the problem of answering complex information needs by conversing conversations with search engines, in the sense that users can express their queries in natural language, and directly receivethe information they need from a short system response in a conversational manner.

Conversational Response Generation Conversational Search +1