2 code implementations • 22 Jul 2020 • Charles L. A. Clarke, Alexandra Vtyurina, Mark D. Smucker
To measure the performance of a ranker, we compare its output to this preferred ordering by applying a rank similarity measure. We demonstrate the practical feasibility of this approach by crowdsourcing partial preferences for the TREC 2019 Conversational Assistance Track, replacing NDCG with a new measure named "compatibility".
no code implementations • 23 Mar 2018 • Haotian Zhang, Gordon V. Cormack, Maura R. Grossman, Mark D. Smucker
This study uses a novel simulation framework to evaluate whether the time and effort necessary to achieve high recall using active learning is reduced by presenting the reviewer with isolated sentences, as opposed to full documents, for relevance feedback.