no code implementations • 16 Oct 2023 • Adam Roegiest, Radha Chitta, Jonathan Donnelly, Maya Lash, Alexandra Vtyurina, François Longtin
In many legal processes being able to action on the concrete implication of a legal question can be valuable to automating human review or signalling certain conditions (e. g., alerts around automatic renewal).
1 code implementation • 31 Aug 2021 • Negar Arabzadeh, Alexandra Vtyurina, Xinyi Yan, Charles L. A. Clarke
To test this observation, we employed crowdsourced workers to make preference judgments between the top item returned by a modern neural ranking stack and a judged relevant item.
2 code implementations • 22 Jul 2020 • Charles L. A. Clarke, Alexandra Vtyurina, Mark D. Smucker
To measure the performance of a ranker, we compare its output to this preferred ordering by applying a rank similarity measure. We demonstrate the practical feasibility of this approach by crowdsourcing partial preferences for the TREC 2019 Conversational Assistance Track, replacing NDCG with a new measure named "compatibility".