Counterfactual Evaluation of Peer-Review Assignment Policies

1 code implementation • NeurIPS 2023 • Martin Saveski, Steven Jecmen, Nihar B. Shah, Johan Ugander

We consider estimates of (i) the effect on review quality when changing weights in the assignment algorithm, e. g., weighting reviewers' bids vs. textual similarity (between the review's past papers and the submission), and (ii) the "cost of randomization", capturing the difference in expected quality between the perturbed and unperturbed optimal match.

counterfactual Off-policy evaluation +1