ML Reproducibility Challenge 2021

Welcome to the ML Reproducibility Challenge 2021 Fall Edition! This is the fifth edition of this event, and a successor of the ML Reproducibility Challenge 2020 (and previous editions V1, V2, V3), and we are excited this year to broaden our coverage of conferences and papers to cover nine top venues of 2021, including: NeurIPS, ICML, ICLR, ACL-IJCNLP, EMNLP, CVPR, ICCV, AAAI and IJCAI.

The primary goal of this event is to encourage the publishing and sharing of scientific results that are reliable and reproducible. In support of this, the objective of this challenge is to investigate reproducibility of papers accepted for publication at top conferences by inviting members of the community at large to select a paper, and verify the empirical results and claims in the paper by reproducing the computational experiments, either via a new implementation or using code/data or other information provided by the authors.

All submitted reports will be peer reviewed and shown next to the original papers on Papers with Code. Reports will be peer-reviewed via OpenReview. Every year, a small number of these reports, selected for their clarity, thoroughness, correctness and insights, are selected for publication in a special edition of the journal ReScience. (see J1, J2).

News

  • OpenReview Portal of RC2021 Fall Edition is up and open for reprodocibility reports.

Invitation to participate

The challenge is a great event for community members to participate in shaping scientific practices and findings in our field. We particularly encourage participation from:

  • Course instructors of advanced ML, NLP, CV courses, who can use this challenge as a course assignment or project.
  • Organizers of hackathons.
  • Members of ML developer communities
  • ML enthusiasts everywhere!

Key dates for Fall 2021 Challenge

  • Announcement of the challenge : August 31st, 2021
  • Challenge goes LIVE : August 31st, 2021
  • Submission deadline (to be considered for peer review) : February 4th, 2022 (11:59PM AOE)
  • Author Notification deadline for journal special issue: May 15th, 2022

How to participate

Courses Participating in RC2021 Fall Edition

Top participating universities in RC2020

Contact Information

Organizing committee

  • Koustuv Sinha (McGill University / FAIR)
  • Jesse Dodge (Allen Institute for AI)
  • Sasha Luccioni (Université de Montréal / Mila)
  • Jessica Forde (Brown University)
  • Joelle Pineau (McGill University / FAIR)
  • Robert Stojnic (Papers with Code / FAIR)

Acknowledgements

  • Parag Pachpute, Melisa Bok, Celeste Martinez Gomez, Mohit Uniyal, Andrew McCallum (OpenReview / University of Massachusetts Amherst)
  • Nicolas Rougier, Konrad Hinsen (ReScience)