ML Reproducibility Challenge 2021
Welcome to the ML Reproducibility Challenge 2021 Fall Edition! This is the fifth edition of this
event, and a successor of the ML Reproducibility Challenge 2020 (and previous editions V1,
and we are excited this year to broaden our coverage
of conferences and papers to cover nine top venues of 2021, including:
The primary goal of this event is to encourage the publishing and sharing of scientific results that
are reliable and reproducible. In support of this, the objective of this challenge is to investigate
reproducibility of papers accepted for publication at top conferences by inviting members of the
community at large to select a paper, and verify the empirical results and claims in the paper by
reproducing the computational experiments, either via a new implementation or using code/data or
other information provided by the authors.
All submitted reports will be peer reviewed and shown next to the original papers on
Papers with Code.
Reports will be peer-reviewed via OpenReview. Every year, a small number of these reports,
selected for their clarity, thoroughness, correctness and insights, are selected for publication
in a special edition of the journal ReScience.
Invitation to participate
The challenge is a great event for community members to participate in shaping scientific practices
and findings in our field. We particularly encourage participation from:
- Course instructors of advanced ML, NLP, CV courses, who can use this challenge as a
course assignment or project.
- Organizers of hackathons.
- Members of ML developer communities
- ML enthusiasts everywhere!
Key dates for Fall 2021 Challenge
Announcement of the challenge : August 31st, 2021
Challenge goes LIVE : August 31st, 2021
Submission deadline (to be considered for peer review) : February 4th, 2022 (11:59PM AOE)
Author Notification deadline for journal special issue: May 15th, 2022
How to participate
Courses Participating in RC2021 Fall Edition
- DD2412 Deep Learning, Advanced. KTH (Royal Institute of Technology), Stockholm, Sweden
- CISC 867 Deep Learning, Queen's University, Ontario, Canada
- Special Topics in CSE: Advanced ML, Indian Institute of Technology, Gandhinagar, India
- FACT: Fairness, Accountability, Confidentiality and Transparency in AI, University of Amsterdam, Netherlands
- CSCI 662 -- Advanced Natural Language Processing, University of Southern California, USA
- Intelligent Systems and Interfaces, Indian Institute of Technology, Guwahati, India
- Intelligent Information Processing Topics, Tsinghua University, China
- Participating with your course? Drop us a mail to include your course here.
Top participating universities in RC2020
- Fairness, Accountability, Confidentiality and Transparency in AI, University of Amsterdam, Netherlands New! Read about their experience of participating in RC2020 in this blog post.
- CS691 Advanced Machine Learning, Indian Institute of Technology, Gandhinagar, India
- University of Waterloo, Canada
- BITS Pilani, India
- University of Wisconsin Madison, USA
- KTH Royal Institute of Technology, Stockholm, Sweden
- IFT 6268 - Self-supervised Representation Learning, Université de Montréal, Canada
- ... and many more!
Koustuv Sinha (McGill University / FAIR)
Jesse Dodge (Allen Institute for AI)
- Sasha Luccioni (Université de Montréal / Mila)
Jessica Forde (Brown University)
Joelle Pineau (McGill University / FAIR)
Robert Stojnic (Papers with Code / FAIR)
Parag Pachpute, Melisa Bok, Celeste Martinez Gomez, Mohit Uniyal, Andrew McCallum (OpenReview / University of Massachusetts Amherst)
Nicolas Rougier, Konrad Hinsen (ReScience)