Registration for RC2021 Fall Edition

Registration and project submissions will be handled by our OpenReview Portal.

New! You no longer need to submit a claim of your Reproducibility Report!

Preparing your report

As our last edition of the ML Reproducibility Challenge, we use Reproducibility Summary templates for the reports. The first page of your report must incorporate this summary template, which should consist of the following sections:

• Scope of reproducibility: Briefly define the claim of the paper that you intend to tackle in this report
• Methodology: Briefly state the overall methodology used in your approach
• Results: Concisely state the results / findings of the reproducibility study
• What was easy: State which part of the reproducibility study was easy to do.
• What was difficult: State the difficulties that you encountered while performing the reproducibility study.

We provide a sample Reproducibility Summary template here. The first page of the template is mandatory and you can organize the rest of the report as you deem fit. We have also released a recommended outline of the report which you can use to incorporate your findings.

Submitting your report

We will use the same OpenReview Portal to submit your Reproducibility projects. Click on “Add Reproducibility Challenge Report” to submit your work. In the field of “Paper URL”, paste the paper URL of the paper (from the appropriate conference proceedings). In the Abstract, copy paste the Reproducibility Summary from your report.

Please check Important Dates for the deadline for submission of your report. Your report should be of maximum 8 pages excluding references, in NeurIPS style format. You can also submit supplementary materials. Code submission is mandatory along with your supplementary materials.

Reviewing Criteria

We will be selecting reproducibility reports which present a clear, well motivated, and thorough reproducibility effort. The report should very briefly discuss the main ideas of the paper and not simply reiterate the main work. Your report should clearly mention the key findings of your effort, and if necessary include discussion / commentary / suggestions to the readers re-implementing this work on its replicability / complexity. Your report must contain the Reproducibility Summary in the first page as defined in the previous section.

Reports will be reviewed on the following criteria:

• Concise Problem Statement
• Reproducibility code
• Communication with authors
• Hyperparameter Search
• Ablation Study
• Discussion on results
• Recommendations for reproducibility
• Clarity of writing