Registration for RC2021 Spring

Registration and project submissions will be handled by our OpenReview Portal.

Claiming a paper

For this challenge, you first have to submit a claim on a paper from the list of accepted papers from NeurIPS 2020, ICLR 2020, ICML 2020, ACL 2020, EMNLP 2020, CVPR 2020 and ECCV 2020. In the portal, you will be able to see a list of papers which have been “claimed” and a list of papers which have not been claimed, or “unclaimed” for the challenge. We encourage you to select papers which are yet to be claimed, however, you can also claim a paper which has been claimed by a team.

Once you submit the form to claim a paper, we will post an anonymous comment to the paper which will show the claim made by your team. Thus, while claiming a paper, you can see how many teams are working on it.

Expect to complete the following information when you submit your claim:

• University / Institution (public)
• Reproducibility Plan: We expect you to write a short proposal (500 words max) of your reproducibility plan of action for the selected paper - see the Task Description for more details. In this proposal, write concisely how you would approach the problem. This proposal essentially helps to narrow down your deliverables regarding the challenge. This proposal will not be made public, neither will it be used in our reviews.

Preparing your report

From this edition of the ML Reproducibility Challenge, we introduce Reproducibility Summary templates for the reports. The first page of your report must incorporate this summary template, which should consist of the following sections:

• Scope of reproducibility: Briefly define the claim of the paper that you intend to tackle in this report
• Methodology: Briefly state the overall methodology used in your approach
• Results: Concisely state the results / findings of the reproducibility study
• What was easy: State which part of the reproducibility study was easy to do.
• What was difficult: State the difficulties that you encountered while performing the reproducibility study.

We provide a sample Reproducibility Summary template here. The first page of the template is mandatory and you can organize the rest of the report as you deem fit. We have also released a recommended outline of the report which you can use to incorporate your findings.

Submitting your report

We will use the same OpenReview Portal to submit your Reproducibility projects. Click on “Add Reproducibility Challenge Report” to submit your work. In the field of “Paper URL”, paste the paper URL of the forum for the paper. In the Abstract, copy paste the Reproducibility Summary from your report.

Please check Important Dates for the deadline for submission of your report. Your report should be of maximum 8 pages excluding references, in NeurIPS style format. You can also submit supplementary materials. Code submission is mandatory along with your supplementary materials.

Reviewing Criteria

We will be selecting reproducibility reports which present a clear, well motivated, and thorough reproducibility effort. The report should very briefly discuss the main ideas of the paper and not simply reiterate the main work. Your report should clearly mention the key findings of your effort, and if necessary include discussion / commentary / suggestions to the readers re-implementing this work on its replicability / complexity. Your report must contain the Reproducibility Summary in the first page as defined in the previous section.

Reports will be reviewed on the following criteria:

• Concise Problem Statement
• Reproducibility code
• Communication with authors
• Hyperparameter Search
• Ablation Study
• Discussion on results
• Recommendations for reproducibility
• Clarity of writing