Registration and project submissions will be handled by our OpenReview Portal.
For this challenge, you first have to submit a claim on a paper from the list of accepted papers from NeurIPS 2020, ICLR 2020, ICML 2020, ACL 2020, EMNLP 2020, CVPR 2020 and ECCV 2020. In the portal, you will be able to see a list of papers which have been “claimed” and a list of papers which have not been claimed, or “unclaimed” for the challenge. We encourage you to select papers which are yet to be claimed, however, you can also claim a paper which has been claimed by a team.
Once you submit the form to claim a paper, we will post an anonymous comment to the paper which will show the claim made by your team. Thus, while claiming a paper, you can see how many teams are working on it.
Expect to complete the following information when you submit your claim:
We provide a sample Reproducibility Summary template here. The first page of the template is mandatory and you can organize the rest of the report as you deem fit. We have also released a recommended outline of the report which you can use to incorporate your findings.
We will use the same OpenReview Portal to submit your Reproducibility projects. Click on “Add Reproducibility Challenge Report” to submit your work. In the field of “Paper ID”, paste the paper ID of the forum for the paper. (For example, if the link is https://openreview.net/forum?id=B1V7ccl2Br, then the corresponding paper ID is B1V7ccl2Br).
Please check Important Dates for the deadline for submission of your report. Your report should be of maximum 8 pages excluding references, in NeurIPS style format. You can also submit supplementary materials. Code submission is mandatory along with your supplementary materials.
We will be selecting reproducibility reports which present a clear, well motivated, and thorough reproducibility effort. The report should very briefly discuss the main ideas of the paper and not simply reiterate the main work. Your report should clearly mention the key findings of your effort, and if necessary include discussion / commentary / suggestions to the readers re-implementing this work on its replicability / complexity. Your report must contain the Reproducibility Summary in the first page as defined in the previous section.
Reports will be reviewed on the following criteria: