Resources

This page lists some useful resources which you can use for the challenge.

Reproducible Code

Code submission are mandatory for all submitted reports. We recommend that you:

• Publish your code in a public repository (e.g. on GitHub, GitLab, BitBucket)
• Document your code appropriately
• Have a README.md file which describes the exact steps to run your code. You can refer to the ML Code Completeness Checklist to write the README file and make sure your code submission is complete.
• See this blog post on best practices for reproducibility

Compute Resources

• Code Ocean provides 10hrs/month of GPU accelerated platform free to academics. Code Ocean is a cloud-based research collaboration platform.
• Instructors can apply for Google Cloud credits for their students. Each student will be given a fixed amount of starter credits to use Google Cloud Compute (GCP).
• By default, Google Cloud accounts don’t come with a GPU quota, but you can find instructions on how to request GPUs, including links on how to check and increase quotas, at this link.
• If necessary, instructors can ask for much more computing credits by contacting: CloudEDUGrants@google.com.
• Students can also request a $300 credit from Google Cloud Compute.
• Anyone can use Google Colaboratory which provides free GPU backed Jupyter Notebooks
• If you are a company that can offer cloud computing credits, please contact reproducibility.challenge@gmail.com or koustuv.sinha@mail.mcgill.ca.

Suggested Readings

• ICLR 2019 Reproducibility Challenge Accepted Papers at ReScience Journal
• ICLR 2019 Reproducibility Challenge Submissions
• NeurIPS 2019 Reproducibility Challenge Accepted Papers at ReScience Journal
• NeurIPS 2019 Reproducibility Challenge Submissions
• ML Reproducibility Checklist
• ML Code Completeness Checklist
• ML reproducibility tools and best practices
• One example class where the reproducibility challenge was part of the coursework.
• Joelle Pineau’s Keynote talk on Reproducibility at NeurIPS 2018
• Check this list of related work