RepoBench is a benchmark designed for evaluating repository-level code auto-completion systems, focusing on more complex, real-world programming scenarios involving multiple files. It comprises three tasks: RepoBench-R (Retrieval), measuring the system's ability to retrieve relevant code snippets; RepoBench-C (Code Completion), assessing the prediction of the next line of code with both in-file and cross-file context; and RepoBench-P (Pipeline), evaluating complex tasks requiring both retrieval and prediction. RepoBench aims to provide a comprehensive performance comparison to foster continuous improvement in auto-completion systems.