DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.
PDF AbstractCode
Spaces
MMLU
GSM8K
HumanEval
MATH
HellaSwag
SVAMP
APPS
DS-1000
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Code Generation | APPS | deepseek-ai/deepseek-coder-6.7b-instruct | Introductory Pass@1 | 33.80 | # 4 | ||
| Interview Pass@1 | 19.70 | # 3 | |||||
| Competition Pass@1 | 11.09 | # 4 | |||||
| Code Generation | MBPP | GPT-3.5 Turbo (few-shot) | Accuracy | 70.8 | # 26 | ||
| Code Generation | MBPP | GPT-4 (few-shot) | Accuracy | 80 | # 21 | ||
| Code Generation | MBPP | DeepSeek-Coder-Instruct 1.3B (few-shot) | Accuracy | 49.4 | # 58 | ||
| Code Generation | MBPP | DeepSeek-Coder-Base 1.3B (few-shot) | Accuracy | 46.2 | # 70 | ||
| Code Generation | MBPP | DeepSeek-Coder-Instruct 33B (few-shot) | Accuracy | 70 | # 28 | ||
| Code Generation | MBPP | DeepSeek-Coder-Base 6.7B (few-shot) | Accuracy | 60.6 | # 46 | ||
| Code Generation | MBPP | DeepSeek-Coder-Instruct 6.7B (few-shot) | Accuracy | 65.4 | # 38 | ||
| Code Generation | MBPP | DeepSeek-Coder-Base 33B (few-shot) | Accuracy | 66 | # 36 |
