Measuring Mathematical Problem Solving With the MATH Dataset

Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.

PDF Abstract

Datasets


Introduced in the Paper:

MATH

Used in the Paper:

Mathematics Dataset HOList

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Math Word Problem Solving MATH GPT-2 (1.5B) Accuracy 6.9 # 119
Parameters (Billions) 1.5 # 101
Math Word Problem Solving MATH GPT-3 2.7B Accuracy 2.9 # 131
Parameters (Billions) 2.7 # 100
Math Word Problem Solving MATH GPT-2 (0.1B) Accuracy 5.4 # 125
Parameters (Billions) 0.1 # 106
Math Word Problem Solving MATH GPT-2 (0.3B) Accuracy 6.2 # 122
Parameters (Billions) 0.3 # 105
Math Word Problem Solving MATH GPT-2 (0.7B) Accuracy 6.4 # 121
Parameters (Billions) 0.7 # 104
Math Word Problem Solving MATH GPT-3-175B (few-shot) Accuracy 5.2 # 126
Parameters (Billions) 175 # 5
Math Word Problem Solving MATH GPT-3 13B Accuracy 5.6 # 123
Parameters (Billions) 13 # 44
Math Word Problem Solving MATH GPT-3-13B (few-shot) Accuracy 3.0 # 130
Parameters (Billions) 13 # 44

Methods