Benchmarking Algorithms from Machine Learning for Low-Budget Black-Box Optimization

Machine learning has invaded various domains of computer science, including black-box optimization. Recent research is particularly concerned with Bayesian Optimization and/or Monte Carlo Tree Search. However, the experiments are usually performed on rather small benchmarks and there are visible issues in the experimental setup, such as poor initialization of baselines, overfitting by specifying hyperparameters specifically for each test function, and low statistical significance. In addition, the interface is sometimes very problem-specific and has more impact on the results than the algorithm itself. We compare several black-box optimization tools from the machine learning world and benchmark them on the classical BBOB benchmark, which is well known in the black-box optimization field, and on Direct Policy Search for OpenAI Gym. In particular, the benchmarks in this work include randomization of the optimum: BBOB considers 15 random instances per test function and dimension, i.e., 24 functions $\times$ 6 dimensionalities $\times$ 15 random instances $=2160$ cases. For OpenAI Gym, we consider tiny and larger neural networks, on a total number of 13 problems $\times$ 8 budgets $\times$ 10 repetitions $=1040$ and 18 problems $\times$ 8 budgets $\times$ 10 repetitions $=1440$ instances, respectively.