Search Results for author: Frank Hutter

Found 152 papers, 103 papers with code

HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models

1 code implementation16 May 2024 Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Jorg K. H. Franke, Frank Hutter

To this end, we propose HW-GPT-Bench, a hardware-aware language model surrogate benchmark, where we leverage weight-sharing techniques from Neural Architecture Search (NAS) to efficiently train a supernet proxy, encompassing language models of varying scales in a single model.

Language Modelling Neural Architecture Search

Don't Waste Your Time: Early Stopping Cross-Validation

1 code implementation6 May 2024 Edward Bergman, Lennart Purucker, Frank Hutter

In addition, we investigate the impact of early stopping with Bayesian optimization instead of random search and also repeated cross-validation.

Bayesian Optimization Model Selection

Surprisingly Strong Performance Prediction with Neural Graph Features

no code implementations25 Apr 2024 Gabriela Kadlecová, Jovita Lukasik, Martin Pilát, Petra Vidnerová, Mahmoud Safari, Roman Neruda, Frank Hutter

Performance prediction has been a key part of the neural architecture search (NAS) process, allowing to speed up NAS algorithms by avoiding resource-consuming network training.

Neural Architecture Search

Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks

2 code implementations4 Mar 2024 Shuhei Watanabe, Neeratyoy Mallik, Edward Bergman, Frank Hutter

While deep learning has celebrated many successes, its results often hinge on the meticulous selection of hyperparameters (HPs).

Benchmarking

Multi-objective Differentiable Neural Architecture Search

1 code implementation28 Feb 2024 Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Samuel Dooley, Josif Grabocka, Frank Hutter

Pareto front profiling in multi-objective optimization (MOO), i. e. finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives like neural network training.

Machine Translation Neural Architecture Search

Diffusion-based Neural Network Weights Generation

no code implementations28 Feb 2024 Bedionita Soro, Bruno Andreis, Hayeon Lee, Song Chong, Frank Hutter, Sung Ju Hwang

By learning the distribution of a neural network on a variety pretrained models, our approach enables adaptive sampling weights for unseen datasets achieving faster convergence and reaching competitive performance.

Transfer Learning

TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks

2 code implementations17 Feb 2024 Benjamin Feuer, Robin Tibor Schirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, Colin White

Similar to large language models, PFNs make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass.

Fairness In-Context Learning +1

Is Mamba Capable of In-Context Learning?

1 code implementation5 Feb 2024 Riccardo Grazzi, Julien Siems, Simon Schrodi, Thomas Brox, Frank Hutter

State of the art foundation models such as GPT-4 perform surprisingly well at in-context learning (ICL), a variant of meta-learning concerning the learned ability to solve tasks during a neural network forward pass, exploiting contextual information provided as input to the model.

AutoML In-Context Learning +1

Weight-Entanglement Meets Gradient-Based Neural Architecture Search

no code implementations16 Dec 2023 Rhea Sanjay Sukthanker, Arjun Krishnakumar, Mahmoud Safari, Frank Hutter

%Due to the inherent differences in the structure of these search spaces, these Since weight-entanglement poses compatibility challenges for gradient-based NAS methods, these two paradigms have largely developed independently in parallel sub-communities.

Neural Architecture Search

Rethinking Performance Measures of RNA Secondary Structure Problems

no code implementations4 Dec 2023 Frederic Runge, Jörg K. H. Franke, Daniel Fertmann, Frank Hutter

Accurate RNA secondary structure prediction is vital for understanding cellular regulation and disease mechanisms.

A General Framework for User-Guided Bayesian Optimization

1 code implementation24 Nov 2023 Carl Hvarfner, Frank Hutter, Luigi Nardi

The optimization of expensive-to-evaluate black-box functions is prevalent in various scientific disciplines.

Bayesian Optimization

Constrained Parameter Regularization

1 code implementation15 Nov 2023 Jörg K. H. Franke, Michael Hefenbrock, Gregor Koehler, Frank Hutter

Instead of applying a single constant penalty to all parameters, we enforce an upper bound on a statistical measure (e. g., the L$_2$-norm) of parameter groups.

Image Classification Language Modelling

Hard View Selection for Self-Supervised Learning

no code implementations5 Oct 2023 Fabio Ferreira, Ivo Rapant, Frank Hutter

Many Self-Supervised Learning (SSL) methods train their models to be invariant to different "views" of an image input for which a good data augmentation pipeline is crucial.

Contrastive Learning Image Augmentation +2

Towards Automated Design of Riboswitches

no code implementations17 Jul 2023 Frederic Runge, Jörg K. H. Franke, Frank Hutter

Experimental screening and selection pipelines for the discovery of novel riboswitches are expensive, time-consuming, and inefficient.

Scalable Deep Learning for RNA Secondary Structure Prediction

1 code implementation14 Jul 2023 Jörg K. H. Franke, Frederic Runge, Frank Hutter

The field of RNA secondary structure prediction has made significant progress with the adoption of deep learning techniques.

Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How

1 code implementation6 Jun 2023 Sebastian Pineda Arango, Fabio Ferreira, Arlind Kadra, Frank Hutter, Josif Grabocka

With the ever-increasing number of pretrained models, machine learning practitioners are continuously faced with which pretrained model to use, and how to finetune it for a new dataset.

Hyperparameter Optimization Image Classification

PFNs4BO: In-Context Learning for Bayesian Optimization

1 code implementation27 May 2023 Samuel Müller, Matthias Feurer, Noah Hollmann, Frank Hutter

In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO).

Bayesian Optimization Hyperparameter Optimization +1

MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization

no code implementations8 May 2023 Noor Awad, Ayushi Sharma, Philipp Muller, Janek Thomas, Frank Hutter

Hyperparameter optimization (HPO) is a powerful technique for automating the tuning of machine learning (ML) models.

Fairness Hyperparameter Optimization +1

PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces

1 code implementation20 Apr 2023 Shuhei Watanabe, Archit Bansal, Frank Hutter

The recent rise in popularity of Hyperparameter Optimization (HPO) for deep learning has highlighted the role that good hyperparameter (HP) space design can play in training strong models.

Hyperparameter Optimization

Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML

no code implementations15 Mar 2023 Hilde Weerts, Florian Pfisterer, Matthias Feurer, Katharina Eggensperger, Edward Bergman, Noor Awad, Joaquin Vanschoren, Mykola Pechenizkiy, Bernd Bischl, Frank Hutter

The field of automated machine learning (AutoML) introduces techniques that automate parts of the development of machine learning (ML) systems, accelerating the process and reducing barriers for novices.

AutoML Fairness

Mind the Gap: Measuring Generalization Performance Across Multiple Objectives

1 code implementation8 Dec 2022 Matthias Feurer, Katharina Eggensperger, Edward Bergman, Florian Pfisterer, Bernd Bischl, Frank Hutter

Modern machine learning models are often constructed taking into account multiple objectives, e. g., minimizing inference time while also maximizing accuracy.

Hyperparameter Optimization

c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization

1 code implementation26 Nov 2022 Shuhei Watanabe, Frank Hutter

In this work, we propose constrained TPE (c-TPE), an extension of the widely-used versatile Bayesian optimization method, tree-structured Parzen estimator (TPE), to handle these constraints.

Bayesian Optimization Hyperparameter Optimization

Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars

2 code implementations NeurIPS 2023 Simon Schrodi, Danny Stoll, Binxin Ru, Rhea Sukthanker, Thomas Brox, Frank Hutter

In this work, we introduce a unifying search space design framework based on context-free grammars that can naturally and compactly generate expressive hierarchical search spaces that are 100s of orders of magnitude larger than common spaces from the literature.

Bayesian Optimization Neural Architecture Search

Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition

3 code implementations NeurIPS 2023 Samuel Dooley, Rhea Sanjay Sukthanker, John P. Dickerson, Colin White, Frank Hutter, Micah Goldblum

Our search outputs a suite of models which Pareto-dominate all other high-performance architectures and existing bias mitigation methods in terms of accuracy and fairness, often by large margins, on the two most widely used datasets for face identification, CelebA and VGGFace2.

Face Identification Face Recognition +2

NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies

1 code implementation6 Oct 2022 Arjun Krishnakumar, Colin White, Arber Zela, Renbo Tu, Mahmoud Safari, Frank Hutter

Zero-cost proxies (ZC proxies) are a recent architecture performance prediction technique aiming to significantly speed up algorithms for neural architecture search (NAS).

Neural Architecture Search

T3VIP: Transformation-based 3D Video Prediction

1 code implementation19 Sep 2022 Iman Nematollahi, Erick Rosete-Beas, Seyed Mahdi B. Azad, Raghu Rajan, Frank Hutter, Wolfram Burgard

To the best of our knowledge, our model is the first generative model that provides an RGB-D video prediction of the future for a static camera.

Hyperparameter Optimization Video Prediction

On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learning

no code implementations16 Jul 2022 Diane Wagner, Fabio Ferreira, Danny Stoll, Robin Tibor Schirrmeister, Samuel Müller, Frank Hutter

Self-Supervised Learning (SSL) has become a very active area of Deep Learning research where it is heavily used as a pre-training method for classification and other tasks.

Bayesian Optimization Data Augmentation +2

TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second

6 code implementations5 Jul 2022 Noah Hollmann, Samuel Müller, Katharina Eggensperger, Frank Hutter

We present TabPFN, a trained Transformer that can do supervised classification for small tabular datasets in less than a second, needs no hyperparameter tuning and is competitive with state-of-the-art classification methods.

AutoML Bayesian Inference +4

Zero-Shot AutoML with Pretrained Models

1 code implementation16 Jun 2022 Ekrem Öztürk, Fabio Ferreira, Hadi S. Jomaa, Lars Schmidt-Thieme, Josif Grabocka, Frank Hutter

Given a new dataset D and a low compute budget, how should we choose a pre-trained model to fine-tune to D, and set the fine-tuning hyperparameters without risking overfitting, particularly if D is small?

AutoML Meta-Learning

Joint Entropy Search for Maximally-Informed Bayesian Optimization

2 code implementations9 Jun 2022 Carl Hvarfner, Frank Hutter, Luigi Nardi

As a light-weight approach with superior results, JES provides a new go-to acquisition function for Bayesian optimization.

Bayesian Optimization Decision Making

DeepCAVE: An Interactive Analysis Tool for Automated Machine Learning

2 code implementations7 Jun 2022 René Sass, Eddie Bergman, André Biedenkapp, Frank Hutter, Marius Lindauer

Automated Machine Learning (AutoML) is used more than ever before to support users in determining efficient hyperparameters, neural architectures, or even full machine learning pipelines.

AutoML BIG-bench Machine Learning +1

Automated Dynamic Algorithm Configuration

1 code implementation27 May 2022 Steven Adriaensen, André Biedenkapp, Gresa Shala, Noor Awad, Theresa Eimer, Marius Lindauer, Frank Hutter

The performance of an algorithm often critically depends on its parameter configuration.

Efficient Automated Deep Learning for Time Series Forecasting

1 code implementation11 May 2022 Difan Deng, Florian Karl, Frank Hutter, Bernd Bischl, Marius Lindauer

In contrast to common NAS search spaces, we designed a novel neural architecture search space covering various state-of-the-art architectures, allowing for an efficient macro-search over different DL approaches.

Bayesian Optimization Neural Architecture Search +2

$π$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization

1 code implementation23 Apr 2022 Carl Hvarfner, Danny Stoll, Artur Souza, Marius Lindauer, Frank Hutter, Luigi Nardi

To address this issue, we propose $\pi$BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user.

Bayesian Optimization Hyperparameter Optimization

Practitioner Motives to Select Hyperparameter Optimization Methods

no code implementations3 Mar 2022 Niklas Hasebrook, Felix Morsbach, Niclas Kannengießer, Marc Zöller, Jörg Franke, Marius Lindauer, Frank Hutter, Ali Sunyaev

Advanced programmatic hyperparameter optimization (HPO) methods, such as Bayesian optimization, have high sample efficiency in reproducibly finding optimal hyperparameter values of machine learning (ML) models.

Bayesian Optimization BIG-bench Machine Learning +1

Contextualize Me -- The Case for Context in Reinforcement Learning

1 code implementation9 Feb 2022 Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, Sebastian Döhler, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, Marius Lindauer

While Reinforcement Learning ( RL) has made great strides towards solving increasingly complicated problems, many algorithms are still brittle to even slight environmental changes.

reinforcement-learning Reinforcement Learning (RL) +1

Theory-inspired Parameter Control Benchmarks for Dynamic Algorithm Configuration

1 code implementation7 Feb 2022 André Biedenkapp, Nguyen Dang, Martin S. Krejca, Frank Hutter, Carola Doerr

We extend this benchmark by analyzing optimal control policies that can select the parameters only from a given portfolio of possible values.

Benchmarking Evolutionary Algorithms

Learning Synthetic Environments and Reward Networks for Reinforcement Learning

1 code implementation ICLR 2022 Fabio Ferreira, Thomas Nierhoff, Andreas Saelinger, Frank Hutter

In a one-to-one comparison, learning an SE proxy requires more interactions with the real environment than training agents only on the real environment.

reinforcement-learning Reinforcement Learning (RL)

NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy

1 code implementation ICLR 2022 Yash Mehta, Colin White, Arber Zela, Arjun Krishnakumar, Guri Zabergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, Frank Hutter

The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS).

Image Classification Neural Architecture Search +4

Automated Reinforcement Learning (AutoRL): A Survey and Open Problems

no code implementations11 Jan 2022 Jack Parker-Holder, Raghu Rajan, Xingyou Song, André Biedenkapp, Yingjie Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, Frank Hutter, Marius Lindauer

The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable agents.

AutoML Meta-Learning +2

Transformers Can Do Bayesian Inference

1 code implementation ICLR 2022 Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, Frank Hutter

Our method restates the objective of posterior approximation as a supervised classification problem with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their labels from it, masks one of the labels and learns to make probabilistic predictions for it based on the set-valued input of the rest of the data points.

AutoML Bayesian Inference +2

NAS-Bench-x11 and the Power of Learning Curves

1 code implementation NeurIPS 2021 Shen Yan, Colin White, Yash Savani, Frank Hutter

While early research in neural architecture search (NAS) required extreme computational resources, the recent releases of tabular and surrogate benchmarks have greatly increased the speed and reproducibility of NAS research.

Neural Architecture Search

CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning

1 code implementation5 Oct 2021 Carolin Benjamins, Theresa Eimer, Frederik Schubert, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, Marius Lindauer

While Reinforcement Learning has made great strides towards solving ever more complicated tasks, many algorithms are still brittle to even slight changes in their environment.

Physical Simulations reinforcement-learning +2

$\pi$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization

no code implementations ICLR 2022 Carl Hvarfner, Danny Stoll, Artur Souza, Luigi Nardi, Marius Lindauer, Frank Hutter

To address this issue, we propose $\pi$BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user.

Bayesian Optimization Hyperparameter Optimization

Multi-headed Neural Ensemble Search

no code implementations9 Jul 2021 Ashwin Raaghav Narayanan, Arber Zela, Tonmoy Saikia, Thomas Brox, Frank Hutter

Ensembles of CNN models trained with different seeds (also known as Deep Ensembles) are known to achieve superior performance over a single copy of the CNN.

Bag of Tricks for Neural Architecture Search

no code implementations8 Jul 2021 Thomas Elsken, Benedikt Staffler, Arber Zela, Jan Hendrik Metzen, Frank Hutter

While neural architecture search methods have been successful in previous years and led to new state-of-the-art performance on various problems, they have also been criticized for being unstable, being highly sensitive with respect to their hyperparameters, and often not performing better than random search.

Neural Architecture Search

Well-tuned Simple Nets Excel on Tabular Datasets

1 code implementation NeurIPS 2021 Arlind Kadra, Marius Lindauer, Frank Hutter, Josif Grabocka

Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures.

Self-Paced Context Evaluation for Contextual Reinforcement Learning

1 code implementation9 Jun 2021 Theresa Eimer, André Biedenkapp, Frank Hutter, Marius Lindauer

Reinforcement learning (RL) has made a lot of advances for solving a single problem in a given environment; but learning policies that generalize to unseen variations of a problem remains challenging.

reinforcement-learning Reinforcement Learning (RL)

TempoRL: Learning When to Act

1 code implementation9 Jun 2021 André Biedenkapp, Raghu Rajan, Frank Hutter, Marius Lindauer

Reinforcement learning is a powerful approach to learn behaviour through interactions with an environment.

Q-Learning

DEHB: Evolutionary Hyperband for Scalable, Robust and Efficient Hyperparameter Optimization

2 code implementations20 May 2021 Noor Awad, Neeratyoy Mallik, Frank Hutter

Modern machine learning algorithms crucially rely on several design decisions to achieve strong performance, making the problem of Hyperparameter Optimization (HPO) more important than ever.

Hyperparameter Optimization Neural Architecture Search

DACBench: A Benchmark Library for Dynamic Algorithm Configuration

1 code implementation18 May 2021 Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriaensen, Frank Hutter, Marius Lindauer

Dynamic Algorithm Configuration (DAC) aims to dynamically control a target algorithm's hyperparameters in order to improve its performance.

Benchmarking

Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization

1 code implementation ICML Workshop AutoML 2021 Julia Guerrero-Viu, Sven Hauns, Sergio Izquierdo, Guilherme Miotto, Simon Schrodi, Andre Biedenkapp, Thomas Elsken, Difan Deng, Marius Lindauer, Frank Hutter

Neural architecture search (NAS) and hyperparameter optimization (HPO) make deep learning accessible to non-experts by automatically finding the architecture of the deep neural network to use and tuning the hyperparameters of the used training pipeline.

Hyperparameter Optimization Neural Architecture Search

How Powerful are Performance Predictors in Neural Architecture Search?

1 code implementation NeurIPS 2021 Colin White, Arber Zela, Binxin Ru, Yang Liu, Frank Hutter

Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks.

Neural Architecture Search

On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning

1 code implementation26 Feb 2021 Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, Roberto Calandra

We demonstrate that this problem can be tackled effectively with automated HPO, which we demonstrate to yield significantly improved performance compared to human experts.

Hyperparameter Optimization Model-based Reinforcement Learning +2

Regularization Cocktails

no code implementations1 Jan 2021 Arlind Kadra, Marius Lindauer, Frank Hutter, Josif Grabocka

The regularization of prediction models is arguably the most crucial ingredient that allows Machine Learning solutions to generalize well on unseen data.

Hyperparameter Optimization

NASLib: A Modular and Flexible Neural Architecture Search Library

1 code implementation1 Jan 2021 Michael Ruchte, Arber Zela, Julien Niklas Siems, Josif Grabocka, Frank Hutter

Neural Architecture Search (NAS) is one of the focal points for the Deep Learning community, but reproducing NAS methods is extremely challenging due to numerous low-level implementation details.

Neural Architecture Search

Differential Evolution for Neural Architecture Search

1 code implementation11 Dec 2020 Noor Awad, Neeratyoy Mallik, Frank Hutter

Neural architecture search (NAS) methods rely on a search strategy for deciding which architectures to evaluate next and a performance estimation strategy for assessing their performance (e. g., using full evaluations, multi-fidelity evaluations, or the one-shot model).

Bayesian Optimization Neural Architecture Search

Convergence Analysis of Homotopy-SGD for non-convex optimization

no code implementations20 Nov 2020 Matilde Gargiani, Andrea Zanelli, Quoc Tran-Dinh, Moritz Diehl, Frank Hutter

In this work, we present a first-order stochastic algorithm based on a combination of homotopy methods and SGD, called Homotopy-Stochastic Gradient Descent (H-SGD), which finds interesting connections with some proposed heuristics in the literature, e. g. optimization by Gaussian continuation, training by diffusion, mollifying networks.

Hyperparameter Transfer Across Developer Adjustments

1 code implementation25 Oct 2020 Danny Stoll, Jörg K. H. Franke, Diane Wagner, Simon Selg, Frank Hutter

After developer adjustments to a machine learning (ML) algorithm, how can the results of an old hyperparameter optimization (HPO) automatically be used to speedup a new HPO?

Hyperparameter Optimization

On the Importance of Domain Model Configuration for Automated Planning Engines

no code implementations15 Oct 2020 Mauro Vallati, Lukas Chrpa, Thomas L. McCluskey, Frank Hutter

The development of domain-independent planners within the AI Planning community is leading to "off-the-shelf" technology that can be used in a wide range of applications.

Smooth Variational Graph Embeddings for Efficient Neural Architecture Search

2 code implementations9 Oct 2020 Jovita Lukasik, David Friede, Arber Zela, Frank Hutter, Margret Keuper

We evaluate the proposed approach on neural architectures defined by the ENAS approach, the NAS-Bench-101 and the NAS-Bench-201 search space and show that our smooth embedding space allows to directly extrapolate the performance prediction to architectures outside the seen domain (e. g. with more operations).

Bayesian Optimization Neural Architecture Search

Neural Model-based Optimization with Right-Censored Observations

no code implementations29 Sep 2020 Katharina Eggensperger, Kai Haase, Philipp Müller, Marius Lindauer, Frank Hutter

When fitting a regression model to predict the distribution of the outcomes, we cannot simply drop these right-censored observations, but need to properly model them.

regression Thompson Sampling

MDP Playground: Controlling Orthogonal Dimensions of Hardness in Toy Environments

no code implementations28 Sep 2020 Raghu Rajan, Jessica Lizeth Borja Diaz, Suresh Guttikonda, Fabio Ferreira, André Biedenkapp, Frank Hutter

We present MDP Playground, an efficient benchmark for Reinforcement Learning (RL) algorithms with various dimensions of hardness that can be controlled independently to challenge algorithms in different ways and to obtain varying degrees of hardness in generated environments.

OpenAI Gym Reinforcement Learning (RL)

Prior-guided Bayesian Optimization

no code implementations28 Sep 2020 Artur Souza, Luigi Nardi, Leonardo Oliveira, Kunle Olukotun, Marius Lindauer, Frank Hutter

While Bayesian Optimization (BO) is a very popular method for optimizing expensive black-box functions, it fails to leverage the experience of domain experts.

Bayesian Optimization

Sample-Efficient Automated Deep Reinforcement Learning

1 code implementation ICLR 2021 Jörg K. H. Franke, Gregor Köhler, André Biedenkapp, Frank Hutter

Despite significant progress in challenging problems across various domains, applying state-of-the-art deep reinforcement learning (RL) algorithms remains challenging due to their sensitivity to the choice of hyperparameters.

Hyperparameter Optimization reinforcement-learning +1

Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks

1 code implementation ICLR 2022 Arber Zela, Julien Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, Frank Hutter

We show that surrogate NAS benchmarks can model the true performance of architectures better than tabular benchmarks (at a small fraction of the cost), that they lead to faithful estimates of how well different NAS methods work on the original non-surrogate benchmark, and that they can generate new scientific insight.

Neural Architecture Search

Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning

4 code implementations8 Jul 2020 Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, Frank Hutter

Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success.

AutoML BIG-bench Machine Learning +1

Bayesian Optimization with a Prior for the Optimum

no code implementations25 Jun 2020 Artur Souza, Luigi Nardi, Leonardo B. Oliveira, Kunle Olukotun, Marius Lindauer, Frank Hutter

We show that BOPrO is around 6. 67x faster than state-of-the-art methods on a common suite of benchmarks, and achieves a new state-of-the-art performance on a real-world hardware design application.

Bayesian Optimization

Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

2 code implementations24 Jun 2020 Lucas Zimmer, Marius Lindauer, Frank Hutter

While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search.

Neural Architecture Search

Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

1 code implementation NeurIPS 2021 Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, Yee Whye Teh

On a variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift.

Image Classification Neural Architecture Search

Learning Heuristic Selection with Dynamic Algorithm Configuration

1 code implementation15 Jun 2020 David Speck, André Biedenkapp, Frank Hutter, Robert Mattmüller, Marius Lindauer

We show that dynamic algorithm configuration can be used for dynamic heuristic selection which takes into account the internal search dynamics of a planning system.

On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNs

1 code implementation3 Jun 2020 Matilde Gargiani, Andrea Zanelli, Moritz Diehl, Frank Hutter

This enables researchers to further study and improve this promising optimization technique and hopefully reconsider stochastic second-order methods as competitive optimization techniques for training DNNs; we also hope that the promise of SGN may lead to forward automatic differentiation being added to Tensorflow or Pytorch.

Second-order methods

Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework

1 code implementation1 Jun 2020 André Biedenkapp, H. Furkan Bozkurt, Theresa Eimer, Frank Hutter, Marius Lindauer

The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on parameter tuning.

General Reinforcement Learning

Transferring Optimality Across Data Distributions via Homotopy Methods

no code implementations ICLR 2020 Matilde Gargiani, Andrea Zanelli, Quoc Tran Dinh, Moritz Diehl, Frank Hutter

Homotopy methods, also known as continuation methods, are a powerful mathematical tool to efficiently solve various problems in numerical analysis, including complex non-convex optimization problems where no or only little prior knowledge regarding the localization of the solutions is available.

Machine-Learning-Based Diagnostics of EEG Pathology

1 code implementation11 Feb 2020 Lukas Alexander Wilhelm Gemein, Robin Tibor Schirrmeister, Patryk Chrabąszcz, Daniel Wilson, Joschka Boedecker, Andreas Schulze-Bonhage, Frank Hutter, Tonio Ball

The results demonstrate that the proposed feature-based decoding framework can achieve accuracies on the same level as state-of-the-art deep neural networks.

BIG-bench Machine Learning EEG

NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search

1 code implementation ICLR 2020 Arber Zela, Julien Siems, Frank Hutter

One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice.

Benchmarking Neural Architecture Search

Meta-Learning of Neural Architectures for Few-Shot Learning

2 code implementations CVPR 2020 Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, Frank Hutter

The recent progress in neural architecture search (NAS) has allowed scaling the automated design of neural architectures to real-world domains, such as object detection and semantic segmentation.

Few-Shot Learning Neural Architecture Search +3

OpenML-Python: an extensible Python API for OpenML

1 code implementation6 Nov 2019 Matthias Feurer, Jan N. van Rijn, Arlind Kadra, Pieter Gijsbers, Neeratyoy Mallik, Sahithya Ravi, Andreas Müller, Joaquin Vanschoren, Frank Hutter

It also provides functionality to conduct machine learning experiments, upload the results to OpenML, and reproduce results which are stored on OpenML.

BIG-bench Machine Learning

Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings

1 code implementation10 Oct 2019 Matilde Gargiani, Aaron Klein, Stefan Falkner, Frank Hutter

We propose probabilistic models that can extrapolate learning curves of iterative machine learning algorithms, such as stochastic gradient descent for training deep networks, based on training data with variable-length learning curves.

BIG-bench Machine Learning Hyperparameter Optimization

Understanding and Robustifying Differentiable Architecture Search

1 code implementation ICLR 2020 Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, Frank Hutter

Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem.

Disparity Estimation Image Classification +1

MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning

1 code implementation17 Sep 2019 Raghu Rajan, Jessica Lizeth Borja Diaz, Suresh Guttikonda, Fabio Ferreira, André Biedenkapp, Jan Ole von Hartz, Frank Hutter

We define a parameterised collection of fast-to-run toy environments in OpenAI Gym by varying these dimensions and propose to use these to understand agents better.

OpenAI Gym reinforcement-learning +1

Best Practices for Scientific Research on Neural Architecture Search

no code implementations5 Sep 2019 Marius Lindauer, Frank Hutter

Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS).

BIG-bench Machine Learning Neural Architecture Search

BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters

1 code implementation16 Aug 2019 Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Joshua Marben, Philipp Müller, Frank Hutter

Hyperparameter optimization and neural architecture search can become prohibitively expensive for regular black-box Bayesian optimization because the training and evaluation of a single model can easily take several hours.

Bayesian Optimization Hyperparameter Optimization +1

Towards White-box Benchmarks for Algorithm Control

no code implementations18 Jun 2019 André Biedenkapp, H. Furkan Bozkurt, Frank Hutter, Marius Lindauer

The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on tuned hyperparameter configurations.

Reinforcement Learning (RL) valid

Meta-Surrogate Benchmarking for Hyperparameter Optimization

1 code implementation NeurIPS 2019 Aaron Klein, Zhenwen Dai, Frank Hutter, Neil Lawrence, Javier Gonzalez

Despite the recent progress in hyperparameter optimization (HPO), available benchmarks that resemble real-world scenarios consist of a few and very large problem instances that are expensive to solve.

Benchmarking Hyperparameter Optimization

Towards Automatically-Tuned Deep Neural Networks

2 code implementations18 May 2019 Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, Matthias Urban, Michael Burkart, Maximilian Dippel, Marius Lindauer, Frank Hutter

Recent advances in AutoML have led to automated tools that can compete with machine learning experts on supervised learning tasks.

AutoML BIG-bench Machine Learning

AutoDispNet: Improving Disparity Estimation With AutoML

1 code implementation ICCV 2019 Tonmoy Saikia, Yassine Marrakchi, Arber Zela, Frank Hutter, Thomas Brox

In this work, we show how to use and extend existing AutoML techniques to efficiently optimize large-scale U-Net-like encoder-decoder architectures.

Bayesian Optimization Decoder +3

Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization

1 code implementation13 May 2019 Aaron Klein, Frank Hutter

Due to the high computational demands executing a rigorous comparison between hyperparameter optimization (HPO) methods is often cumbersome.

Hyperparameter Optimization

NAS-Bench-101: Towards Reproducible Neural Architecture Search

4 code implementations25 Feb 2019 Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, Frank Hutter

Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation.

Benchmarking Neural Architecture Search

Learning to Design RNA

5 code implementations ICLR 2019 Frederic Runge, Danny Stoll, Stefan Falkner, Frank Hutter

Designing RNA molecules has garnered recent interest in medicine, synthetic biology, biotechnology and bioinformatics since many functional RNA molecules were shown to be involved in regulatory processes for transcription, epigenetics and translation.

Meta-Learning

Neural Architecture Search: A Survey

1 code implementation16 Aug 2018 Thomas Elsken, Jan Hendrik Metzen, Frank Hutter

Deep Learning has enabled remarkable progress over the last years on a variety of tasks, such as image recognition, speech recognition, and machine translation.

Machine Translation Neural Architecture Search +3

Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search

3 code implementations18 Jul 2018 Arber Zela, Aaron Klein, Stefan Falkner, Frank Hutter

While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal.

Bayesian Optimization Neural Architecture Search

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

4 code implementations ICML 2018 Stefan Falkner, Aaron Klein, Frank Hutter

Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible.

Bayesian Optimization Hyperparameter Optimization

Training Generative Reversible Networks

1 code implementation5 Jun 2018 Robin Tibor Schirrmeister, Patryk Chrabąszcz, Frank Hutter, Tonio Ball

This first attempt to use RevNets inside the adversarial autoencoder framework slightly underperformed relative to recent advanced generative models using an autoencoder component on CelebA, but this gap may diminish with further optimization of the training setup of generative RevNets.

Decoder

Maximizing acquisition functions for Bayesian optimization

1 code implementation NeurIPS 2018 James T. Wilson, Frank Hutter, Marc Peter Deisenroth

Bayesian optimization is a sample-efficient approach to global optimization that relies on theoretically motivated value heuristics (acquisition functions) to guide its search process.

Bayesian Optimization

Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution

no code implementations ICLR 2019 Thomas Elsken, Jan Hendrik Metzen, Frank Hutter

Neural Architecture Search aims at automatically finding neural architectures that are competitive with architectures designed by human experts.

Neural Architecture Search

Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari

1 code implementation24 Feb 2018 Patryk Chrabaszcz, Ilya Loshchilov, Frank Hutter

Evolution Strategies (ES) have recently been demonstrated to be a viable alternative to reinforcement learning (RL) algorithms on a set of challenging deep RL problems, including Atari games and MuJoCo humanoid locomotion benchmarks.

Atari Games Benchmarking +1

Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow

1 code implementation ECCV 2018 Eddy Ilg, Özgün Çiçek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, Thomas Brox

Optical flow estimation can be formulated as an end-to-end supervised learning problem, which yields estimates with a superior accuracy-runtime tradeoff compared to alternative methodology.

Optical Flow Estimation

Practical Transfer Learning for Bayesian Optimization

2 code implementations6 Feb 2018 Matthias Feurer, Benjamin Letham, Frank Hutter, Eytan Bakshy

When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset.

Bayesian Optimization Gaussian Processes +3

Fixing Weight Decay Regularization in Adam

no code implementations ICLR 2018 Ilya Loshchilov, Frank Hutter

We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.

Image Classification

The reparameterization trick for acquisition functions

1 code implementation1 Dec 2017 James T. Wilson, Riccardo Moriconi, Frank Hutter, Marc Peter Deisenroth

Bayesian optimization is a sample-efficient approach to solving global optimization problems.

Bayesian Optimization

Decoupled Weight Decay Regularization

20 code implementations ICLR 2019 Ilya Loshchilov, Frank Hutter

L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradient algorithms, such as Adam.

Image Classification

Neural Networks for Predicting Algorithm Runtime Distributions

no code implementations22 Sep 2017 Katharina Eggensperger, Marius Lindauer, Frank Hutter

Many state-of-the-art algorithms for solving hard combinatorial problems in artificial intelligence (AI) include elements of stochasticity that lead to high variations in runtime, even for a fixed problem instance.

Warmstarting of Model-based Algorithm Configuration

no code implementations14 Sep 2017 Marius Lindauer, Frank Hutter

The performance of many hard combinatorial problem solvers depends strongly on their parameter settings, and since manual parameter tuning is both tedious and suboptimal the AI community has recently developed several algorithm configuration (AC) methods to automatically address this problem.

Deep learning with convolutional neural networks for decoding and visualization of EEG pathology

2 code implementations26 Aug 2017 Robin Tibor Schirrmeister, Lukas Gemein, Katharina Eggensperger, Frank Hutter, Tonio Ball

We apply convolutional neural networks (ConvNets) to the task of distinguishing pathological from normal EEG recordings in the Temple University Hospital EEG Abnormal Corpus.

EEG

Pitfalls and Best Practices in Algorithm Configuration

2 code implementations17 May 2017 Katharina Eggensperger, Marius Lindauer, Frank Hutter

Good parameter settings are crucial to achieve high performance in many areas of artificial intelligence (AI), such as propositional satisfiability solving, AI planning, scheduling, and machine learning (in particular deep learning).

Experimental Design Scheduling

Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates

no code implementations30 Mar 2017 Katharina Eggensperger, Marius Lindauer, Holger H. Hoos, Frank Hutter, Kevin Leyton-Brown

In our experiments, we construct and evaluate surrogate benchmarks for hyperparameter optimization as well as for AC problems that involve performance optimization of solvers for hard combinatorial problems, drawing training data from the runs of existing AC procedures.

Benchmarking Hyperparameter Optimization

Deep learning with convolutional neural networks for EEG decoding and visualization

5 code implementations15 Mar 2017 Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, Tonio Ball

PLEASE READ AND CITE THE REVISED VERSION at Human Brain Mapping: http://onlinelibrary. wiley. com/doi/10. 1002/hbm. 23730/full Code available here: https://github. com/robintibor/braindecode

EEG Eeg Decoding

Asynchronous Stochastic Gradient MCMC with Elastic Coupling

no code implementations2 Dec 2016 Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, Frank Hutter

We consider parallel asynchronous Markov Chain Monte Carlo (MCMC) sampling for problems where we can leverage (stochastic) gradients to define continuous dynamics which explore the target distribution.

Bayesian Optimization with Robust Bayesian Neural Networks

1 code implementation NeurIPS 2016 Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, Frank Hutter

Bayesian optimization is a prominent method for optimizing expensive to evaluate black-box functions that is prominently applied to tuning the hyperparameters of machine learning algorithms.

Bayesian Optimization Hyperparameter Optimization +1

SGDR: Stochastic Gradient Descent with Warm Restarts

17 code implementations13 Aug 2016 Ilya Loshchilov, Frank Hutter

Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions.

EEG Stochastic Optimization

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

1 code implementation23 May 2016 Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, Frank Hutter

Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks.

Bayesian Optimization BIG-bench Machine Learning +1

CMA-ES for Hyperparameter Optimization of Deep Neural Networks

no code implementations25 Apr 2016 Ilya Loshchilov, Frank Hutter

Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization.

Bayesian Optimization Hyperparameter Optimization

Efficient and Robust Automated Machine Learning

2 code implementations NeurIPS 2015 Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, Frank Hutter

The success of machine learning in a broad range of applications has led to an ever-growing demand for machine learning systems that can be used off the shelf by non-experts.

Bayesian Optimization BIG-bench Machine Learning +1

Online Batch Selection for Faster Training of Neural Networks

1 code implementation19 Nov 2015 Ilya Loshchilov, Frank Hutter

We investigate online batch selection strategies for two state-of-the-art methods of stochastic gradient-based optimization, AdaDelta and Adam.

ASlib: A Benchmark Library for Algorithm Selection

2 code implementations8 Jun 2015 Bernd Bischl, Pascal Kerschke, Lars Kotthoff, Marius Lindauer, Yuri Malitsky, Alexandre Frechette, Holger Hoos, Frank Hutter, Kevin Leyton-Brown, Kevin Tierney, Joaquin Vanschoren

To address this problem, we introduce a standardized format for representing algorithm selection scenarios and a repository that contains a growing number of data sets from the literature.

The Configurable SAT Solver Challenge (CSSC)

no code implementations5 May 2015 Frank Hutter, Marius Lindauer, Adrian Balint, Sam Bayless, Holger Hoos, Kevin Leyton-Brown

It is well known that different solution strategies work well for different types of instances of hard combinatorial problems.

ParamILS: An Automatic Algorithm Configuration Framework

no code implementations15 Jan 2014 Frank Hutter, Thomas Stuetzle, Kevin Leyton-Brown, Holger H. Hoos

The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms.

Hyperparameter Optimization

A Kernel for Hierarchical Parameter Spaces

no code implementations21 Oct 2013 Frank Hutter, Michael A. Osborne

We define a family of kernels for mixed continuous/discrete hierarchical parameter spaces and show that they are positive definite.

Bayesian Optimization With Censored Response Data

no code implementations7 Oct 2013 Frank Hutter, Holger Hoos, Kevin Leyton-Brown

Bayesian optimization (BO) aims to minimize a given blackbox function using a model that is updated whenever new evidence about the function becomes available.

Bayesian Optimization

Bayesian Optimization in a Billion Dimensions via Random Embeddings

1 code implementation9 Jan 2013 Ziyu Wang, Frank Hutter, Masrour Zoghi, David Matheson, Nando de Freitas

Bayesian optimization techniques have been successfully applied to robotics, planning, sensor placement, recommendation, advertising, intelligent user interfaces and automatic algorithm configuration.

Bayesian Optimization

Algorithm Runtime Prediction: Methods & Evaluation

no code implementations5 Nov 2012 Frank Hutter, Lin Xu, Holger H. Hoos, Kevin Leyton-Brown

We also comprehensively describe new and existing features for predicting algorithm runtime for propositional satisfiability (SAT), travelling salesperson (TSP) and mixed integer programming (MIP) problems.

Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms

1 code implementation18 Aug 2012 Chris Thornton, Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown

Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall.

Bayesian Optimization BIG-bench Machine Learning +3

Sequential Model-Based Optimization for General Algorithm Configuration

1 code implementation LION 2011 2011 Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown

State-of-the-art algorithms for hard computational problems often expose many parameters that can be modified to improve empirical performance.

Hyperparameter Optimization

Cannot find the paper you are looking for? You can Submit a new open access paper.