1 code implementation • 27 May 2023 • Samuel Müller, Matthias Feurer, Noah Hollmann, Frank Hutter
In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO).
no code implementations • 8 May 2023 • Noor Awad, Ayushi Sharma, Philipp Muller, Janek Thomas, Frank Hutter
Hyperparameter optimization (HPO) is a powerful technique for automating the tuning of machine learning (ML) models.
1 code implementation • 5 May 2023 • Noah Hollmann, Samuel Müller, Frank Hutter
Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset.
no code implementations • 21 Apr 2023 • Carl Hvarfner, Erik Hellsten, Frank Hutter, Luigi Nardi
Gaussian processes are cemented as the model of choice in Bayesian optimization and active learning.
1 code implementation • 20 Apr 2023 • Shuhei Watanabe, Archit Bansal, Frank Hutter
The recent rise in popularity of Hyperparameter Optimization (HPO) for deep learning has highlighted the role that good hyperparameter (HP) space design can play in training strong models.
no code implementations • 15 Mar 2023 • Hilde Weerts, Florian Pfisterer, Matthias Feurer, Katharina Eggensperger, Edward Bergman, Noor Awad, Joaquin Vanschoren, Mykola Pechenizkiy, Bernd Bischl, Frank Hutter
The field of automated machine learning (AutoML) introduces techniques that automate parts of the development of machine learning (ML) systems, accelerating the process and reducing barriers for novices.
no code implementations • 20 Jan 2023 • Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, Frank Hutter
Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas.
Natural Language Understanding
Neural Architecture Search
+2
1 code implementation • 13 Dec 2022 • Shuhei Watanabe, Noor Awad, Masaki Onishi, Frank Hutter
Hyperparameter optimization (HPO) is a vital step in improving performance in deep learning (DL).
1 code implementation • 8 Dec 2022 • Matthias Feurer, Katharina Eggensperger, Edward Bergman, Florian Pfisterer, Bernd Bischl, Frank Hutter
Modern machine learning models are often constructed taking into account multiple objectives, e. g., minimizing inference time while also maximizing accuracy.
1 code implementation • 26 Nov 2022 • Shuhei Watanabe, Frank Hutter
In this work, we propose constrained TPE (c-TPE), an extension of the widely-used versatile Bayesian optimization method, tree-structured Parzen estimator (TPE), to handle these constraints.
1 code implementation • 3 Nov 2022 • Simon Schrodi, Danny Stoll, Binxin Ru, Rhea Sukthanker, Thomas Brox, Frank Hutter
The discovery of neural architectures from scratch is the long-standing goal of Neural Architecture Search (NAS).
1 code implementation • 18 Oct 2022 • Rhea Sukthanker, Samuel Dooley, John P. Dickerson, Colin White, Frank Hutter, Micah Goldblum
Motivated by our findings, we run the first neural architecture search for fairness, jointly with a search for hyperparameters.
1 code implementation • 6 Oct 2022 • Arjun Krishnakumar, Colin White, Arber Zela, Renbo Tu, Mahmoud Safari, Frank Hutter
Zero-cost proxies (ZC proxies) are a recent architecture performance prediction technique aiming to significantly speed up algorithms for neural architecture search (NAS).
1 code implementation • 19 Sep 2022 • Iman Nematollahi, Erick Rosete-Beas, Seyed Mahdi B. Azad, Raghu Rajan, Frank Hutter, Wolfram Burgard
To the best of our knowledge, our model is the first generative model that provides an RGB-D video prediction of the future for a static camera.
no code implementations • 16 Jul 2022 • Diane Wagner, Fabio Ferreira, Danny Stoll, Robin Tibor Schirrmeister, Samuel Müller, Frank Hutter
Self-Supervised Learning (SSL) has become a very active area of Deep Learning research where it is heavily used as a pre-training method for classification and other tasks.
3 code implementations • 5 Jul 2022 • Noah Hollmann, Samuel Müller, Katharina Eggensperger, Frank Hutter
We present TabPFN, a trained Transformer that can do supervised classification for small tabular datasets in less than a second, needs no hyperparameter tuning and is competitive with state-of-the-art classification methods.
1 code implementation • 16 Jun 2022 • Ekrem Öztürk, Fabio Ferreira, Hadi S. Jomaa, Lars Schmidt-Thieme, Josif Grabocka, Frank Hutter
Given a new dataset D and a low compute budget, how should we choose a pre-trained model to fine-tune to D, and set the fine-tuning hyperparameters without risking overfitting, particularly if D is small?
no code implementations • 15 Jun 2022 • Adrian El Baz, Ihsan Ullah, Edesio Alcobaça, André C. P. L. F. Carvalho, Hong Chen, Fabio Ferreira, Henry Gouk, Chaoyu Guan, Isabelle Guyon, Timothy Hospedales, Shell Hu, Mike Huisman, Frank Hutter, Zhengying Liu, Felix Mohr, Ekrem Öztürk, Jan N. van Rijn, Haozhe Sun, Xin Wang, Wenwu Zhu
Although deep neural networks are capable of achieving performance superior to humans on various tasks, they are notorious for requiring large amounts of data and computing resources, restricting their success to domains where such resources are available.
2 code implementations • 9 Jun 2022 • Carl Hvarfner, Frank Hutter, Luigi Nardi
As a light-weight approach with superior results, JES provides a new go-to acquisition function for Bayesian optimization.
2 code implementations • 7 Jun 2022 • René Sass, Eddie Bergman, André Biedenkapp, Frank Hutter, Marius Lindauer
Automated Machine Learning (AutoML) is used more than ever before to support users in determining efficient hyperparameters, neural architectures, or even full machine learning pipelines.
1 code implementation • 27 May 2022 • Steven Adriaensen, André Biedenkapp, Gresa Shala, Noor Awad, Theresa Eimer, Marius Lindauer, Frank Hutter
The performance of an algorithm often critically depends on its parameter configuration.
1 code implementation • 27 May 2022 • Jörg K. H. Franke, Frederic Runge, Frank Hutter
Our world is ambiguous and this is reflected in the data we use to train our algorithms.
1 code implementation • 11 May 2022 • Difan Deng, Florian Karl, Frank Hutter, Bernd Bischl, Marius Lindauer
In contrast to common NAS search spaces, we designed a novel neural architecture search space covering various state-of-the-art architectures, allowing for an efficient macro-search over different DL approaches.
no code implementations • 23 Apr 2022 • Carl Hvarfner, Danny Stoll, Artur Souza, Marius Lindauer, Frank Hutter, Luigi Nardi
To address this issue, we propose $\pi$BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user.
no code implementations • 3 Mar 2022 • Niklas Hasebrook, Felix Morsbach, Niclas Kannengießer, Jörg Franke, Frank Hutter, Ali Sunyaev
Current advanced hyperparameter optimization (HPO) methods, such as Bayesian optimization, have high sampling efficiency and facilitate replicability.
no code implementations • 15 Feb 2022 • Thomas Elsken, Arber Zela, Jan Hendrik Metzen, Benedikt Staffler, Thomas Brox, Abhinav Valada, Frank Hutter
The success of deep learning in recent years has lead to a rising demand for neural network architecture engineering.
no code implementations • 9 Feb 2022 • Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, Sebastian Döhler, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, Marius Lindauer
While Reinforcement Learning ( RL) has made great strides towards solving increasingly complicated problems, many algorithms are still brittle to even slight environmental changes.
1 code implementation • 7 Feb 2022 • André Biedenkapp, Nguyen Dang, Martin S. Krejca, Frank Hutter, Carola Doerr
We extend this benchmark by analyzing optimal control policies that can select the parameters only from a given portfolio of possible values.
1 code implementation • ICLR 2022 • Fabio Ferreira, Thomas Nierhoff, Andreas Saelinger, Frank Hutter
In a one-to-one comparison, learning an SE proxy requires more interactions with the real environment than training agents only on the real environment.
1 code implementation • ICLR 2022 • Yash Mehta, Colin White, Arber Zela, Arjun Krishnakumar, Guri Zabergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, Frank Hutter
The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS).
no code implementations • 11 Jan 2022 • Zhengying Liu, Adrien Pavao, Zhen Xu, Sergio Escalera, Fabio Ferreira, Isabelle Guyon, Sirui Hong, Frank Hutter, Rongrong Ji, Julio C. S. Jacques Junior, Ge Li, Marius Lindauer, Zhipeng Luo, Meysam Madadi, Thomas Nierhoff, Kangning Niu, Chunguang Pan, Danny Stoll, Sebastien Treguer, Jin Wang, Peng Wang, Chenglin Wu, Youcheng Xiong, Arbe r Zela, Yang Zhang
Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly.
no code implementations • 11 Jan 2022 • Jack Parker-Holder, Raghu Rajan, Xingyou Song, André Biedenkapp, Yingjie Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, Frank Hutter, Marius Lindauer
The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable agents.
1 code implementation • ICLR 2022 • Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, Frank Hutter
Our method restates the objective of posterior approximation as a supervised classification problem with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their labels from it, masks one of the labels and learns to make probabilistic predictions for it based on the set-valued input of the rest of the data points.
1 code implementation • NeurIPS 2021 • Shen Yan, Colin White, Yash Savani, Frank Hutter
While early research in neural architecture search (NAS) required extreme computational resources, the recent releases of tabular and surrogate benchmarks have greatly increased the speed and reproducibility of NAS research.
1 code implementation • 5 Oct 2021 • Carolin Benjamins, Theresa Eimer, Frederik Schubert, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, Marius Lindauer
While Reinforcement Learning has made great strides towards solving ever more complicated tasks, many algorithms are still brittle to even slight changes in their environment.
no code implementations • ICLR 2022 • Carl Hvarfner, Danny Stoll, Artur Souza, Luigi Nardi, Marius Lindauer, Frank Hutter
To address this issue, we propose $\pi$BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user.
1 code implementation • 20 Sep 2021 • Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhopf, René Sass, Frank Hutter
Algorithm parameters, in particular hyperparameters of machine learning algorithms, can substantially impact their performance.
2 code implementations • 14 Sep 2021 • Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor Awad, Marius Lindauer, Frank Hutter
To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications.
no code implementations • 9 Jul 2021 • Ashwin Raaghav Narayanan, Arber Zela, Tonmoy Saikia, Thomas Brox, Frank Hutter
Ensembles of CNN models trained with different seeds (also known as Deep Ensembles) are known to achieve superior performance over a single copy of the CNN.
no code implementations • 8 Jul 2021 • Thomas Elsken, Benedikt Staffler, Arber Zela, Jan Hendrik Metzen, Frank Hutter
While neural architecture search methods have been successful in previous years and led to new state-of-the-art performance on various problems, they have also been criticized for being unstable, being highly sensitive with respect to their hyperparameters, and often not performing better than random search.
1 code implementation • NeurIPS 2021 • Arlind Kadra, Marius Lindauer, Frank Hutter, Josif Grabocka
Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures.
1 code implementation • 9 Jun 2021 • André Biedenkapp, Raghu Rajan, Frank Hutter, Marius Lindauer
Reinforcement learning is a powerful approach to learn behaviour through interactions with an environment.
1 code implementation • 9 Jun 2021 • Theresa Eimer, André Biedenkapp, Frank Hutter, Marius Lindauer
Reinforcement learning (RL) has made a lot of advances for solving a single problem in a given environment; but learning policies that generalize to unseen variations of a problem remains challenging.
1 code implementation • 20 May 2021 • Noor Awad, Neeratyoy Mallik, Frank Hutter
Modern machine learning algorithms crucially rely on several design decisions to achieve strong performance, making the problem of Hyperparameter Optimization (HPO) more important than ever.
1 code implementation • 18 May 2021 • Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriaensen, Frank Hutter, Marius Lindauer
Dynamic Algorithm Configuration (DAC) aims to dynamically control a target algorithm's hyperparameters in order to improve its performance.
1 code implementation • ICML Workshop AutoML 2021 • Julia Guerrero-Viu, Sven Hauns, Sergio Izquierdo, Guilherme Miotto, Simon Schrodi, Andre Biedenkapp, Thomas Elsken, Difan Deng, Marius Lindauer, Frank Hutter
Neural architecture search (NAS) and hyperparameter optimization (HPO) make deep learning accessible to non-experts by automatically finding the architecture of the deep neural network to use and tuning the hyperparameters of the used training pipeline.
1 code implementation • NeurIPS 2021 • Colin White, Arber Zela, Binxin Ru, Yang Liu, Frank Hutter
Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks.
2 code implementations • ICCV 2021 • Samuel G. Müller, Frank Hutter
Automatic augmentation methods have recently become a crucial pillar for strong model performance in vision tasks.
Ranked #6 on
Data Augmentation
on ImageNet
1 code implementation • 26 Feb 2021 • Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, Roberto Calandra
We demonstrate that this problem can be tackled effectively with automated HPO, which we demonstrate to yield significantly improved performance compared to human experts.
Hyperparameter Optimization
Model-based Reinforcement Learning
+2
no code implementations • 5 Feb 2021 • Samuel Müller, André Biedenkapp, Frank Hutter
To do this, we optimize the loss of the next training step.
1 code implementation • 24 Jan 2021 • Fabio Ferreira, Thomas Nierhoff, Frank Hutter
This work explores learning agent-agnostic synthetic environments (SEs) for Reinforcement Learning.
no code implementations • 1 Jan 2021 • Arlind Kadra, Marius Lindauer, Frank Hutter, Josif Grabocka
The regularization of prediction models is arguably the most crucial ingredient that allows Machine Learning solutions to generalize well on unseen data.
no code implementations • 1 Jan 2021 • Michael Ruchte, Arber Zela, Julien Niklas Siems, Josif Grabocka, Frank Hutter
Neural Architecture Search (NAS) is one of the focal points for the Deep Learning community, but reproducing NAS methods is extremely challenging due to numerous low-level implementation details.
1 code implementation • 15 Dec 2020 • Noor Awad, Gresa Shala, Difan Deng, Neeratyoy Mallik, Matthias Feurer, Katharina Eggensperger, Andre' Biedenkapp, Diederick Vermetten, Hao Wang, Carola Doerr, Marius Lindauer, Frank Hutter
In this short note, we describe our submission to the NeurIPS 2020 BBO challenge.
1 code implementation • 11 Dec 2020 • Noor Awad, Neeratyoy Mallik, Frank Hutter
Neural architecture search (NAS) methods rely on a search strategy for deciding which architectures to evaluate next and a performance estimation strategy for assessing their performance (e. g., using full evaluations, multi-fidelity evaluations, or the one-shot model).
no code implementations • 20 Nov 2020 • Matilde Gargiani, Andrea Zanelli, Quoc Tran-Dinh, Moritz Diehl, Frank Hutter
In this work, we present a first-order stochastic algorithm based on a combination of homotopy methods and SGD, called Homotopy-Stochastic Gradient Descent (H-SGD), which finds interesting connections with some proposed heuristics in the literature, e. g. optimization by Gaussian continuation, training by diffusion, mollifying networks.
1 code implementation • 25 Oct 2020 • Danny Stoll, Jörg K. H. Franke, Diane Wagner, Simon Selg, Frank Hutter
After developer adjustments to a machine learning (ML) algorithm, how can the results of an old hyperparameter optimization (HPO) automatically be used to speedup a new HPO?
no code implementations • 15 Oct 2020 • Mauro Vallati, Lukas Chrpa, Thomas L. McCluskey, Frank Hutter
The development of domain-independent planners within the AI Planning community is leading to "off-the-shelf" technology that can be used in a wide range of applications.
2 code implementations • 9 Oct 2020 • Jovita Lukasik, David Friede, Arber Zela, Frank Hutter, Margret Keuper
We evaluate the proposed approach on neural architectures defined by the ENAS approach, the NAS-Bench-101 and the NAS-Bench-201 search space and show that our smooth embedding space allows to directly extrapolate the performance prediction to architectures outside the seen domain (e. g. with more operations).
no code implementations • 29 Sep 2020 • Katharina Eggensperger, Kai Haase, Philipp Müller, Marius Lindauer, Frank Hutter
When fitting a regression model to predict the distribution of the outcomes, we cannot simply drop these right-censored observations, but need to properly model them.
no code implementations • 28 Sep 2020 • Artur Souza, Luigi Nardi, Leonardo Oliveira, Kunle Olukotun, Marius Lindauer, Frank Hutter
While Bayesian Optimization (BO) is a very popular method for optimizing expensive black-box functions, it fails to leverage the experience of domain experts.
no code implementations • 28 Sep 2020 • Raghu Rajan, Jessica Lizeth Borja Diaz, Suresh Guttikonda, Fabio Ferreira, André Biedenkapp, Frank Hutter
We present MDP Playground, an efficient benchmark for Reinforcement Learning (RL) algorithms with various dimensions of hardness that can be controlled independently to challenge algorithms in different ways and to obtain varying degrees of hardness in generated environments.
1 code implementation • ICLR 2021 • Jörg K. H. Franke, Gregor Köhler, André Biedenkapp, Frank Hutter
Despite significant progress in challenging problems across various domains, applying state-of-the-art deep reinforcement learning (RL) algorithms remains challenging due to their sensitivity to the choice of hyperparameters.
1 code implementation • ICLR 2022 • Arber Zela, Julien Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, Frank Hutter
We show that surrogate NAS benchmarks can model the true performance of architectures better than tabular benchmarks (at a small fraction of the cost), that they lead to faithful estimates of how well different NAS methods work on the original non-surrogate benchmark, and that they can generate new scientific insight.
4 code implementations • 8 Jul 2020 • Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, Frank Hutter
Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success.
no code implementations • 25 Jun 2020 • Artur Souza, Luigi Nardi, Leonardo B. Oliveira, Kunle Olukotun, Marius Lindauer, Frank Hutter
We show that BOPrO is around 6. 67x faster than state-of-the-art methods on a common suite of benchmarks, and achieves a new state-of-the-art performance on a real-world hardware design application.
2 code implementations • 24 Jun 2020 • Lucas Zimmer, Marius Lindauer, Frank Hutter
While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search.
1 code implementation • NeurIPS 2021 • Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, Yee Whye Teh
On a variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift.
1 code implementation • 15 Jun 2020 • David Speck, André Biedenkapp, Frank Hutter, Robert Mattmüller, Marius Lindauer
We show that dynamic algorithm configuration can be used for dynamic heuristic selection which takes into account the internal search dynamics of a planning system.
1 code implementation • 3 Jun 2020 • Matilde Gargiani, Andrea Zanelli, Moritz Diehl, Frank Hutter
This enables researchers to further study and improve this promising optimization technique and hopefully reconsider stochastic second-order methods as competitive optimization techniques for training DNNs; we also hope that the promise of SGN may lead to forward automatic differentiation being added to Tensorflow or Pytorch.
1 code implementation • 1 Jun 2020 • André Biedenkapp, H. Furkan Bozkurt, Theresa Eimer, Frank Hutter, Marius Lindauer
The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on parameter tuning.
no code implementations • ICLR 2020 • Matilde Gargiani, Andrea Zanelli, Quoc Tran Dinh, Moritz Diehl, Frank Hutter
Homotopy methods, also known as continuation methods, are a powerful mathematical tool to efficiently solve various problems in numerical analysis, including complex non-convex optimization problems where no or only little prior knowledge regarding the localization of the solutions is available.
1 code implementation • 11 Feb 2020 • Lukas Alexander Wilhelm Gemein, Robin Tibor Schirrmeister, Patryk Chrabąszcz, Daniel Wilson, Joschka Boedecker, Andreas Schulze-Bonhage, Frank Hutter, Tonio Ball
The results demonstrate that the proposed feature-based decoding framework can achieve accuracies on the same level as state-of-the-art deep neural networks.
1 code implementation • ICLR 2020 • Arber Zela, Julien Siems, Frank Hutter
One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice.
2 code implementations • CVPR 2020 • Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, Frank Hutter
The recent progress in neural architecture search (NAS) has allowed scaling the automated design of neural architectures to real-world domains, such as object detection and semantic segmentation.
1 code implementation • 6 Nov 2019 • Matthias Feurer, Jan N. van Rijn, Arlind Kadra, Pieter Gijsbers, Neeratyoy Mallik, Sahithya Ravi, Andreas Müller, Joaquin Vanschoren, Frank Hutter
It also provides functionality to conduct machine learning experiments, upload the results to OpenML, and reproduce results which are stored on OpenML.
no code implementations • 28 Oct 2019 • Jörg K. H. Franke, Gregor Köhler, Noor Awad, Frank Hutter
Current Deep Reinforcement Learning algorithms still heavily rely on handcrafted neural network architectures.
1 code implementation • 10 Oct 2019 • Matilde Gargiani, Aaron Klein, Stefan Falkner, Frank Hutter
We propose probabilistic models that can extrapolate learning curves of iterative machine learning algorithms, such as stochastic gradient descent for training deep networks, based on training data with variable-length learning curves.
1 code implementation • ICLR 2020 • Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, Frank Hutter
Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem.
1 code implementation • 17 Sep 2019 • Raghu Rajan, Jessica Lizeth Borja Diaz, Suresh Guttikonda, Fabio Ferreira, André Biedenkapp, Jan Ole von Hartz, Frank Hutter
We present \emph{MDP Playground}, an efficient testbed for Reinforcement Learning (RL) agents with \textit{orthogonal} dimensions that can be controlled independently to challenge agents in different ways and obtain varying degrees of hardness in generated environments.
no code implementations • 5 Sep 2019 • Marius Lindauer, Frank Hutter
Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS).
no code implementations • 19 Aug 2019 • Marius Lindauer, Matthias Feurer, Katharina Eggensperger, André Biedenkapp, Frank Hutter
Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning.
1 code implementation • 16 Aug 2019 • Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Joshua Marben, Philipp Müller, Frank Hutter
Hyperparameter optimization and neural architecture search can become prohibitively expensive for regular black-box Bayesian optimization because the training and evaluation of a single model can easily take several hours.
no code implementations • 18 Jun 2019 • André Biedenkapp, H. Furkan Bozkurt, Frank Hutter, Marius Lindauer
The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on tuned hyperparameter configurations.
1 code implementation • NeurIPS 2019 • Aaron Klein, Zhenwen Dai, Frank Hutter, Neil Lawrence, Javier Gonzalez
Despite the recent progress in hyperparameter optimization (HPO), available benchmarks that resemble real-world scenarios consist of a few and very large problem instances that are expensive to solve.
2 code implementations • 18 May 2019 • Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, Matthias Urban, Michael Burkart, Maximilian Dippel, Marius Lindauer, Frank Hutter
Recent advances in AutoML have led to automated tools that can compete with machine learning experts on supervised learning tasks.
1 code implementation • ICCV 2019 • Tonmoy Saikia, Yassine Marrakchi, Arber Zela, Frank Hutter, Thomas Brox
In this work, we show how to use and extend existing AutoML techniques to efficiently optimize large-scale U-Net-like encoder-decoder architectures.
1 code implementation • 13 May 2019 • Aaron Klein, Frank Hutter
Due to the high computational demands executing a rigorous comparison between hyperparameter optimization (HPO) methods is often cumbersome.
2 code implementations • ICLR 2020 • Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, Christian Daniel
Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the field of global black-box optimization.
4 code implementations • 25 Feb 2019 • Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, Frank Hutter
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation.
1 code implementation • ICLR 2019 • Frederic Runge, Danny Stoll, Stefan Falkner, Frank Hutter
Designing RNA molecules has garnered recent interest in medicine, synthetic biology, biotechnology and bioinformatics since many functional RNA molecules were shown to be involved in regulatory processes for transcription, epigenetics and translation.
1 code implementation • IDA 2018: Advances in Intelligent Data Analysis XVII 2018 • Benjamin Strang, Peter van der Putten, Jan N. van Rijn, Frank Hutter
A basic step for each data-mining or machine learning task is to determine which model to choose based on the problem and the data at hand.
1 code implementation • 16 Aug 2018 • Thomas Elsken, Jan Hendrik Metzen, Frank Hutter
Deep Learning has enabled remarkable progress over the last years on a variety of tasks, such as image recognition, speech recognition, and machine translation.
3 code implementations • 18 Jul 2018 • Arber Zela, Aaron Klein, Stefan Falkner, Frank Hutter
While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal.
4 code implementations • ICML 2018 • Stefan Falkner, Aaron Klein, Frank Hutter
Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible.
1 code implementation • 5 Jun 2018 • Robin Tibor Schirrmeister, Patryk Chrabąszcz, Frank Hutter, Tonio Ball
This first attempt to use RevNets inside the adversarial autoencoder framework slightly underperformed relative to recent advanced generative models using an autoencoder component on CelebA, but this gap may diminish with further optimization of the training setup of generative RevNets.
1 code implementation • NeurIPS 2018 • James T. Wilson, Frank Hutter, Marc Peter Deisenroth
Bayesian optimization is a sample-efficient approach to global optimization that relies on theoretically motivated value heuristics (acquisition functions) to guide its search process.
no code implementations • ICLR 2019 • Thomas Elsken, Jan Hendrik Metzen, Frank Hutter
Neural Architecture Search aims at automatically finding neural architectures that are competitive with architectures designed by human experts.
no code implementations • 9 Mar 2018 • Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J. Bentley, Samuel Bernard, Guillaume Beslon, David M. Bryson, Patryk Chrabaszcz, Nick Cheney, Antoine Cully, Stephane Doncieux, Fred C. Dyer, Kai Olav Ellefsen, Robert Feldt, Stephan Fischer, Stephanie Forrest, Antoine Frénoy, Christian Gagné, Leni Le Goff, Laura M. Grabowski, Babak Hodjat, Frank Hutter, Laurent Keller, Carole Knibbe, Peter Krcah, Richard E. Lenski, Hod Lipson, Robert MacCurdy, Carlos Maestre, Risto Miikkulainen, Sara Mitri, David E. Moriarty, Jean-Baptiste Mouret, Anh Nguyen, Charles Ofria, Marc Parizeau, David Parsons, Robert T. Pennock, William F. Punch, Thomas S. Ray, Marc Schoenauer, Eric Shulte, Karl Sims, Kenneth O. Stanley, François Taddei, Danesh Tarapore, Simon Thibault, Westley Weimer, Richard Watson, Jason Yosinski
Biological evolution provides a creative fount of complex and subtle adaptations, often surprising the scientists who discover them.
1 code implementation • 24 Feb 2018 • Patryk Chrabaszcz, Ilya Loshchilov, Frank Hutter
Evolution Strategies (ES) have recently been demonstrated to be a viable alternative to reinforcement learning (RL) algorithms on a set of challenging deep RL problems, including Atari games and MuJoCo humanoid locomotion benchmarks.
1 code implementation • ECCV 2018 • Eddy Ilg, Özgün Çiçek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, Thomas Brox
Optical flow estimation can be formulated as an end-to-end supervised learning problem, which yields estimates with a superior accuracy-runtime tradeoff compared to alternative methodology.
2 code implementations • 6 Feb 2018 • Matthias Feurer, Benjamin Letham, Frank Hutter, Eytan Bakshy
When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset.
no code implementations • ICLR 2018 • Ilya Loshchilov, Frank Hutter
We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.
1 code implementation • 1 Dec 2017 • James T. Wilson, Riccardo Moriconi, Frank Hutter, Marc Peter Deisenroth
Bayesian optimization is a sample-efficient approach to solving global optimization problems.
20 code implementations • ICLR 2019 • Ilya Loshchilov, Frank Hutter
L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradient algorithms, such as Adam.
3 code implementations • ICLR 2018 • Thomas Elsken, Jan-Hendrik Metzen, Frank Hutter
Neural networks have recently had a lot of success for many tasks.
no code implementations • 22 Sep 2017 • Katharina Eggensperger, Marius Lindauer, Frank Hutter
Many state-of-the-art algorithms for solving hard combinatorial problems in artificial intelligence (AI) include elements of stochasticity that lead to high variations in runtime, even for a fixed problem instance.
no code implementations • 14 Sep 2017 • Marius Lindauer, Frank Hutter
The performance of many hard combinatorial problem solvers depends strongly on their parameter settings, and since manual parameter tuning is both tedious and suboptimal the AI community has recently developed several algorithm configuration (AC) methods to automatically address this problem.
2 code implementations • 26 Aug 2017 • Robin Tibor Schirrmeister, Lukas Gemein, Katharina Eggensperger, Frank Hutter, Tonio Ball
We apply convolutional neural networks (ConvNets) to the task of distinguishing pathological from normal EEG recordings in the Temple University Hospital EEG Abnormal Corpus.
4 code implementations • 11 Aug 2017 • Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G. Mantovani, Jan N. van Rijn, Joaquin Vanschoren
Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks.
6 code implementations • 27 Jul 2017 • Patryk Chrabaszcz, Ilya Loshchilov, Frank Hutter
The original ImageNet dataset is a popular large-scale benchmark for training Deep Neural Networks.
2 code implementations • 17 May 2017 • Katharina Eggensperger, Marius Lindauer, Frank Hutter
Good parameter settings are crucial to achieve high performance in many areas of artificial intelligence (AI), such as propositional satisfiability solving, AI planning, scheduling, and machine learning (in particular deep learning).
no code implementations • 30 Mar 2017 • Katharina Eggensperger, Marius Lindauer, Holger H. Hoos, Frank Hutter, Kevin Leyton-Brown
In our experiments, we construct and evaluate surrogate benchmarks for hyperparameter optimization as well as for AC problems that involve performance optimization of solvers for hard combinatorial problems, drawing training data from the runs of existing AC procedures.
5 code implementations • 15 Mar 2017 • Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, Tonio Ball
PLEASE READ AND CITE THE REVISED VERSION at Human Brain Mapping: http://onlinelibrary. wiley. com/doi/10. 1002/hbm. 23730/full Code available here: https://github. com/robintibor/braindecode
1 code implementation • NIPS 2017 2017 • Aaron Klein, Stefan Falkner, Numair Mansur, Frank Hutter
Bayesian optimization is a powerful approach for the global derivative-free optimization of non-convex expensive functions.
no code implementations • 2 Dec 2016 • Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, Frank Hutter
We consider parallel asynchronous Markov Chain Monte Carlo (MCMC) sampling for problems where we can leverage (stochastic) gradients to define continuous dynamics which explore the target distribution.
1 code implementation • NeurIPS 2016 • Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, Frank Hutter
Bayesian optimization is a prominent method for optimizing expensive to evaluate black-box functions that is prominently applied to tuning the hyperparameters of machine learning algorithms.
no code implementations • 2 Sep 2016 • Markus Wagner, Marius Lindauer, Mustafa Misir, Samadhi Nallaperuma, Frank Hutter
Many real-world problems are composed of several interacting components.
17 code implementations • 13 Aug 2016 • Ilya Loshchilov, Frank Hutter
Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions.
1 code implementation • 23 May 2016 • Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, Frank Hutter
Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks.
no code implementations • 25 Apr 2016 • Ilya Loshchilov, Frank Hutter
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization.
2 code implementations • NeurIPS 2015 • Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, Frank Hutter
The success of machine learning in a broad range of applications has led to an ever-growing demand for machine learning systems that can be used off the shelf by non-experts.
1 code implementation • 19 Nov 2015 • Ilya Loshchilov, Frank Hutter
We investigate online batch selection strategies for two state-of-the-art methods of stochastic gradient-based optimization, AdaDelta and Adam.
2 code implementations • 8 Jun 2015 • Bernd Bischl, Pascal Kerschke, Lars Kotthoff, Marius Lindauer, Yuri Malitsky, Alexandre Frechette, Holger Hoos, Frank Hutter, Kevin Leyton-Brown, Kevin Tierney, Joaquin Vanschoren
To address this problem, we introduce a standardized format for representing algorithm selection scenarios and a repository that contains a growing number of data sets from the literature.
no code implementations • 5 May 2015 • Frank Hutter, Marius Lindauer, Adrian Balint, Sam Bayless, Holger Hoos, Kevin Leyton-Brown
It is well known that different solution strategies work well for different types of instances of hard combinatorial problems.
1 code implementation • NIPS 2015 2015 • Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, Frank Hutter
Supplementary Material for Efficient and Robust Automated Machine Learning
no code implementations • 14 Sep 2014 • Kevin Swersky, David Duvenaud, Jasper Snoek, Frank Hutter, Michael A. Osborne
In practical Bayesian optimization, we must often search over structures with differing numbers of parameters.
no code implementations • 15 Jan 2014 • Frank Hutter, Thomas Stuetzle, Kevin Leyton-Brown, Holger H. Hoos
The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms.
no code implementations • 21 Oct 2013 • Frank Hutter, Michael A. Osborne
We define a family of kernels for mixed continuous/discrete hierarchical parameter spaces and show that they are positive definite.
no code implementations • 7 Oct 2013 • Frank Hutter, Holger Hoos, Kevin Leyton-Brown
Bayesian optimization (BO) aims to minimize a given blackbox function using a model that is updated whenever new evidence about the function becomes available.
1 code implementation • 9 Jan 2013 • Ziyu Wang, Frank Hutter, Masrour Zoghi, David Matheson, Nando de Freitas
Bayesian optimization techniques have been successfully applied to robotics, planning, sensor placement, recommendation, advertising, intelligent user interfaces and automatic algorithm configuration.
no code implementations • 5 Nov 2012 • Frank Hutter, Lin Xu, Holger H. Hoos, Kevin Leyton-Brown
We also comprehensively describe new and existing features for predicting algorithm runtime for propositional satisfiability (SAT), travelling salesperson (TSP) and mixed integer programming (MIP) problems.
1 code implementation • 18 Aug 2012 • Chris Thornton, Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown
Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall.
1 code implementation • LION 2011 2011 • Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown
State-of-the-art algorithms for hard computational problems often expose many parameters that can be modified to improve empirical performance.