Search Results for author:
Found 134 papers, 90 papers with code
PFNs Are Flexible Models for Real-World Bayesian Optimization
In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO).
MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization
Hyperparameter optimization (HPO) is a powerful technique for automating the tuning of machine learning (ML) models.
LLMs for Semi-Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset.
Self-Correcting Bayesian Optimization through Bayesian Active Learning
Gaussian processes are cemented as the model of choice in Bayesian optimization and active learning.
PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces
The recent rise in popularity of Hyperparameter Optimization (HPO) for deep learning has highlighted the role that good hyperparameter (HP) space design can play in training strong models.
Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
The field of automated machine learning (AutoML) introduces techniques that automate parts of the development of machine learning (ML) systems, accelerating the process and reducing barriers for novices.
Neural Architecture Search: Insights from 1000 Papers
Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas.
Natural Language Understanding
Neural Architecture Search
+2
Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator
Hyperparameter optimization (HPO) is a vital step in improving performance in deep learning (DL).
Mind the Gap: Measuring Generalization Performance Across Multiple Objectives
Modern machine learning models are often constructed taking into account multiple objectives, e. g., minimizing inference time while also maximizing accuracy.
c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization
In this work, we propose constrained TPE (c-TPE), an extension of the widely-used versatile Bayesian optimization method, tree-structured Parzen estimator (TPE), to handle these constraints.
Towards Discovering Neural Architectures from Scratch
The discovery of neural architectures from scratch is the long-standing goal of Neural Architecture Search (NAS).
On the Importance of Architectures and Hyperparameters for Fairness in Face Recognition
Motivated by our findings, we run the first neural architecture search for fairness, jointly with a search for hyperparameters.
NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
Zero-cost proxies (ZC proxies) are a recent architecture performance prediction technique aiming to significantly speed up algorithms for neural architecture search (NAS).
T3VIP: Transformation-based 3D Video Prediction
To the best of our knowledge, our model is the first generative model that provides an RGB-D video prediction of the future for a static camera.
On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learning
Self-Supervised Learning (SSL) has become a very active area of Deep Learning research where it is heavily used as a pre-training method for classification and other tasks.
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
We present TabPFN, a trained Transformer that can do supervised classification for small tabular datasets in less than a second, needs no hyperparameter tuning and is competitive with state-of-the-art classification methods.
Zero-Shot AutoML with Pretrained Models
Given a new dataset D and a low compute budget, how should we choose a pre-trained model to fine-tune to D, and set the fine-tuning hyperparameters without risking overfitting, particularly if D is small?
Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification
Although deep neural networks are capable of achieving performance superior to humans on various tasks, they are notorious for requiring large amounts of data and computing resources, restricting their success to domains where such resources are available.
Joint Entropy Search for Maximally-Informed Bayesian Optimization
As a light-weight approach with superior results, JES provides a new go-to acquisition function for Bayesian optimization.
DeepCAVE: An Interactive Analysis Tool for Automated Machine Learning
Automated Machine Learning (AutoML) is used more than ever before to support users in determining efficient hyperparameters, neural architectures, or even full machine learning pipelines.
Automated Dynamic Algorithm Configuration
The performance of an algorithm often critically depends on its parameter configuration.
Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design
Our world is ambiguous and this is reflected in the data we use to train our algorithms.
Efficient Automated Deep Learning for Time Series Forecasting
In contrast to common NAS search spaces, we designed a novel neural architecture search space covering various state-of-the-art architectures, allowing for an efficient macro-search over different DL approaches.
$π$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization
To address this issue, we propose $\pi$BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user.
Why Do Machine Learning Practitioners Still Use Manual Tuning? A Qualitative Study
Current advanced hyperparameter optimization (HPO) methods, such as Bayesian optimization, have high sampling efficiency and facilitate replicability.
Neural Architecture Search for Dense Prediction Tasks in Computer Vision
The success of deep learning in recent years has lead to a rising demand for neural network architecture engineering.
Contextualize Me -- The Case for Context in Reinforcement Learning
While Reinforcement Learning ( RL) has made great strides towards solving increasingly complicated problems, many algorithms are still brittle to even slight environmental changes.
Theory-inspired Parameter Control Benchmarks for Dynamic Algorithm Configuration
We extend this benchmark by analyzing optimal control policies that can select the parameters only from a given portfolio of possible values.
Learning Synthetic Environments and Reward Networks for Reinforcement Learning
In a one-to-one comparison, learning an SE proxy requires more interactions with the real environment than training agents only on the real environment.
NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy
The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS).
Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019
Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly.
Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable agents.
Transformers Can Do Bayesian Inference
Our method restates the objective of posterior approximation as a supervised classification problem with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their labels from it, masks one of the labels and learns to make probabilistic predictions for it based on the set-valued input of the rest of the data points.
NAS-Bench-x11 and the Power of Learning Curves
While early research in neural architecture search (NAS) required extreme computational resources, the recent releases of tabular and surrogate benchmarks have greatly increased the speed and reproducibility of NAS research.
CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
While Reinforcement Learning has made great strides towards solving ever more complicated tasks, many algorithms are still brittle to even slight changes in their environment.
$\pi$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization
To address this issue, we propose $\pi$BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user.
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
Algorithm parameters, in particular hyperparameters of machine learning algorithms, can substantially impact their performance.
HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications.
Multi-headed Neural Ensemble Search
Ensembles of CNN models trained with different seeds (also known as Deep Ensembles) are known to achieve superior performance over a single copy of the CNN.
Bag of Tricks for Neural Architecture Search
While neural architecture search methods have been successful in previous years and led to new state-of-the-art performance on various problems, they have also been criticized for being unstable, being highly sensitive with respect to their hyperparameters, and often not performing better than random search.
Well-tuned Simple Nets Excel on Tabular Datasets
Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures.
TempoRL: Learning When to Act
Reinforcement learning is a powerful approach to learn behaviour through interactions with an environment.
Self-Paced Context Evaluation for Contextual Reinforcement Learning
Reinforcement learning (RL) has made a lot of advances for solving a single problem in a given environment; but learning policies that generalize to unseen variations of a problem remains challenging.
DEHB: Evolutionary Hyperband for Scalable, Robust and Efficient Hyperparameter Optimization
Modern machine learning algorithms crucially rely on several design decisions to achieve strong performance, making the problem of Hyperparameter Optimization (HPO) more important than ever.
DACBench: A Benchmark Library for Dynamic Algorithm Configuration
Dynamic Algorithm Configuration (DAC) aims to dynamically control a target algorithm's hyperparameters in order to improve its performance.
Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization
Neural architecture search (NAS) and hyperparameter optimization (HPO) make deep learning accessible to non-experts by automatically finding the architecture of the deep neural network to use and tuning the hyperparameters of the used training pipeline.
How Powerful are Performance Predictors in Neural Architecture Search?
Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks.
TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation
Automatic augmentation methods have recently become a crucial pillar for strong model performance in vision tasks.
Ranked #6 on
Data Augmentation
on ImageNet
On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
We demonstrate that this problem can be tackled effectively with automated HPO, which we demonstrate to yield significantly improved performance compared to human experts.
Hyperparameter Optimization
Model-based Reinforcement Learning
+2
In-Loop Meta-Learning with Gradient-Alignment Reward
To do this, we optimize the loss of the next training step.
Learning Synthetic Environments for Reinforcement Learning with Evolution Strategies
This work explores learning agent-agnostic synthetic environments (SEs) for Reinforcement Learning.
Regularization Cocktails
The regularization of prediction models is arguably the most crucial ingredient that allows Machine Learning solutions to generalize well on unseen data.
NASLib: A Modular and Flexible Neural Architecture Search Library
Neural Architecture Search (NAS) is one of the focal points for the Deep Learning community, but reproducing NAS methods is extremely challenging due to numerous low-level implementation details.
Squirrel: A Switching Hyperparameter Optimizer
In this short note, we describe our submission to the NeurIPS 2020 BBO challenge.
Differential Evolution for Neural Architecture Search
Neural architecture search (NAS) methods rely on a search strategy for deciding which architectures to evaluate next and a performance estimation strategy for assessing their performance (e. g., using full evaluations, multi-fidelity evaluations, or the one-shot model).
Convergence Analysis of Homotopy-SGD for non-convex optimization
In this work, we present a first-order stochastic algorithm based on a combination of homotopy methods and SGD, called Homotopy-Stochastic Gradient Descent (H-SGD), which finds interesting connections with some proposed heuristics in the literature, e. g. optimization by Gaussian continuation, training by diffusion, mollifying networks.
Hyperparameter Transfer Across Developer Adjustments
After developer adjustments to a machine learning (ML) algorithm, how can the results of an old hyperparameter optimization (HPO) automatically be used to speedup a new HPO?
On the Importance of Domain Model Configuration for Automated Planning Engines
The development of domain-independent planners within the AI Planning community is leading to "off-the-shelf" technology that can be used in a wide range of applications.
Smooth Variational Graph Embeddings for Efficient Neural Architecture Search
We evaluate the proposed approach on neural architectures defined by the ENAS approach, the NAS-Bench-101 and the NAS-Bench-201 search space and show that our smooth embedding space allows to directly extrapolate the performance prediction to architectures outside the seen domain (e. g. with more operations).
Neural Model-based Optimization with Right-Censored Observations
When fitting a regression model to predict the distribution of the outcomes, we cannot simply drop these right-censored observations, but need to properly model them.
Prior-guided Bayesian Optimization
While Bayesian Optimization (BO) is a very popular method for optimizing expensive black-box functions, it fails to leverage the experience of domain experts.
MDP Playground: Controlling Orthogonal Dimensions of Hardness in Toy Environments
We present MDP Playground, an efficient benchmark for Reinforcement Learning (RL) algorithms with various dimensions of hardness that can be controlled independently to challenge algorithms in different ways and to obtain varying degrees of hardness in generated environments.
Sample-Efficient Automated Deep Reinforcement Learning
Despite significant progress in challenging problems across various domains, applying state-of-the-art deep reinforcement learning (RL) algorithms remains challenging due to their sensitivity to the choice of hyperparameters.
Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks
We show that surrogate NAS benchmarks can model the true performance of architectures better than tabular benchmarks (at a small fraction of the cost), that they lead to faithful estimates of how well different NAS methods work on the original non-surrogate benchmark, and that they can generate new scientific insight.
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning
Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success.
Bayesian Optimization with a Prior for the Optimum
We show that BOPrO is around 6. 67x faster than state-of-the-art methods on a common suite of benchmarks, and achieves a new state-of-the-art performance on a real-world hardware design application.
Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search.
Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
On a variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift.
Learning Heuristic Selection with Dynamic Algorithm Configuration
We show that dynamic algorithm configuration can be used for dynamic heuristic selection which takes into account the internal search dynamics of a planning system.
On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNs
This enables researchers to further study and improve this promising optimization technique and hopefully reconsider stochastic second-order methods as competitive optimization techniques for training DNNs; we also hope that the promise of SGN may lead to forward automatic differentiation being added to Tensorflow or Pytorch.
Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework
The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on parameter tuning.
Transferring Optimality Across Data Distributions via Homotopy Methods
Homotopy methods, also known as continuation methods, are a powerful mathematical tool to efficiently solve various problems in numerical analysis, including complex non-convex optimization problems where no or only little prior knowledge regarding the localization of the solutions is available.
Machine-Learning-Based Diagnostics of EEG Pathology
The results demonstrate that the proposed feature-based decoding framework can achieve accuracies on the same level as state-of-the-art deep neural networks.
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice.
Meta-Learning of Neural Architectures for Few-Shot Learning
The recent progress in neural architecture search (NAS) has allowed scaling the automated design of neural architectures to real-world domains, such as object detection and semantic segmentation.
OpenML-Python: an extensible Python API for OpenML
It also provides functionality to conduct machine learning experiments, upload the results to OpenML, and reproduce results which are stored on OpenML.
Neural Architecture Evolution in Deep Reinforcement Learning for Continuous Control
Current Deep Reinforcement Learning algorithms still heavily rely on handcrafted neural network architectures.
Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings
We propose probabilistic models that can extrapolate learning curves of iterative machine learning algorithms, such as stochastic gradient descent for training deep networks, based on training data with variable-length learning curves.
Understanding and Robustifying Differentiable Architecture Search
Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem.
MDP Playground: A Design and Debug Testbed for Reinforcement Learning
We present \emph{MDP Playground}, an efficient testbed for Reinforcement Learning (RL) agents with \textit{orthogonal} dimensions that can be controlled independently to challenge agents in different ways and obtain varying degrees of hardness in generated environments.
Best Practices for Scientific Research on Neural Architecture Search
Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS).
Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters
Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning.
BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters
Hyperparameter optimization and neural architecture search can become prohibitively expensive for regular black-box Bayesian optimization because the training and evaluation of a single model can easily take several hours.
Towards White-box Benchmarks for Algorithm Control
The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on tuned hyperparameter configurations.
Meta-Surrogate Benchmarking for Hyperparameter Optimization
Despite the recent progress in hyperparameter optimization (HPO), available benchmarks that resemble real-world scenarios consist of a few and very large problem instances that are expensive to solve.
Towards Automatically-Tuned Deep Neural Networks
Recent advances in AutoML have led to automated tools that can compete with machine learning experts on supervised learning tasks.
AutoDispNet: Improving Disparity Estimation With AutoML
In this work, we show how to use and extend existing AutoML techniques to efficiently optimize large-scale U-Net-like encoder-decoder architectures.
Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
Due to the high computational demands executing a rigorous comparison between hyperparameter optimization (HPO) methods is often cumbersome.
Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization
Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the field of global black-box optimization.
NAS-Bench-101: Towards Reproducible Neural Architecture Search
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation.
Learning to Design RNA
Designing RNA molecules has garnered recent interest in medicine, synthetic biology, biotechnology and bioinformatics since many functional RNA molecules were shown to be involved in regulatory processes for transcription, epigenetics and translation.
Don't Rule Out Simple Models Prematurely: A Large Scale Benchmark Comparing Linear and Non-linear Classifiers in OpenML
A basic step for each data-mining or machine learning task is to determine which model to choose based on the problem and the data at hand.
Neural Architecture Search: A Survey
Deep Learning has enabled remarkable progress over the last years on a variety of tasks, such as image recognition, speech recognition, and machine translation.
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal.
BOHB: Robust and Efficient Hyperparameter Optimization at Scale
Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible.
Training Generative Reversible Networks
This first attempt to use RevNets inside the adversarial autoencoder framework slightly underperformed relative to recent advanced generative models using an autoencoder component on CelebA, but this gap may diminish with further optimization of the training setup of generative RevNets.
Maximizing acquisition functions for Bayesian optimization
Bayesian optimization is a sample-efficient approach to global optimization that relies on theoretically motivated value heuristics (acquisition functions) to guide its search process.
Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution
Neural Architecture Search aims at automatically finding neural architectures that are competitive with architectures designed by human experts.
The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities
Biological evolution provides a creative fount of complex and subtle adaptations, often surprising the scientists who discover them.
Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari
Evolution Strategies (ES) have recently been demonstrated to be a viable alternative to reinforcement learning (RL) algorithms on a set of challenging deep RL problems, including Atari games and MuJoCo humanoid locomotion benchmarks.
Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow
Optical flow estimation can be formulated as an end-to-end supervised learning problem, which yields estimates with a superior accuracy-runtime tradeoff compared to alternative methodology.
Practical Transfer Learning for Bayesian Optimization
When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset.
Fixing Weight Decay Regularization in Adam
We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.
The reparameterization trick for acquisition functions
Bayesian optimization is a sample-efficient approach to solving global optimization problems.
Decoupled Weight Decay Regularization
L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradient algorithms, such as Adam.
Simple And Efficient Architecture Search for Convolutional Neural Networks
Neural networks have recently had a lot of success for many tasks.
Neural Networks for Predicting Algorithm Runtime Distributions
Many state-of-the-art algorithms for solving hard combinatorial problems in artificial intelligence (AI) include elements of stochasticity that lead to high variations in runtime, even for a fixed problem instance.
Warmstarting of Model-based Algorithm Configuration
The performance of many hard combinatorial problem solvers depends strongly on their parameter settings, and since manual parameter tuning is both tedious and suboptimal the AI community has recently developed several algorithm configuration (AC) methods to automatically address this problem.
Deep learning with convolutional neural networks for decoding and visualization of EEG pathology
We apply convolutional neural networks (ConvNets) to the task of distinguishing pathological from normal EEG recordings in the Temple University Hospital EEG Abnormal Corpus.
OpenML Benchmarking Suites
Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks.
A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
The original ImageNet dataset is a popular large-scale benchmark for training Deep Neural Networks.
Pitfalls and Best Practices in Algorithm Configuration
Good parameter settings are crucial to achieve high performance in many areas of artificial intelligence (AI), such as propositional satisfiability solving, AI planning, scheduling, and machine learning (in particular deep learning).
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates
In our experiments, we construct and evaluate surrogate benchmarks for hyperparameter optimization as well as for AC problems that involve performance optimization of solvers for hard combinatorial problems, drawing training data from the runs of existing AC procedures.
Deep learning with convolutional neural networks for EEG decoding and visualization
PLEASE READ AND CITE THE REVISED VERSION at Human Brain Mapping: http://onlinelibrary. wiley. com/doi/10. 1002/hbm. 23730/full Code available here: https://github. com/robintibor/braindecode
RoBO: A Flexible and Robust Bayesian Optimization Framework in Python
Bayesian optimization is a powerful approach for the global derivative-free optimization of non-convex expensive functions.
Asynchronous Stochastic Gradient MCMC with Elastic Coupling
We consider parallel asynchronous Markov Chain Monte Carlo (MCMC) sampling for problems where we can leverage (stochastic) gradients to define continuous dynamics which explore the target distribution.
Bayesian Optimization with Robust Bayesian Neural Networks
Bayesian optimization is a prominent method for optimizing expensive to evaluate black-box functions that is prominently applied to tuning the hyperparameters of machine learning algorithms.
A case study of algorithm selection for the traveling thief problem
Many real-world problems are composed of several interacting components.
SGDR: Stochastic Gradient Descent with Warm Restarts
Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions.
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks.
CMA-ES for Hyperparameter Optimization of Deep Neural Networks
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization.
Efficient and Robust Automated Machine Learning
The success of machine learning in a broad range of applications has led to an ever-growing demand for machine learning systems that can be used off the shelf by non-experts.
Online Batch Selection for Faster Training of Neural Networks
We investigate online batch selection strategies for two state-of-the-art methods of stochastic gradient-based optimization, AdaDelta and Adam.
ASlib: A Benchmark Library for Algorithm Selection
To address this problem, we introduce a standardized format for representing algorithm selection scenarios and a repository that contains a growing number of data sets from the literature.
The Configurable SAT Solver Challenge (CSSC)
It is well known that different solution strategies work well for different types of instances of hard combinatorial problems.
Supplementary Material for Efficient and Robust Automated Machine Learning
Supplementary Material for Efficient and Robust Automated Machine Learning
Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
In practical Bayesian optimization, we must often search over structures with differing numbers of parameters.
ParamILS: An Automatic Algorithm Configuration Framework
The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms.
A Kernel for Hierarchical Parameter Spaces
We define a family of kernels for mixed continuous/discrete hierarchical parameter spaces and show that they are positive definite.
Bayesian Optimization With Censored Response Data
Bayesian optimization (BO) aims to minimize a given blackbox function using a model that is updated whenever new evidence about the function becomes available.
Bayesian Optimization in a Billion Dimensions via Random Embeddings
Bayesian optimization techniques have been successfully applied to robotics, planning, sensor placement, recommendation, advertising, intelligent user interfaces and automatic algorithm configuration.
Algorithm Runtime Prediction: Methods & Evaluation
We also comprehensively describe new and existing features for predicting algorithm runtime for propositional satisfiability (SAT), travelling salesperson (TSP) and mixed integer programming (MIP) problems.
Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms
Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall.
Sequential Model-Based Optimization for General Algorithm Configuration
State-of-the-art algorithms for hard computational problems often expose many parameters that can be modified to improve empirical performance.
