Search Results for author:
Found 180 papers, 122 papers with code
Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization
As model sizes grow, finding efficient and cost-effective hyperparameter optimization (HPO) methods becomes increasingly crucial for deep learning pipelines.
Unreflected Use of Tabular Data Repositories Can Undermine Research Quality
Data repositories have accumulated a large number of tabular datasets from various domains.
DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling, offering efficient training and linear-time inference.
EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Networks
Recent foundational models for tabular data, such as TabPFN, have demonstrated remarkable effectiveness in adapting to new tasks through in-context learning.
Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
The GoLU activation leverages the asymmetry in the Gompertz function to reduce variance in the latent space more effectively compared to GELU and Swish, while preserving robust gradient flow.
Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes
While matching or surpassing the performance of the transformer at sufficiently small dataset sizes and GBDTs at sufficiently large sizes, LLM-Boost and PFN-Boost outperform both standalone components on a wide range of dataset sizes in between.
Tuning LLM Judges Hyperparameters
To address this, LLM-based judges have been proposed, which compare the outputs of two LLMs enabling the ranking of models without human intervention.
The Tabular Foundation Model TabPFN Outperforms Specialized Time Series Forecasting Models Based on Simple Features
Foundation models have become popular in forecasting due to their ability to make accurate predictions, even with minimal fine-tuning on specific datasets.
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
Our experiments confirm that extending the eigenvalue range of Mamba and DeltaNet to include negative values not only enables them to solve parity but consistently improves their performance on state-tracking tasks.
Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data
To model shifts of these causal models, we use a secondary SCM, that specifies changes in the primary model parameters.
Warmstarting for Scaling Language Models
We investigate the aspects that contribute to the speedup in convergence and the preservation of stable training dynamics under warmstarting with {\mu}Transfer.
Transfer Learning for Finetuning Large Language Models
To reduce the complexity for practitioners, we investigate transfer learning for finetuning large language models and aim to transfer knowledge about configurations from related finetuning tasks to a new task.
Ensembling Finetuned Language Models for Text Classification
Finetuning is a common practice widespread across different communities to adapt pretrained models to particular tasks.
Large Language Models Engineer Too Many Simple Features For Tabular Data
Furthermore, the bias can negatively impact the predictive performance when using LLM-generated features.
A Human-in-the-Loop Fairness-Aware Model Selection Framework for Complex Fairness Objective Landscapes
In this socio-technical and empirical study, we frame fairness as a many-objective (MaO) problem by treating fairness metrics as conflicting objectives.
Large Language Model Compression with Neural Architecture Search
Large language models (LLMs) exhibit remarkable reasoning abilities, allowing them to generalize across a wide range of downstream tasks, such as commonsense reasoning or instruction following.
GAMformer: In-Context Learning for Generalized Additive Models
Generalized Additive Models (GAMs) are widely recognized for their ability to create fully interpretable machine learning models for tabular data.
Dynamic Post-Hoc Neural Ensemblers
In this study, we explore employing neural networks as ensemble methods, emphasizing the significance of dynamic ensembling to leverage diverse model predictions adaptively.
Bayes' Power for Explaining In-Context Learning Generalizations
In this paper, we argue that a more useful interpretation of neural network behavior in this era is as an approximation of the true posterior, as defined by the data-generating process.
ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning
With the extensive and large-scale dataset on hyperparameter landscapes that our selection is based on, ARLBench is an efficient, flexible, and future-oriented foundation for research on AutoRL.
One-shot World Models Using a Transformer Trained on a Synthetic Prior
We propose One-Shot World Model (OSWM), a transformer world model that is learned in an in-context learning fashion from purely synthetic data sampled from a prior distribution.
Efficient Search for Customized Activation Functions with Gradient Descent
Different activation functions work best for different deep learning models.
LMEMs for post-hoc analysis of HPO Benchmarking
The importance of tuning hyperparameters in Machine Learning (ML) and Deep Learning (DL) is established through empirical research and applications, evident from the increase in new hyperparameter optimization (HPO) algorithms and benchmarks steadily added by the community.
FairPFN: Transformers Can do Counterfactual Fairness
Machine Learning systems are increasingly prevalent across healthcare, law enforcement, and finance but often operate on historical data, which may carry biases against certain demographic groups.
Fast Optimizer Benchmark
In this paper, we present the Fast Optimizer Benchmark (FOB), a tool designed for evaluating deep learning optimizers during their development.
Position: A Call to Action for a Human-Centered AutoML Paradigm
Automated machine learning (AutoML) was formed around the fundamental objectives of automatically and efficiently configuring machine learning (ML) workflows, aiding the research of new ML algorithms, and contributing to the democratization of ML by making it accessible to a broader audience.
HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models
The increasing size of language models necessitates a thorough analysis across multiple dimensions to assess trade-offs among crucial hardware metrics such as latency, energy consumption, GPU memory usage, and performance.
Don't Waste Your Time: Early Stopping Cross-Validation
In addition, we investigate the impact of early stopping with Bayesian optimization instead of random search and also repeated cross-validation.
In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization
In this work, we propose FT-PFN, a novel surrogate for Freeze-thaw style BO.
Surprisingly Strong Performance Prediction with Neural Graph Features
Performance prediction has been a key part of the neural architecture search (NAS) process, allowing to speed up NAS algorithms by avoiding resource-consuming network training.
Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks
While deep learning has celebrated many successes, its results often hinge on the meticulous selection of hyperparameters (HPs).
Multi-objective Differentiable Neural Architecture Search
Pareto front profiling in multi-objective optimization (MOO), i. e. finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives like neural network training.
Diffusion-Based Neural Network Weights Generation
Transfer learning has gained significant attention in recent deep learning research due to its ability to accelerate convergence and enhance performance on new tasks.
TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks
Notably, TabPFN achieves very strong performance on small tabular datasets but is not designed to make predictions for datasets of size larger than 1000.
Is Mamba Capable of In-Context Learning?
State of the art foundation models such as GPT-4 perform surprisingly well at in-context learning (ICL), a variant of meta-learning concerning the learned ability to solve tasks during a neural network forward pass, exploiting contextual information provided as input to the model.
Weight-Entanglement Meets Gradient-Based Neural Architecture Search
%Due to the inherent differences in the structure of these search spaces, these Since weight-entanglement poses compatibility challenges for gradient-based NAS methods, these two paradigms have largely developed independently in parallel sub-communities.
Rethinking Performance Measures of RNA Secondary Structure Problems
Accurate RNA secondary structure prediction is vital for understanding cellular regulation and disease mechanisms.
A General Framework for User-Guided Bayesian Optimization
The optimization of expensive-to-evaluate black-box functions is prevalent in various scientific disciplines.
Improving Deep Learning Optimization through Constrained Parameter Regularization
Unlike the uniform application of a single penalty, CPR enforces an upper bound on a statistical measure, such as the L2-norm, of individual parameter matrices.
Managing extreme AI risks amid rapid progress
Artificial Intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and pursue goals.
Beyond Random Augmentations: Pretraining with Hard Views
To achieve this invariance, conventional approaches make use of random sampling operations within the image augmentation pipeline.
Towards Automated Design of Riboswitches
Experimental screening and selection pipelines for the discovery of novel riboswitches are expensive, time-consuming, and inefficient.
Scalable Deep Learning for RNA Secondary Structure Prediction
The field of RNA secondary structure prediction has made significant progress with the adoption of deep learning techniques.
PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning
Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance.
Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How
With the ever-increasing number of pretrained models, machine learning practitioners are continuously faced with which pretrained model to use, and how to finetune it for a new dataset.
PFNs4BO: In-Context Learning for Bayesian Optimization
In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO).
MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization
Hyperparameter optimization (HPO) is a powerful technique for automating the tuning of machine learning (ML) models.
Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset.
Self-Correcting Bayesian Optimization through Bayesian Active Learning
Gaussian processes are the model of choice in Bayesian optimization and active learning.
PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces
The recent rise in popularity of Hyperparameter Optimization (HPO) for deep learning has highlighted the role that good hyperparameter (HP) space design can play in training strong models.
Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
The field of automated machine learning (AutoML) introduces techniques that automate parts of the development of machine learning (ML) systems, accelerating the process and reducing barriers for novices.
Neural Architecture Search: Insights from 1000 Papers
Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas.
Natural Language Understanding
Neural Architecture Search
+2
Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator
Hyperparameter optimization (HPO) is a vital step in improving performance in deep learning (DL).
Mind the Gap: Measuring Generalization Performance Across Multiple Objectives
Modern machine learning models are often constructed taking into account multiple objectives, e. g., minimizing inference time while also maximizing accuracy.
c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization
In this work, we propose constrained TPE (c-TPE), an extension of the widely-used versatile Bayesian optimization method, tree-structured Parzen estimator (TPE), to handle these constraints.
Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars
In this work, we introduce a unifying search space design framework based on context-free grammars that can naturally and compactly generate expressive hierarchical search spaces that are 100s of orders of magnitude larger than common spaces from the literature.
Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition
Our search outputs a suite of models which Pareto-dominate all other high-performance architectures and existing bias mitigation methods in terms of accuracy and fairness, often by large margins, on the two most widely used datasets for face identification, CelebA and VGGFace2.
NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
Zero-cost proxies (ZC proxies) are a recent architecture performance prediction technique aiming to significantly speed up algorithms for neural architecture search (NAS).
T3VIP: Transformation-based 3D Video Prediction
To the best of our knowledge, our model is the first generative model that provides an RGB-D video prediction of the future for a static camera.
On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learning
Self-Supervised Learning (SSL) has become a very active area of Deep Learning research where it is heavily used as a pre-training method for classification and other tasks.
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
We present TabPFN, a trained Transformer that can do supervised classification for small tabular datasets in less than a second, needs no hyperparameter tuning and is competitive with state-of-the-art classification methods.
Zero-Shot AutoML with Pretrained Models
Given a new dataset D and a low compute budget, how should we choose a pre-trained model to fine-tune to D, and set the fine-tuning hyperparameters without risking overfitting, particularly if D is small?
Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification
Although deep neural networks are capable of achieving performance superior to humans on various tasks, they are notorious for requiring large amounts of data and computing resources, restricting their success to domains where such resources are available.
Joint Entropy Search for Maximally-Informed Bayesian Optimization
As a light-weight approach with superior results, JES provides a new go-to acquisition function for Bayesian optimization.
DeepCAVE: An Interactive Analysis Tool for Automated Machine Learning
Automated Machine Learning (AutoML) is used more than ever before to support users in determining efficient hyperparameters, neural architectures, or even full machine learning pipelines.
Automated Dynamic Algorithm Configuration
The performance of an algorithm often critically depends on its parameter configuration.
Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design
Our world is ambiguous and this is reflected in the data we use to train our algorithms.
Efficient Automated Deep Learning for Time Series Forecasting
In contrast to common NAS search spaces, we designed a novel neural architecture search space covering various state-of-the-art architectures, allowing for an efficient macro-search over different DL approaches.
$π$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization
To address this issue, we propose $\pi$BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user.
Practitioner Motives to Select Hyperparameter Optimization Methods
Advanced programmatic hyperparameter optimization (HPO) methods, such as Bayesian optimization, have high sample efficiency in reproducibly finding optimal hyperparameter values of machine learning (ML) models.
Neural Architecture Search for Dense Prediction Tasks in Computer Vision
The success of deep learning in recent years has lead to a rising demand for neural network architecture engineering.
Contextualize Me -- The Case for Context in Reinforcement Learning
While Reinforcement Learning ( RL) has made great strides towards solving increasingly complicated problems, many algorithms are still brittle to even slight environmental changes.
Theory-inspired Parameter Control Benchmarks for Dynamic Algorithm Configuration
We extend this benchmark by analyzing optimal control policies that can select the parameters only from a given portfolio of possible values.
Learning Synthetic Environments and Reward Networks for Reinforcement Learning
In a one-to-one comparison, learning an SE proxy requires more interactions with the real environment than training agents only on the real environment.
NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy
The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS).
Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable agents.
Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019
Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly.
Transformers Can Do Bayesian Inference
Our method restates the objective of posterior approximation as a supervised classification problem with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their labels from it, masks one of the labels and learns to make probabilistic predictions for it based on the set-valued input of the rest of the data points.
NAS-Bench-x11 and the Power of Learning Curves
While early research in neural architecture search (NAS) required extreme computational resources, the recent releases of tabular and surrogate benchmarks have greatly increased the speed and reproducibility of NAS research.
CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
While Reinforcement Learning has made great strides towards solving ever more complicated tasks, many algorithms are still brittle to even slight changes in their environment.
$\pi$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization
To address this issue, we propose $\pi$BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user.
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
Algorithm parameters, in particular hyperparameters of machine learning algorithms, can substantially impact their performance.
HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications.
Multi-headed Neural Ensemble Search
Ensembles of CNN models trained with different seeds (also known as Deep Ensembles) are known to achieve superior performance over a single copy of the CNN.
Bag of Tricks for Neural Architecture Search
While neural architecture search methods have been successful in previous years and led to new state-of-the-art performance on various problems, they have also been criticized for being unstable, being highly sensitive with respect to their hyperparameters, and often not performing better than random search.
Well-tuned Simple Nets Excel on Tabular Datasets
Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures.
TempoRL: Learning When to Act
Reinforcement learning is a powerful approach to learn behaviour through interactions with an environment.
Self-Paced Context Evaluation for Contextual Reinforcement Learning
Reinforcement learning (RL) has made a lot of advances for solving a single problem in a given environment; but learning policies that generalize to unseen variations of a problem remains challenging.
DEHB: Evolutionary Hyperband for Scalable, Robust and Efficient Hyperparameter Optimization
Modern machine learning algorithms crucially rely on several design decisions to achieve strong performance, making the problem of Hyperparameter Optimization (HPO) more important than ever.
DACBench: A Benchmark Library for Dynamic Algorithm Configuration
Dynamic Algorithm Configuration (DAC) aims to dynamically control a target algorithm's hyperparameters in order to improve its performance.
Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization
Neural architecture search (NAS) and hyperparameter optimization (HPO) make deep learning accessible to non-experts by automatically finding the architecture of the deep neural network to use and tuning the hyperparameters of the used training pipeline.
How Powerful are Performance Predictors in Neural Architecture Search?
Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks.
TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation
Automatic augmentation methods have recently become a crucial pillar for strong model performance in vision tasks.
Ranked #8 on
Data Augmentation
on ImageNet
On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
We demonstrate that this problem can be tackled effectively with automated HPO, which we demonstrate to yield significantly improved performance compared to human experts.
Hyperparameter Optimization
Model-based Reinforcement Learning
+2
In-Loop Meta-Learning with Gradient-Alignment Reward
To do this, we optimize the loss of the next training step.
Learning Synthetic Environments for Reinforcement Learning with Evolution Strategies
This work explores learning agent-agnostic synthetic environments (SEs) for Reinforcement Learning.
NASLib: A Modular and Flexible Neural Architecture Search Library
Neural Architecture Search (NAS) is one of the focal points for the Deep Learning community, but reproducing NAS methods is extremely challenging due to numerous low-level implementation details.
Regularization Cocktails
The regularization of prediction models is arguably the most crucial ingredient that allows Machine Learning solutions to generalize well on unseen data.
Squirrel: A Switching Hyperparameter Optimizer
In this short note, we describe our submission to the NeurIPS 2020 BBO challenge.
Differential Evolution for Neural Architecture Search
Neural architecture search (NAS) methods rely on a search strategy for deciding which architectures to evaluate next and a performance estimation strategy for assessing their performance (e. g., using full evaluations, multi-fidelity evaluations, or the one-shot model).
Convergence Analysis of Homotopy-SGD for non-convex optimization
In this work, we present a first-order stochastic algorithm based on a combination of homotopy methods and SGD, called Homotopy-Stochastic Gradient Descent (H-SGD), which finds interesting connections with some proposed heuristics in the literature, e. g. optimization by Gaussian continuation, training by diffusion, mollifying networks.
Hyperparameter Transfer Across Developer Adjustments
After developer adjustments to a machine learning (ML) algorithm, how can the results of an old hyperparameter optimization (HPO) automatically be used to speedup a new HPO?
On the Importance of Domain Model Configuration for Automated Planning Engines
The development of domain-independent planners within the AI Planning community is leading to "off-the-shelf" technology that can be used in a wide range of applications.
Smooth Variational Graph Embeddings for Efficient Neural Architecture Search
We evaluate the proposed approach on neural architectures defined by the ENAS approach, the NAS-Bench-101 and the NAS-Bench-201 search space and show that our smooth embedding space allows to directly extrapolate the performance prediction to architectures outside the seen domain (e. g. with more operations).
Neural Model-based Optimization with Right-Censored Observations
When fitting a regression model to predict the distribution of the outcomes, we cannot simply drop these right-censored observations, but need to properly model them.
MDP Playground: Controlling Orthogonal Dimensions of Hardness in Toy Environments
We present MDP Playground, an efficient benchmark for Reinforcement Learning (RL) algorithms with various dimensions of hardness that can be controlled independently to challenge algorithms in different ways and to obtain varying degrees of hardness in generated environments.
Prior-guided Bayesian Optimization
While Bayesian Optimization (BO) is a very popular method for optimizing expensive black-box functions, it fails to leverage the experience of domain experts.
Sample-Efficient Automated Deep Reinforcement Learning
Despite significant progress in challenging problems across various domains, applying state-of-the-art deep reinforcement learning (RL) algorithms remains challenging due to their sensitivity to the choice of hyperparameters.
Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks
We show that surrogate NAS benchmarks can model the true performance of architectures better than tabular benchmarks (at a small fraction of the cost), that they lead to faithful estimates of how well different NAS methods work on the original non-surrogate benchmark, and that they can generate new scientific insight.
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning
Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success.
Bayesian Optimization with a Prior for the Optimum
We show that BOPrO is around 6. 67x faster than state-of-the-art methods on a common suite of benchmarks, and achieves a new state-of-the-art performance on a real-world hardware design application.
Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search.
Learning Heuristic Selection with Dynamic Algorithm Configuration
We show that dynamic algorithm configuration can be used for dynamic heuristic selection which takes into account the internal search dynamics of a planning system.
Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
On a variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift.
On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNs
This enables researchers to further study and improve this promising optimization technique and hopefully reconsider stochastic second-order methods as competitive optimization techniques for training DNNs; we also hope that the promise of SGN may lead to forward automatic differentiation being added to Tensorflow or Pytorch.
Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework
The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on parameter tuning.
Transferring Optimality Across Data Distributions via Homotopy Methods
Homotopy methods, also known as continuation methods, are a powerful mathematical tool to efficiently solve various problems in numerical analysis, including complex non-convex optimization problems where no or only little prior knowledge regarding the localization of the solutions is available.
Machine-Learning-Based Diagnostics of EEG Pathology
The results demonstrate that the proposed feature-based decoding framework can achieve accuracies on the same level as state-of-the-art deep neural networks.
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice.
Meta-Learning of Neural Architectures for Few-Shot Learning
The recent progress in neural architecture search (NAS) has allowed scaling the automated design of neural architectures to real-world domains, such as object detection and semantic segmentation.
OpenML-Python: an extensible Python API for OpenML
It also provides functionality to conduct machine learning experiments, upload the results to OpenML, and reproduce results which are stored on OpenML.
Neural Architecture Evolution in Deep Reinforcement Learning for Continuous Control
Current Deep Reinforcement Learning algorithms still heavily rely on handcrafted neural network architectures.
Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings
We propose probabilistic models that can extrapolate learning curves of iterative machine learning algorithms, such as stochastic gradient descent for training deep networks, based on training data with variable-length learning curves.
Understanding and Robustifying Differentiable Architecture Search
Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem.
MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning
We define a parameterised collection of fast-to-run toy environments in OpenAI Gym by varying these dimensions and propose to use these to understand agents better.
Best Practices for Scientific Research on Neural Architecture Search
Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS).
Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters
Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning.
BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters
Hyperparameter optimization and neural architecture search can become prohibitively expensive for regular black-box Bayesian optimization because the training and evaluation of a single model can easily take several hours.
Towards White-box Benchmarks for Algorithm Control
The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on tuned hyperparameter configurations.
Meta-Surrogate Benchmarking for Hyperparameter Optimization
Despite the recent progress in hyperparameter optimization (HPO), available benchmarks that resemble real-world scenarios consist of a few and very large problem instances that are expensive to solve.
Towards Automatically-Tuned Deep Neural Networks
Recent advances in AutoML have led to automated tools that can compete with machine learning experts on supervised learning tasks.
AutoDispNet: Improving Disparity Estimation With AutoML
In this work, we show how to use and extend existing AutoML techniques to efficiently optimize large-scale U-Net-like encoder-decoder architectures.
Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
Due to the high computational demands executing a rigorous comparison between hyperparameter optimization (HPO) methods is often cumbersome.
Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization
Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the field of global black-box optimization.
NAS-Bench-101: Towards Reproducible Neural Architecture Search
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation.
Learning to Design RNA
Designing RNA molecules has garnered recent interest in medicine, synthetic biology, biotechnology and bioinformatics since many functional RNA molecules were shown to be involved in regulatory processes for transcription, epigenetics and translation.
Don't Rule Out Simple Models Prematurely: A Large Scale Benchmark Comparing Linear and Non-linear Classifiers in OpenML
A basic step for each data-mining or machine learning task is to determine which model to choose based on the problem and the data at hand.
Neural Architecture Search: A Survey
Deep Learning has enabled remarkable progress over the last years on a variety of tasks, such as image recognition, speech recognition, and machine translation.
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal.
BOHB: Robust and Efficient Hyperparameter Optimization at Scale
Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible.
Training Generative Reversible Networks
This first attempt to use RevNets inside the adversarial autoencoder framework slightly underperformed relative to recent advanced generative models using an autoencoder component on CelebA, but this gap may diminish with further optimization of the training setup of generative RevNets.
Maximizing acquisition functions for Bayesian optimization
Bayesian optimization is a sample-efficient approach to global optimization that relies on theoretically motivated value heuristics (acquisition functions) to guide its search process.
Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution
Neural Architecture Search aims at automatically finding neural architectures that are competitive with architectures designed by human experts.
The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities
Biological evolution provides a creative fount of complex and subtle adaptations, often surprising the scientists who discover them.
Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari
Evolution Strategies (ES) have recently been demonstrated to be a viable alternative to reinforcement learning (RL) algorithms on a set of challenging deep RL problems, including Atari games and MuJoCo humanoid locomotion benchmarks.
Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow
Optical flow estimation can be formulated as an end-to-end supervised learning problem, which yields estimates with a superior accuracy-runtime tradeoff compared to alternative methodology.
Practical Transfer Learning for Bayesian Optimization
When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset.
Fixing Weight Decay Regularization in Adam
We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.
The reparameterization trick for acquisition functions
Bayesian optimization is a sample-efficient approach to solving global optimization problems.
Decoupled Weight Decay Regularization
L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradient algorithms, such as Adam.
Simple And Efficient Architecture Search for Convolutional Neural Networks
Neural networks have recently had a lot of success for many tasks.
Neural Networks for Predicting Algorithm Runtime Distributions
Many state-of-the-art algorithms for solving hard combinatorial problems in artificial intelligence (AI) include elements of stochasticity that lead to high variations in runtime, even for a fixed problem instance.
Warmstarting of Model-based Algorithm Configuration
The performance of many hard combinatorial problem solvers depends strongly on their parameter settings, and since manual parameter tuning is both tedious and suboptimal the AI community has recently developed several algorithm configuration (AC) methods to automatically address this problem.
Deep learning with convolutional neural networks for decoding and visualization of EEG pathology
We apply convolutional neural networks (ConvNets) to the task of distinguishing pathological from normal EEG recordings in the Temple University Hospital EEG Abnormal Corpus.
OpenML Benchmarking Suites
Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks.
A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
The original ImageNet dataset is a popular large-scale benchmark for training Deep Neural Networks.
Ranked #1 on
Image Classification
on ImageNet-32
Pitfalls and Best Practices in Algorithm Configuration
Good parameter settings are crucial to achieve high performance in many areas of artificial intelligence (AI), such as propositional satisfiability solving, AI planning, scheduling, and machine learning (in particular deep learning).
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates
In our experiments, we construct and evaluate surrogate benchmarks for hyperparameter optimization as well as for AC problems that involve performance optimization of solvers for hard combinatorial problems, drawing training data from the runs of existing AC procedures.
Deep learning with convolutional neural networks for EEG decoding and visualization
PLEASE READ AND CITE THE REVISED VERSION at Human Brain Mapping: http://onlinelibrary. wiley. com/doi/10. 1002/hbm. 23730/full Code available here: https://github. com/robintibor/braindecode
RoBO: A Flexible and Robust Bayesian Optimization Framework in Python
Bayesian optimization is a powerful approach for the global derivative-free optimization of non-convex expensive functions.
Asynchronous Stochastic Gradient MCMC with Elastic Coupling
We consider parallel asynchronous Markov Chain Monte Carlo (MCMC) sampling for problems where we can leverage (stochastic) gradients to define continuous dynamics which explore the target distribution.
Bayesian Optimization with Robust Bayesian Neural Networks
Bayesian optimization is a prominent method for optimizing expensive to evaluate black-box functions that is prominently applied to tuning the hyperparameters of machine learning algorithms.
A case study of algorithm selection for the traveling thief problem
Many real-world problems are composed of several interacting components.
SGDR: Stochastic Gradient Descent with Warm Restarts
Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions.
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks.
CMA-ES for Hyperparameter Optimization of Deep Neural Networks
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization.
Efficient and Robust Automated Machine Learning
The success of machine learning in a broad range of applications has led to an ever-growing demand for machine learning systems that can be used off the shelf by non-experts.
Online Batch Selection for Faster Training of Neural Networks
We investigate online batch selection strategies for two state-of-the-art methods of stochastic gradient-based optimization, AdaDelta and Adam.
ASlib: A Benchmark Library for Algorithm Selection
To address this problem, we introduce a standardized format for representing algorithm selection scenarios and a repository that contains a growing number of data sets from the literature.
The Configurable SAT Solver Challenge (CSSC)
It is well known that different solution strategies work well for different types of instances of hard combinatorial problems.
Supplementary Material for Efficient and Robust Automated Machine Learning
Supplementary Material for Efficient and Robust Automated Machine Learning
Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
In practical Bayesian optimization, we must often search over structures with differing numbers of parameters.
ParamILS: An Automatic Algorithm Configuration Framework
The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms.
A Kernel for Hierarchical Parameter Spaces
We define a family of kernels for mixed continuous/discrete hierarchical parameter spaces and show that they are positive definite.
Bayesian Optimization With Censored Response Data
Bayesian optimization (BO) aims to minimize a given blackbox function using a model that is updated whenever new evidence about the function becomes available.
Bayesian Optimization in a Billion Dimensions via Random Embeddings
Bayesian optimization techniques have been successfully applied to robotics, planning, sensor placement, recommendation, advertising, intelligent user interfaces and automatic algorithm configuration.
Algorithm Runtime Prediction: Methods & Evaluation
We also comprehensively describe new and existing features for predicting algorithm runtime for propositional satisfiability (SAT), travelling salesperson (TSP) and mixed integer programming (MIP) problems.
Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms
Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall.
Sequential Model-Based Optimization for General Algorithm Configuration
State-of-the-art algorithms for hard computational problems often expose many parameters that can be modified to improve empirical performance.
