Search Results for author:
Found 148 papers, 100 papers with code
SGDR: Stochastic Gradient Descent with Warm Restarts
Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions.
TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation
Automatic augmentation methods have recently become a crucial pillar for strong model performance in vision tasks.
Ranked #8 on
Data Augmentation
on ImageNet
Efficient and Robust Automated Machine Learning
The success of machine learning in a broad range of applications has led to an ever-growing demand for machine learning systems that can be used off the shelf by non-experts.
Supplementary Material for Efficient and Robust Automated Machine Learning
Supplementary Material for Efficient and Robust Automated Machine Learning
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning
Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success.
Online Batch Selection for Faster Training of Neural Networks
We investigate online batch selection strategies for two state-of-the-art methods of stochastic gradient-based optimization, AdaDelta and Adam.
Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search.
Towards Automatically-Tuned Deep Neural Networks
Recent advances in AutoML have led to automated tools that can compete with machine learning experts on supervised learning tasks.
Efficient Automated Deep Learning for Time Series Forecasting
In contrast to common NAS search spaces, we designed a novel neural architecture search space covering various state-of-the-art architectures, allowing for an efficient macro-search over different DL approaches.
Decoupled Weight Decay Regularization
L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradient algorithms, such as Adam.
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
We present TabPFN, a trained Transformer that can do supervised classification for small tabular datasets in less than a second, needs no hyperparameter tuning and is competitive with state-of-the-art classification methods.
Sequential Model-Based Optimization for General Algorithm Configuration
State-of-the-art algorithms for hard computational problems often expose many parameters that can be modified to improve empirical performance.
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
Algorithm parameters, in particular hyperparameters of machine learning algorithms, can substantially impact their performance.
DeepCAVE: An Interactive Analysis Tool for Automated Machine Learning
Automated Machine Learning (AutoML) is used more than ever before to support users in determining efficient hyperparameters, neural architectures, or even full machine learning pipelines.
Deep learning with convolutional neural networks for EEG decoding and visualization
PLEASE READ AND CITE THE REVISED VERSION at Human Brain Mapping: http://onlinelibrary. wiley. com/doi/10. 1002/hbm. 23730/full Code available here: https://github. com/robintibor/braindecode
NAS-Bench-101: Towards Reproducible Neural Architecture Search
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation.
OpenML Benchmarking Suites
Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks.
BOHB: Robust and Efficient Hyperparameter Optimization at Scale
Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible.
HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications.
Meta-Surrogate Benchmarking for Hyperparameter Optimization
Despite the recent progress in hyperparameter optimization (HPO), available benchmarks that resemble real-world scenarios consist of a few and very large problem instances that are expensive to solve.
NASLib: A Modular and Flexible Neural Architecture Search Library
Neural Architecture Search (NAS) is one of the focal points for the Deep Learning community, but reproducing NAS methods is extremely challenging due to numerous low-level implementation details.
How Powerful are Performance Predictors in Neural Architecture Search?
Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks.
NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy
The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS).
NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
Zero-cost proxies (ZC proxies) are a recent architecture performance prediction technique aiming to significantly speed up algorithms for neural architecture search (NAS).
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks.
RoBO: A Flexible and Robust Bayesian Optimization Framework in Python
Bayesian optimization is a powerful approach for the global derivative-free optimization of non-convex expensive functions.
Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms
Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall.
Don't Rule Out Simple Models Prematurely: A Large Scale Benchmark Comparing Linear and Non-linear Classifiers in OpenML
A basic step for each data-mining or machine learning task is to determine which model to choose based on the problem and the data at hand.
OpenML-Python: an extensible Python API for OpenML
It also provides functionality to conduct machine learning experiments, upload the results to OpenML, and reproduce results which are stored on OpenML.
A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
The original ImageNet dataset is a popular large-scale benchmark for training Deep Neural Networks.
Ranked #1 on
Image Classification
on ImageNet-32
Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition
Our search outputs a suite of models which Pareto-dominate all other high-performance architectures and existing bias mitigation methods in terms of accuracy and fairness, often by large margins, on the two most widely used datasets for face identification, CelebA and VGGFace2.
Transformers Can Do Bayesian Inference
Our method restates the objective of posterior approximation as a supervised classification problem with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their labels from it, masks one of the labels and learns to make probabilistic predictions for it based on the set-valued input of the rest of the data points.
Understanding and Robustifying Differentiable Architecture Search
Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem.
Bayesian Optimization with Robust Bayesian Neural Networks
Bayesian optimization is a prominent method for optimizing expensive to evaluate black-box functions that is prominently applied to tuning the hyperparameters of machine learning algorithms.
CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
While Reinforcement Learning has made great strides towards solving ever more complicated tasks, many algorithms are still brittle to even slight changes in their environment.
Contextualize Me -- The Case for Context in Reinforcement Learning
While Reinforcement Learning ( RL) has made great strides towards solving increasingly complicated problems, many algorithms are still brittle to even slight environmental changes.
Bayesian Optimization in a Billion Dimensions via Random Embeddings
Bayesian optimization techniques have been successfully applied to robotics, planning, sensor placement, recommendation, advertising, intelligent user interfaces and automatic algorithm configuration.
Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
Due to the high computational demands executing a rigorous comparison between hyperparameter optimization (HPO) methods is often cumbersome.
Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari
Evolution Strategies (ES) have recently been demonstrated to be a viable alternative to reinforcement learning (RL) algorithms on a set of challenging deep RL problems, including Atari games and MuJoCo humanoid locomotion benchmarks.
Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks
We show that surrogate NAS benchmarks can model the true performance of architectures better than tabular benchmarks (at a small fraction of the cost), that they lead to faithful estimates of how well different NAS methods work on the original non-surrogate benchmark, and that they can generate new scientific insight.
Well-tuned Simple Nets Excel on Tabular Datasets
Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures.
BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters
Hyperparameter optimization and neural architecture search can become prohibitively expensive for regular black-box Bayesian optimization because the training and evaluation of a single model can easily take several hours.
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice.
DEHB: Evolutionary Hyperband for Scalable, Robust and Efficient Hyperparameter Optimization
Modern machine learning algorithms crucially rely on several design decisions to achieve strong performance, making the problem of Hyperparameter Optimization (HPO) more important than ever.
Learning to Design RNA
Designing RNA molecules has garnered recent interest in medicine, synthetic biology, biotechnology and bioinformatics since many functional RNA molecules were shown to be involved in regulatory processes for transcription, epigenetics and translation.
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal.
Zero-Shot AutoML with Pretrained Models
Given a new dataset D and a low compute budget, how should we choose a pre-trained model to fine-tune to D, and set the fine-tuning hyperparameters without risking overfitting, particularly if D is small?
Deep learning with convolutional neural networks for decoding and visualization of EEG pathology
We apply convolutional neural networks (ConvNets) to the task of distinguishing pathological from normal EEG recordings in the Temple University Hospital EEG Abnormal Corpus.
Sample-Efficient Automated Deep Reinforcement Learning
Despite significant progress in challenging problems across various domains, applying state-of-the-art deep reinforcement learning (RL) algorithms remains challenging due to their sensitivity to the choice of hyperparameters.
$π$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization
To address this issue, we propose $\pi$BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user.
PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning
Hyperparameters of Deep Learning (DL) pipelines are crucial for their downstream performance.
Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars
In this work, we introduce a unifying search space design framework based on context-free grammars that can naturally and compactly generate expressive hierarchical search spaces that are 100s of orders of magnitude larger than common spaces from the literature.
Training Generative Reversible Networks
This first attempt to use RevNets inside the adversarial autoencoder framework slightly underperformed relative to recent advanced generative models using an autoencoder component on CelebA, but this gap may diminish with further optimization of the training setup of generative RevNets.
Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization
Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the field of global black-box optimization.
Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
On a variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift.
Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow
Optical flow estimation can be formulated as an end-to-end supervised learning problem, which yields estimates with a superior accuracy-runtime tradeoff compared to alternative methodology.
DACBench: A Benchmark Library for Dynamic Algorithm Configuration
Dynamic Algorithm Configuration (DAC) aims to dynamically control a target algorithm's hyperparameters in order to improve its performance.
MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning
We define a parameterised collection of fast-to-run toy environments in OpenAI Gym by varying these dimensions and propose to use these to understand agents better.
Learning Synthetic Environments for Reinforcement Learning with Evolution Strategies
This work explores learning agent-agnostic synthetic environments (SEs) for Reinforcement Learning.
Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework
The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on parameter tuning.
Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization
Neural architecture search (NAS) and hyperparameter optimization (HPO) make deep learning accessible to non-experts by automatically finding the architecture of the deep neural network to use and tuning the hyperparameters of the used training pipeline.
Learning Synthetic Environments and Reward Networks for Reinforcement Learning
In a one-to-one comparison, learning an SE proxy requires more interactions with the real environment than training agents only on the real environment.
Scalable Deep Learning for RNA Secondary Structure Prediction
The field of RNA secondary structure prediction has made significant progress with the adoption of deep learning techniques.
Machine-Learning-Based Diagnostics of EEG Pathology
The results demonstrate that the proposed feature-based decoding framework can achieve accuracies on the same level as state-of-the-art deep neural networks.
Differential Evolution for Neural Architecture Search
Neural architecture search (NAS) methods rely on a search strategy for deciding which architectures to evaluate next and a performance estimation strategy for assessing their performance (e. g., using full evaluations, multi-fidelity evaluations, or the one-shot model).
NAS-Bench-x11 and the Power of Learning Curves
While early research in neural architecture search (NAS) required extreme computational resources, the recent releases of tabular and surrogate benchmarks have greatly increased the speed and reproducibility of NAS research.
AutoDispNet: Improving Disparity Estimation With AutoML
In this work, we show how to use and extend existing AutoML techniques to efficiently optimize large-scale U-Net-like encoder-decoder architectures.
Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design
Our world is ambiguous and this is reflected in the data we use to train our algorithms.
PFNs4BO: In-Context Learning for Bayesian Optimization
In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO).
Meta-Learning of Neural Architectures for Few-Shot Learning
The recent progress in neural architecture search (NAS) has allowed scaling the automated design of neural architectures to real-world domains, such as object detection and semantic segmentation.
TempoRL: Learning When to Act
Reinforcement learning is a powerful approach to learn behaviour through interactions with an environment.
Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How
With the ever-increasing number of pretrained models, machine learning practitioners are continuously faced with which pretrained model to use, and how to finetune it for a new dataset.
Smooth Variational Graph Embeddings for Efficient Neural Architecture Search
We evaluate the proposed approach on neural architectures defined by the ENAS approach, the NAS-Bench-101 and the NAS-Bench-201 search space and show that our smooth embedding space allows to directly extrapolate the performance prediction to architectures outside the seen domain (e. g. with more operations).
On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
We demonstrate that this problem can be tackled effectively with automated HPO, which we demonstrate to yield significantly improved performance compared to human experts.
Hyperparameter Optimization
Model-based Reinforcement Learning
+2
Squirrel: A Switching Hyperparameter Optimizer
In this short note, we describe our submission to the NeurIPS 2020 BBO challenge.
Pitfalls and Best Practices in Algorithm Configuration
Good parameter settings are crucial to achieve high performance in many areas of artificial intelligence (AI), such as propositional satisfiability solving, AI planning, scheduling, and machine learning (in particular deep learning).
Maximizing acquisition functions for Bayesian optimization
Bayesian optimization is a sample-efficient approach to global optimization that relies on theoretically motivated value heuristics (acquisition functions) to guide its search process.
Practical Transfer Learning for Bayesian Optimization
When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset.
The reparameterization trick for acquisition functions
Bayesian optimization is a sample-efficient approach to solving global optimization problems.
On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNs
This enables researchers to further study and improve this promising optimization technique and hopefully reconsider stochastic second-order methods as competitive optimization techniques for training DNNs; we also hope that the promise of SGN may lead to forward automatic differentiation being added to Tensorflow or Pytorch.
T3VIP: Transformation-based 3D Video Prediction
To the best of our knowledge, our model is the first generative model that provides an RGB-D video prediction of the future for a static camera.
ASlib: A Benchmark Library for Algorithm Selection
To address this problem, we introduce a standardized format for representing algorithm selection scenarios and a repository that contains a growing number of data sets from the literature.
Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings
We propose probabilistic models that can extrapolate learning curves of iterative machine learning algorithms, such as stochastic gradient descent for training deep networks, based on training data with variable-length learning curves.
Learning Heuristic Selection with Dynamic Algorithm Configuration
We show that dynamic algorithm configuration can be used for dynamic heuristic selection which takes into account the internal search dynamics of a planning system.
c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization
In this work, we propose constrained TPE (c-TPE), an extension of the widely-used versatile Bayesian optimization method, tree-structured Parzen estimator (TPE), to handle these constraints.
TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks
Similar to large language models, PFNs make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass.
Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks
While deep learning has celebrated many successes, its results often hinge on the meticulous selection of hyperparameters (HPs).
Simple And Efficient Architecture Search for Convolutional Neural Networks
Neural networks have recently had a lot of success for many tasks.
Self-Paced Context Evaluation for Contextual Reinforcement Learning
Reinforcement learning (RL) has made a lot of advances for solving a single problem in a given environment; but learning policies that generalize to unseen variations of a problem remains challenging.
Joint Entropy Search for Maximally-Informed Bayesian Optimization
As a light-weight approach with superior results, JES provides a new go-to acquisition function for Bayesian optimization.
Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator
Hyperparameter optimization (HPO) is a vital step in improving performance in deep learning (DL).
PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces
The recent rise in popularity of Hyperparameter Optimization (HPO) for deep learning has highlighted the role that good hyperparameter (HP) space design can play in training strong models.
Neural Architecture Search: A Survey
Deep Learning has enabled remarkable progress over the last years on a variety of tasks, such as image recognition, speech recognition, and machine translation.
Automated Dynamic Algorithm Configuration
The performance of an algorithm often critically depends on its parameter configuration.
A General Framework for User-Guided Bayesian Optimization
The optimization of expensive-to-evaluate black-box functions is prevalent in various scientific disciplines.
Multi-objective Differentiable Neural Architecture Search
Pareto front profiling in multi-objective optimization (MOO), i. e. finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives like neural network training.
Hyperparameter Transfer Across Developer Adjustments
After developer adjustments to a machine learning (ML) algorithm, how can the results of an old hyperparameter optimization (HPO) automatically be used to speedup a new HPO?
Theory-inspired Parameter Control Benchmarks for Dynamic Algorithm Configuration
We extend this benchmark by analyzing optimal control policies that can select the parameters only from a given portfolio of possible values.
Mind the Gap: Measuring Generalization Performance Across Multiple Objectives
Modern machine learning models are often constructed taking into account multiple objectives, e. g., minimizing inference time while also maximizing accuracy.
Neural Networks for Predicting Algorithm Runtime Distributions
Many state-of-the-art algorithms for solving hard combinatorial problems in artificial intelligence (AI) include elements of stochasticity that lead to high variations in runtime, even for a fixed problem instance.
Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution
Neural Architecture Search aims at automatically finding neural architectures that are competitive with architectures designed by human experts.
The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities
Biological evolution provides a creative fount of complex and subtle adaptations, often surprising the scientists who discover them.
Warmstarting of Model-based Algorithm Configuration
The performance of many hard combinatorial problem solvers depends strongly on their parameter settings, and since manual parameter tuning is both tedious and suboptimal the AI community has recently developed several algorithm configuration (AC) methods to automatically address this problem.
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates
In our experiments, we construct and evaluate surrogate benchmarks for hyperparameter optimization as well as for AC problems that involve performance optimization of solvers for hard combinatorial problems, drawing training data from the runs of existing AC procedures.
Asynchronous Stochastic Gradient MCMC with Elastic Coupling
We consider parallel asynchronous Markov Chain Monte Carlo (MCMC) sampling for problems where we can leverage (stochastic) gradients to define continuous dynamics which explore the target distribution.
A case study of algorithm selection for the traveling thief problem
Many real-world problems are composed of several interacting components.
The Configurable SAT Solver Challenge (CSSC)
It is well known that different solution strategies work well for different types of instances of hard combinatorial problems.
CMA-ES for Hyperparameter Optimization of Deep Neural Networks
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization.
Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
In practical Bayesian optimization, we must often search over structures with differing numbers of parameters.
ParamILS: An Automatic Algorithm Configuration Framework
The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms.
Algorithm Runtime Prediction: Methods & Evaluation
We also comprehensively describe new and existing features for predicting algorithm runtime for propositional satisfiability (SAT), travelling salesperson (TSP) and mixed integer programming (MIP) problems.
A Kernel for Hierarchical Parameter Spaces
We define a family of kernels for mixed continuous/discrete hierarchical parameter spaces and show that they are positive definite.
Bayesian Optimization With Censored Response Data
Bayesian optimization (BO) aims to minimize a given blackbox function using a model that is updated whenever new evidence about the function becomes available.
Fixing Weight Decay Regularization in Adam
We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.
Towards White-box Benchmarks for Algorithm Control
The performance of many algorithms in the fields of hard combinatorial problem solving, machine learning or AI in general depends on tuned hyperparameter configurations.
Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters
Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning.
Best Practices for Scientific Research on Neural Architecture Search
Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS).
Neural Architecture Evolution in Deep Reinforcement Learning for Continuous Control
Current Deep Reinforcement Learning algorithms still heavily rely on handcrafted neural network architectures.
Bayesian Optimization with a Prior for the Optimum
We show that BOPrO is around 6. 67x faster than state-of-the-art methods on a common suite of benchmarks, and achieves a new state-of-the-art performance on a real-world hardware design application.
Transferring Optimality Across Data Distributions via Homotopy Methods
Homotopy methods, also known as continuation methods, are a powerful mathematical tool to efficiently solve various problems in numerical analysis, including complex non-convex optimization problems where no or only little prior knowledge regarding the localization of the solutions is available.
Neural Model-based Optimization with Right-Censored Observations
When fitting a regression model to predict the distribution of the outcomes, we cannot simply drop these right-censored observations, but need to properly model them.
Regularization Cocktails
The regularization of prediction models is arguably the most crucial ingredient that allows Machine Learning solutions to generalize well on unseen data.
On the Importance of Domain Model Configuration for Automated Planning Engines
The development of domain-independent planners within the AI Planning community is leading to "off-the-shelf" technology that can be used in a wide range of applications.
Convergence Analysis of Homotopy-SGD for non-convex optimization
In this work, we present a first-order stochastic algorithm based on a combination of homotopy methods and SGD, called Homotopy-Stochastic Gradient Descent (H-SGD), which finds interesting connections with some proposed heuristics in the literature, e. g. optimization by Gaussian continuation, training by diffusion, mollifying networks.
In-Loop Meta-Learning with Gradient-Alignment Reward
To do this, we optimize the loss of the next training step.
Bag of Tricks for Neural Architecture Search
While neural architecture search methods have been successful in previous years and led to new state-of-the-art performance on various problems, they have also been criticized for being unstable, being highly sensitive with respect to their hyperparameters, and often not performing better than random search.
Multi-headed Neural Ensemble Search
Ensembles of CNN models trained with different seeds (also known as Deep Ensembles) are known to achieve superior performance over a single copy of the CNN.
$\pi$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization
To address this issue, we propose $\pi$BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user.
MDP Playground: Controlling Orthogonal Dimensions of Hardness in Toy Environments
We present MDP Playground, an efficient benchmark for Reinforcement Learning (RL) algorithms with various dimensions of hardness that can be controlled independently to challenge algorithms in different ways and to obtain varying degrees of hardness in generated environments.
Prior-guided Bayesian Optimization
While Bayesian Optimization (BO) is a very popular method for optimizing expensive black-box functions, it fails to leverage the experience of domain experts.
Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019
Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly.
Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable agents.
Neural Architecture Search for Dense Prediction Tasks in Computer Vision
The success of deep learning in recent years has lead to a rising demand for neural network architecture engineering.
Practitioner Motives to Select Hyperparameter Optimization Methods
Advanced programmatic hyperparameter optimization (HPO) methods, such as Bayesian optimization, have high sample efficiency in reproducibly finding optimal hyperparameter values of machine learning (ML) models.
Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification
Although deep neural networks are capable of achieving performance superior to humans on various tasks, they are notorious for requiring large amounts of data and computing resources, restricting their success to domains where such resources are available.
On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learning
Self-Supervised Learning (SSL) has become a very active area of Deep Learning research where it is heavily used as a pre-training method for classification and other tasks.
Neural Architecture Search: Insights from 1000 Papers
Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas.
Natural Language Understanding
Neural Architecture Search
+2
Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
The field of automated machine learning (AutoML) introduces techniques that automate parts of the development of machine learning (ML) systems, accelerating the process and reducing barriers for novices.
Self-Correcting Bayesian Optimization through Bayesian Active Learning
Gaussian processes are the model of choice in Bayesian optimization and active learning.
MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization
Hyperparameter optimization (HPO) is a powerful technique for automating the tuning of machine learning (ML) models.
Towards Automated Design of Riboswitches
Experimental screening and selection pipelines for the discovery of novel riboswitches are expensive, time-consuming, and inefficient.
Hard View Selection for Self-Supervised Learning
Many Self-Supervised Learning (SSL) methods train their models to be invariant to different "views" of an image input for which a good data augmentation pipeline is crucial.
Managing AI Risks in an Era of Rapid Progress
In this short consensus paper, we outline risks from upcoming, advanced AI systems.
Constrained Parameter Regularization
Instead of applying a single constant penalty to all parameters, we enforce an upper bound on a statistical measure (e. g., the L$_2$-norm) of parameter groups.
Weight-Entanglement Meets Gradient-Based Neural Architecture Search
%Due to the inherent differences in the structure of these search spaces, these Since weight-entanglement poses compatibility challenges for gradient-based NAS methods, these two paradigms have largely developed independently in parallel sub-communities.
Rethinking Performance Measures of RNA Secondary Structure Problems
Accurate RNA secondary structure prediction is vital for understanding cellular regulation and disease mechanisms.
Is Mamba Capable of In-Context Learning?
This work provides empirical evidence that Mamba, a newly proposed selective structured state space model, has similar in-context learning (ICL) capabilities as transformers.
Diffusion-based Neural Network Weights Generation
By learning the distribution of a neural network on a variety pretrained models, our approach enables adaptive sampling weights for unseen datasets achieving faster convergence and reaching competitive performance.
