Search Results for author:
Found 53 papers, 28 papers with code
Personalized Benchmarking with the Ludwig Benchmarking Toolkit
LBT provides a configurable interface for controlling training and customizing evaluation, a standardized training framework for eliminating confounding variables, and support for multi-objective evaluation.
Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
However, this loss restricts the scope of applicable regression models, limits the ability to target many desirable properties (e. g. calibration, sharpness, centered intervals), and may produce poor conditional quantiles.
Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification
With increasing deployment of machine learning systems in various real-world tasks, there is a greater need for accurate quantification of predictive uncertainty.
Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
We compare Dragonfly to a suite of other packages and algorithms for global optimisation and demonstrate that when the above methods are integrated, they enable significant improvements in the performance of BO.
Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
Some recent schedulers choose job resources for users, but do so without awareness of how DL training can be re-optimized to better utilize the provided resources.
Betty: An Automatic Differentiation Library for Multilevel Optimization
Gradient-based multilevel optimization (MLO) has gained attention as a framework for studying numerous problems, ranging from hyperparameter optimization and meta-learning to neural architecture search and reinforcement learning.
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
Bayesian optimization (BO), which has long had success in hyperparameter optimization, has recently emerged as a very promising strategy for NAS when it is coupled with a neural predictor.
A Study on Encodings for Neural Architecture Search
First we formally define architecture encodings and give a theoretical characterization on the scalability of the encodings we study Then we identify the main encoding-dependent subroutines which NAS algorithms employ, running experiments to show which encodings work best with each subroutine for many popular algorithms.
Neural Architecture Search with Bayesian Optimisation and Optimal Transport
A common use case for BO in machine learning is model selection, where it is not possible to analytically model the generalisation performance of a statistical model, and we resort to noisy and expensive training and validation procedures to choose the best model.
LLM360: Towards Fully Transparent Open-Source LLMs
The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers.
Synthetic Benchmarks for Scientific Research in Explainable Machine Learning
In this work, we address this issue by releasing XAI-Bench: a suite of synthetic datasets along with a library for benchmarking feature attribution algorithms.
Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mutual Information
Given such an $\mathcal{A}$, and a prior distribution over $f$, we refer to the problem of inferring the output of $\mathcal{A}$ using $T$ evaluations as Bayesian Algorithm Execution (BAX).
A General Recipe for Likelihood-free Bayesian Optimization
To extend BO to a broader class of models and utilities, we propose likelihood-free BO (LFBO), an approach based on likelihood-free inference.
Methods for comparing uncertainty quantifications for material property predictions
These uncertainty estimates are instrumental for determining which materials to screen next, but there is not yet a standard procedure for judging the quality of such uncertainty estimates objectively.
Materials Science Computational Physics
Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling
Our experiments demonstrate that only a small number of feedback iterations are needed to train models that achieve highly competitive test set performance without access to ground truth training labels.
IS-COUNT: Large-scale Object Counting from Satellite Images with Covariate-based Importance Sampling
We show empirically that the proposed framework achieves strong performance on estimating the number of buildings in the United States and Africa, cars in Kenya, brick kilns in Bangladesh, and swimming pools in the U. S., while requiring as few as 0. 01% of satellite images compared to an exhaustive approach.
An Experimental Design Perspective on Model-Based Reinforcement Learning
In particular, we leverage ideas from Bayesian optimal experimental design to guide the selection of state-action queries for efficient learning.
Exploration via Planning for Information about the Optimal Trajectory
In this work, we develop a method that allows us to plan for exploration while taking both the task and the current knowledge about the dynamics into account.
ProBO: Versatile Bayesian Optimization Using Any Probabilistic Programming Language
Optimizing an expensive-to-query function is a common task in science and engineering, where it is beneficial to keep the number of queries to a minimum.
AutoML for Climate Change: A Call to Action
The challenge that climate change poses to humanity has spurred a rapidly developing field of artificial intelligence research focused on climate change applications.
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis
In particular, there are various considerations behind the pipeline: (1) the choice and (2) the size of PLM, (3) the choice of uncertainty quantifier, (4) the choice of fine-tuning loss, and many more.
Uncertainty quantification using martingales for misspecified Gaussian processes
There is a necessary cost to achieving robustness: if the prior was correct, posterior GP bands are narrower than our CS.
Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolution
Deep learning-based surrogate models have demonstrated remarkable advantages over classical solvers in terms of speed, often achieving speedups of 10 to 1000 times over traditional partial differential equation (PDE) solvers.
Offline Contextual Bayesian Optimization
In black-box optimization, an agent repeatedly chooses a configuration to test, so as to find an optimal configuration.
Amortized Auto-Tuning: Cost-Efficient Bayesian Transfer Optimization for Hyperparameter Recommendation
With the surge in the number of hyperparameters and training times of modern machine learning models, hyperparameter tuning is becoming increasingly expensive.
Generative Modeling Helps Weak Supervision (and Vice Versa)
The model outperforms baseline weak supervision label models on a number of multiclass image classification datasets, improves the quality of generated images, and further improves end-model performance through data augmentation with synthetic samples.
Myopic Bayesian Design of Experiments via Posterior Sampling and Probabilistic Programming
We design a new myopic strategy for a wide class of sequential design of experiment (DOE) problems, where the goal is to collect data in order to to fulfil a certain problem specific goal.
ChemBO: Bayesian Optimization of Small Organic Molecules with Synthesizable Recommendations
In applications such as molecule design or drug discovery, it is desirable to have an algorithm which recommends new candidate molecules based on the results of past tests.
Post-Inference Prior Swapping
However, we demonstrate that IS will fail for many choices of the target prior, depending on its parametric form and similarity to the false prior.
Performance Bounds for Graphical Record Linkage
Record linkage involves merging records in large, noisy databases to remove duplicate entities.
Parallel and Distributed Block-Coordinate Frank-Wolfe Algorithms
We develop parallel and distributed Frank-Wolfe algorithms; the former on shared memory machines with mini-batching, and the latter in a delayed update framework.
Embarrassingly Parallel Variational Inference in Nonconjugate Models
We develop a parallel variational inference (VI) procedure for use in data-distributed settings, where each machine only has access to a subset of data and runs VI independently, without communicating with other machines.
Fast Function to Function Regression
Function to function regression (FFR) covers a large range of interesting applications including time-series prediction problems, and also more general tasks like studying a mapping between two separate types of distributions.
Asymptotically Exact, Embarrassingly Parallel MCMC
This embarrassingly parallel algorithm allows each machine to act independently on a subset of the data (without communication) until the final combination stage.
Fast Distribution To Real Regression
We study the problem of distribution to real-value regression, where one aims to regress a mapping $f$ that takes in a distribution input covariate $P\in \mathcal{I}$ (for a non-parametric family of distributions $\mathcal{I}$) and outputs a real-valued response $Y=f(P) + \epsilon$.
Geometric Generalization Based Zero-Shot Learning Dataset Infinite World: Simple Yet Powerful
Analogously, this paper introduces geometric generalization based zero-shot learning tests to measure the rapid learning ability and the internal consistency of deep generative models.
Offline Contextual Bayesian Optimization for Nuclear Fusion
One obstacle in utilizing fusion as a feasible energy source is the stability of the reaction.
Neural Dynamical Systems: Balancing Structure and Flexibility in Physical Prediction
We introduce Neural Dynamical Systems (NDS), a method of learning dynamical models in various gray-box settings which incorporates prior knowledge in the form of systems of ordinary differential equations.
Computational catalyst discovery: Active classification through myopic multiscale sampling
We address this issue by relaxing the catalyst discovery goal into a classification problem: "What is the set of catalysts that is worth testing experimentally?"
Chemical Physics
An Experimental Design Perspective on Exploration in Reinforcement Learning
In particular, we leverage ideas from Bayesian optimal experimental design to guide the selection of state-action queries for efficient learning.
H-Entropy Search: Generalizing Bayesian Optimization with a Decision-theoretic Uncertainty Measure
For special cases of the loss and design space, we develop gradient-based methods to efficiently optimize our proposed family of acquisition functions, and demonstrate that the resulting BO procedure shows strong empirical performance on a diverse set of optimization tasks.
BANANAS: Bayesian Optimization with Neural Networks for Neural Architecture Search
We develop a path-based encoding scheme to featurize the neural architectures that are used to train the neural network model.
Neural Dynamical Systems
We introduce Neural Dynamical Systems (NDS), a method of learning dynamical models which incorporates prior knowledge in the form of systems of ordinary differential equations.
Modular Conformal Calibration
We introduce a versatile class of algorithms for recalibration in regression that we call Modular Conformal Calibration (MCC).
Multipoint-BAX: A New Approach for Efficiently Tuning Particle Accelerator Emittance via Virtual Objectives
Traditional black-box optimizers such as Bayesian optimization are slow and inefficient when dealing with such objectives as they must acquire the full series of measurements, but return only the emittance, with each query.
Generalizing Bayesian Optimization with Decision-theoretic Entropies
Bayesian optimization (BO) is a popular method for efficiently inferring optima of an expensive black-box function via a sequence of queries.
Near-optimal Policy Identification in Active Reinforcement Learning
Many real-world reinforcement learning tasks require control of complex dynamical systems that involve both costly data acquisition processes and large state spaces.
Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
In this case, a well-known issue is the distribution shift between the learned policy and the behavior policy that collects the offline data.
Kernelized Offline Contextual Dueling Bandits
Preference-based feedback is important for many applications where direct evaluation of a reward function is not feasible.
SlimPajama-DC: Understanding Data Combinations for LLM Training
This paper aims to understand the impacts of various data combinations (e. g., web text, wikipedia, github, books) on the training of large language models using SlimPajama.
Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration
Preference-based feedback is important for many applications in reinforcement learning where direct evaluation of a reward function is not feasible.
DeLLMa: A Framework for Decision Making Under Uncertainty with Large Language Models
Large language models (LLMs) are increasingly used across society, including in domains like business, engineering, and medicine.
IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations
Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs.
