Search Results for author: Mohammad Emtiyaz Khan

Found 46 papers, 28 papers with code

Variational Imitation Learning with Diverse-quality Demonstrations

1 code implementation ICML 2020 Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, Masashi Sugiyama

Learning from demonstrations can be challenging when the quality of demonstrations is diverse, and even more so when the quality is unknown and there is no additional information to estimate the quality.

Continuous Control Imitation Learning +2

Conformal Prediction via Regression-as-Classification

no code implementations12 Apr 2024 Etash Guha, Shlok Natarajan, Thomas Möllenhoff, Mohammad Emtiyaz Khan, Eugene Ndiaye

Conformal prediction (CP) for regression can be challenging, especially when the output distribution is heteroscedastic, multimodal, or skewed.

Classification Conformal Prediction +1

Variational Learning is Effective for Large Deep Networks

1 code implementation27 Feb 2024 Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, Thomas Möllenhoff

We give extensive empirical evidence against the common belief that variational learning is ineffective for large neural networks.

The Memory Perturbation Equation: Understanding Model's Sensitivity to Data

1 code implementation30 Oct 2023 Peter Nickl, Lu Xu, Dharmesh Tailor, Thomas Möllenhoff, Mohammad Emtiyaz Khan

Understanding model's sensitivity to its training data is crucial but can also be challenging and costly, especially during training.

Model Merging by Uncertainty-Based Gradient Matching

no code implementations19 Oct 2023 Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych, Mohammad Emtiyaz Khan

Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail?

Exploiting Inferential Structure in Neural Processes

1 code implementation27 Jun 2023 Dharmesh Tailor, Mohammad Emtiyaz Khan, Eric Nalisnick

Neural Processes (NPs) are appealing due to their ability to perform fast adaptation based on a context set.

Memory-Based Dual Gaussian Processes for Sequential Learning

1 code implementation6 Jun 2023 Paul E. Chang, Prakhar Verma, S. T. John, Arno Solin, Mohammad Emtiyaz Khan

Sequential learning with Gaussian processes (GPs) is challenging when access to past data is limited, for example, in continual and active learning.

Active Learning Bayesian Optimization +2

Variational Bayes Made Easy

no code implementations27 Apr 2023 Mohammad Emtiyaz Khan

Variational Bayes is a popular method for approximate inference but its derivation can be cumbersome.

The Lie-Group Bayesian Learning Rule

no code implementations8 Mar 2023 Eren Mehmet Kıral, Thomas Möllenhoff, Mohammad Emtiyaz Khan

This simplifies all three difficulties for many cases, providing flexible parametrizations through group's action, simple gradient computation through reparameterization, and updates that always stay on the manifold.

Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning

1 code implementation20 Feb 2023 Wu Lin, Valentin Duruisseaux, Melvin Leok, Frank Nielsen, Mohammad Emtiyaz Khan, Mark Schmidt

Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult differential equations.

Bridging the Gap Between Target Networks and Functional Regularization

no code implementations21 Oct 2022 Alexandre Piche, Valentin Thomas, Joseph Marino, Rafael Pardinas, Gian Maria Marconi, Christopher Pal, Mohammad Emtiyaz Khan

However, learning the value function via bootstrapping often leads to unstable training due to fast-changing target values.

Can Calibration Improve Sample Prioritization?

no code implementations12 Oct 2022 Ganesh Tata, Gautham Krishna Gudur, Gopinath Chennupati, Mohammad Emtiyaz Khan

Calibration can reduce overconfident predictions of deep neural networks, but can calibration also accelerate training?

SAM as an Optimal Relaxation of Bayes

1 code implementation4 Oct 2022 Thomas Möllenhoff, Mohammad Emtiyaz Khan

Sharpness-aware minimization (SAM) and related adversarial deep-learning methods can drastically improve generalization, but their underlying mechanisms are not yet fully understood.

Dual Parameterization of Sparse Variational Gaussian Processes

1 code implementation NeurIPS 2021 Vincent Adam, Paul E. Chang, Mohammad Emtiyaz Khan, Arno Solin

Sparse variational Gaussian process (SVGP) methods are a common choice for non-conjugate Gaussian process inference because of their computational benefits.

Computational Efficiency Gaussian Processes

Structured second-order methods via natural gradient descent

no code implementations22 Jul 2021 Wu Lin, Frank Nielsen, Mohammad Emtiyaz Khan, Mark Schmidt

In this paper, we propose new structured second-order methods and structured adaptive-gradient methods obtained by performing natural-gradient descent on structured parameter spaces.

Second-order methods

Subset-of-Data Variational Inference for Deep Gaussian-Processes Regression

1 code implementation17 Jul 2021 Ayush Jain, P. K. Srijith, Mohammad Emtiyaz Khan

Deep Gaussian Processes (DGPs) are multi-layer, flexible extensions of Gaussian processes but their training remains challenging.

Gaussian Processes regression +1

The Bayesian Learning Rule

no code implementations9 Jul 2021 Mohammad Emtiyaz Khan, Håvard Rue

The rule, derived from Bayesian principles, yields a wide-range of algorithms from fields such as optimization, deep learning, and graphical models.


Knowledge-Adaptation Priors

1 code implementation NeurIPS 2021 Mohammad Emtiyaz Khan, Siddharth Swaroop

Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch.

Bridging the Gap Between Target Networks and Functional Regularization

1 code implementation4 Jun 2021 Alexandre Piché, Valentin Thomas, Rafael Pardinas, Joseph Marino, Gian Maria Marconi, Christopher Pal, Mohammad Emtiyaz Khan

Our findings emphasize that Functional Regularization can be used as a drop-in replacement for Target Networks and result in performance improvement.


Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning

1 code implementation11 Apr 2021 Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, Mohammad Emtiyaz Khan

Marginal-likelihood based model-selection, even though promising, is rarely used in deep learning due to estimation difficulties.

Image Classification Model Selection +2

Tractable structured natural gradient descent using local parameterizations

no code implementations15 Feb 2021 Wu Lin, Frank Nielsen, Mohammad Emtiyaz Khan, Mark Schmidt

Natural-gradient descent (NGD) on structured parameter spaces (e. g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations.

Variational Inference

Fast Variational Learning in State-Space Gaussian Process Models

1 code implementation9 Jul 2020 Paul E. Chang, William J. Wilkinson, Mohammad Emtiyaz Khan, Arno Solin

Gaussian process (GP) regression with 1D inputs can often be performed in linear time via a stochastic differential equation formulation.

Time Series Time Series Analysis +1

Continual Deep Learning by Functional Regularisation of Memorable Past

1 code implementation NeurIPS 2020 Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard E. Turner, Mohammad Emtiyaz Khan

Continually learning new skills is important for intelligent systems, yet standard deep learning methods suffer from catastrophic forgetting of the past.

Training Binary Neural Networks using the Bayesian Learning Rule

4 code implementations ICML 2020 Xiangming Meng, Roman Bachmann, Mohammad Emtiyaz Khan

Our work provides a principled approach for training binary neural networks which justifies and extends existing approaches.

Continual Learning

Handling the Positive-Definite Constraint in the Bayesian Learning Rule

1 code implementation ICML 2020 Wu Lin, Mark Schmidt, Mohammad Emtiyaz Khan

The Bayesian learning rule is a natural-gradient variational inference method, which not only contains many existing learning algorithms as special cases but also enables the design of new algorithms.

valid Variational Inference

Stein's Lemma for the Reparameterization Trick with Exponential Family Mixtures

1 code implementation29 Oct 2019 Wu Lin, Mohammad Emtiyaz Khan, Mark Schmidt

Our generalization enables us to establish a connection between Stein's lemma and the reparamterization trick to derive gradients of expectations of a large class of functions under weak assumptions.


VILD: Variational Imitation Learning with Diverse-quality Demonstrations

no code implementations15 Sep 2019 Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, Masashi Sugiyama

However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs.

Continuous Control Imitation Learning

Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations

1 code implementation7 Jun 2019 Wu Lin, Mohammad Emtiyaz Khan, Mark Schmidt

Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to \emph{minimal} exponential-family (EF) approximations.

Bayesian Inference Variational Inference

Practical Deep Learning with Bayesian Principles

1 code implementation NeurIPS 2019 Kazuki Osawa, Siddharth Swaroop, Anirudh Jain, Runa Eschenhagen, Richard E. Turner, Rio Yokota, Mohammad Emtiyaz Khan

Importantly, the benefits of Bayesian principles are preserved: predictive probabilities are well-calibrated, uncertainties on out-of-distribution data are improved, and continual-learning performance is boosted.

Continual Learning Data Augmentation +1

Approximate Inference Turns Deep Networks into Gaussian Processes

1 code implementation NeurIPS 2019 Mohammad Emtiyaz Khan, Alexander Immer, Ehsan Abedi, Maciej Korzepa

Deep neural networks (DNN) and Gaussian processes (GP) are two powerful models with several theoretical connections relating them, but the relationship between their training methods is not well understood.

Gaussian Processes

Scalable Training of Inference Networks for Gaussian-Process Models

2 code implementations27 May 2019 Jiaxin Shi, Mohammad Emtiyaz Khan, Jun Zhu

Inference in Gaussian process (GP) models is computationally challenging for large data, and often difficult to approximate with a small number of inducing points.

A Generalization Bound for Online Variational Inference

no code implementations8 Apr 2019 Badr-Eddine Chérief-Abdellatif, Pierre Alquier, Mohammad Emtiyaz Khan

Our work in this paper presents theoretical justifications in favor of online algorithms relying on approximate Bayesian methods.

Bayesian Inference Generalization Bounds +1

TD-Regularized Actor-Critic Methods

1 code implementation19 Dec 2018 Simone Parisi, Voot Tangkaratt, Jan Peters, Mohammad Emtiyaz Khan

Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability.

reinforcement-learning Reinforcement Learning (RL)

SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient

2 code implementations NeurIPS 2018 Aaron Mishkin, Frederik Kunstner, Didrik Nielsen, Mark Schmidt, Mohammad Emtiyaz Khan

Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution.

Variational Inference

Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Models

1 code implementation12 Jul 2018 Mohammad Emtiyaz Khan, Didrik Nielsen

Bayesian inference plays an important role in advancing machine learning, but faces computational challenges when applied to complex models such as deep neural networks.

Bayesian Inference Variational Inference

Beyond Unfolding: Exact Recovery of Latent Convex Tensor Decomposition under Reshuffling

no code implementations22 May 2018 Chao Li, Mohammad Emtiyaz Khan, Zhun Sun, Gang Niu, Bo Han, Shengli Xie, Qibin Zhao

Exact recovery of tensor decomposition (TD) methods is a desirable property in both unsupervised learning and scientific data analysis.

Image Steganography Tensor Decomposition

Variational Message Passing with Structured Inference Networks

1 code implementation ICLR 2018 Wu Lin, Nicolas Hubacher, Mohammad Emtiyaz Khan

Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret.

Variational Inference

Vprop: Variational Inference using RMSprop

no code implementations4 Dec 2017 Mohammad Emtiyaz Khan, Zuozhu Liu, Voot Tangkaratt, Yarin Gal

Overall, this paper presents Vprop as a principled, computationally-efficient, and easy-to-implement method for Bayesian deep learning.

Variational Inference

Variational Adaptive-Newton Method for Explorative Learning

no code implementations15 Nov 2017 Mohammad Emtiyaz Khan, Wu Lin, Voot Tangkaratt, Zuozhu Liu, Didrik Nielsen

We present the Variational Adaptive Newton (VAN) method which is a black-box optimization method especially suitable for explorative-learning tasks such as active learning and reinforcement learning.

Active Learning reinforcement-learning +2

Bayesian Nonparametric Poisson-Process Allocation for Time-Sequence Modeling

1 code implementation19 May 2017 Hongyi Ding, Mohammad Emtiyaz Khan, Issei Sato, Masashi Sugiyama

We model the intensity of each sequence as an infinite mixture of latent functions, each of which is obtained using a function drawn from a Gaussian process.

Variational Inference

Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models

2 code implementations13 Mar 2017 Mohammad Emtiyaz Khan, Wu Lin

In this paper, we propose a new algorithm called Conjugate-computation Variational Inference (CVI) which brings the best of the two worlds together -- it uses conjugate computations for the conjugate terms and employs stochastic gradients for the rest.

Variational Inference

Cannot find the paper you are looking for? You can Submit a new open access paper.