Search Results for author: Diederik P. Kingma

Found 28 papers, 18 papers with code

Understanding the Diffusion Objective as a Weighted Integral of ELBOs

no code implementations1 Mar 2023 Diederik P. Kingma, Ruiqi Gao

Diffusion models in the literature are optimized with various objectives that are special cases of a weighted loss, where the weighting function specifies the weight per noise level.

Data Augmentation

On Distillation of Guided Diffusion Models

1 code implementation6 Oct 2022 Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, Tim Salimans

For standard diffusion models trained on the pixel-space, our approach is able to generate images visually comparable to that of the original model using as few as 4 sampling steps on ImageNet 64x64 and CIFAR-10, achieving FID/IS scores comparable to that of the original model while being up to 256 times faster to sample from.

Denoising Image Generation +1

Variational Diffusion Models

4 code implementations1 Jul 2021 Diederik P. Kingma, Tim Salimans, Ben Poole, Jonathan Ho

In addition, we show that the continuous-time VLB is invariant to the noise schedule, except for the signal-to-noise ratio at its endpoints.

 Ranked #1 on Image Generation on CIFAR-10 (bits/dimension metric)

Density Estimation Image Generation

How to Train Your Energy-Based Models

2 code implementations9 Jan 2021 Yang song, Diederik P. Kingma

Energy-Based Models (EBMs), also known as non-normalized probabilistic models, specify probability density or mass functions up to an unknown normalizing constant.

Learning Energy-Based Models by Diffusion Recovery Likelihood

2 code implementations ICLR 2021 Ruiqi Gao, Yang song, Ben Poole, Ying Nian Wu, Diederik P. Kingma

Inspired by recent progress on diffusion probabilistic models, we present a diffusion recovery likelihood method to tractably learn and sample from a sequence of EBMs trained on increasingly noisy versions of a dataset.

Image Generation

Score-Based Generative Modeling through Stochastic Differential Equations

8 code implementations ICLR 2021 Yang song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole

Combined with multiple architectural improvements, we achieve record-breaking performance for unconditional image generation on CIFAR-10 with an Inception score of 9. 89 and FID of 2. 20, a competitive likelihood of 2. 99 bits/dim, and demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.

Colorization Image Inpainting +1

On Linear Identifiability of Learned Representations

no code implementations1 Jul 2020 Geoffrey Roeder, Luke Metz, Diederik P. Kingma

Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data.

Representation Learning

ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA

1 code implementation NeurIPS 2020 Ilyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma, Aapo Hyvärinen

We consider the identifiability theory of probabilistic models and establish sufficient conditions under which the representations learned by a very broad family of conditional energy-based models are unique in function space, up to a simple transformation.

Transfer Learning

Variational Autoencoders and Nonlinear ICA: A Unifying Framework

2 code implementations10 Jul 2019 Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, Aapo Hyvärinen

We address this issue by showing that for a broad family of deep latent-variable models, identification of the true joint distribution over observed and latent variables is actually possible up to very simple transformations, thus achieving a principled and powerful form of disentanglement.


An Introduction to Variational Autoencoders

6 code implementations6 Jun 2019 Diederik P. Kingma, Max Welling

Variational autoencoders provide a principled framework for learning deep latent-variable models and corresponding inference models.

Glow: Generative Flow with Invertible 1x1 Convolutions

27 code implementations NeurIPS 2018 Diederik P. Kingma, Prafulla Dhariwal

Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis.

Image Generation

Learning Sparse Neural Networks through L_0 Regularization

no code implementations ICLR 2018 Christos Louizos, Max Welling, Diederik P. Kingma

We further propose the \emph{hard concrete} distribution for the gates, which is obtained by ``stretching'' a binary concrete distribution and then transforming its samples with a hard-sigmoid.

Model Selection

Learning Sparse Neural Networks through $L_0$ Regularization

4 code implementations4 Dec 2017 Christos Louizos, Max Welling, Diederik P. Kingma

We further propose the \emph{hard concrete} distribution for the gates, which is obtained by "stretching" a binary concrete distribution and then transforming its samples with a hard-sigmoid.

Model Selection

PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

7 code implementations19 Jan 2017 Tim Salimans, Andrej Karpathy, Xi Chen, Diederik P. Kingma

1) We use a discretized logistic mixture likelihood on the pixels, rather than a 256-way softmax, which we find to speed up training.

Image Generation

Variational Lossy Autoencoder

no code implementations8 Nov 2016 Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, Pieter Abbeel

Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification.

Density Estimation Image Generation +1

Improving Variational Inference with Inverse Autoregressive Flow

8 code implementations15 Jun 2016 Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, Max Welling

The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables.

Variational Inference

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

9 code implementations NeurIPS 2016 Tim Salimans, Diederik P. Kingma

We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction.

Image Classification reinforcement-learning +1

Variational Dropout and the Local Reparameterization Trick

11 code implementations NeurIPS 2015 Diederik P. Kingma, Tim Salimans, Max Welling

Our method allows inference of more flexibly parameterized posteriors; specifically, we propose variational dropout, a generalization of Gaussian dropout where the dropout rates are learned, often leading to better models.

Bayesian Inference

Note on Equivalence Between Recurrent Neural Network Time Series Models and Variational Bayesian Models

no code implementations29 Apr 2015 Jascha Sohl-Dickstein, Diederik P. Kingma

We observe that the standard log likelihood training objective for a Recurrent Neural Network (RNN) model of time series data is equivalent to a variational Bayesian training objective, given the proper choice of generative and inference models.

Time Series Analysis

Adam: A Method for Stochastic Optimization

77 code implementations22 Dec 2014 Diederik P. Kingma, Jimmy Ba

We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.

Stochastic Optimization

Markov Chain Monte Carlo and Variational Inference: Bridging the Gap

no code implementations23 Oct 2014 Tim Salimans, Diederik P. Kingma, Max Welling

Recent advances in stochastic gradient variational inference have made it possible to perform variational Bayesian inference with posterior approximations containing auxiliary random variables.

Bayesian Inference Variational Inference

Semi-Supervised Learning with Deep Generative Models

17 code implementations NeurIPS 2014 Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, Max Welling

The ever-increasing size of modern data sets combined with the difficulty of obtaining label information has made semi-supervised learning one of the problems of significant practical importance in modern data analysis.

Bayesian Inference

Efficient Gradient-Based Inference through Transformations between Bayes Nets and Neural Nets

no code implementations3 Feb 2014 Diederik P. Kingma, Max Welling

Hierarchical Bayesian networks and neural networks with stochastic hidden units are commonly perceived as two separate types of models.

Auto-Encoding Variational Bayes

129 code implementations20 Dec 2013 Diederik P. Kingma, Max Welling

First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods.

Image Clustering Variational Inference

Fast Gradient-Based Inference with Continuous Latent Variable Models in Auxiliary Form

no code implementations4 Jun 2013 Diederik P. Kingma

We propose a technique for increasing the efficiency of gradient-based inference and learning in Bayesian networks with multiple layers of continuous latent vari- ables.

Cannot find the paper you are looking for? You can Submit a new open access paper.