FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network. Likelihood-based training of these models requires restricting their architectures to allow cheap computation of Jacobian determinants. Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation. In this paper, we use Hutchinson's trace estimator to give a scalable unbiased estimate of the log-density. The result is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures. We demonstrate our approach on high-dimensional density estimation, image generation, and variational inference, achieving the state-of-the-art among exact likelihood methods with efficient sampling.

PDF Abstract ICLR 2019 PDF ICLR 2019 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Density Estimation BSDS300 FFJORD NLL -157.4 # 1
Density Estimation Caltech-101 FFJORD Negative ELBO 104.03 # 1
Image Generation CIFAR-10 FFJORD bits/dimension 3.4 # 43
Density Estimation CIFAR-10 FFJORD NLL 3.4 # 1
Density Estimation Freyfaces FFJORD Negative ELBO 4.39 # 1
Density Estimation MNIST FFJORD NLL 0.99 # 2
Negative ELBO 82.82 # 1
Density Estimation OMNIGLOT FFJORD Negative ELBO 98.33 # 1
Density Estimation UCI GAS FFJORD Log-likelihood 8.59 # 3
Density Estimation UCI HEPMASS FFJORD Log-likelihood -14.92 # 1
NLL 14.92 # 1
Density Estimation UCI MINIBOONE FFJORD Log-likelihood -10.43 # 1
NLL 10.43 # 1
Density Estimation UCI POWER FFJORD Log-likelihood 0.46 # 4
NLL -0.46 # 1

Methods


No methods listed for this paper. Add relevant methods here