Search Results for author:
Found 58 papers, 38 papers with code
Improving Mutual Information Estimation with Annealed and Energy-Based Bounds
Since accurate estimation of MI without density information requires a sample size exponential in the true MI, we assume either a single marginal or the full joint density information is known.
Efficient Parametric Approximations of Neural Network Function Space Distance
It is often useful to compactly summarize important properties of model parameters and training data so that they can be used later without storing and/or iterating over the entire dataset.
On Implicit Bias in Overparameterized Bilevel Optimization
Bilevel problems consist of two nested sub-problems, called the outer and inner problems, respectively.
Discovering Language Model Behaviors with Model-Written Evaluations
We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse.
Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve
Variational autoencoders (VAEs) are powerful tools for learning latent representations of data used in a wide range of applications.
Similarity-based Cooperation
Moreover, it is challenging for agents to learn their way to cooperation in the full transparency setting.
Path Independent Equilibrium Models Can Better Exploit Test-Time Computation
Designing networks capable of attaining better performance with an increased inference budget is important to facilitate generalization to harder problem instances.
Toy Models of Superposition
Neural networks often pack many unrelated concepts into a single neuron - a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging.
If Influence Functions are the Answer, Then What is the Question?
Influence functions efficiently estimate the effect of removing a single training data point on a model's learned parameters.
Amortized Proximal Optimization
Using APO to adapt a structured preconditioning matrix generally results in optimization performance competitive with second-order methods.
Learning to Give Checkable Answers with Prover-Verifier Games
We develop instantiations of the PVG for two algorithmic tasks, and show that in practice, the verifier learns a robust decision rule that is able to receive useful and reliable information from an untrusted prover.
Differentiable Annealed Importance Sampling and the Perils of Gradient Noise
To this end, we propose Differentiable AIS (DAIS), a variant of AIS which ensures differentiability by abandoning the Metropolis-Hastings corrections.
Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition
We introduce a new scalable variational Gaussian process approximation which provides a high fidelity approximation while retaining general applicability.
Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decrease in the training objective.
Near-optimal Local Convergence of Alternating Gradient Descent-Ascent for Minimax Optimization
Smooth minimax games often proceed by simultaneous or alternating gradient updates.
LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning
While designing inductive bias in neural architectures has been widely studied, we hypothesize that transformer networks are flexible enough to learn inductive bias from suitable generic tasks.
Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?
While uncertainty estimation is a well-studied topic in deep learning, most such work focuses on marginal uncertainty estimates, i. e. the predictive mean and variance at individual input locations.
Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
Hyperparameter optimization of neural networks can be elegantly formulated as a bilevel optimization problem.
A Unified Analysis of First-Order Methods for Smooth Games via Integral Quadratic Constraints
The theory of integral quadratic constraints (IQCs) allows the certification of exponential convergence of interconnected systems containing nonlinear or uncertain elements.
Evaluating Lossy Compression Rates of Deep Generative Models
The field of deep generative modeling has succeeded in producing astonishingly realistic-seeming images and audio, but quantitative evaluation remains a challenge.
Regularized linear autoencoders recover the principal components, eventually
Our understanding of learning input-output relationships with neural nets has improved rapidly in recent years, but little is known about the convergence of the underlying representations, even in the simple case of linear autoencoders (LAEs).
The Scattering Compositional Learner: Discovering Objects, Attributes, Relationships in Analogical Reasoning
In this work, we focus on an analogical reasoning task that contains rich compositional structures, Raven's Progressive Matrices (RPM).
Learning Branching Heuristics for Propositional Model Counting
In addition to step count improvements, Neuro# can also achieve orders of magnitude wall-clock speedups over the vanilla solver on larger instances in some problem families, despite the runtime overhead of querying the model.
INT: An Inequality Benchmark for Evaluating Generalization in Theorem Proving
In learning-assisted theorem proving, one of the most critical challenges is to generalize to theorems unlike those seen at training time.
When Does Preconditioning Help or Hurt Generalization?
While second order optimizers such as natural gradient descent (NGD) often speed up optimization, their effect on generalization has been called into question.
Understanding and Mitigating Exploding Inverses in Invertible Neural Networks
For problems where global invertibility is necessary, such as applying normalizing flows on OOD data, we show the importance of designing stable INN building blocks.
Picking Winning Tickets Before Training by Preserving Gradient Flow
Overparameterization has been shown to benefit both the optimization and generalization of neural networks, but large networks are resource hungry at both training and test time.
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables.
Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks
Our BCOP parameterization allows us to train large convolutional networks with provable Lipschitz bounds.
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns.
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
In this work, we analyze for the first time the speed of convergence of natural gradient descent on nonlinear neural networks with squared-error loss.
EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
Reducing the test time resource requirements of a neural network while preserving test accuracy is crucial for running inference on resource-constrained devices.
Online Hyperparameter Adaptation via Amortized Proximal Optimization
Effective performance of neural networks depends critically on effective tuning of optimization hyperparameters, especially learning rates (and schedules thereof).
Understanding Posterior Collapse in Generative Latent Variable Models
Posterior collapse in Variational Autoencoders (VAEs) arises when the variational distribution closely matches the uninformative prior for a subset of latent variables.
Functional Variational Bayesian Neural Networks
We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defined directly on stochastic processes, i. e. distributions over functions.
Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems.
Eigenvalue Corrected Noisy Natural Gradient
A recently proposed method, noisy natural gradient, is a surprisingly simple method to fit expressive posteriors by adding weight noise to regular natural gradient updates.
Sorting out Lipschitz function approximation
We identify a necessary property for such an architecture: each of the layers must preserve the gradient norm during backpropagation.
Three Mechanisms of Weight Decay Regularization
Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization.
Reversible Recurrent Neural Networks
Reversible RNNs---RNNs for which the hidden-to-hidden transition can be reversed---offer a path to reduce the memory requirements of training, as hidden states need not be stored and instead can be recomputed during backpropagation.
A Coordinate-Free Construction of Scalable Natural Gradient
Most neural networks are trained using first-order optimization methods, which are sensitive to the parameterization of the model.
Distilling the Posterior in Bayesian Neural Networks
We propose a framework, Adversarial Posterior Distillation, to distill the SGLD samples using a Generative Adversarial Network (GAN).
Adversarial Distillation of Bayesian Neural Network Posteriors
We propose a framework, Adversarial Posterior Distillation, to distill the SGLD samples using a Generative Adversarial Network (GAN).
Differentiable Compositional Kernel Learning for Gaussian Processes
The NKN architecture is based on the composition rules for kernels, so that each unit of the network corresponds to a valid kernel.
Aggregated Momentum: Stability Through Passive Damping
Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions.
Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies.
Understanding Short-Horizon Bias in Stochastic Meta-Optimization
Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training.
Isolating Sources of Disentanglement in Variational Autoencoders
We decompose the evidence lower bound to show the existence of a term measuring the total correlation between latent variables.
Noisy Natural Gradient as Variational Inference
Variational Bayesian neural nets combine the flexibility of deep learning with Bayesian uncertainty estimation.
Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
In this work, we propose to apply trust region optimization to deep reinforcement learning using a recently proposed Kronecker-factored approximation to the curvature.
On the Quantitative Analysis of Decoder-Based Generative Models
The past several years have seen remarkable progress in generative models which produce convincing samples of images and other modalities.
A Kronecker-factored approximate Fisher matrix for convolution layers
Second-order optimization methods such as natural gradient descent have the potential to speed up training of neural networks by correcting for the curvature of the loss function.
Learning Wake-Sleep Recurrent Attention Models
Despite their success, convolutional neural networks are computationally expensive because they must examine all image locations.
Statistical Inference, Learning and Models in Big Data
The need for new methods to deal with big data is a common theme in most scientific fields, although its definition tends to vary with the context.
Importance Weighted Autoencoders
The variational autoencoder (VAE; Kingma, Welling (2014)) is a recently proposed generative model pairing a top-down generative network with a bottom-up recognition network which approximates posterior inference.
Optimizing Neural Networks with Kronecker-factored Approximate Curvature
This is because the cost of storing and inverting K-FAC's approximation to the curvature matrix does not depend on the amount of data used to estimate it, which is a feature typically associated only with diagonal or low-rank approximations to the curvature matrix.
Automatic Construction and Natural-Language Description of Nonparametric Regression Models
This paper presents the beginnings of an automatic statistician, focusing on regression problems.
Structure Discovery in Nonparametric Regression through Compositional Kernel Search
Despite its importance, choosing the structural form of the kernel in nonparametric regression remains a black art.
