Search Results for author:
Found 25 papers, 11 papers with code
Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation
Skip connections and normalisation layers form two standard architectural components that are ubiquitous for the training of Deep Neural Networks (DNNs), but whose precise roles are poorly understood.
Approximate optimality and the risk/reward tradeoff in a class of bandit problems
This paper studies a multi-armed bandit problem where payoff distributions are known but where the riskiness of payoffs matters.
Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers
However, this method (called Deep Kernel Shaping) isn't fully compatible with ReLUs, and produces networks that overfit significantly more than ResNets on ImageNet.
On the Application of Data-Driven Deep Neural Networks in Linear and Nonlinear Structural Dynamics
For nonlinear dynamics, it is shown that sparsity in network layers is lost, and efficient DNNs architectures with fully-connected and convolutional network layers are explored.
Learning to Give Checkable Answers with Prover-Verifier Games
We develop instantiations of the PVG for two algorithmic tasks, and show that in practice, the verifier learns a robust decision rule that is able to receive useful and reliable information from an untrusted prover.
Differentiable Annealed Importance Sampling and the Perils of Gradient Noise
To this end, we propose Differentiable AIS (DAIS), a variant of AIS which ensures differentiability by abandoning the Metropolis-Hastings corrections.
A Central Limit Theorem, Loss Aversion and Multi-Armed Bandits
This paper studies a multi-armed bandit problem where the decision-maker is loss averse, in particular she is risk averse in the domain of gains and risk loving in the domain of losses.
Near-optimal Local Convergence of Alternating Gradient Descent-Ascent for Minimax Optimization
Smooth minimax games often proceed by simultaneous or alternating gradient updates.
A Unified Analysis of First-Order Methods for Smooth Games via Integral Quadratic Constraints
The theory of integral quadratic constraints (IQCs) allows the certification of exponential convergence of interconnected systems containing nonlinear or uncertain elements.
On the Suboptimality of Negative Momentum for Minimax Optimization
Smooth game optimization has recently attracted great interest in machine learning as it generalizes the single-objective optimization paradigm.
Picking Winning Tickets Before Training by Preserving Gradient Flow
Overparameterization has been shown to benefit both the optimization and generalization of neural networks, but large networks are resource hungry at both training and test time.
Fast Convergence of Natural Gradient Descent for Over-Parameterized Neural Networks
For two-layer ReLU neural networks (i. e. with one hidden layer), we prove that these two conditions do hold throughout the training under the assumptions that the inputs do not degenerate and the network is over-parameterized.
On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach
Many tasks in modern machine learning can be formulated as finding equilibria in \emph{sequential} games.
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns.
Benchmarking Model-Based Reinforcement Learning
Model-based reinforcement learning (MBRL) is widely seen as having the potential to be significantly more sample efficient than model-free RL.
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
In this work, we analyze for the first time the speed of convergence of natural gradient descent on nonlinear neural networks with squared-error loss.
EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
Reducing the test time resource requirements of a neural network while preserving test accuracy is crucial for running inference on resource-constrained devices.
Functional Variational Bayesian Neural Networks
We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defined directly on stochastic processes, i. e. distributions over functions.
An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
We demonstrate that the learning performance of our method is more accurately captured by the structure of the covariance matrix of the noise rather than by the variance of gradients.
Eigenvalue Corrected Noisy Natural Gradient
A recently proposed method, noisy natural gradient, is a surprisingly simple method to fit expressive posteriors by adding weight noise to regular natural gradient updates.
Three Mechanisms of Weight Decay Regularization
Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization.
Exploring Curvature Noise in Large-Batch Stochastic Optimization
Unfortunately, a major drawback is the so-called generalization gap: large-batch training typically leads to a degradation in generalization performance of the model as compared to small-batch training.
Differentiable Compositional Kernel Learning for Gaussian Processes
The NKN architecture is based on the composition rules for kernels, so that each unit of the network corresponds to a valid kernel.
Noisy Natural Gradient as Variational Inference
Variational Bayesian neural nets combine the flexibility of deep learning with Bayesian uncertainty estimation.
Deformable Convolutional Networks
Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in its building modules.
Ranked #3 on
Vessel Detection
on Vessel detection Dateset
