Search Results for author:
Found 33 papers, 7 papers with code
Analyzing the effect of neural network architecture on training performance
Through novel theoretical and experimental results, we show how the neural net architecture affects gradient confusion, and thus the efficiency of training.
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models
We introduce RecurrentGemma, an open language model which uses Google's novel Griffin architecture.
Gemma: Open Models Based on Gemini Research and Technology
This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models.
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
Recurrent neural networks (RNNs) have fast inference and scale efficiently on long sequences, but they are difficult to train and hard to scale.
ConvNets Match Vision Transformers at Scale
Many researchers believe that ConvNets perform well on small or moderately sized datasets, but are not competitive with Vision Transformers when given access to datasets on the web-scale.
Unlocking Accuracy and Fairness in Differentially Private Image Classification
The poor performance of classifiers trained with DP has prevented the widespread adoption of privacy preserving machine learning in industry.
Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues
Deep neural networks based on linear complex-valued RNNs interleaved with position-wise MLPs are gaining traction as competitive approaches to sequence modeling.
Resurrecting Recurrent Neural Networks for Long Sequences
Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train.
Differentially Private Diffusion Models Generate Useful Synthetic Images
By privately fine-tuning ImageNet pre-trained diffusion models with more than 80M parameters, we obtain SOTA results on CIFAR-10 and Camelyon17 in terms of both FID and the accuracy of downstream classifiers trained on synthetic data.
Unlocking High-Accuracy Differentially Private Image Classification through Scale
Differential Privacy (DP) provides a formal privacy guarantee preventing adversaries with access to a machine learning model from extracting information about individual training points.
Classification
Image Classification with Differential Privacy
+1
Regularising for invariance to data augmentation improves supervised learning
Data augmentation is used in machine learning to make the classifier invariant to label-preserving transformations.
A study on the plasticity of neural networks
One aim shared by multiple settings, such as continual learning or transfer learning, is to leverage previously acquired knowledge to converge faster on the current task.
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error
In this work, we provide a detailed empirical evaluation of how the number of augmentation samples per unique image influences model performance on held out data when training deep ResNets.
Ranked #124 on
Image Classification
on ImageNet
High-Performance Large-Scale Image Recognition Without Normalization
Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples.
Ranked #30 on
Image Classification
on ImageNet
On the Origin of Implicit Regularization in Stochastic Gradient Descent
To interpret this phenomenon we prove that for SGD with random shuffling, the mean SGD iterate also stays close to the path of gradient flow if the learning rate is small and finite, but on a modified loss.
Characterizing signal propagation to close the performance gap in unnormalized ResNets
Batch Normalization is a key component in almost all state-of-the-art image classifiers, but it also introduces practical challenges: it breaks the independence between training examples within a batch, can incur compute and memory overhead, and often results in unexpected bugs.
BYOL works even without batch statistics
Bootstrap Your Own Latent (BYOL) is a self-supervised learning approach for image representation.
On the Generalization Benefit of Noise in Stochastic Gradient Descent
It has long been argued that minibatch stochastic gradient descent can generalize better than large batch gradient descent in deep neural networks.
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
Batch normalization dramatically increases the largest trainable depth of residual networks, and this benefit has been crucial to the empirical success of deep residual networks on a wide range of benchmarks.
Hyperparameter Tuning and Implicit Regularization in Minibatch SGD
First, we argue that stochastic gradient descent exhibits two regimes with different behaviours; a noise dominated regime which typically arises for small or moderate batch sizes, and a curvature dominated regime which typically arises when the batch size is large.
The Effect of Neural Net Architecture on Gradient Confusion & Training Performance
Through novel theoretical and experimental results, we show how the neural net architecture affects gradient confusion, and thus the efficiency of training.
Batch Normalization has Multiple Benefits: An Empirical Study on Residual Networks
This initialization scheme outperforms batch normalization when the batch size is very small, and is competitive with batch normalization for batch sizes that are not too large.
Adversarial Robustness through Local Linearization
Using this regularizer, we exceed current state of the art and achieve 47% adversarial accuracy for ImageNet with l-infinity adversarial perturbations of radius 4/255 under an untargeted, strong, white-box attack.
The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
Our results show that, for popular initialization techniques, increasing the width of neural networks leads to lower gradient confusion, and thus faster model training.
Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
Through these experiments we demonstrate the interesting sensitivity that ADAM has to its momentum parameter $\beta_1$.
Training Quantized Nets: A Deeper Understanding
Currently, deep neural networks are deployed on low-power portable devices by first training a full-precision model using powerful hardware, and then deriving a corresponding low-precision model for efficient inference on such systems.
An Empirical Study of ADMM for Nonconvex Problems
The alternating direction method of multipliers (ADMM) is a common optimization tool for solving constrained and non-differentiable problems.
Big Batch SGD: Automated Inference using Adaptive Batch Sizes
The high fidelity gradients enable automated learning rate selection and do not require stepsize decay.
Efficient Distributed SGD with Variance Reduction
Stochastic Gradient Descent (SGD) has become one of the most popular optimization methods for training machine learning models on massive datasets.
Variance Reduction for Distributed Stochastic Gradient Descent
Variance reduction (VR) methods boost the performance of stochastic gradient descent (SGD) by enabling the use of larger, constant stepsizes and preserving linear convergence rates.
Layer-Specific Adaptive Learning Rates for Deep Networks
In this paper, we attempt to overcome the two above problems by proposing an optimization method for training deep neural networks which uses learning rates which are both specific to each layer in the network and adaptive to the curvature of the function, increasing the learning rate at low curvature points.
Plagiarism Detection in Polyphonic Music using Monaural Signal Separation
Given the large number of new musical tracks released each year, automated approaches to plagiarism detection are essential to help us track potential violations of copyright.
