Search Results for author:
Found 20 papers, 8 papers with code
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
Recurrent neural networks (RNNs) have fast inference and scale efficiently on long sequences, but they are difficult to train and hard to scale.
ConvNets Match Vision Transformers at Scale
Many researchers believe that ConvNets perform well on small or moderately sized datasets, but are not competitive with Vision Transformers when given access to datasets on the web-scale.
Unlocking Accuracy and Fairness in Differentially Private Image Classification
The poor performance of classifiers trained with DP has prevented the widespread adoption of privacy preserving machine learning in industry.
Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues
Deep neural networks based on linear complex-valued RNNs interleaved with position-wise MLPs are gaining traction as competitive approaches to sequence modeling.
Differentially Private Diffusion Models Generate Useful Synthetic Images
By privately fine-tuning ImageNet pre-trained diffusion models with more than 80M parameters, we obtain SOTA results on CIFAR-10 and Camelyon17 in terms of both FID and the accuracy of downstream classifiers trained on synthetic data.
Unlocking High-Accuracy Differentially Private Image Classification through Scale
Differential Privacy (DP) provides a formal privacy guarantee preventing adversaries with access to a machine learning model from extracting information about individual training points.
Classification
Image Classification with Differential Privacy
+1
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error
In this work, we provide a detailed empirical evaluation of how the number of augmentation samples per unique image influences model performance on held out data when training deep ResNets.
Ranked #124 on
Image Classification
on ImageNet
High-Performance Large-Scale Image Recognition Without Normalization
Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples.
Ranked #31 on
Image Classification
on ImageNet
On the Origin of Implicit Regularization in Stochastic Gradient Descent
To interpret this phenomenon we prove that for SGD with random shuffling, the mean SGD iterate also stays close to the path of gradient flow if the learning rate is small and finite, but on a modified loss.
Characterizing signal propagation to close the performance gap in unnormalized ResNets
Batch Normalization is a key component in almost all state-of-the-art image classifiers, but it also introduces practical challenges: it breaks the independence between training examples within a batch, can incur compute and memory overhead, and often results in unexpected bugs.
Cold Posteriors and Aleatoric Uncertainty
Recent work has observed that one can outperform exact inference in Bayesian neural networks by tuning the "temperature" of the posterior on a validation set (the "cold posterior" effect).
On the Generalization Benefit of Noise in Stochastic Gradient Descent
It has long been argued that minibatch stochastic gradient descent can generalize better than large batch gradient descent in deep neural networks.
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
Batch normalization dramatically increases the largest trainable depth of residual networks, and this benefit has been crucial to the empirical success of deep residual networks on a wide range of benchmarks.
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
We find that the optimal SGD hyper-parameters are determined by a "normalized noise scale," which is a function of the batch size, learning rate, and initialization conditions.
Stochastic natural gradient descent draws posterior samples in function space
Recent work has argued that stochastic gradient descent can approximate the Bayesian uncertainty in model parameters near local minima.
Decoding Decoders: Finding Optimal Representation Spaces for Unsupervised Similarity Tasks
Experimental evidence indicates that simple models outperform complex deep networks on many unsupervised similarity tasks.
Don't Decay the Learning Rate, Increase the Batch Size
We can further reduce the number of parameter updates by increasing the learning rate $\epsilon$ and scaling the batch size $B \propto \epsilon$.
A Bayesian Perspective on Generalization and Stochastic Gradient Descent
Interpreting stochastic gradient descent as a stochastic differential equation, we identify the "noise scale" $g = \epsilon (\frac{N}{B} - 1) \approx \epsilon N/B$, where $\epsilon$ is the learning rate, $N$ the training set size and $B$ the batch size.
Offline bilingual word vectors, orthogonal transformations and the inverted softmax
We introduce a novel "inverted softmax" for identifying translation pairs, with which we improve the precision @1 of Mikolov's original mapping from 34% to 43%, when translating a test set composed of both common and rare English words into Italian.
Monte Carlo Sort for unreliable human comparisons
We develop a novel sorting algorithm, where each pairwise comparison reflects a subjective human judgement about which element is bigger or better.
