Search Results for author:
Found 14 papers, 7 papers with code
What’s Hidden in a One-layer Randomly Weighted Transformer?
Hidden within a one-layer randomly weighted Transformer, we find that subnetworks that can achieve 29. 45/17. 29 BLEU on IWSLT14/WMT14.
Error Estimation for Sketched SVD
In order to compute fast approximations to the singular value decompositions (SVD) of very large matrices, randomized sketching algorithms have become a leading approach.
GACT: Activation Compressed Training for Generic Network Architectures
Training large neural network (NN) models requires extensive memory resources, and Activation Compressed Training (ACT) is a promising approach to reduce training memory footprint.
AutoIP: A United Framework to Integrate Physics into Gaussian Processes
Physical modeling is critical for many modern science and engineering applications.
LocalNewton: Reducing Communication Bottleneck for Distributed Learning
To enhance practicability, we devise an adaptive scheme to choose L, and we show that this reduces the number of local iterations in worker machines between two model synchronizations as the training proceeds, successively refining the model quality at the master.
PyHessian: Neural Networks Through the Lens of the Hessian
To illustrate this, we analyze the effect of residual connections and Batch Normalization layers on the trainability of neural networks.
ANODEV2: A Coupled Neural ODE Evolution Framework
It has been observed that residual networks can be viewed as the explicit Euler discretization of an Ordinary Differential Equation (ODE).
HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
Another challenge is a similar factorial complexity for determining block-wise fine-tuning order when quantizing the model to a target precision.
Trust Region Based Adversarial Attack on Neural Networks
To address this problem, we present a new family of trust region based adversarial attacks, with the goal of computing adversarial perturbations efficiently.
Parameter Re-Initialization through Cyclical Batch Size Schedules
We demonstrate the ability of our method to improve language modeling performance by up to 7. 91 perplexity and reduce training iterations by up to $61\%$, in addition to its flexibility in enabling snapshot ensembling and use with adversarial training.
Ranked #51 on
Natural Language Inference
on SNLI
Large batch size training of neural networks with adversarial training and second-order information
Our method exceeds the performance of existing solutions in terms of both accuracy and the number of SGD iterations (up to 1\% and $5\times$, respectively).
Statistical and Algorithmic Perspectives on Randomized Sketching for Ordinary Least-Squares -- ICML
We then consider the statistical prediction efficiency (PE) and the statistical residual efficiency (RE) of the sketched LS estimator; and we use our framework to provide upper bounds for several types of random projection and random sampling algorithms.
Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernels
These approximate feature maps arise as Monte Carlo approximations to integral representations of shift-invariant kernel functions (e. g., Gaussian kernel).
A Statistical Perspective on Randomized Sketching for Ordinary Least-Squares
Prior results show that, when using sketching matrices such as random projections and leverage-score sampling algorithms, with $p < r \ll n$, the WC error is the same as solving the original problem, up to a small constant.
