Search Results for author:
Found 13 papers, 8 papers with code
Quantization Aware Factorization for Deep Neural Network Compression
Tensor decomposition of convolutional and fully-connected layers is an effective way to reduce parameters and FLOP in neural networks.
Rockmate: an Efficient, Fast, Automatic and Generic Tool for Re-materialization in PyTorch
We show through experiments on many models that Rockmate is as fast as Rotor and as efficient as Checkmate, and that it allows in many cases to obtain a significantly lower memory consumption for activations (by a factor of 2 to 5) for a rather negligible overhead (of the order of 10% to 20%).
Efficient GPT Model Pre-training using Tensor Train Matrix Representation
On the downstream tasks, including language understanding and text summarization, the model performs similarly to the original GPT-2 model.
Survey on Large Scale Neural Network Training
Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training.
Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction
Every modern neural network model has quite a few pointwise nonlinearities in its architecture, and such operation induces additional memory costs which -- as we show -- can be significantly reduced by quantization of the gradients.
Memory-Efficient Backpropagation through Large Linear Layers
Also, we investigate the variance of the gradient estimate induced by the randomized matrix multiplication.
Meta-Solver for Neural Ordinary Differential Equations
Moreover, we show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks.
Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network
Most state of the art deep neural networks are overparameterized and exhibit a high computational cost.
Towards Understanding Normalization in Neural ODEs
Normalization is an important and vastly investigated technique in deep learning.
Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs
We propose a simple interpolation-based method for the efficient approximation of gradients in neural ODE models.
Active Subspace of Neural Networks: Structural Analysis and Universal Attacks
Secondly, we propose analyzing the vulnerability of a neural network using active subspace and finding an additive universal adversarial attack vector that can misclassify a dataset with a high probability.
Reduced-Order Modeling of Deep Neural Networks
We introduce a new method for speeding up the inference of deep neural networks.
MUSCO: Multi-Stage Compression of neural networks
The low-rank tensor approximation is very promising for the compression of deep neural networks.
