Search Results for author: Julia Gusak

Found 13 papers, 8 papers with code

Quantization Aware Factorization for Deep Neural Network Compression

no code implementations8 Aug 2023 Daria Cherniuk, Stanislav Abukhovich, Anh-Huy Phan, Ivan Oseledets, Andrzej Cichocki, Julia Gusak

Tensor decomposition of convolutional and fully-connected layers is an effective way to reduce parameters and FLOP in neural networks.

Neural Network Compression Quantization +1

Rockmate: an Efficient, Fast, Automatic and Generic Tool for Re-materialization in PyTorch

1 code implementation3 Jul 2023 Xunyi Zhao, Théotime Le Hellard, Lionel Eyraud, Julia Gusak, Olivier Beaumont

We show through experiments on many models that Rockmate is as fast as Rotor and as efficient as Checkmate, and that it allows in many cases to obtain a significantly lower memory consumption for activations (by a factor of 2 to 5) for a rather negligible overhead (of the order of 10% to 20%).

Efficient GPT Model Pre-training using Tensor Train Matrix Representation

no code implementations5 Jun 2023 Viktoriia Chekalina, Georgii Novikov, Julia Gusak, Ivan Oseledets, Alexander Panchenko

On the downstream tasks, including language understanding and text summarization, the model performs similarly to the original GPT-2 model.

Language Modelling Text Summarization

Survey on Large Scale Neural Network Training

no code implementations21 Feb 2022 Julia Gusak, Daria Cherniuk, Alena Shilova, Alexander Katrutsa, Daniel Bershatsky, Xunyi Zhao, Lionel Eyraud-Dubois, Oleg Shlyazhko, Denis Dimitrov, Ivan Oseledets, Olivier Beaumont

Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training.

Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction

2 code implementations1 Feb 2022 Georgii Novikov, Daniel Bershatsky, Julia Gusak, Alex Shonenkov, Denis Dimitrov, Ivan Oseledets

Every modern neural network model has quite a few pointwise nonlinearities in its architecture, and such operation induces additional memory costs which -- as we show -- can be significantly reduced by quantization of the gradients.

Neural Network Compression Quantization

Memory-Efficient Backpropagation through Large Linear Layers

2 code implementations31 Jan 2022 Daniel Bershatsky, Aleksandr Mikhalev, Alexandr Katrutsa, Julia Gusak, Daniil Merkulov, Ivan Oseledets

Also, we investigate the variance of the gradient estimate induced by the randomized matrix multiplication.

Model Compression

Meta-Solver for Neural Ordinary Differential Equations

1 code implementation15 Mar 2021 Julia Gusak, Alexandr Katrutsa, Talgat Daulbaev, Andrzej Cichocki, Ivan Oseledets

Moreover, we show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks.

Active Subspace of Neural Networks: Structural Analysis and Universal Attacks

1 code implementation29 Oct 2019 Chunfeng Cui, Kaiqi Zhang, Talgat Daulbaev, Julia Gusak, Ivan Oseledets, Zheng Zhang

Secondly, we propose analyzing the vulnerability of a neural network using active subspace and finding an additive universal adversarial attack vector that can misclassify a dataset with a high probability.

Adversarial Attack Uncertainty Quantification

Reduced-Order Modeling of Deep Neural Networks

no code implementations15 Oct 2019 Julia Gusak, Talgat Daulbaev, Evgeny Ponomarev, Andrzej Cichocki, Ivan Oseledets

We introduce a new method for speeding up the inference of deep neural networks.

Cannot find the paper you are looking for? You can Submit a new open access paper.