Search Results for author:
Found 18 papers, 0 papers with code
Gradient Sparsification For Masked Fine-Tuning of Transformers
We introduce GradDrop and variants thereof, a class of gradient sparsification methods that mask gradients during the backward pass, acting as gradient noise.
Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models
We investigate the effects of post-training quantization and quantization-aware training on the generalization of Transformer language models.
Aligned Weight Regularizers for Pruning Pretrained Neural Networks
While various avenues of research have been explored for iterative pruning, little is known what effect pruning has on zero-shot test performance and its potential implications on the choice of pruning criteria.
Deep Neural Compression Via Concurrent Pruning and Self-Distillation
Pruning aims to reduce the number of parameters while maintaining performance close to the original network.
Self-Distilled Pruning Of Neural Networks
Pruning aims to reduce the number of parameters while maintaining performance close to the original network.
Semantically-Conditioned Negative Samples for Efficient Contrastive Learning
In the knowledge distillation setting, (1) the performance of student networks increase by 4. 56\% percentage points on Tiny-ImageNet-200 and 3. 29\% on CIFAR-100 over student networks trained with no teacher and (2) 1. 23\% and 1. 72\% respectively over a \textit{hard-to-beat} baseline (Hinton et al., 2015).
$k$-Neighbor Based Curriculum Sampling for Sequence Prediction
At test time, a sequence predictor is required to make predictions given past predictions as the input, instead of the past targets that are provided during training.
Compressing Deep Neural Networks via Layer Fusion
This paper proposes \textit{layer fusion} - a model compression technique that discovers which weights to combine and then fuses weights of similar fully-connected, convolutional and attention layers.
An Overview of Neural Network Compression
Thus, in recent years there has been a resurgence in model compression techniques, particularly for deep convolutional neural networks and self-attention based networks such as the Transformer.
Transfer Reward Learning for Policy Gradient-Based Text Generation
However, we argue that current n-gram overlap based measures that are used as rewards can be improved by using model-based rewards transferred from tasks that directly compare the similarity of sentence pairs.
Learning To Avoid Negative Transfer in Few Shot Transfer Learning
However, transferring all parameters, some of which irrelevant for a target task, can lead to sub-optimal results and can have a negative effect on performance, referred to as \textit{negative} transfer.
Error-Correcting Neural Sequence Prediction
We propose a novel neural sequence prediction method based on \textit{error-correcting output codes} that avoids exact softmax normalization and allows for a tradeoff between speed and performance.
Analysing Dropout and Compounding Errors in Neural Language Models
Moreover, we propose an extension of variational dropout to concrete dropout and curriculum dropout with varying schedules.
Curriculum-Based Neighborhood Sampling For Sequence Prediction
At test time, a language model is required to make predictions given past predictions as input, instead of the past targets that are provided during training.
Meta-Embedding as Auxiliary Task Regularization
For intrinsic task evaluation, supervision comes from various labeled word similarity datasets.
Angular-Based Word Meta-Embedding Learning
This work compares meta-embeddings trained for different losses, namely loss functions that account for angular distance between the reconstructed embedding and the target and those that account normalized distances based on the vector length.
Siamese Capsule Networks
Capsule Networks have shown encouraging results on \textit{defacto} benchmark computer vision datasets such as MNIST, CIFAR and smallNORB.
Dropping Networks for Transfer Learning
We also compare against models that are fully trained on the target task in the standard supervised learning setup.
