Search Results for author:
Found 41 papers, 7 papers with code
Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation
Intermediate layer matching is shown as an effective approach for improving knowledge distillation (KD).
How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge in a large neural network into a smaller one.
RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation
Knowledge Distillation (KD) is extensively used in Natural Language Processing to compress the pre-training and task-specific fine-tuning phases of large neural language models.
Towards Understanding Label Regularization for Fine-tuning Pre-trained Language Models
Knowledge Distillation (KD) is a prominent neural model compression technique which heavily relies on teacher network predictions to guide the training of a student model.
Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding
There is a growing body of work in recent years to develop pre-trained language models (PLMs) for the Arabic language.
Dynamic Position Encoding for Transformers
To solve this problem, position embeddings are defined exclusively for each time step to enrich word information.
CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillation
Knowledge distillation (KD) is an efficient framework for compressing large-scale pre-trained language models.
When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation
From a pre-generated pool of augmented samples, Glitter adaptively selects a subset of worst-case samples with maximal loss, analogous to adversarial DA.
JABER and SABER: Junior and Senior Arabic BERt
Language-specific pre-trained models have proven to be more accurate than multilingual ones in a monolingual evaluation setting, Arabic is no exception.
NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation
We apply NATURE to common slot-filling and intent detection benchmarks and demonstrate that simple perturbations from the standard evaluation set by NATURE can deteriorate model performance significantly.
A Short Study on Compressing Decoder-Based Language Models
Pre-trained Language Models (PLMs) have been successful for a wide range of natural language processing (NLP) tasks.
Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher
A case in point is that the best performing checkpoint of the teacher might not necessarily be the best teacher for training the student in KD.
Kronecker Decomposition for GPT Compression
GPT is an auto-regressive Transformer-based pre-trained language model which has attracted a lot of attention in the natural language processing (NLP) domain due to its state-of-the-art performance in several downstream tasks.
Pseudo Knowledge Distillation: Towards Learning Optimal Instance-specific Label Smoothing Regularization
Knowledge Distillation (KD) is an algorithm that transfers the knowledge of a trained, typically larger, neural network into another model under training.
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation
To address these problems, we propose a RAndom Intermediate Layer Knowledge Distillation (RAIL-KD) approach in which, intermediate layers from the teacher model are selected randomly to be distilled into the intermediate layers of the student model.
Knowledge Distillation with Noisy Labels for Natural Language Understanding
Knowledge Distillation (KD) is extensively used to compress and deploy large pre-trained language models on edge devices for real-world applications.
KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation
We present our KroneckerBERT, a compressed version of the BERT_BASE model obtained using this framework.
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one.
End-to-End Self-Debiasing Framework for Robust NLU Training
Existing Natural Language Understanding (NLU) models have been shown to incorporate dataset biases leading to strong performance on in-distribution (ID) test sets but poor performance on out-of-distribution (OOD) ones.
Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition
In this work, we examine the ability of NER models to use contextual information when predicting the type of an ambiguous entity.
Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMax
We exploit a semi-supervised approach based on KD to train a model on augmented data.
MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation
We present, MATE-KD, a novel text-based adversarial training algorithm which improves the performance of knowledge distillation.
From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables
Embedding matrices are key components in neural natural language processing (NLP) models that are responsible to provide numerical representations of input tokens.\footnote{In this paper words and subwords are referred to as \textit{tokens} and the term \textit{embedding} only refers to embeddings of inputs.}
Robust Embeddings Via Distributions
Despite recent monumental advances in the field, many Natural Language Processing (NLP) models still struggle to perform adequately on noisy domains.
Annealing Knowledge Distillation
Knowledge distillation (KD) is a prominent model compression technique for deep neural networks in which the knowledge of a trained large teacher model is transferred to a smaller student model.
Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation
End-to-end automatic speech recognition (ASR), unlike conventional ASR, does not have modules to learn the semantic representation from speech encoder.
Ranked #9 on
Speech Recognition
on LibriSpeech test-clean
Fine-tuning of Pre-trained End-to-end Speech Recognition with Generative Adversarial Networks
In this paper, we introduce a novel framework for fine-tuning a pre-trained ASR model using the GAN objective where the ASR model acts as a generator and a discriminator tries to distinguish the ASR output from the real data.
Improved knowledge distillation by utilizing backward pass knowledge in neural networks
Augmenting the training set by adding this auxiliary improves the performance of KD significantly and leads to a closer match between the student and the teacher.
Towards Zero-Shot Knowledge Distillation for Natural Language Processing
Knowledge Distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions.
ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
Knowledge distillation is considered as a training and compression strategy in which two neural networks, namely a teacher and a student, are coupled together during training.
From Unsupervised Machine Translation To Adversarial Text Generation
B-GAN is able to generate a distributed latent space representation which can be paired with an attention based decoder to generate fluent sentences.
A Simplified Fully Quantized Transformer for End-to-end Speech Recognition
While significant improvements have been made in recent years in terms of end-to-end automatic speech recognition (ASR) performance, such improvements were obtained through the use of very large neural networks, unfit for embedded use on edge devices.
Fully Quantized Transformer for Machine Translation
State-of-the-art neural machine translation methods employ massive amounts of parameters.
Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition
Word-embeddings are vital components of Natural Language Processing (NLP) models and have been extensively explored.
Distilled embedding: non-linear embedding factorization using knowledge distillation
Word-embeddings are a vital component of Natural Language Processing (NLP) systems and have been extensively researched.
EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing
We present the first sentence simplification model that learns explicit edit operations (ADD, DELETE, and KEEP) via a neural programmer-interpreter approach.
Ranked #2 on
Text Simplification
on PWKP / WikiSmall
(SARI metric)
TextKD-GAN: Text Generation using KnowledgeDistillation and Generative Adversarial Networks
Text generation is of particular interest in many NLP applications such as machine translation, language modeling, and text summarization.
Latent Code and Text-based Generative Adversarial Networks for Soft-text Generation
This soft representation will be used in GAN discrimination to synthesize similar soft-texts.
Bilingual-GAN: A Step Towards Parallel Text Generation
Latent space based GAN methods and attention based sequence to sequence models have achieved impressive results in text generation and unsupervised machine translation respectively.
Semi-Supervised Regression with Generative Adverserial Networks for End to End Learning in Autonomous Driving
We performed several experiments on a publicly available driving dataset to evaluate our proposed method, and the results are very promising.
SALSA-TEXT : self attentive latent space based adversarial text generation
Inspired by the success of self attention mechanism and Transformer architecture in sequence transduction and image generation applications, we propose novel self attention-based architectures to improve the performance of adversarial latent code- based schemes in text generation.
