Search Results for author:
Found 25 papers, 5 papers with code
How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge in a large neural network into a smaller one.
RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation
Knowledge Distillation (KD) is extensively used in Natural Language Processing to compress the pre-training and task-specific fine-tuning phases of large neural language models.
Preventing Arbitrarily High Confidence on Far-Away Data in Point-Estimated Discriminative Neural Networks
For instance, ReLU networks - a popular class of neural network architectures - have been shown to almost always yield high confidence predictions when the test data are far away from the training set, even when they are trained with OOD data.
Attribute Controlled Dialogue Prompting
Prompt-tuning has become an increasingly popular parameter-efficient method for adapting large pretrained language models to downstream tasks.
LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization
Label Smoothing (LS) is another simple, versatile and efficient regularization which can be applied to various supervised classification tasks.
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
Moreover, we observe that this simple optimization technique is able to outperform the state-of-the-art KD methods for compact models.
Learning Functions on Multiple Sets using Multi-Set Transformers
We propose a general deep architecture for learning functions on multiple permutation-invariant sets.
Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding
There is a growing body of work in recent years to develop pre-trained language models (PLMs) for the Arabic language.
JABER and SABER: Junior and Senior Arabic BERt
Language-specific pre-trained models have proven to be more accurate than multilingual ones in a monolingual evaluation setting, Arabic is no exception.
NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation
We apply NATURE to common slot-filling and intent detection benchmarks and demonstrate that simple perturbations from the standard evaluation set by NATURE can deteriorate model performance significantly.
A Short Study on Compressing Decoder-Based Language Models
Pre-trained Language Models (PLMs) have been successful for a wide range of natural language processing (NLP) tasks.
Kronecker Decomposition for GPT Compression
GPT is an auto-regressive Transformer-based pre-trained language model which has attracted a lot of attention in the natural language processing (NLP) domain due to its state-of-the-art performance in several downstream tasks.
Pseudo Knowledge Distillation: Towards Learning Optimal Instance-specific Label Smoothing Regularization
Knowledge Distillation (KD) is an algorithm that transfers the knowledge of a trained, typically larger, neural network into another model under training.
Knowledge Distillation with Noisy Labels for Natural Language Understanding
Knowledge Distillation (KD) is extensively used to compress and deploy large pre-trained language models on edge devices for real-world applications.
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one.
End-to-End Self-Debiasing Framework for Robust NLU Training
Existing Natural Language Understanding (NLU) models have been shown to incorporate dataset biases leading to strong performance on in-distribution (ID) test sets but poor performance on out-of-distribution (OOD) ones.
Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition
In this work, we examine the ability of NER models to use contextual information when predicting the type of an ambiguous entity.
MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation
We present, MATE-KD, a novel text-based adversarial training algorithm which improves the performance of knowledge distillation.
Robust Embeddings Via Distributions
Despite recent monumental advances in the field, many Natural Language Processing (NLP) models still struggle to perform adequately on noisy domains.
Towards Zero-Shot Knowledge Distillation for Natural Language Processing
Knowledge Distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions.
From Unsupervised Machine Translation To Adversarial Text Generation
B-GAN is able to generate a distributed latent space representation which can be paired with an attention based decoder to generate fluent sentences.
Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition
Word-embeddings are vital components of Natural Language Processing (NLP) models and have been extensively explored.
Distilled embedding: non-linear embedding factorization using knowledge distillation
Word-embeddings are a vital component of Natural Language Processing (NLP) systems and have been extensively researched.
Latent Code and Text-based Generative Adversarial Networks for Soft-text Generation
This soft representation will be used in GAN discrimination to synthesize similar soft-texts.
Bilingual-GAN: A Step Towards Parallel Text Generation
Latent space based GAN methods and attention based sequence to sequence models have achieved impressive results in text generation and unsupervised machine translation respectively.
