Search Results for author:
Found 70 papers, 16 papers with code
JABER and SABER: Junior and Senior Arabic BERt
Language-specific pre-trained models have proven to be more accurate than multilingual ones in a monolingual evaluation setting, Arabic is no exception.
On the importance of Data Scale in Pretraining Arabic Language Models
Pretraining monolingual language models have been proven to be vital for performance in Arabic Natural Language Processing (NLP) tasks.
DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation
Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training.
Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages
MIRACL (Multilingual Information Retrieval Across a Continuum of Languages) is a multilingual dataset we have built for the WSDM 2023 Cup challenge that focuses on ad hoc retrieval across 18 different languages, which collectively encompass over three billion native speakers around the world.
Annealing Knowledge Distillation
Knowledge distillation (KD) is a prominent model compression technique for deep neural networks in which the knowledge of a trained large teacher model is transferred to a smaller student model.
MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation
We present, MATE-KD, a novel text-based adversarial training algorithm which improves the performance of knowledge distillation.
Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMax
We exploit a semi-supervised approach based on KD to train a model on augmented data.
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one.
When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation
From a pre-generated pool of augmented samples, Glitter adaptively selects a subset of worst-case samples with maximal loss, analogous to adversarial DA.
EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing
We present the first sentence simplification model that learns explicit edit operations (ADD, DELETE, and KEEP) via a neural programmer-interpreter approach.
Ranked #2 on
Text Simplification
on PWKP / WikiSmall
(SARI metric)
Resonance RoPE: Improving Context Length Generalization of Large Language Models
This paper addresses the challenge of train-short-test-long (TSTL) scenarios in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE), where models pre-trained on shorter sequences face difficulty with out-of-distribution (OOD) token positions in longer sequences.
NoMIRACL: Knowing When You Don't Know for Robust Multilingual Retrieval-Augmented Generation
We measure LLM robustness using two metrics: (i) hallucination rate, measuring model tendency to hallucinate an answer, when the answer is not present in passages in the non-relevant subset, and (ii) error rate, measuring model inaccuracy to recognize relevant passages in the relevant subset.
TextKD-GAN: Text Generation using KnowledgeDistillation and Generative Adversarial Networks
Text generation is of particular interest in many NLP applications such as machine translation, language modeling, and text summarization.
An Efficient End-to-End Approach to Noise Invariant Speech Features via Multi-Task Learning
Lastly, we show that the proposed recipe can be applied to other distillation methodologies, such as the recent DPWavLM.
Learning Functions on Multiple Sets using Multi-Set Transformers
We propose a general deep architecture for learning functions on multiple permutation-invariant sets.
A Simplified Fully Quantized Transformer for End-to-end Speech Recognition
While significant improvements have been made in recent years in terms of end-to-end automatic speech recognition (ASR) performance, such improvements were obtained through the use of very large neural networks, unfit for embedded use on edge devices.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SALSA-TEXT : self attentive latent space based adversarial text generation
Inspired by the success of self attention mechanism and Transformer architecture in sequence transduction and image generation applications, we propose novel self attention-based architectures to improve the performance of adversarial latent code- based schemes in text generation.
Bilingual-GAN: A Step Towards Parallel Text Generation
Latent space based GAN methods and attention based sequence to sequence models have achieved impressive results in text generation and unsupervised machine translation respectively.
Latent Code and Text-based Generative Adversarial Networks for Soft-text Generation
This soft representation will be used in GAN discrimination to synthesize similar soft-texts.
Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition
Word-embeddings are vital components of Natural Language Processing (NLP) models and have been extensively explored.
Fully Quantized Transformer for Machine Translation
State-of-the-art neural machine translation methods employ massive amounts of parameters.
From Unsupervised Machine Translation To Adversarial Text Generation
B-GAN is able to generate a distributed latent space representation which can be paired with an attention based decoder to generate fluent sentences.
ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
Knowledge distillation is considered as a training and compression strategy in which two neural networks, namely a teacher and a student, are coupled together during training.
Towards Zero-Shot Knowledge Distillation for Natural Language Processing
Knowledge Distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions.
Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation
End-to-end automatic speech recognition (ASR), unlike conventional ASR, does not have modules to learn the semantic representation from speech encoder.
Ranked #12 on
Speech Recognition
on LibriSpeech test-clean
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Fine-tuning of Pre-trained End-to-end Speech Recognition with Generative Adversarial Networks
In this paper, we introduce a novel framework for fine-tuning a pre-trained ASR model using the GAN objective where the ASR model acts as a generator and a discriminator tries to distinguish the ASR output from the real data.
Semi-Supervised Regression with Generative Adverserial Networks for End to End Learning in Autonomous Driving
We performed several experiments on a publicly available driving dataset to evaluate our proposed method, and the results are very promising.
From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables
Embedding matrices are key components in neural natural language processing (NLP) models that are responsible to provide numerical representations of input tokens.\footnote{In this paper words and subwords are referred to as \textit{tokens} and the term \textit{embedding} only refers to embeddings of inputs.}
Robust Embeddings Via Distributions
Despite recent monumental advances in the field, many Natural Language Processing (NLP) models still struggle to perform adequately on noisy domains.
Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition
In this work, we examine the ability of NER models to use contextual information when predicting the type of an ambiguous entity.
End-to-End Self-Debiasing Framework for Robust NLU Training
Existing Natural Language Understanding (NLU) models have been shown to incorporate dataset biases leading to strong performance on in-distribution (ID) test sets but poor performance on out-of-distribution (OOD) ones.
KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation
We present our KroneckerBERT, a compressed version of the BERT_BASE model obtained using this framework.
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation
To address these problems, we propose a RAndom Intermediate Layer Knowledge Distillation (RAIL-KD) approach in which, intermediate layers from the teacher model are selected randomly to be distilled into the intermediate layers of the student model.
Knowledge Distillation with Noisy Labels for Natural Language Understanding
Knowledge Distillation (KD) is extensively used to compress and deploy large pre-trained language models on edge devices for real-world applications.
Pseudo Knowledge Distillation: Towards Learning Optimal Instance-specific Label Smoothing Regularization
Knowledge Distillation (KD) is an algorithm that transfers the knowledge of a trained, typically larger, neural network into another model under training.
Kronecker Decomposition for GPT Compression
GPT is an auto-regressive Transformer-based pre-trained language model which has attracted a lot of attention in the natural language processing (NLP) domain due to its state-of-the-art performance in several downstream tasks.
Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher
A case in point is that the best performing checkpoint of the teacher might not necessarily be the best teacher for training the student in KD.
A Short Study on Compressing Decoder-Based Language Models
Pre-trained Language Models (PLMs) have been successful for a wide range of natural language processing (NLP) tasks.
How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge in a large neural network into a smaller one.
RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation
Knowledge Distillation (KD) is extensively used in Natural Language Processing to compress the pre-training and task-specific fine-tuning phases of large neural language models.
Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation
Intermediate layer matching is shown as an effective approach for improving knowledge distillation (KD).
NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation
We apply NATURE to common slot-filling and intent detection benchmarks and demonstrate that simple perturbations from the standard evaluation set by NATURE can deteriorate model performance significantly.
Distilled embedding: non-linear embedding factorization using knowledge distillation
Word-embeddings are a vital component of Natural Language Processing (NLP) systems and have been extensively researched.
Dynamic Position Encoding for Transformers
To solve this problem, position embeddings are defined exclusively for each time step to enrich word information.
CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillation
Knowledge distillation (KD) is an efficient framework for compressing large-scale pre-trained language models.
Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding
There is a growing body of work in recent years to develop pre-trained language models (PLMs) for the Arabic language.
Do we need Label Regularization to Fine-tune Pre-trained Language Models?
Knowledge Distillation (KD) is a prominent neural model compression technique that heavily relies on teacher network predictions to guide the training of a student model.
KroneckerBERT: Significant Compression of Pre-trained Language Models Through Kronecker Decomposition and Knowledge Distillation
We push the limits of state-of-the-art Transformer-based pre-trained language model compression using Kronecker decomposition.
Towards Fine-tuning Pre-trained Language Models with Integer Forward and Backward Propagation
Previously researchers were focused on lower bit-width integer data types for the forward propagation of language models to save memory and computation.
Improving the Robustness of DistilHuBERT to Unseen Noisy Conditions via Data Augmentation, Curriculum Learning, and Multi-Task Enhancement
Self-supervised speech representation learning aims to extract meaningful factors from the speech signal that can later be used across different downstream tasks, such as speech and/or emotion recognition.
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization
Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowledge from a larger model (a teacher).
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
Moreover, we observe that this simple optimization technique is able to outperform the state-of-the-art KD methods for compact models.
KronA: Parameter Efficient Tuning with Kronecker Adapter
We apply the proposed methods for fine-tuning T5 on the GLUE benchmark to show that incorporating the Kronecker-based modules can outperform state-of-the-art PET methods.
Improved knowledge distillation by utilizing backward pass knowledge in neural networks
Augmenting the training set by adding this auxiliary improves the performance of KD significantly and leads to a closer match between the student and the teacher.
RobustDistiller: Compressing Universal Speech Representations for Enhanced Environment Robustness
The proposed layer-wise distillation recipe is evaluated on top of three well-established universal representations, as well as with three downstream tasks.
Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval
The advent of multilingual language models has generated a resurgence of interest in cross-lingual information retrieval (CLIR), which is the task of searching documents in one language with queries from another.
LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization
Label Smoothing (LS) is another simple, versatile and efficient regularization which can be applied to various supervised classification tasks.
An Exploration into the Performance of Unsupervised Cross-Task Speech Representations for "In the Wild'' Edge Applications
Later, these representations serve as input to downstream models to solve a number of tasks, such as keyword spotting or emotion recognition.
Evaluating Embedding APIs for Information Retrieval
The ever-increasing size of language models curtails their widespread availability to the community, thereby galvanizing many companies into offering access to large language models through APIs.
On the Transferability of Whisper-based Representations for "In-the-Wild" Cross-Task Downstream Speech Applications
Large self-supervised pre-trained speech models have achieved remarkable success across various speech-processing tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models
We propose a systematic framework to measure parametric knowledge utilization in PLMs.
Multimodal Audio-textual Architecture for Robust Spoken Language Understanding
In this work, we investigate impacts of this ASR error propagation on state-of-the-art NLU systems based on pre-trained language models (PLM), such as BERT and RoBERTa.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing
Developing monolingual large Pre-trained Language Models (PLMs) is shown to be very successful in handling different tasks in Natural Language Processing (NLP).
Attribute Controlled Dialogue Prompting
Prompt-tuning has become an increasingly popular parameter-efficient method for adapting large pretrained language models to downstream tasks.
SortedNet, a Place for Every Network and Every Network in its Place: Towards a Generalized Solution for Training Many-in-One Neural Networks
Deep neural networks (DNNs) must cater to a variety of users with different performance needs and budgets, leading to the costly practice of training, storing, and maintaining numerous specific models.
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference
We extend SortedNet to generative NLP tasks, making large language models dynamic without any Pre-Training and by only replacing Standard Fine-Tuning (SFT) with Sorted Fine-Tuning (SoFT).
On the Impact of Quantization and Pruning of Self-Supervised Speech Models for Downstream Speech Recognition Tasks "In-the-Wild''
Recent advances with self-supervised learning have allowed speech recognition systems to achieve state-of-the-art (SOTA) word error rates (WER) while requiring only a fraction of the labeled training data needed by its predecessors.
Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
Recently, large language models (LLMs) have shown remarkable capabilities including understanding context, engaging in logical reasoning, and generating responses.
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
Finetuning large language models requires huge GPU memory, restricting the choice to acquire Larger models.
Towards Practical Tool Usage for Continually Learning LLMs
Large language models (LLMs) show an innate skill for solving language based tasks.
