Dynamic Corrective Self-Distillation for Better Fine-Tuning of Pretrained Models

no code implementations • 12 Dec 2023 • Ibtihel Amara, Vinija Jain, Aman Chadha

We tackle the challenging issue of aggressive fine-tuning encountered during the process of transfer learning of pre-trained language models (PLMs) with limited labeled downstream data.

Transfer Learning

Improved Contextual Recognition In Automatic Speech Recognition Systems By Semantic Lattice Rescoring

no code implementations • 14 Oct 2023 • Ankitha Sudarshan, Vinay Samuel, Parth Patwa, Ibtihel Amara, Aman Chadha

Automatic Speech Recognition (ASR) has witnessed a profound research interest.

Automatic Speech Recognition Automatic Speech Recognition (ASR) +1

BD-KD: Balancing the Divergences for Online Knowledge Distillation

no code implementations • 25 Dec 2022 • Ibtihel Amara, Nazanin Sepahvand, Brett H. Meyer, Warren J. Gross, James J. Clark

We show that adaptively balancing between the reverse and forward divergences shifts the focus of the training strategy to the compact student network without limiting the teacher network's learning process.

Knowledge Distillation Model Compression +1

CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation

no code implementations • 15 Sep 2022 • Ibtihel Amara, Maryam Ziaeefard, Brett H. Meyer, Warren Gross, James J. Clark

Knowledge distillation (KD) is an effective tool for compressing deep classification models for edge devices.

Knowledge Distillation