Search Results for author:
Found 3 papers, 3 papers with code
Token-Scaled Logit Distillation for Ternary Weight Generative Language Models
Generative Language Models (GLMs) have shown impressive performance in tasks such as text generation, understanding, and reasoning.
Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective
Knowledge distillation (KD) is a highly promising method for mitigating the computational problems of pre-trained language models (PLMs).
Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoders
In particular, KD has been employed in quantization-aware training (QAT) of Transformer encoders like BERT to improve the accuracy of the student model with the reduced-precision weight parameters.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
