Search Results for author:
Found 4 papers, 0 papers with code
Maximizing Discrimination Capability of Knowledge Distillation with Energy Function
To distill optimal knowledge by adjusting non-target class predictions, we apply a higher temperature to low energy samples to create smoother distributions and a lower temperature to high energy samples to achieve sharper distributions.
Cosine Similarity Knowledge Distillation for Individual Class Information Transfer
Furthermore, we propose a method called cosine similarity weighted temperature (CSWT) to improve the performance.
Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning
The improvement in the performance of efficient and lightweight models (i. e., the student model) is achieved through knowledge distillation (KD), which involves transferring knowledge from more complex models (i. e., the teacher model).
A$^3$: Accelerating Attention Mechanisms in Neural Networks with Approximation
The attention mechanism is widely adopted by many state-of-the-art neural networks for computer vision, natural language processing, and machine translation, and accounts for a large portion of total execution time.
