Search Results for author:
Found 10 papers, 8 papers with code
Cumulative Spatial Knowledge Distillation for Vision Transformers
(2) Distilling knowledge from CNN limits the network convergence in the later training period since ViT's capability of integrating global information is suppressed by CNN's local-inductive-bias supervision.
DOT: A Distillation-Oriented Trainer
In this paper, we observe a trade-off between task and distillation losses, i. e., introducing distillation loss limits the convergence of task loss.
Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?
Diffusion models have recently achieved astonishing performance in generating high-fidelity photo-realistic images.
A Survey of Historical Learning: Learning Models with Learning History
The various types of elements, deposited in the training history, are a large amount of wealth for improving learning deep models.
Boosting Semi-Supervised Learning by Exploiting All Unlabeled Data
ANL introduces the additional negative pseudo-label for all unlabeled data to leverage low-confidence examples.
Curriculum Temperature for Knowledge Distillation
In this paper, we propose a simple curriculum-based technique, termed Curriculum Temperature for Knowledge Distillation (CTKD), which controls the task difficulty level during the student's learning career through a dynamic and learnable temperature.
Decoupled Knowledge Distillation
To provide a novel viewpoint to study logit distillation, we reformulate the classical KD loss into two parts, i. e., target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD).
RecursiveMix: Mixed Learning with History
In semantic segmentation, RM also surpasses the baseline and CutMix by 1. 9 and 1. 1 mIoU points under UperNet on ADE20K, respectively.
Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective
This work simultaneously considers the discriminability and transferability properties of deep representations in the typical supervised learning task, i. e., image classification.
Dynamic MLP for Fine-Grained Image Classification by Leveraging Geographical and Temporal Information
Therefore, it is helpful to leverage additional information, e. g., the locations and dates for data shooting, which can be easily accessible but rarely exploited.
