Search Results for author:
Found 10 papers, 7 papers with code
Cumulative Spatial Knowledge Distillation for Vision Transformers
(2) Distilling knowledge from CNN limits the network convergence in the later training period since ViT's capability of integrating global information is suppressed by CNN's local-inductive-bias supervision.
DOT: A Distillation-Oriented Trainer
In this paper, we observe a trade-off between task and distillation losses, i. e., introducing distillation loss limits the convergence of task loss.
Boosting Semi-Supervised Learning by Exploiting All Unlabeled Data
ANL introduces the additional negative pseudo-label for all unlabeled data to leverage low-confidence examples.
Curriculum Temperature for Knowledge Distillation
In this paper, we propose a simple curriculum-based technique, termed Curriculum Temperature for Knowledge Distillation (CTKD), which controls the task difficulty level during the student's learning career through a dynamic and learnable temperature.
Efficient One Pass Self-distillation with Zipf's Label Smoothing
This paper proposes an efficient self-distillation method named Zipf's Label Smoothing (Zipf's LS), which uses the on-the-fly prediction of a network to generate soft supervision that conforms to Zipf distribution without using any contrastive samples or auxiliary parameters.
Decoupled Knowledge Distillation
To provide a novel viewpoint to study logit distillation, we reformulate the classical KD loss into two parts, i. e., target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD).
RecursiveMix: Mixed Learning with History
In semantic segmentation, RM also surpasses the baseline and CutMix by 1. 9 and 1. 1 mIoU points under UperNet on ADE20K, respectively.
Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective
This work simultaneously considers the discriminability and transferability properties of deep representations in the typical supervised learning task, i. e., image classification.
Dynamic MLP for Fine-Grained Image Classification by Leveraging Geographical and Temporal Information
Therefore, it is helpful to leverage additional information, e. g., the locations and dates for data shooting, which can be easily accessible but rarely exploited.
Hierarchical Context Embedding for Region-based Object Detection
To address this issue, we present a simple but effective Hierarchical Context Embedding (HCE) framework, which can be applied as a plug-and-play component, to facilitate the classification ability of a series of region-based detectors by mining contextual cues.
