torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation

25 Nov 2020 Yoshitomo Matsubara

While knowledge distillation (transfer) has been attracting attentions from the research community, the recent development in the fields has heightened the need for reproducible studies and highly generalized frameworks to lower barriers to such high-quality, reproducible deep learning research. Several researchers voluntarily published frameworks used in their knowledge distillation studies to help other interested researchers reproduce their original work... (read more)

PDF Abstract
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Object Detection COCO test-dev Faster R-CNN (Bottleneck-injected ResNet-50 and FPN) box AP 35.9 # 153
Object Detection COCO test-dev Mask R-CNN (Bottleneck-injected ResNet-50, FPN) box AP 36.9 # 148
Instance Segmentation COCO test-dev Mask R-CNN (Bottleneck-injected ResNet-50, FPN) mask AP 33.6 # 40
Image Classification ImageNet ResNet-18 (tf-KD w/ ResNet-18 teacher) Top 1 Accuracy 70.52% # 352
Image Classification ImageNet ResNet-18 (CRD w/ ResNet-34 teacher) Top 1 Accuracy 70.93% # 349
Image Classification ImageNet ResNet-18 (KD w/ ResNet-34 teacher) Top 1 Accuracy 71.37% # 346
Image Classification ImageNet ResNet-18 (SSKD w/ ResNet-34 teacher) Top 1 Accuracy 70.09% # 353
Image Classification ImageNet ResNet-18 (L2 w/ ResNet-34 teacher) Top 1 Accuracy 71.08% # 348
Image Classification ImageNet ResNet-18 (PAD-L2 w/ ResNet-34 teacher) Top 1 Accuracy 71.71% # 343
Image Classification ImageNet ResNet-18 (FT w/ ResNet-34 teacher) Top 1 Accuracy 71.56% # 345

Methods used in the Paper


METHOD TYPE
🤖 No Methods Found Help the community by adding them if they're not listed; e.g. Deep Residual Learning for Image Recognition uses ResNet