TSPipe: Learn from Teacher Faster with Pipelines

1 code implementation • ICML 2022 • Hwijoon Lim, Yechan Kim, Sukmin Yun, Jinwoo Shin, Dongsu Han

The teacher-student (TS) framework, training a (student) network by utilizing an auxiliary superior (teacher) network, has been adopted as a popular training paradigm in many machine learning schemes, since the seminal work---Knowledge distillation (KD) for model compression and transfer learning.

Knowledge Distillation Self-Supervised Learning +1