Building Variable-sized Models via Learngene Pool

no code implementations • 10 Dec 2023 • Boyu Shi, Shiyu Xia, Xu Yang, Haokun Chen, Zhiqiang Kou, Xin Geng

To overcome these challenges, motivated by the recently proposed Learngene framework, we propose a novel method called Learngene Pool.

Transformer as Linear Expansion of Learngene

1 code implementation • 9 Dec 2023 • Shiyu Xia, Miaosen Zhang, Xu Yang, Ruiming Chen, Haokun Chen, Xin Geng

Under the situation where we need to produce models of varying depths adapting for different resource constraints, TLEG achieves comparable results while reducing around 19x parameters stored to initialize these models and around 5x pre-training costs, in contrast to the pre-training and fine-tuning approach.

Towards Effective Visual Representations for Partial-Label Learning

1 code implementation • CVPR 2023 • Shiyu Xia, Jiaqi Lv, Ning Xu, Gang Niu, Xin Geng

Under partial-label learning (PLL) where, for each training instance, only a set of ambiguous candidate labels containing the unknown true label is accessible, contrastive learning has recently boosted the performance of PLL on vision tasks, attributed to representations learned by contrasting the same/different classes of entities.

Contrastive Learning Image Classification +3

Learngene: From Open-World to Your Learning Task

1 code implementation • 12 Jun 2021 • Qiufeng Wang, Xin Geng, Shuxia Lin, Shiyu Xia, Lei Qi, Ning Xu

Moreover, the learngene, i. e., the gene for learning initialization rules of the target model, is proposed to inherit the meta-knowledge from the collective model and reconstruct a lightweight individual model on the target task.