MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts

no code implementations • 31 Jan 2024 • Zhitian Xie, Yinger Zhang, Chenyi Zhuang, Qitao Shi, Zhining Liu, Jinjie Gu, Guannan Zhang

However, the gate's routing mechanism also gives rise to narrow vision: the individual MoE's expert fails to use more samples in learning the allocated sub-task, which in turn limits the MoE to further improve its generalization ability.

Mixture-of-Experts

ALT: An Automatic System for Long Tail Scenario Modeling

no code implementations • 19 May 2023 • Ya-Lin Zhang, Jun Zhou, Yankun Ren, Yue Zhang, Xinxing Yang, Meng Li, Qitao Shi, Longfei Li

In this paper, we consider the problem of long tail scenario modeling with budget limitation, i. e., insufficient human resources for model training stage and limited time and computing resources for model inference stage.

Meta-Learning Neural Architecture Search +1

SAFE: Scalable Automatic Feature Engineering Framework for Industrial Tasks

no code implementations • 5 Mar 2020 • Qitao Shi, Ya-Lin Zhang, Longfei Li, Xinxing Yang, Meng Li, Jun Zhou

Machine learning techniques have been widely applied in Internet companies for various tasks, acting as an essential driving force, and feature engineering has been generally recognized as a crucial tache when constructing machine learning systems.

BIG-bench Machine Learning Feature Engineering