Mind's Mirror: Distilling Self-Evaluation Capability and Comprehensive Thinking from Large Language Models

1 code implementation • 15 Nov 2023 • Weize Liu, Guocong Li, Kai Zhang, Bang Du, Qiyuan Chen, Xuming Hu, Hongxia Xu, Jintai Chen, Jian Wu

While techniques such as chain-of-thought (CoT) distillation have displayed promise in distilling LLMs into small language models (SLMs), there is a risk that distilled SLMs may still inherit flawed reasoning and hallucinations from LLMs.

Transfer Learning