Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models

no code implementations • 28 May 2024 • Longze Chen, Ziqiang Liu, Wanwei He, Yunshui Li, Run Luo, Min Yang

In this study, we propose a data mining framework \textbf{ProLong} that can assign each training sample with a long dependency score, which can be used to rank and filter samples that are more advantageous for enhancing long-context modeling abilities in LLM training.

DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception

no code implementations • 24 May 2024 • Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, Min Yang, Binyuan Hui

Delving into the modeling capabilities of diffusion models for images naturally prompts the question: Can diffusion models serve as the eyes of large language models for image perception?

COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning

no code implementations • 26 Mar 2024 • Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang, Ruibin Yuan, Haihong Wu, Hongquan Lin, Wenhao Huang, Jiajun Zhang, Wenhu Chen, Chenghua Lin, Jie Fu, Min Yang, Shiwen Ni, Ge Zhang

To bridge this gap, we introduce COIG-CQIA, a high-quality Chinese instruction tuning dataset.

Marathon: A Race Through the Realm of Long Context with Large Language Models

no code implementations • 15 Dec 2023 • Lei Zhang, Yunshui Li, Ziqiang Liu, Jiaxi Yang, Junhao Liu, Min Yang

Although there are currently many benchmarks available for evaluating the long context understanding and reasoning capability of large language models, with the expansion of the context window in these models, the existing long context benchmarks are no longer sufficient for evaluating the long context understanding and reasoning capability of large language models.

Long-Context Understanding Multiple-choice