A Comprehensive Survey on Long Context Language Modeling

1 code implementation • 20 Mar 2025 • Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Zhang, Zhuo Chen, Hangyu Guo, Shilong Li, Ziqiang Liu, Yong Shan, YiFan Song, Jiayi Tian, Wenhao Wu, Zhejian Zhou, Ruijie Zhu, Junlan Feng, Yang Gao, Shizhu He, Zhoujun Li, Tianyu Liu, Fanyu Meng, Wenbo Su, Yingshui Tan, Zili Wang, Jian Yang, Wei Ye, Bo Zheng, Wangchunshu Zhou, Wenhao Huang, Sujian Li, Zhaoxiang Zhang

With the growing number of long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (LCLMs) that can process and analyze extensive inputs in an effective and efficient way.

Language Modeling Language Modelling +1

InternLM2.5-StepProver: Advancing Automated Theorem Proving via Expert Iteration on Large-Scale LEAN Problems

1 code implementation • 21 Oct 2024 • Zijian Wu, Suozhi Huang, Zhejian Zhou, Huaiyuan Ying, Jiayu Wang, Dahua Lin, Kai Chen

We propose to use large scale LEAN problem datasets Lean-workbook for expert iteration with more than 20, 000 CPU days.

Automated Theorem Proving Math

Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia

no code implementations • 25 Sep 2024 • Zhejian Zhou, Jiayu Wang, Dahua Lin, Kai Chen

Numbers can be tokenized into tokens in various ways by different LLMs and affect the numeric operations performance.

Attribute

StackSight: Unveiling WebAssembly through Large Language Models and Neurosymbolic Chain-of-Thought Decompilation

no code implementations • 7 Jun 2024 • Weike Fang, Zhejian Zhou, Junzhou He, Weihang Wang

WebAssembly enables near-native execution in web applications and is increasingly adopted for tasks that demand high performance and robust security.

InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

1 code implementation • 9 Feb 2024 • Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, Dahua Lin

We further explore how to use LEAN to solve math problems and study its performance under the setting of multi-task learning which shows the possibility of using LEAN as a unified platform for solving and proving in math.

Data Augmentation GSM8K +3

ConFiguRe: Exploring Discourse-level Chinese Figures of Speech

1 code implementation • COLING 2022 • Dawei Zhu, Qiusi Zhan, Zhejian Zhou, YiFan Song, Jiebin Zhang, Sujian Li

Different from previous token-level or sentence-level counterparts, ConFiguRe aims at extracting a figurative unit from discourse-level context, and classifying the figurative unit into the right figure type.

Natural Language Understanding Sentence