Search Results for author: Guangtao Zeng

Found 16 papers, 12 papers with code

Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering

1 code implementation29 May 2025 Guangtao Zeng, Maohao Shen, Delin Chen, Zhenting Qi, Subhro Das, Dan Gutfreund, David Cox, Gregory Wornell, Wei Lu, Zhang-Wei Hong, Chuang Gan

Language models (LMs) perform well on standardized coding benchmarks but struggle with real-world software engineering tasks such as resolving GitHub issues in SWE-Bench, especially when model parameters are less than 100B.

Reinforcement Learning (RL)

SailCompass: Towards Reproducible and Robust Evaluation for Southeast Asian Languages

1 code implementation2 Dec 2024 Jia Guo, Longxu Dou, Guangtao Zeng, Stanley Kok, Wei Lu, Qian Liu

In this paper, we introduce SailCompass, a reproducible and robust evaluation benchmark for assessing Large Language Models (LLMs) on Southeast Asian Languages (SEA).

Multiple-choice

Scaling up Masked Diffusion Models on Text

1 code implementation24 Oct 2024 Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, Chongxuan Li

Masked diffusion models (MDMs) have shown promise in language modeling, yet their scalability and effectiveness in core language tasks, such as text generation and language understanding, remain underexplored.

GSM8K Language Modeling +3

RegMix: Data Mixture as Regression for Language Model Pre-training

1 code implementation1 Jul 2024 Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, Min Lin

RegMix trains many small models on diverse data mixtures, uses regression to predict performance of unseen mixtures, and applies the best predicted mixture to train a large-scale model with orders of magnitude more compute.

Common Sense Reasoning Language Modeling +3

Long Context Transfer from Language to Vision

2 code implementations24 Jun 2024 Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, Ziwei Liu

By simply extrapolating the context length of the language backbone, we enable LMMs to comprehend orders of magnitude more visual tokens without any video training.

Language Modeling Language Modelling +4

MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation

1 code implementation19 May 2024 Jianbo Dai, Jianqiao Lu, Yunlong Feng, Dong Huang, Guangtao Zeng, Rongju Ruan, Ming Cheng, Haochen Tan, Zhijiang Guo

Our study analyzed two common benchmarks, HumanEval and MBPP, and found that these might not thoroughly evaluate LLMs' code generation capacities due to limitations in quality, difficulty, and granularity.

Code Generation HumanEval +1

Sailor: Open Language Models for South-East Asia

3 code implementations4 Apr 2024 Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Wei Lu, Min Lin

We present Sailor, a family of open language models ranging from 0. 5B to 7B parameters, tailored for South-East Asian (SEA) languages.

Language Modeling Language Modelling +2

TinyLlama: An Open-Source Small Language Model

2 code implementations4 Jan 2024 Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, Wei Lu

We present TinyLlama, a compact 1. 1B language model pretrained on around 1 trillion tokens for approximately 3 epochs.

Computational Efficiency Language Modeling +3

Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models

2 code implementations23 Oct 2023 Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, Mrinmaya Sachan

We show that MechanisticProbe is able to detect the information of the reasoning tree from the model's attentions for most examples, suggesting that the LM indeed is going through a process of multi-step reasoning within its architecture in many cases.

AI2 Reasoning Challenge

One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning

1 code implementation28 May 2023 Guangtao Zeng, Peiyuan Zhang, Wei Lu

Fine-tuning pre-trained language models for multiple tasks tends to be expensive in terms of storage.

Transfer Learning

Unsupervised Non-transferable Text Classification

1 code implementation23 Oct 2022 Guangtao Zeng, Wei Lu

Training a good deep learning model requires substantial data and computing resources, which makes the resulting neural model a valuable intellectual property.

text-classification Text Classification

Cannot find the paper you are looking for? You can Submit a new open access paper.