Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models

no code implementations • 23 Jan 2025 • Zhenghao Lin, Zihao Tang, Xiao Liu, Yeyun Gong, Yi Cheng, Qi Chen, Hang Li, Ying Xin, Ziyue Yang, Kailai Yang, Yu Yan, Xiao Liang, Shuai Lu, Yiming Huang, Zheheng Luo, Lei Qu, Xuan Feng, Yaoxiang Wang, Yuqing Xia, Feiyang Chen, Yuting Jiang, Yasen Hu, Hao Ni, Binyang Li, Guoshuai Zhao, Jui-Hao Chiang, Zhongxin Guo, Chen Lin, Kun Kuang, Wenjie Li, Yelong Shen, Jian Jiao, Peng Cheng, Mao Yang

We introduce Sigma, an efficient large language model specialized for the system domain, empowered by a novel architecture including DiffQKV attention, and pre-trained on our meticulously collected system domain data.

Large Language Model

Collaborative Optimization in Financial Data Mining Through Deep Learning and ResNeXt

no code implementations • 23 Dec 2024 • Pengbin Feng, Yankaiqi Li, Yijiashun Qi, Xiaojun Guo, Zhenghao Lin

This study proposes a multi-task learning framework based on ResNeXt, aiming to solve the problem of feature extraction and task collaborative optimization in financial data mining.

Multi-Task Learning

Integrative Analysis of Financial Market Sentiment Using CNN and GRU for Risk Prediction and Alert Systems

no code implementations • 13 Dec 2024 • You Wu, Mengfang Sun, Hongye Zheng, Jinxin Hu, Yingbin Liang, Zhenghao Lin

This document presents an in-depth examination of stock market sentiment through the integration of Convolutional Neural Networks (CNN) and Gated Recurrent Units (GRU), enabling precise risk alerts.

Time Series

Revolutionizing Database Q&A with Large Language Models: Comprehensive Benchmark and Evaluation

1 code implementation • 5 Sep 2024 • Yihang Zheng, Bo Li, Zhenghao Lin, Yi Luo, Xuanhe Zhou, Chen Lin, Jinsong Su, Guoliang Li, Shifu Li

However, there is still a lack of a comprehensive benchmark to evaluate the capabilities of different LLMs and their modular components in database QA.

Fairness RAG

Rho-1: Not All Tokens Are What You Need

3 code implementations • 11 Apr 2024 • Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, Weizhu Chen

Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution.

Continual Pretraining Language Modeling +2

Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models

1 code implementation • 18 Mar 2024 • Yi Luo, Zhenghao Lin, Yuhao Zhang, Jiashuo Sun, Chen Lin, Chengjin Xu, Xiangdong Su, Yelong Shen, Jian Guo, Yeyun Gong

Subsequently, the retrieval model correlates new inputs with relevant guidelines, which guide LLMs in response generation to ensure safe and high-quality outputs, thereby aligning with human values.

Response Generation Retrieval

Competition-Level Problems are Effective LLM Evaluators

no code implementations • 4 Dec 2023 • Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, Weizhu Chen

Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about these abilities and the potential data contamination problem recently.

On the Distributed Evaluation of Generative Models

no code implementations • 18 Oct 2023 • Zixiao Wang, Farzan Farnia, Zhenghao Lin, Yunheng Shen, Bei Yu

On the other hand, several applications of generative models concern distributed settings, e. g. the federated learning setting, where the reference data for conducting evaluation are provided by several clients in a network.

Avg Federated Learning

AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators

2 code implementations • 29 Mar 2023 • Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen

Many natural language processing (NLP) tasks rely on labeled data to train machine learning models with high performance.

Information Retrieval Retrieval

Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise

1 code implementation • 22 Dec 2022 • Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, Weizhu Chen

In this paper, we introduce a novel dIffusion language modEl pre-training framework for text generation, which we call GENIE.

Decoder Denoising +3

Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis

1 code implementation • 18 Oct 2022 • Shuai Fan, Chen Lin, Haonan Li, Zhenghao Lin, Jinsong Su, Hang Zhang, Yeyun Gong, Jian Guo, Nan Duan

Most existing pre-trained language representation models (PLMs) are sub-optimal in sentiment analysis tasks, as they capture the sentiment information from word-level while under-considering sentence-level information.

Contrastive Learning Language Modeling +4

PROD: Progressive Distillation for Dense Retrieval

1 code implementation • 27 Sep 2022 • Zhenghao Lin, Yeyun Gong, Xiao Liu, Hang Zhang, Chen Lin, Anlei Dong, Jian Jiao, Jingwen Lu, Daxin Jiang, Rangan Majumder, Nan Duan

It is common that a better teacher model results in a bad student via distillation due to the nonnegligible gap between teacher and student.

Knowledge Distillation Natural Questions +1