ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation

1 code implementation • 20 Jun 2024 • Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, Yi Wu

To overcome this limitation, we propose a novel technique named parameter ReaLlocation, which dynamically adapts the parallelization strategies for different workloads during training by redistributing LLM parameters across the training cluster.

Language Modelling Large Language Model

SaberLDA: Sparsity-Aware Learning of Topic Models on GPUs

no code implementations • 8 Oct 2016 • Kaiwei Li, Jianfei Chen, WenGuang Chen, Jun Zhu

Latent Dirichlet Allocation (LDA) is a popular tool for analyzing discrete count data such as text and images.

Topic Models

WarpLDA: a Cache Efficient O(1) Algorithm for Latent Dirichlet Allocation

no code implementations • 29 Oct 2015 • Jianfei Chen, Kaiwei Li, Jun Zhu, WenGuang Chen

We then develop WarpLDA, an LDA sampler which achieves both the best O(1) time complexity per token and the best O(K) scope of random access.