Search Results for author:
Found 10 papers, 7 papers with code
Scaling Laws for Linear Complexity Language Models
In this study, we present the scaling laws for linear complexity language models to establish a foundation for their scalability.
Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective
Finally, the memory state is projected back to a low-dimensional space in the Shrink stage.
Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention
This eliminates the need for cumsum in the linear attention calculation.
HGRN2: Gated Linear RNNs with State Expansion
Hierarchically gated linear RNN (HGRN, \citealt{HGRN}) has demonstrated competitive training speed and performance in language modeling while offering efficient inference.
Linear Attention Sequence Parallelism
In this paper, we introduce Linear Attention Sequence Parallel (LASP), an efficient SP method tailored to linear attention-based language models.
MS-Net: A Multi-Path Sparse Model for Motion Prediction in Multi-Scenes
In the training stage, the motion prediction task under differentiated scenes is abstracted as a multi-task learning problem, an evolutionary algorithm is designed to encourage the network search of the optimal parameters for each scene while sharing common knowledge between different scenes.
CO2: Efficient Distributed Training with Full Communication-Computation Overlap
CO2 is able to attain a high scalability even on extensive multi-node clusters constrained by very limited communication bandwidth.
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
With its ability to process tokens in linear computational complexities, linear attention, in theory, can handle sequences of unlimited length without sacrificing speed, i. e., maintaining a constant training speed for various sequence lengths with a fixed memory consumption.
TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer
TransNormerLLM evolves from the previous linear attention architecture TransNormer by making advanced modifications that include positional embedding, linear attention acceleration, gating mechanisms, tensor normalization, and inference acceleration and stabilization.
PowerSGD: Powered Stochastic Gradient Descent Methods for Accelerated Non-Convex Optimization
In this paper, we propose a novel technique for improving the stochastic gradient descent (SGD) method to train deep networks, which we term \emph{PowerSGD}.
