Search Results for author:
Found 10 papers, 9 papers with code
Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis
State-of-the-art LLMs often rely on scale with high computational costs, which has sparked a research agenda to reduce parameter counts and costs without significantly impacting performance.
Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers
Additionally, we propose a novel training regime, called \textit{self-guided training}, aimed at improving the poor training dynamics that these approximations exhibit when used from initialization.
Selective Focus: Investigating Semantics Sensitivity in Post-training Quantization for Lane Detection
In this paper, we pioneeringly investigate semantic sensitivity to post-processing for lane detection with a novel Lane Distortion Score.
Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes
Previous methods for finding sparsity rates mainly focus on the training-aware scenario, which usually fails to converge stably under the PTS setting with limited data and much less training cost.
QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
Additionally, an adaptive strategy is designed to autonomously determine the optimal number of sub-channels for channel disassembly.
Lossy and Lossless (L$^2$) Post-training Model Size Compression
Then, a dedicated differentiable counter is introduced to guide the optimization of lossy compression to arrive at a more suitable point for later lossless compression.
Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling
The channel-wise shifting aligns the center of each channel for removal of outlier asymmetry.
Lossy and Lossless (L2) Post-training Model Size Compression
Then, a dedicated differentiable counter is introduced to guide the optimization of lossy compression to arrive at a more suitable point for later lossless compression.
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models
With the trends of large NLP models, the increasing memory and computation costs hinder their efficient deployment on resource-limited devices.
QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization
With QDROP, the limit of PTQ is pushed to the 2-bit activation for the first time and the accuracy boost can be up to 51. 49%.
