Search Results for author:
Found 6 papers, 5 papers with code
Genetic Quantization-Aware Approximation for Non-Linear Operations in Transformers
Non-linear functions are prevalent in Transformers and their lightweight variants, incurring substantial and frequently underestimated hardware costs.
DoRA: Weight-Decomposed Low-Rank Adaptation
By employing DoRA, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead.
CMOSE: Comprehensive Multi-Modality Online Student Engagement Dataset with High-Quality Labels
We also developed a training mechanism, MocoRank, to handle the intra-class variation, the ordinal relationship between different classes, and the data imbalance problem.
LLM-FP4: 4-Bit Floating-Point Quantized Transformers
Our method, for the first time, can quantize both weights and activations in the LLaMA-13B to only 4-bit and achieves an average score of 63. 1 on the common sense zero-shot reasoning tasks, which is only 5. 8 lower than the full-precision model, significantly outperforming the previous state-of-the-art by 12. 7 points.
Efficient Quantization-aware Training with Adaptive Coreset Selection
Compared with previous coreset selection methods, our method significantly improves QAT performance with different dataset fractions.
Oscillation-free Quantization for Low-bit Vision Transformers
In addition, we also found that the interdependence between quantized weights in $\textit{query}$ and $\textit{key}$ of a self-attention layer makes ViT vulnerable to oscillation.
