Search Results for author:
Found 11 papers, 5 papers with code
LUT Tensor Core: Lookup Table Enables Efficient Low-Bit LLM Inference Acceleration
Then, LUT Tensor Core proposes the hardware design featuring an elongated tiling shape design to enhance table reuse and a bit-serial design to support various precision combinations in mpGEMM.
T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge
Weight quantization is crucial for reducing the memory footprint of LLMs on devices.
BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation
The upscaling of Large Language Models (LLMs) has yielded impressive advances in natural language processing, yet it also poses significant deployment challenges.
AFPQ: Asymmetric Floating Point Quantization for LLMs
Large language models (LLMs) show great performance in various tasks, but face deployment challenges from limited memory capacity and bandwidth.
Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
To tackle the high compute requirements of LLMs, the Mixture-of-Experts (MoE) architecture was introduced which is able to scale its model size without proportionally scaling up its computational requirements.
Accurate and Structured Pruning for Efficient Automatic Speech Recognition
In this paper, we propose a novel compression strategy that leverages structured pruning and knowledge distillation to reduce the model size and inference cost of the Conformer model while preserving high recognition performance.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Adam Accumulation to Reduce Memory Footprints of both Activations and Gradients for Large-scale DNN Training
We find that previous gradient accumulation reduces activation memory but fails to be compatible with gradient memory reduction due to a contradiction between preserving gradients and releasing gradients.
Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models
In this study, we conduct a comparative analysis of INT and FP quantization with the same bit-width, revealing that the optimal quantization format varies across different layers due to the complexity and diversity of tensor distribution.
EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate
Then it diversifies the experts and continues to train the MoE with a novel Dense-to-Sparse gate (DTS-Gate).
SeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity Through Low-Bit Quantization
In this paper we present a novel and general method to accelerate convolutional neural network (CNN) inference by taking advantage of feature map sparsity.
Balanced Sparsity for Efficient DNN Inference on GPU
However, it requires the customization of hardwares to speed up practical inference.
