Search Results for author:
Found 12 papers, 9 papers with code
Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
Specifically, we propose, for the first time to our best knowledge, post-training approaches for task-agnostic and task-specific expert pruning and skipping of MoE LLMs, tailored to improve deployment efficiency while maintaining model performance across a wide range of tasks.
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
Recently years have witnessed a rapid development of large language models (LLMs).
Batch Normalization with Enhanced Linear Transformation
Batch normalization (BN) is a fundamental unit in modern deep networks, in which a linear transformation module was designed for improving BN's flexibility of fitting complex data distributions.
Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
Neural architecture search (NAS) has attracted increasing attentions in both academia and industry.
TRP: Trained Rank Pruning for Efficient Deep Neural Networks
The TRP trained network inherently has a low-rank structure, and is approximated with negligible performance loss, thus eliminating the fine-tuning process after low rank decomposition.
Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
We alleviate this issue by training a graph convolutional network to fit the performance of sampled sub-networks so that the impact of random errors becomes minimal.
Latency-Aware Differentiable Neural Architecture Search
However, these methods suffer the difficulty in optimizing network, so that the searched network is often unfriendly to hardware.
Trained Rank Pruning for Efficient Deep Neural Networks
To accelerate DNNs inference, low-rank approximation has been widely adopted because of its solid theoretical rationale and efficient implementations.
PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-network and searching for an optimal architecture.
Ranked #20 on
Neural Architecture Search
on CIFAR-10
Trained Rank Pruning for Efficient Deep Neural Networks
We propose Trained Rank Pruning (TRP), which iterates low rank approximation and training.
DNQ: Dynamic Network Quantization
Network quantization is an effective method for the deployment of neural networks on memory and energy constrained mobile devices.
Deep Neural Network Compression with Single and Multiple Level Quantization
In this paper, we propose two novel network quantization approaches, single-level network quantization (SLQ) for high-bit quantization and multi-level network quantization (MLQ) for extremely low-bit quantization (ternary). We are the first to consider the network quantization from both width and depth level.
