Search Results for author:
Found 21 papers, 11 papers with code
Separated Contrastive Learning for Matching in Cross-domain Recommendation with Curriculum Scheduling
Cross-domain recommendation (CDR) is a task that aims to improve the recommendation performance in a target domain by leveraging the information from source domains.
Reward Models Identify Consistency, Not Causality
Reward models (RMs) play a crucial role in aligning large language models (LLMs) with human preferences and enhancing reasoning quality.
Reward-Guided Speculative Decoding for Efficient LLM Reasoning
We introduce Reward-Guided Speculative Decoding (RSD), a novel framework aimed at improving the efficiency of inference in large language models (LLMs).
GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection
To further enhance performance, we propose sparsely coded residuals to reduce the errors caused by low-rank approximation on the first- and second-order moments of the optimizers and weight updates.
MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs
Although some recent benchmarks have been developed to evaluate the long-context capabilities of LLMs, there is a lack of benchmarks evaluating the mathematical reasoning abilities of LLMs over long contexts, which is crucial for LLMs' application in real-world scenarios.
ThinK: Thinner Key Cache by Query-Driven Pruning
Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.
One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments
Large Language Models (LLMs) have advanced rapidly but face significant memory demands.
SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
Large Language Models (LLMs) have become pivotal in advancing the field of artificial intelligence, yet their immense sizes pose significant challenges for both fine-tuning and deployment.
TerDiT: Ternary Diffusion Models with Transformers
Recent developments in large-scale pre-trained text-to-image diffusion models have significantly improved the generation of high-fidelity images, particularly with the emergence of diffusion models based on transformer architecture (DiTs).
Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
Specifically, we propose, for the first time to our best knowledge, post-training approaches for task-agnostic and task-specific expert pruning and skipping of MoE LLMs, tailored to improve deployment efficiency while maintaining model performance across a wide range of tasks.
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
Recently years have witnessed a rapid development of large language models (LLMs).
Batch Normalization with Enhanced Linear Transformation
Batch normalization (BN) is a fundamental unit in modern deep networks, in which a linear transformation module was designed for improving BN's flexibility of fitting complex data distributions.
Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
Neural architecture search (NAS) has attracted increasing attentions in both academia and industry.
TRP: Trained Rank Pruning for Efficient Deep Neural Networks
The TRP trained network inherently has a low-rank structure, and is approximated with negligible performance loss, thus eliminating the fine-tuning process after low rank decomposition.
Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
We alleviate this issue by training a graph convolutional network to fit the performance of sampled sub-networks so that the impact of random errors becomes minimal.
Latency-Aware Differentiable Neural Architecture Search
However, these methods suffer the difficulty in optimizing network, so that the searched network is often unfriendly to hardware.
Trained Rank Pruning for Efficient Deep Neural Networks
To accelerate DNNs inference, low-rank approximation has been widely adopted because of its solid theoretical rationale and efficient implementations.
PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-network and searching for an optimal architecture.
Ranked #20 on
Neural Architecture Search
on CIFAR-10
Trained Rank Pruning for Efficient Deep Neural Networks
We propose Trained Rank Pruning (TRP), which iterates low rank approximation and training.
DNQ: Dynamic Network Quantization
Network quantization is an effective method for the deployment of neural networks on memory and energy constrained mobile devices.
Deep Neural Network Compression with Single and Multiple Level Quantization
In this paper, we propose two novel network quantization approaches, single-level network quantization (SLQ) for high-bit quantization and multi-level network quantization (MLQ) for extremely low-bit quantization (ternary). We are the first to consider the network quantization from both width and depth level.
