Search Results for author:
Found 35 papers, 21 papers with code
GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning
Large Language Models (LLMs) fine-tuning technologies have achieved remarkable results.
Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis
Then, we conduct extensive experiments with the baseline within each class, covering models with various sizes (7B-70B), bitwidths, training levels (LLaMA1/2/3/3. 1), architectures (Mixtral, DeepSeekMoE and Mamba) and modality (LLaVA1. 5 and VILA1. 5) on a wide range of evaluation metrics. Through comparative analysis on the results, we summarize the superior of each PTQ strategy and modelsize-bitwidth trade-off considering the performance.
PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models
To explore the real limit of PTQ, we propose an extremely low-bit PTQ method called PTQ1. 61, which enables weight quantization to 1. 61-bit for the first time.
DLFR-VAE: Dynamic Latent Frame Rate VAE for Video Generation
In this paper, we propose the Dynamic Latent Frame Rate VAE (DLFR-VAE), a training-free paradigm that can make use of adaptive temporal compression in latent space.
E-CAR: Efficient Continuous Autoregressive Image Generation via Multistage Modeling
Our approach not only enhances computational efficiency but also aligns naturally with image generation principles by operating in continuous token space and following a hierarchical generation process from coarse to fine details.
freePruner: A Training-free Approach for Large Multimodal Model Acceleration
Large Multimodal Models (LMMs) have demonstrated impressive capabilities in visual-language tasks but face significant deployment challenges due to their high computational demands.
Prompt Diffusion Robustifies Any-Modality Prompt Learning
This paper introduces prompt diffusion, which uses a diffusion model to gradually refine the prompts to obtain a customized prompt for each sample.
TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
TemporalBench consists of ~10K video question-answer pairs, derived from ~2K high-quality human annotations detailing the temporal dynamics in video clips.
Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning
RIG generates two key instruction data: 1) the Adversarial Instruction-following data, which features mixed negative and positive samples to enhance the model's discriminative understanding.
Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Manner
In this paper, we first identify the primary challenges in interpolating Video-LLMs: (1) the video encoder and modality alignment projector are fixed, preventing the integration of additional frames into Video-LLMs, and (2) the LLM backbone is limited in its content length capabilities, which complicates the processing of an increased number of video tokens.
DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture
Furthermore, considering that the source data is either unaccessible or too enormous to store for current generative models, we introduce a new paradigm for their distillation without source data, termed Data-Free Knowledge Distillation for Diffusion Models (DKDM).
Distilling Long-tailed Datasets
It reduces the distance between the student and the biased expert trajectories and prevents the tail class bias from being distilled to the synthetic dataset.
Dataset Quantization with Active Learning based Adaptive Sampling
In addition, we introduce a novel pipeline for dataset quantization, utilizing feature space from the final stage of dataset quantization to generate more precise dataset bins.
A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training
Training diffusion models is always a computation-intensive task.
PTQ4DiT: Post-training Quantization for Diffusion Transformers
SSC extends this approach by dynamically adjusting the balanced salience to capture the temporal variations in activation.
Efficient Multitask Dense Predictor via Binarization
Moreover, we introduce a knowledge distillation mechanism to correct the direction of information flow in backward propagation.
LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models
In response, we propose PruMerge, a novel adaptive visual token reduction strategy that significantly reduces the number of visual tokens without compromising the performance of LMMs.
FBPT: A Fully Binary Point Transformer
This paper presents a novel Fully Binary Point Cloud Transformer (FBPT) model which has the potential to be widely applied and expanded in the fields of robotics and mobile devices.
Online Multi-spectral Neuron Tracing
In this paper, we propose an online multi-spectral neuron tracing method with uniquely designed modules, where no offline training are required.
LLM Inference Unveiled: Survey and Roofline Model Insights
Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model for systematic analysis of LLM inference techniques.
QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning
We empirically verify that our approach modifies the activation distribution and provides meaningful temporal information, facilitating easier and more accurate quantization.
Enhancing Post-training Quantization Calibration through Contrastive Learning
We aim to calibrate the quantized activations by maximizing the mutual information between the pre- and post-quantized activations.
MIM4DD: Mutual Information Maximization for Dataset Distillation
Thus, we introduce mutual information (MI) as the metric to quantify the shared information between the synthetic and the real datasets, and devise MIM4DD numerically maximizing the MI via a newly designed optimizable objective within a contrastive learning framework to update the synthetic dataset.
ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
Based on the success of the low-rank decomposition of projection matrices in the self-attention module, we further introduce ASVD to compress the KV cache.
PB-LLM: Partially Binarized Large Language Models
This paper explores network binarization, a radical form of quantization, compressing model weights to a single bit, specifically for Large Language Models (LLMs) compression.
Causal-DFQ: Causality Guided Data-free Network Quantization
Inspired by the causal understanding, we propose the Causality-guided Data-free Network Quantization method, Causal-DFQ, to eliminate the reliance on data via approaching an equilibrium of causality-driven intervened distributions.
RPTQ: Reorder-based Post-training Quantization for Large Language Models
In this paper, we identify that the challenge in quantizing activations in LLMs arises from varying ranges across channels, rather than solely the presence of outliers.
BPT: Binary Point Cloud Transformer for Place Recognition
To solve this issue, we propose a binary point cloud transformer for place recognition.
Post-training Quantization on Diffusion Models
These approaches define a forward diffusion process for transforming data into noise and a backward denoising process for sampling data from noise.
Lipschitz Continuity Retained Binary Neural Network
Relying on the premise that the performance of a binary neural network can be largely restored with eliminated quantization error between full-precision weight vectors and their corresponding binary vectors, existing works of network binarization frequently adopt the idea of model robustness to reach the aforementioned objective.
Network Binarization via Contrastive Learning
Neural network binarization accelerates deep models by quantizing their weights and activations into 1-bit.
Supplementing Missing Visions via Dialog for Scene Graph Generations
Most current AI systems rely on the premise that the input visual data are sufficient to achieve competitive performance in various computer vision tasks.
Win the Lottery Ticket via Fourier Analysis: Frequencies Guided Network Pruning
Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate the superiority of our novel Fourier analysis based MBP compared to other traditional MBP algorithms.
Contrastive Mutual Information Maximization for Binary Neural Networks
Neural network binarization accelerates deep models by quantizing their weights and activations into 1-bit.
Lipschitz Continuity Guided Knowledge Distillation
Knowledge distillation has become one of the most important model compression techniques by distilling knowledge from larger teacher networks to smaller student ones.
