Search Results for author:
Found 77 papers, 43 papers with code
Neighboring Autoregressive Modeling for Efficient Visual Generation
In this paper, we propose Neighboring Autoregressive Modeling (NAR), a novel paradigm that formulates autoregressive visual generation as a progressive outpainting procedure, following a near-to-far ``next-neighbor prediction" mechanism.
Motion Anything: Any to Motion Generation
Conditional motion generation has been extensively studied in computer vision, yet two critical challenges remain.
Ranked #1 on
Motion Synthesis
on TMD
Channel Merging: Preserving Specialization for Merged Experts
To address the inefficiency, model merging strategies have emerged, merging all LLMs into one model to reduce the memory footprint during inference.
ZipAR: Accelerating Auto-regressive Image Generation through Spatial Locality
By decoding multiple tokens simultaneously in a single forward pass, the number of forward passes required to generate an image is significantly reduced, resulting in a substantial improvement in generation efficiency.
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
By merging the parameters of language models from these MLLMs, VisionFuse allows a single language model to align with various vision encoders, significantly reducing deployment overhead.
Evaluating and Advancing Multimodal Large Language Models in Ability Lens
To address this, we introduce \textbf{AbilityLens}, a unified benchmark designed to evaluate MLLMs across six key perception abilities, focusing on both accuracy and stability, with each ability encompassing diverse question types, domains, and metrics.
KMM: Key Frame Mask Mamba for Extended Motion Generation
The recent Mamba architecture shows promising results in efficiently modeling long and complex sequences, yet two significant challenges remain: Firstly, directly applying Mamba to extended motion generation is ineffective, as the limited capacity of the implicit memory leads to memory decay.
MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views
To evaluate MVSplat360's performance, we introduce a new benchmark using the challenging DL3DV-10K dataset, where MVSplat360 achieves superior visual quality compared to state-of-the-art methods on wide-sweeping or even 360{\deg} NVS tasks.
ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
The efficiency of large vision-language models (LVLMs) is constrained by the computational bottleneck of the attention mechanism during the prefill phase and the memory bottleneck of fetching the key-value (KV) cache in the decoding phase, particularly in scenarios involving high-resolution images or videos.
McCaD: Multi-Contrast MRI Conditioned, Adaptive Adversarial Diffusion Model for High-Fidelity MRI Synthesis
Moreover, existing methods for multi-contrast MRI synthesis often fail to accurately map feature-level information across multiple imaging contrasts.
GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI
Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields.
InfiniMotion: Mamba Boosts Memory in Transformer for Arbitrary Long Motion Generation
Text-to-motion generation holds potential for film, gaming, and robotics, yet current methods often prioritize short motion generation, making it challenging to produce long motion sequences effectively: (1) Current methods struggle to handle long motion sequences as a single input due to prohibitively high computational cost; (2) Breaking down the generation of long motion sequences into shorter segments can result in inconsistent transitions and requires interpolation or inpainting, which lacks entire sequence modeling.
SAM-Med3D-MoE: Towards a Non-Forgetting Segment Anything Model via Mixture of Experts for 3D Medical Image Segmentation
Supervised Finetuning (SFT) serves as an effective way to adapt such foundation models for task-specific downstream tasks but at the cost of degrading the general knowledge previously stored in the original foundation model. To address this, we propose SAM-Med3D-MoE, a novel framework that seamlessly integrates task-specific finetuned models with the foundational model, creating a unified model at minimal additional training expense for an extra gating network.
ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models
LLM development involves pre-training a foundation model on massive data, followed by fine-tuning on task-specific data to create specialized experts.
Streaming Video Diffusion: Online Video Editing with Diffusion Models
We present a novel task called online video editing, which is designed to edit \textbf{streaming} frames while maintaining temporal consistency.
ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
In terms of efficiency, ZipCache also showcases a $37. 3\%$ reduction in prefill-phase latency, a $56. 9\%$ reduction in decoding-phase latency, and a $19. 8\%$ reduction in GPU memory usage when evaluating LLaMA3-8B model with a input length of $4096$.
MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference.
LongVLM: Efficient Long Video Understanding via Large Language Models
In this way, we encode video representations that incorporate both local and global information, enabling the LLM to generate comprehensive responses for long-term videos.
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images
We introduce MVSplat, an efficient model that, given sparse multi-view images as input, predicts clean feed-forward 3D Gaussians.
Ranked #1 on
Generalizable Novel View Synthesis
on ACID
Motion Mamba: Efficient and Long Sequence Motion Generation
Human motion generation stands as a significant pursuit in generative computer vision, while achieving long-sequence and efficient motion generation remains challenging.
Ranked #12 on
Motion Synthesis
on KIT Motion-Language
T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching
Sampling from diffusion probabilistic models (DPMs) is often expensive for high-quality image generation and typically requires many steps with a large model.
ModaVerse: Efficiently Transforming Modalities with LLMs
This alignment process, which synchronizes a language model trained on textual data with encoders and decoders trained on multi-modal data, often necessitates extensive training of several projection layers in multiple stages.
Efficient Stitchable Task Adaptation
In this work, we present a novel framework, Efficient Stitchable Task Adaptation (ESTA), to efficiently produce a palette of fine-tuned models that adhere to diverse resource constraints.
Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction
Then, we adapt the SR model by implementing feature-level reconstruction learning from the initial test image to its second-order degraded counterparts, which helps the SR model generate plausible HR images.
Mask Propagation for Efficient Video Semantic Segmentation
By reusing predictions from key frames, we circumvent the need to process a large volume of video frames individually with resource-intensive segmentors, alleviating temporal redundancy and significantly reducing computational costs.
Object-aware Inversion and Reassembly for Image Editing
Subsequently, we propose an additional reassembly step to seamlessly integrate the respective editing results and the non-editing region to obtain the final edited image.
QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
Additionally, an adaptive strategy is designed to autonomously determine the optimal number of sub-channels for channel disassembly.
EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models
In this paper, we introduce a data-free and parameter-efficient fine-tuning framework for low-bit diffusion models, dubbed EfficientDM, to achieve QAT-level performance with PTQ-like efficiency.
SwitchGPT: Adapting Large Language Models for Non-Text Outputs
To bridge this gap, we propose a novel approach, \methodname, from a modality conversion perspective that evolves a text-based LLM into a multi-modal one.
Stitched ViTs are Flexible Vision Backbones
With extensive experiments on ImageNet-1K, ADE20K, COCO-Stuff-10K and NYUv2, SN-Netv2 demonstrates superior performance over SN-Netv1 on downstream dense predictions and shows strong ability as a flexible vision backbone, achieving great advantages in both training efficiency and deployment flexibility.
LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning
To this end, we propose LoRAPrune, a new framework that delivers an accurate structured pruned model in a highly memory-efficient manner.
PTQD: Accurate Post-Training Quantization for Diffusion Models
To address these challenges, we propose a unified formulation for the quantization noise and diffusion perturbed noise in the quantized denoising process.
Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning
Visual Parameter-Efficient Fine-Tuning (PEFT) has become a powerful alternative for full fine-tuning so as to adapt pre-trained vision models to downstream tasks, which only tunes a small number of parameters while freezing the vast majority ones to ease storage burden and optimization difficulty.
Stitchable Neural Networks
As each model family consists of pretrained models with diverse scales (e. g., DeiT-Ti/S/B), it naturally arises a fundamental question of how to efficiently assemble these readily available models in a family for dynamic accuracy-efficiency trade-offs at runtime.
A Survey on Efficient Training of Transformers
Recent advances in Transformers have come with a huge requirement on computing resources, highlighting the importance of developing efficient training techniques to make Transformer training faster, at lower cost, and to higher accuracy by the efficient use of computation and memory resources.
BiViT: Extremely Compressed Binary Vision Transformers
To solve this, we propose Softmax-aware Binarization, which dynamically adapts to the data distribution and reduces the error caused by binarization.
BiViT: Extremely Compressed Binary Vision Transformer
To solve this, we propose Softmax-aware Binarization, which dynamically adapts to the data distribution and reduces the error caused by binarization.
EcoFormer: Energy-Saving Attention with Linear Complexity
To this end, we propose a new binarization paradigm customized to high-dimensional softmax attention via kernelized hashing, called EcoFormer, to map the original queries and keys into low-dimensional binary codes in Hamming space.
FocusFormer: Focusing on What We Need via Architecture Sampler
During architecture search, these methods focus on finding architectures on the Pareto frontier of performance and resource consumption, which forms a gap between training and deployment.
An Efficient Spatio-Temporal Pyramid Transformer for Action Detection
The task of action detection aims at deducing both the action category and localization of the start and end moment for each action instance in a long, untrimmed video.
Fast Vision Transformers with HiLo Attention
Therefore, we propose to disentangle the high/low frequency patterns in an attention layer by separating the heads into two groups, where one group encodes high frequencies via self-attention within each local window, and another group encodes low frequencies by performing global attention between the average-pooled low-frequency keys and values from each window and each query position in the input feature map.
Ranked #190 on
Image Classification
on ImageNet
(GFLOPs metric)
Dynamic Focus-aware Positional Queries for Semantic Segmentation
In this paper, we propose a simple yet effective query design for semantic segmentation termed Dynamic Focus-aware Positional Queries (DFPQ), which dynamically generates positional queries conditioned on the cross-attention scores from the preceding decoder block and the positional encodings for the corresponding image features, simultaneously.
Ranked #24 on
Semantic Segmentation
on ADE20K
Automated Progressive Learning for Efficient Training of Vision Transformers
First, we develop a strong manual baseline for progressive learning of ViTs, by introducing momentum growth (MoGrow) to bridge the gap brought by model growth.
Sharpness-aware Quantization for Deep Neural Networks
However, the abrupt changes in quantized weights during training often lead to severe loss fluctuations and result in a sharp loss landscape, making the gradients unstable and thus degrading the performance.
Ranked #205 on
Image Classification
on CIFAR-100
Pruning Self-attentions into Convolutional Layers in Single Path
Relying on the single-path space, we introduce learnable binary gates to encode the operation choices in MSA layers.
Ranked #18 on
Efficient ViTs
on ImageNet-1K (with DeiT-T)
Mesa: A Memory-saving Training Framework for Transformers
While Transformers have delivered significant performance improvements, training such networks is extremely memory intensive owing to storing all intermediate activations that are needed for gradient computation during backpropagation, especially for long sequences.
Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints
More critically, EAS is able to find compact architectures within 0. 1 second for 50 deployment scenarios.
Less is More: Pay Less Attention in Vision Transformers
Transformers have become one of the dominant architectures in deep learning, particularly as a powerful alternative to convolutional neural networks (CNNs) in computer vision.
End-to-end One-shot Human Parsing
To address three main challenges in OSHP, i. e., small sizes, testing bias, and similar parts, we devise an End-to-end One-shot human Parsing Network (EOP-Net).
Scalable Vision Transformers with Hierarchical Pooling
However, the routine of the current ViT model is to maintain a full-length patch sequence during inference, which is redundant and lacks hierarchical representation.
Ranked #22 on
Efficient ViTs
on ImageNet-1K (with DeiT-T)
Single-path Bit Sharing for Automatic Loss-aware Model Compression
By jointly training the binary gates in conjunction with network parameters, the compression configurations of each layer can be automatically determined.
Fully Quantized Image Super-Resolution Networks
With the rising popularity of intelligent mobile devices, it is of great practical significance to develop accurate, realtime and energy-efficient image Super-Resolution (SR) inference methods.
FATNN: Fast and Accurate Ternary Neural Networks
Ternary Neural Networks (TNNs) have received much attention due to being potentially orders of magnitude faster in inference, as well as more power efficient, than full-precision counterparts.
AQD: Towards Accurate Fully-Quantized Object Detection
Network quantization allows inference to be conducted using low-precision arithmetic for improved inference efficiency of deep neural networks on edge devices.
Role-Wise Data Augmentation for Knowledge Distillation
To our knowledge, existing methods overlook the fact that although the student absorbs extra knowledge from the teacher, both models share the same input data -- and this data is the only medium by which the teacher's knowledge can be demonstrated.
Generative Low-bitwidth Data Free Quantization
More critically, our method achieves much higher accuracy on 4-bit quantization than the existing data free quantization method.
Ranked #2 on
Data Free Quantization
on CIFAR-100
Switchable Precision Neural Networks
Instantaneous and on demand accuracy-efficiency trade-off has been recently explored in the context of neural networks slimming.
Automatic Pruning for Quantized Neural Networks
In particular, for ResNet-18 on ImageNet, we prune 26. 12% of the model size with Binarized Neural Network quantization, achieving a top-1 classification accuracy of 47. 32% in a model of 2. 47 MB and 59. 30% with a 2-bit DoReFa-Net in 4. 36 MB.
Discrimination-aware Network Pruning for Deep Model Compression
In this paper, we propose a simple-yet-effective method called discrimination-aware channel pruning (DCP) to choose the channels that actually contribute to the discriminative power.
Structured Binary Neural Networks for Image Recognition
Experiments on both classification, semantic segmentation and object detection tasks demonstrate the superior performance of the proposed methods over various quantized networks in the literature.
Auxiliary Learning for Deep Multi-task Learning
The most current methods can be categorized as either: (i) hard parameter sharing where a subset of the parameters is shared among tasks while other parameters are task-specific; or (ii) soft parameter sharing where all parameters are task-specific but they are jointly regularized.
RBCN: Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
Binarized convolutional neural networks (BCNNs) are widely used to improve memory and computation efficiency of deep convolutional neural networks (DCNNs) for mobile and AI chips based applications.
Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
Furthermore, we propose a second progressive quantization scheme which gradually decreases the bit-width from high-precision to low-precision during training.
Training Quantized Neural Networks with a Full-precision Auxiliary Module
In this paper, we seek to tackle a challenge in training low-precision networks: the notorious difficulty in propagating gradient through a low-precision network due to the non-differentiable quantization function.
Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
In this paper, we propose to train convolutional neural networks (CNNs) with both binarized weights and activations, leading to quantized models specifically} for mobile devices with limited power capacity and computation resources.
Discrimination-aware Channel Pruning for Deep Neural Networks
Channel pruning is one of the predominant approaches for deep model compression.
Training Compact Neural Networks with Binary Weights and Low Precision Activations
In this paper, we propose to train a network with binary weights and low-bitwidth activations, designed especially for mobile devices with limited power consumption.
Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
To this end we propose a unified framework, the ParalleL AttentioN (PLAN) network, to discover the object in an image that is being referred to in variable length natural expression descriptions, from short phrases query to long multi-round dialogs.
Towards Effective Low-bitwidth Convolutional Neural Networks
This paper tackles the problem of training a deep convolutional neural network with both low-precision weights and low-bitwidth activations.
Towards Context-Aware Interaction Recognition for Visual Relationship Detection
The proposed method still builds one classifier for one interaction (as per type (ii) above), but the classifier built is adaptive to context via weights which are context dependent.
TasselNet: Counting maize tassels in the wild via local counts regression network
To our knowledge, this is the first time that a plant-related counting problem is considered using computer vision technologies under unconstrained field-based environment.
Care about you: towards large-scale human-centric visual relationship detection
In addressing this problem we first construct a large-scale human-centric visual relationship detection dataset (HCVRD), which provides many more types of relationship annotation (nearly 10K categories) than the previous released datasets.
Human-Object Interaction Detection
Relationship Detection
+1
Towards Context-aware Interaction Recognition
Recognizing how objects interact with each other is a crucial task in visual recognition.
Sequential Person Recognition in Photo Albums with a Recurrent Network
In this work, we propose to model the relational information between people as a sequence prediction task.
Attend in groups: a weakly-supervised deep learning framework for learning from web data
Large-scale datasets have driven the rapid development of deep neural networks for visual recognition.
Visual Tracking via Shallow and Deep Collaborative Model
In the discriminative model, we exploit the advances of deep learning architectures to learn generic features which are robust to both background clutters and foreground appearance variations.
Fast Training of Triplet-based Deep Binary Embedding Networks
To solve the first stage, we design a large-scale high-order binary codes inference algorithm to reduce the high-order objective to a standard binary quadratic problem such that graph cuts can be used to efficiently infer the binary code which serve as the label of each training datum.
