Search Results for author:
Found 113 papers, 83 papers with code
NTIRE 2025 Challenge on Day and Night Raindrop Removal for Dual-Focused Images: Methods and Results
This paper reviews the NTIRE 2025 Challenge on Day and Night Raindrop Removal for Dual-Focused Images.
DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistency
In this work, we propose the Dual Consistency SAM (DC-SAM) method based on prompt-tuning to adapt SAM and SAM2 for in-context segmentation of both images and videos.
PVUW 2025 Challenge Report: Advances in Pixel-level Understanding of Complex Videos in the Wild
This report provides a comprehensive overview of the 4th Pixel-level Video Understanding in the Wild (PVUW) Challenge, held in conjunction with CVPR 2025.
Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding
We present Pixel-SAIL, a single transformer for pixel-wise MLLM tasks.
The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer
This paper introduces SAIL, a single transformer unified multimodal large language model (MLLM) that integrates raw pixel encoding and language decoding within a singular architecture.
An Empirical Study of GPT-4o Image Generation Capabilities
The landscape of image generation has rapidly evolved, from early GAN-based approaches to diffusion models and, most recently, to unified generative architectures that seek to bridge understanding and generation tasks.
Generative Classifier for Domain Generalization
To address these issues, we explore the theoretical implications of relying on domain invariance, revealing the crucial role of domain-specific information in mitigating the target risk for DG.
4th PVUW MeViS 3rd Place Report: Sa2VA
Referring video object segmentation (RVOS) is a challenging task that requires the model to segment the object in a video given the language description.
Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfer
In contrast, state-of-the-art video Diffusion Transformers (DiT) models use 3D full attention, which does not explicitly separate temporal and spatial information.
Unified Dense Prediction of Video Diffusion
We present a unified network for simultaneously generating videos and their corresponding entity segmentation and depth maps from text prompts.
RobuRCDet: Enhancing Robustness of Radar-Camera Fusion in Bird's Eye View for 3D Object Detection
Specifically, we design a 3D Gaussian Expansion (3DGE) module to mitigate inaccuracies in radar points, including position, Radar Cross-Section (RCS), and velocity.
Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs
Recent advancements in multimodal models have shown a strong ability in visual perception, reasoning abilities, and vision-language understanding.
Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos
This work presents Sa2VA, the first unified model for dense grounded understanding of both images and videos.
Ranked #1 on
Referring Video Object Segmentation
on MeViS
DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation
Story visualization, the task of creating visual narratives from textual descriptions, has seen progress with text-to-image generation models.
EMOv2: Pushing 5M Vision Model Frontier
Our goal is to set up the new frontier of the 5M magnitude lightweight model on various downstream tasks.
HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editing
HumanEdit bridges this gap by employing human annotators to construct data pairs and administrators to provide feedback.
SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model
The rapid advancement of generative models in creating highly realistic images poses substantial risks for misinformation dissemination.
DynamicControl: Adaptive Condition Selection for Improved Text-to-Image Generation
To enhance the controllability of text-to-image diffusion models, current ControlNet-like models have explored various control signals to dictate image attributes.
RelationBooth: Towards Relation-Aware Customized Object Generation
Instead, this work addresses that gap by focusing on relation-aware customized image generation, which aims to preserve the identities from image prompts while maintaining the predicate relations described in text prompts.
Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Image
In the visual spatial understanding (VSU) area, spatial image-to-text (SI2T) and spatial text-to-image (ST2I) are two fundamental tasks that appear in dual form.
Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation
However, when it comes to stereo audio generation, the soundscapes often have a complex scene of multiple objects and directions.
Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis
We present Meissonic, which elevates non-autoregressive masked image modeling (MIM) text-to-image to a level comparable with state-of-the-art diffusion models like SDXL.
Video Prediction Transformers without Recurrence or Convolution
Video prediction has witnessed the emergence of RNN-based models led by ConvLSTM, and CNN-based models led by SimVP.
Ranked #1 on
Video Prediction
on Moving MNIST
MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learning
Drawing from this analysis, we introduce a Motion-Identity Modulated Appearance Learning Module (MIA) that modulates CLIP features at both motion and identity levels.
PointDGMamba: Domain Generalization of Point Cloud Classification via Generalized State Space Model
Domain Generalization (DG) has been recently explored to improve the generalizability of point cloud classification (PCC) models toward unseen domains.
You Can't Ignore Either: Unifying Structure and Feature Denoising for Robust Graph Learning
However, the noises and attacks may come from both structures and features in graphs, making the graph denoising a dilemma and challenging problem.
LLAVADI: What Matters For Multimodal Large Language Models Distillation
The recent surge in Multimodal Large Language Models (MLLMs) has showcased their remarkable potential for achieving generalized intelligence by integrating visual understanding into Large Language Models. Nevertheless, the sheer model size of MLLMs leads to substantial memory and computational demands that hinder their widespread deployment.
MEDIC: Zero-shot Music Editing with Disentangled Inversion Control
To counteract these issues, we introduce the Disentangled Inversion technique to disentangle the diffusion process into triple branches, rectifying the deviated path of the source branch caused by DDIM inversion.
Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language
Diffusion models can generate realistic and diverse images, potentially facilitating data availability for data-intensive perception tasks.
Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Model
Based on the benchmark results, our RWKV-SAM achieves outstanding performance in efficiency and segmentation quality compared to transformers and other linear attention models.
OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding
Current universal segmentation methods demonstrate strong capabilities in pixel-level image and video understanding.
MotionBooth: Motion-Aware Customized Text-to-Video Generation
In this work, we present MotionBooth, an innovative framework designed for animating customized subjects with precise control over both object and camera movements.
MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning
We propose the integration of an additional high-resolution visual encoder to capture fine-grained details, which are then fused with base visual features through a Conv-Gate fusion network.
Ranked #75 on
Visual Question Answering
on MM-Vet
Towards Semantic Equivalence of Tokenization in Multimodal LLM
The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features.
Ranked #73 on
Visual Question Answering
on MM-Vet
BACON: Bayesian Optimal Condensation Framework for Dataset Distillation
Our extensive experiments confirm the effectiveness of BACON and its seamless integration with existing methods, thereby enhancing their performance for the DD task.
SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow
For image synthesis, we propose a finite perturbation approach to enhance the diversity of generated results without changing the semantic categories.
Adversarial Attacks on Both Face Recognition and Face Anti-spoofing Models
In particular, we introduce a new attack method, namely Style-aligned Distribution Biasing (SDB), to improve the capacity of black-box attacks on both FR and FAS models.
Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model
This foundational estimation facilitates a detailed, coarse-to-fine segmentation strategy that significantly enhances the precision of object identification and segmentation.
PointRWKV: Efficient RWKV-Like Model for Hierarchical Point Cloud Learning
Transformers have revolutionized the point cloud learning task, but the quadratic complexity hinders its extension to long sequence and makes a burden on limited computational resources.
Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding
This framework is designed to perform understanding tasks for any 3D scene without requiring prior knowledge of the scene.
CPT-Interp: Continuous sPatial and Temporal Motion Modeling for 4D Medical Image Interpolation
Motion information from 4D medical imaging offers critical insights into dynamic changes in patient anatomy for clinical assessments and radiotherapy planning and, thereby, enhances the capabilities of 3D image analysis.
4D Panoptic Scene Graph Generation
To facilitate research in this new area, we build a richly annotated PSG-4D dataset consisting of 3K RGB-D videos with a total of 1M frames, each of which is labeled with 4D panoptic segmentation masks as well as fine-grained, dynamic scene graphs.
Point-In-Context: Understanding Point Cloud via In-Context Learning
With the emergence of large-scale models trained on diverse datasets, in-context learning has emerged as a promising paradigm for multitasking, notably in natural language processing and image processing.
VG4D: Vision-Language Model Goes 4D Video Recognition
By transferring the knowledge of the VLM to the 4D encoder and combining the VLM, our VG4D achieves improved recognition performance.
Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark
Moreover, current metrics such as AU-ROC have nearly reached saturation on simple datasets, which prevents a comprehensive evaluation of different methods.
DGMamba: Domain Generalization via Generalized State Space Model
SPR strives to encourage the model to concentrate more on objects rather than context, consisting of two designs: Prior-Free Scanning~(PFS), and Domain Context Interchange~(DCI).
MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection
Recent advancements in anomaly detection have seen the efficacy of CNN- and transformer-based approaches.
DVIS-DAQ: Improving Video Segmentation via Dynamic Anchor Queries
Modern video segmentation methods adopt object queries to perform inter-frame association and demonstrate satisfactory performance in tracking continuously appearing objects despite large-scale motion and transient occlusion.
Ranked #1 on
Video Instance Segmentation
on OVIS validation
(using extra training data)
GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learning
To tackle these challenges, we present GenView, a controllable framework that augments the diversity of positive views leveraging the power of pretrained generative models while preserving semantics.
Explore In-Context Segmentation via Latent Diffusion Models
In-context segmentation has drawn more attention with the introduction of vision foundation models.
Point Cloud Mamba: Point Cloud Learning via State Space Model
To enable Mamba to process 3-D point cloud data more effectively, we propose a novel Consistent Traverse Serialization method to convert point clouds into 1-D point sequences while ensuring that neighboring points in the sequence are also spatially adjacent.
Generalizable Entity Grounding via Assistance of Large Language Model
In this work, we propose a novel approach to densely ground visual entities from a long caption.
Towards Language-Driven Video Inpainting via Multimodal Large Language Models
We introduce a new task -- language-driven video inpainting, which uses natural language instructions to guide the inpainting process.
ModelNet-O: A Large-Scale Synthetic Dataset for Occlusion-Aware Point Cloud Classification
Through extensive experiments, we demonstrate that our PointMLS achieves state-of-the-art results on ModelNet-O and competitive results on regular datasets, and it is robust and effective.
Ranked #25 on
3D Point Cloud Classification
on ModelNet40
Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively
The CLIP and Segment Anything Model (SAM) are remarkable vision foundation models (VFMs).
BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model
To this end, we propose Scalable Bias-Mode Attention Mask (BA-SAM) to enhance SAM's adaptability to varying image resolutions while eliminating the need for structure modifications.
An Open and Comprehensive Pipeline for Unified Object Grounding and Detection
Grounding-DINO is a state-of-the-art open-set detection model that tackles multiple vision tasks including Open-Vocabulary Detection (OVD), Phrase Grounding (PG), and Referring Expression Comprehension (REC).
Referring Image Editing: Object-level Image Editing via Referring Expressions
Significant advancements have been made in image editing with the recent advance of the Diffusion model.
A Generalist FaceX via Learning Unified Facial Representation
This work presents FaceX framework, a novel facial generalist model capable of handling diverse facial tasks simultaneously.
Exploring Plain ViT Reconstruction for Multi-class Unsupervised Anomaly Detection
\Eg, achieving 85. 4 mAD that surpasses UniAD by +3. 0 for the MVTec AD dataset, and it requires only 1. 1 hours and 2. 3G GPU memory to complete model training on a single V100 that can serve as a strong baseline to facilitate the development of future research.
Multi-class Anomaly Detection
Unsupervised Anomaly Detection
RTMO: Towards High-Performance One-Stage Real-Time Multi-Person Pose Estimation
Real-time multi-person pose estimation presents significant challenges in balancing speed and precision.
Ranked #1 on
Multi-Person Pose Estimation
on CrowdPose
(using extra training data)
EdgeSAM: Prompt-In-the-Loop Distillation for On-Device Deployment of SAM
It is also the first SAM variant that can run at over 30 FPS on an iPhone 14.
Skeleton-in-Context: Unified Skeleton Sequence Modeling with In-Context Learning
Under this setting, the model can perceive tasks from prompts and accomplish them without any extra task-specific head predictions or model fine-tuning.
Effective Adapter for Face Recognition in the Wild
Traditional heuristic approaches-either training models directly on these degraded images or their enhanced counterparts using face restoration techniques-have proven ineffective, primarily due to the degradation of facial features and the discrepancy in image domains.
Panoptic Video Scene Graph Generation
PVSG relates to the existing video scene graph generation (VidSGG) problem, which focuses on temporal interactions between humans and objects grounded with bounding boxes in videos.
Rethinking Evaluation Metrics of Open-Vocabulary Segmentaion
We benchmarked the proposed evaluation metrics on 12 open-vocabulary methods of three segmentation tasks.
OV-VG: A Benchmark for Open-Vocabulary Visual Grounding
While current foundational models excel at various visual language tasks, there's a noticeable absence of models specifically tailored for open-vocabulary visual grounding.
DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detection
We refer to this approach as the self-training strategy, which enhances recall and accuracy for novel classes without requiring extra annotations, datasets, and re-training.
CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction
However, when transferring the vision-language alignment of CLIP from global image representation to local region representation for the open-vocabulary dense prediction tasks, CLIP ViTs suffer from the domain shift from full images to local image regions.
Ranked #4 on
Open Vocabulary Panoptic Segmentation
on ADE20K
MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation
We present MosaicFusion, a simple yet effective diffusion-based data augmentation approach for large vocabulary instance segmentation.
Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants
Beyond the normal case, long-tail class incremental learning and few-shot class incremental learning are also proposed to consider the data imbalance and data scarcity, respectively, which are common in real-world implementations and further exacerbate the well-known problem of catastrophic forgetting.
class-incremental learning
Few-Shot Class-Incremental Learning
+1
Iterative Robust Visual Grounding with Masked Reference based Centerpoint Supervision
The proposed framework is evaluated on five regular VG datasets and two newly constructed robust VG datasets.
Pair then Relation: Pair-Net for Panoptic Scene Graph Generation
Panoptic Scene Graph (PSG) is a challenging task in Scene Graph Generation (SGG) that aims to create a more comprehensive scene graph representation using panoptic segmentation instead of boxes.
Towards Open Vocabulary Learning: A Survey
To our knowledge, this is the first comprehensive literature review of open vocabulary learning.
Explore In-Context Learning for 3D Point Cloud Understanding
With the rise of large-scale models trained on broad data, in-context learning has become a new learning paradigm that has demonstrated significant potential in natural language processing and computer vision tasks.
Change Detection Methods for Remote Sensing in the Last Decade: A Comprehensive Review
We first introduce some preliminary knowledge for the change detection task, such as problem definition, datasets, evaluation metrics, and transformer basics, as well as provide a detailed taxonomy of existing algorithms from three different perspectives: algorithm granularity, supervision modes, and learning frameworks in the methodology section.
Change Detection
Change detection for remote sensing images
+1
Transformer-Based Visual Segmentation: A Survey
Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks.
Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation
Our framework is a near-online approach that takes a short subclip as input and outputs the corresponding spatial-temporal tube masks.
Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning
In this paper, we deal with this misalignment dilemma in FSCIL inspired by the recently discovered phenomenon named neural collapse, which reveals that the last-layer features of the same class will collapse into a vertex, and the vertices of all classes are aligned with the classifier prototypes, which are formed as a simplex equiangular tight frame (ETF).
class-incremental learning
Few-Shot Class-Incremental Learning
+1
Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class-Incremental Learning
In this paper, we deal with this misalignment dilemma in FSCIL inspired by the recently discovered phenomenon named neural collapse, which reveals that the last-layer features of the same class will collapse into a vertex, and the vertices of all classes are aligned with the classifier prototypes, which are formed as a simplex equiangular tight frame (ETF).
Ranked #5 on
Few-Shot Class-Incremental Learning
on mini-Imagenet
class-incremental learning
Few-Shot Class-Incremental Learning
+1
Rethinking Mobile Block for Efficient Attention-based Models
This paper focuses on developing modern, efficient, lightweight models for dense predictions while trading off parameters, FLOPs, and performance.
PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation
Third, inspired by Mask2Former, based on our meta-architecture, we propose Panoptic-PartFormer++ and design a new part-whole cross-attention scheme to boost part segmentation qualities further.
Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation
Few-Shot Instance Segmentation (FSIS) requires detecting and segmenting novel classes with limited support examples.
Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation
Experiments on the COCO dataset with two settings: Open Vocabulary Instance Segmentation (OVIS) and Open Set Panoptic Segmentation (OSPS) demonstrate the superiority of the CGG.
Convolution-enhanced Evolving Attention Networks
To the best of our knowledge, this is the first work that explicitly models the layer-wise evolution of attention maps.
Towards Robust Referring Image Segmentation
It considers the negative sentence inputs besides the regular positive text inputs.
SFNet: Faster, Accurate, and Domain Agnostic Semantic Segmentation via Semantic Flow
In this paper, we focus on exploring effective methods for faster, accurate, and domain agnostic semantic segmentation.
Ranked #12 on
Semantic Segmentation
on BDD100K val
EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm
Motivated by biological evolution, this paper explains the rationality of Vision Transformer by analogy with the proven practical evolutionary algorithm (EA) and derives that both have consistent mathematical formulation.
Multi-Task Learning with Multi-Query Transformer for Dense Prediction
We first model each task with a task-relevant query.
Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation
We hope this simple, yet effective method can serve as a new, flexible baseline in unified video segmentation design.
Ranked #1 on
Video Panoptic Segmentation
on KITTI-STEP
(using extra training data)
Fashionformer: A simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition
This focus on joint human fashion segmentation and attribute recognition.
Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentation
To the best of our knowledge, we are the first to solve the PPS problem via \textit{a unified and end-to-end transformer model.
Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?
Modern deep neural networks for classification usually jointly learn a backbone for representation and a linear classifier to output the logit of each class.
Ranked #28 on
Long-tail Learning
on CIFAR-10-LT (ρ=100)
TransVOD: End-to-End Video Object Detection with Spatial-Temporal Transformers
Detection Transformer (DETR) and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors.
Ranked #6 on
Video Object Detection
on ImageNet VID
(using extra training data)
PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation
The Depth-aware Video Panoptic Segmentation (DVPS) is a new challenging vision problem that aims to predict panoptic segmentation and depth in a video simultaneously.
Improving Video Instance Segmentation via Temporal Pyramid Routing
To incorporate both temporal and scale information, we propose a Temporal Pyramid Routing (TPR) strategy to conditionally align and conduct pixel-level aggregation from a feature pyramid pair of two adjacent frames.
Global Aggregation then Local Distribution for Scene Parsing
Modelling long-range contextual relationships is critical for pixel-wise prediction tasks such as semantic segmentation.
BoundarySqueeze: Image Segmentation as Boundary Squeezing
This module is used to squeeze the object boundary from both inner and outer directions, which contributes to precise mask representation.
Fast and Accurate Scene Parsing via Bi-direction Alignment Networks
Motivated by this, we propose a novel network by aligning two-path information into each other through a learned flow field.
Ranked #12 on
Semantic Segmentation
on BDD100K val
Dynamic Dual Sampling Module for Fine-Grained Semantic Segmentation
Representation of semantic context and local details is the essential issue for building modern semantic segmentation models.
End-to-End Video Object Detection with Spatial-Temporal Transformers
Recently, DETR and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors.
Enhanced Boundary Learning for Glass-like Object Segmentation
We use these two modules to design a decoder that generates accurate and clean segmentation results, especially on the object contours.
Ranked #20 on
Thermal Image Segmentation
on RGB-T-Glass-Segmentation
PointFlow: Flowing Semantics Through Points for Aerial Image Segmentation
Experimental results on three different aerial segmentation datasets suggest that the proposed method is more effective and efficient than state-of-the-art general semantic segmentation methods.
Involution: Inverting the Inherence of Convolution for Visual Recognition
Convolution has been the core ingredient of modern neural networks, triggering the surge of deep learning in vision.
Ranked #768 on
Image Classification
on ImageNet
Towards Efficient Scene Understanding via Squeeze Reasoning
Instead of propagating information on the spatial map, we first learn to squeeze the input feature into a channel-wise global vector and perform reasoning within the single vector where the computation cost can be significantly reduced.
Improving Semantic Segmentation via Decoupled Body and Edge Supervision
Our insight is that appealing performance of semantic segmentation requires \textit{explicitly} modeling the object \textit{body} and \textit{edge}, which correspond to the high and low frequency of the image.
Semantic Flow for Fast and Accurate Scene Parsing
A common practice to improve the performance is to attain high resolution feature maps with strong semantic representation.
Ranked #2 on
Real-Time Semantic Segmentation
on Cityscapes test
Global Aggregation then Local Distribution in Fully Convolutional Networks
GALD is end-to-end trainable and can be easily plugged into existing FCNs with various global aggregation modules for a wide range of vision tasks, and consistently improves the performance of state-of-the-art object detection and instance segmentation approaches.
Ranked #1 on
Semantic Segmentation
on PASCAL VOC 2007
Dual Graph Convolutional Network for Semantic Segmentation
Exploiting long-range contextual information is key for pixel-wise prediction tasks such as semantic segmentation.
Ranked #32 on
Semantic Segmentation
on Cityscapes test
GFF: Gated Fully Fusion for Semantic Segmentation
Semantic segmentation generates comprehensive understanding of scenes through densely predicting the category for each pixel.
Ranked #29 on
Semantic Segmentation
on Cityscapes test
