Search Results for author:
Found 98 papers, 61 papers with code
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models.
LION: Linear Group RNN for 3D Object Detection in Point Clouds
To tackle this problem, we simply introduce a 3D spatial feature descriptor and integrate it into the linear group RNN operators to enhance their spatial features rather than blindly increasing the number of scanning orders for voxel features.
Ranked #1 on
3D Object Detection
on Waymo Open Dataset
Point Transformer V3 Extreme: 1st Place Solution for 2024 Waymo Open Dataset Challenge in Semantic Segmentation
In this technical report, we detail our first-place solution for the 2024 Waymo Open Dataset Challenge's semantic segmentation track.
ViLLa: Video Reasoning Segmentation with Large Language Model
To bridge the gap between image and video, in this work, we propose a new video segmentation task - video reasoning segmentation.
LogoSticker: Inserting Logos into Diffusion Models for Customized Generation
Recent advances in text-to-image model customization have underscored the importance of integrating new concepts with a few examples.
OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
Given the foundational role of multimodal joint representation in understanding and generation pipelines, high-quality omni joint representations would be a step toward co-processing more diverse multimodal information.
HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models
High-resolution inputs enable Large Vision-Language Models (LVLMs) to discern finer visual details, enhancing their comprehension capabilities.
Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Images
Afterward, it proceeds to edit the back views.
Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models
To address this problem efficiently, we propose the Distribution-aware Interference-free Knowledge Integration (DIKI) framework, retaining pre-trained knowledge of VLMs from a perspective of avoiding information interference.
Zero-shot Image Editing with Reference Imitation
Image editing serves as a practical yet challenging task considering the diverse demands from users, where one of the hardest parts is to precisely describe how the edited image should look like.
LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
Due to the need to interact with the real world, embodied agents are required to possess comprehensive prior knowledge, long-horizon planning capability, and a swift response speed.
OV-Uni3DETR: Towards Unified Open-Vocabulary 3D Object Detection via Cycle-Modality Propagation
Specifically, we propose the cycle-modality propagation, aimed at propagating knowledge bridging 2D and 3D modalities, to support the aforementioned functionalities.
Pixel-GS: Density Control with Pixel-aware Gradient for 3D Gaussian Splatting
3D Gaussian Splatting (3DGS) has demonstrated impressive novel view synthesis results while advancing real-time rendering performance.
OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation
This exploration led to the creation of Omni-Adaptive 3D CNNs (OA-CNNs), a family of networks that integrates a lightweight module to greatly enhance the adaptivity of sparse CNNs at minimal computational cost.
Ranked #5 on
3D Semantic Segmentation
on SemanticKITTI
(val mIoU metric)
GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding
To address this issue, we propose GroupContrast, a novel approach that combines segment grouping and semantic-aware contrastive learning.
Towards Unified 3D Object Detection via Algorithm and Data Unification
In this work, we propose to address the challenges from two perspectives, the algorithm perspective and data perspective.
OpenSUN3D: 1st Workshop Challenge on Open-Vocabulary 3D Scene Understanding
This report provides an overview of the challenge hosted at the OpenSUN3D Workshop on Open-Vocabulary 3D Scene Understanding held in conjunction with ICCV 2023.
Memory Consistency Guided Divide-and-Conquer Learning for Generalized Category Discovery
Generalized category discovery (GCD) aims at addressing a more realistic and challenging setting of semi-supervised learning, where only part of the category labels are assigned to certain training samples.
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
To this end, we scale up the dataset by designing a data engine to collect and automatically annotate large-scale unlabeled data (~62M), which significantly enlarges the data coverage and thus is able to reduce the generalization error.
Ranked #3 on
Monocular Depth Estimation
on ETH3D
Point Transformer V3: Simpler Faster Stronger
This paper is not motivated to seek innovation within the attention mechanism.
Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases
We conducted a series of structured experiments to evaluate their performance in various industrial application scenarios, offering a comprehensive perspective on their practical utility.
Self-supervised Learning for Enhancing Geometrical Modeling in 3D-Aware Generative Adversarial Network
To address this, we present a Self-Supervised Learning (SSL) technique tailored as an auxiliary loss for any 3D-GAN, designed to improve its 3D geometrical modeling capabilities.
Point Transformer V3: Simpler, Faster, Stronger
This paper is not motivated to seek innovation within the attention mechanism.
Ranked #1 on
3D Semantic Segmentation
on SemanticKITTI
(using extra training data)
VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation
In combination with the existing text tokenizer and detokenizer, this framework allows for the encoding of interleaved image-text data into a multimodal sequence, which can subsequently be fed into the transformer model.
TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation
What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly train models across the aggregation of datasets to enhance data volume and diversity.
CorresNeRF: Image Correspondence Priors for Neural Radiance Fields
We present CorresNeRF, a novel method that leverages image correspondence priors computed by off-the-shelf methods to supervise NeRF training.
DreamComposer: Controllable 3D Object Generation via Multi-View Conditions
However, due to the lack of information from multiple views, these works encounter difficulties in generating controllable novel views.
LivePhoto: Real Image Animation with Text-guided Motion Control
In particular, considering the facts that (1) text can only describe motions roughly (e. g., regardless of the moving speed) and (2) text may include both content and motion descriptions, we introduce a motion intensity estimation module as well as a text re-weighting module to reduce the ambiguity of text-to-motion mapping.
GPT4Point: A Unified Framework for Point-Language Understanding and Generation
Multimodal Large Language Models (MLLMs) have excelled in 2D image-text comprehension and image generation, but their understanding of the 3D world is notably deficient, limiting progress in 3D language understanding and generation.
A Lightweight Clustering Framework for Unsupervised Semantic Segmentation
We thus propose a lightweight clustering framework for unsupervised semantic segmentation.
Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding
Building on this, we design a visual program that consists of three types of modules, i. e., view-independent, view-dependent, and functional modules.
FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models
Then, we investigate the role of synthetic images by joint training with real images, or pre-training for real images.
UniPAD: A Universal Pre-training Paradigm for Autonomous Driving
In the context of autonomous driving, the significance of effective feature learning is widely acknowledged.
PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm
In this paper, we introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation, thereby establishing a pathway to 3D foundational models.
Ranked #2 on
Semantic Segmentation
on S3DIS
(using extra training data)
Uni3DETR: Unified 3D Detection Transformer
In this paper, we propose Uni3DETR, a unified 3D detector that addresses indoor and outdoor 3D detection within the same framework.
DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
Multimodal large language models (MLLMs) have emerged as a prominent area of interest within the research community, given their proficiency in handling and reasoning with non-textual data, including images and videos.
OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation
In this work, we introduce OpenIns3D, a new 3D-input-only framework for 3D open-vocabulary scene understanding.
Ranked #1 on
Zero-shot 3D Point Cloud Classification
on ScanNetV2
3D Open-Vocabulary Instance Segmentation
3D Open-Vocabulary Object Detection
+6
Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training
In contrast, such privilege has not yet fully benefited 3D deep learning, mainly due to the limited availability of large-scale 3D datasets.
Ranked #3 on
3D Semantic Segmentation
on SemanticKITTI
(val mIoU metric, using extra
training data)
InsMapper: Exploring Inner-instance Information for Vectorized HD Mapping
Vectorized high-definition (HD) maps contain detailed information about surrounding road elements, which are crucial for various downstream tasks in modern autonomous vehicles, such as motion planning and vehicle control.
Shrinking Class Space for Enhanced Certainty in Semi-Supervised Learning
To mitigate potentially incorrect pseudo labels, recent frameworks mostly set a fixed confidence threshold to discard uncertain samples.
AnyDoor: Zero-shot Object-level Image Customization
This work presents AnyDoor, a diffusion-based image generator with the power to teleport target objects to new scenes at user-specified locations in a harmonious way.
GroupLane: End-to-End 3D Lane Detection with Channel-wise Grouping
Besides, GroupLane with ResNet18 still surpasses PersFormer by 4. 9% F1 score, while the inference speed is nearly 7x faster and the FLOPs is only 13. 3% of it.
SAM3D: Segment Anything in 3D Scenes
In this work, we propose SAM3D, a novel framework that is able to predict masks in 3D point clouds by leveraging the Segment-Anything Model (SAM) in RGB images without further training or finetuning.
OCBEV: Object-Centric BEV Transformer for Multi-View 3D Object Detection
Multi-view 3D object detection is becoming popular in autonomous driving due to its high effectiveness and low cost.
LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields
We address this challenge by formulating, to the best of our knowledge, the first differentiable end-to-end LiDAR rendering framework, LiDAR-NeRF, leveraging a neural radiance field (NeRF) to facilitate the joint learning of geometry and the attributes of 3D points.
VoxelFormer: Bird's-Eye-View Feature Generation based on Dual-view Attention for Multi-view 3D Object Detection
In recent years, transformer-based detectors have demonstrated remarkable performance in 2D visual perception tasks.
Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning
As a pioneering work, PointContrast conducts unsupervised 3D representation learning via leveraging contrastive learning over raw RGB-D frames and proves its effectiveness on various downstream tasks.
Ranked #13 on
Semantic Segmentation
on ScanNet
(using extra training data)
Influencer Backdoor Attack on Semantic Segmentation
In this work, we explore backdoor attacks on segmentation models to misclassify all pixels of a victim class by injecting a specific trigger on non-victim pixels during inferences, which is dubbed Influencer Backdoor Attack (IBA).
Detecting Everything in the Open World: Towards Universal Object Detection
In this paper, we formally address universal object detection, which aims to detect every scene and predict every category.
ScribbleSeg: Scribble-based Interactive Image Segmentation
We hope this could serve as a more powerful and general solution for interactive segmentation.
Open-vocabulary Panoptic Segmentation with Embedding Modulation
Open-vocabulary image segmentation is attracting increasing attention due to its critical applications in the real world.
GeoSpark: Sparking up Point Cloud Segmentation with Geometry Clue
For feature aggregation, it improves feature modeling by allowing the network to learn from both local points and neighboring geometry partitions, resulting in an enlarged data-tailored receptive field.
Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentation
Referring image segmentation segments an image from a language expression.
PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformer
As key modules in PhysFormer, the temporal difference transformers first enhance the quasi-periodic rPPG features with temporal difference guided global attention, and then refine the local spatio-temporal representation against interference.
BT^2: Backward-compatible Training with Basis Transformation
In this way, the new representation can be directly compared with the old representation, in principle avoiding the need for any backfilling.
Mod-Squad: Designing Mixtures of Experts As Modular Multi-Task Learners
To address the MTL challenge, we propose Mod-Squad, a new model that is Modularized into groups of experts (a 'Squad').
Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners
To address the MTL challenge, we propose Mod-Squad, a new model that is Modularized into groups of experts (a 'Squad').
General Adversarial Defense Against Black-box Attacks via Pixel Level and Feature Level Distribution Alignments
In this paper, we use Deep Generative Networks (DGNs) with a novel training mechanism to eliminate the distribution gap.
$BT^2$: Backward-compatible Training with Basis Transformation
In this way, the new representation can be directly compared with the old representation, in principle avoiding the need for any backfilling.
Point Transformer V2: Grouped Vector Attention and Partition-based Pooling
In this work, we analyze the limitations of the Point Transformer and propose our powerful and efficient Point Transformer V2 model with novel designs that overcome the limitations of previous work.
Ranked #1 on
3D Semantic Segmentation
on nuScenes
SegPGD: An Effective and Efficient Adversarial Attack for Evaluating and Boosting Segmentation Robustness
Since SegPGD can create more effective adversarial examples, the adversarial training with our SegPGD can boost the robustness of segmentation models.
DecoupleNet: Decoupled Network for Domain Adaptive Semantic Segmentation
Unsupervised domain adaptation in semantic segmentation has been raised to alleviate the reliance on expensive pixel-wise annotations.
Universal Adaptive Data Augmentation
Different from existing methods, UADA would adaptively update DA's parameters according to the target model's gradient information during training: given a pre-defined set of DA operations, we randomly decide types and magnitudes of DA operations for every data batch during training, and adaptively update DA's parameters along the gradient direction of the loss concerning DA's parameters.
FocalClick: Towards Practical Interactive Image Segmentation
To make the model work with preexisting masks, we formulate a sub-task termed Interactive Mask Correction, and propose Progressive Merge as the solution.
Ranked #2 on
Interactive Segmentation
on DAVIS
(using extra training data)
Stratified Transformer for 3D Point Cloud Segmentation
In this paper, we propose Stratified Transformer that is able to capture long-range contexts and demonstrates strong generalization ability and high performance.
Ranked #17 on
Semantic Segmentation
on ScanNet
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
Referring image segmentation is a fundamental vision-language task that aims to segment out an object referred to by a natural language expression from an image.
PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e. g., remote healthcare and affective computing).
Hierarchical interaction network for video object segmentation from referring expressions
In this paper, we investigate the problem of video object segmentation from referring expressions (VOSRE).
Ranked #1 on
Referring Expression Segmentation
on J-HMDB
(Precision@0.9 metric)
Optical Flow Estimation
Referring Expression Segmentation
+3
Adversarial Examples on Segmentation Models Can be Easy to Transfer
The high transferability achieved by our method shows that, in contrast to the observations in previous work, adversarial examples on a segmentation model can be easy to transfer to other segmentation models.
Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision
In particular, Panoptic FCN encodes each object instance or stuff category with the proposed kernel generator and produces the prediction by convolving the high-resolution feature directly.
Open-World Entity Segmentation
By removing the need of class label prediction, the models trained for such task can focus more on improving segmentation quality.
Do Different Tracking Tasks Require Different Appearance Models?
We show how most tracking tasks can be solved within this framework, and that the same appearance model can be successfully used to obtain results that are competitive against specialised methods for most of the tasks considered.
Ranked #2 on
Video Object Segmentation
on DAVIS 2017
(mIoU metric)
Multi-Object Tracking
Multi-Object Tracking and Segmentation
+10
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency
Semantic segmentation has made tremendous progress in recent years.
Dual-Cross Central Difference Network for Face Anti-Spoofing
In this paper, we propose two Cross Central Difference Convolutions (C-CDC), which exploit the difference of the center and surround sparse local features from the horizontal/vertical and diagonal directions, respectively.
Distilling Knowledge via Knowledge Review
Knowledge distillation transfers knowledge from the teacher network to the student one, with the goal of greatly improving the performance of the student network.
Ranked #13 on
Knowledge Distillation
on CIFAR-100
Bidirectional Projection Network for Cross Dimension Scene Understanding
Via the \emph{BPM}, complementary 2D and 3D information can interact with each other in multiple architectural levels, such that advantages in these two visual domains can be combined for better scene recognition.
Ranked #19 on
Semantic Segmentation
on ScanNet
PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds
The key of PAConv is to construct the convolution kernel by dynamically assembling basic weight matrices stored in Weight Bank, where the coefficients of these weight matrices are self-adaptively learned from point positions through ScoreNet.
Ranked #2 on
Point Cloud Segmentation
on PointCloud-C
General Adversarial Defense via Pixel Level and Feature Level Distribution Alignment
Specifically, compared with previous methods, we propose a more efficient pixel-level training constraint to weaken the hardness of aligning adversarial samples to clean samples, which can thus obviously enhance the robustness on adversarial samples.
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
In this paper, we aim to provide an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task.
Ranked #2 on
Semantic Segmentation
on FoodSeg103
(using extra training data)
Point Transformer
For example, on the challenging S3DIS dataset for large-scale semantic scene segmentation, the Point Transformer attains an mIoU of 70. 4% on Area 5, outperforming the strongest prior model by 3. 3 absolute percentage points and crossing the 70% mIoU threshold for the first time.
Ranked #3 on
3D Semantic Segmentation
on STPLS3D
Fully Convolutional Networks for Panoptic Segmentation
In this paper, we present a conceptually simple, strong, and efficient framework for panoptic segmentation, called Panoptic FCN.
Ranked #1 on
Panoptic Segmentation
on COCO minival
(SQ metric)
Generalized Few-shot Semantic Segmentation
Then, since context is essential for semantic segmentation, we propose the Context-Aware Prototype Learning (CAPL) that significantly improves performance by 1) leveraging the co-occurrence prior knowledge from support samples, and 2) dynamically enriching contextual information to the classifier, conditioned on the content of each query image.
Prior Guided Feature Enrichment Network for Few-Shot Segmentation
It consists of novel designs of (1) a training-free prior mask generation method that not only retains generalization power but also improves model performance and (2) Feature Enrichment Module (FEM) that overcomes spatial inconsistency by adaptively enriching query features with support features and prior masks.
Ranked #69 on
Few-Shot Semantic Segmentation
on COCO-20i (1-shot)
Exploring Self-attention for Image Recognition
Recent work has shown that self-attention can serve as a basic building block for image recognition models.
PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation
Instance segmentation is an important task for scene understanding.
Ranked #5 on
3D Instance Segmentation
on STPLS3D
Dynamic Divide-and-Conquer Adversarial Training for Robust Semantic Segmentation
Adversarial training is promising for improving robustness of deep neural networks towards adversarial perturbations, especially on the classification task.
GridMask Data Augmentation
Then we show limitation of existing information dropping algorithms and propose our structured method, which is simple and yet very effective.
Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation
To incorporate point features in the edge branch, we establish a hierarchical graph framework, where the graph is initialized from a coarse layer and gradually enriched along the point decoding process.
Ranked #43 on
Semantic Segmentation
on S3DIS Area5
Region Refinement Network for Salient Object Detection
Albeit intensively studied, false prediction and unclear boundaries are still major issues of salient object detection.
PointWeb: Enhancing Local Neighborhood Features for Point Cloud Processing
Unlike previous work, we densely connect each point with every other in a local neighborhood, aiming to specify feature of each point based on the local region characteristics for better representing the region.
Ranked #2 on
Semantic Segmentation
on S3DIS Area5
(Number of params metric)
UPSNet: A Unified Panoptic Segmentation Network
More importantly, we introduce a parameter-free panoptic head which solves the panoptic segmentation via pixel-wise classification.
Ranked #3 on
Panoptic Segmentation
on Indian Driving Dataset
PSANet: Point-wise Spatial Attention Network for Scene Parsing
We notice information flow in convolutional neural networks is restricted inside local neighborhood regions due to the physical design of convolutional filters, which limits the overall understanding of complex scenes.
Ranked #51 on
Semantic Segmentation
on Cityscapes test
Compositing-aware Image Search
We present a new image search technique that, given a background image, returns compatible foreground objects for image compositing tasks.
SegStereo: Exploiting Semantic Information for Disparity Estimation
Disparity estimation for binocular stereo images finds a wide range of applications.
Ranked #6 on
Semantic Segmentation
on KITTI Semantic Segmentation
Automatic Real-time Background Cut for Portrait Videos
A spatial-temporal refinement network is developed to further refine the segmentation errors in each frame and ensure temporal coherence in the segmentation map.
ICNet for Real-Time Semantic Segmentation on High-Resolution Images
We focus on the challenging task of real-time semantic segmentation in this paper.
Ranked #11 on
Semantic Segmentation
on BDD100K val
Dichotomous Image Segmentation
Real-Time Semantic Segmentation
+3
Pyramid Scene Parsing Network
Scene parsing is challenging for unrestricted open vocabulary and diverse scenes.
Ranked #4 on
Video Semantic Segmentation
on CamVid
