Search Results for author:
Found 82 papers, 41 papers with code
UVRM: A Scalable 3D Reconstruction Model from Unposed Videos
Training 3D reconstruction models with 2D visual data traditionally requires prior knowledge of camera poses for the training samples, a process that is both time-consuming and prone to errors.
Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Composition
For video-to-audio (VTA) tasks, most existing methods require training a timestamp detector to synchronize video events with the generated audio, a process that can be tedious and time-consuming.
ChatCam: Empowering Camera Control through Conversational AI
Cinematographers adeptly capture the essence of the world, crafting compelling visual narratives through intricate camera movements.
VP-LLM: Text-Driven 3D Volume Completion with Large Language Models through Patchification
Recent conditional 3D completion works have mainly relied on CLIP or BERT to encode textual information, which cannot support complex instruction.
Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling
Extensions of Neural Radiance Fields (NeRFs) to model dynamic scenes have enabled their near photo-realistic, free-viewpoint rendering.
Ranked #1 on
Novel View Synthesis
on .
Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs
This is accomplished by encoding and quantizing motions into discrete tokens that align with the language model's vocabulary.
C3LLM: Conditional Multimodal Content Generation Using Large Language Models
C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities, synthesizing the given conditional information, and making multimodal generation in a discrete manner.
FED-NeRF: Achieve High 3D Consistency and Temporal Coherence for Face Video Editing on Dynamic NeRF
However, achieving simultaneously multi-view consistency and temporal coherence while editing video sequences remains a formidable challenge.
Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models
Second and the remaining problem is thus 3D multiview consistency among all completed images, now guided by the seed images and their 3D proxies.
Prompt2NeRF-PIL: Fast NeRF Generation via Pretrained Implicit Latent
This paper explores promptable NeRF generation (e. g., text prompt or single image prompt) for direct conditioning and fast generation of NeRF parameters for the underlying 3D scenes, thus undoing complex intermediate steps while providing full 3D generation with conditional control.
DragVideo: Interactive Drag-style Video Editing
The main issues are: 1) how to perform direct and accurate user control in editing; 2) how to execute editings like changing shape, expression, and layout without unsightly distortion and artifacts to the edited content; and 3) how to maintain spatio-temporal consistency of video after editing.
SANeRF-HQ: Segment Anything for NeRF in High Quality
Recently, the Segment Anything Model (SAM) has showcased remarkable capabilities of zero-shot segmentation, while NeRF (Neural Radiance Fields) has gained popularity as a method for various 3D problems beyond novel view synthesis.
C3Net: Compound Conditioned ControlNet for Multimodal Content Generation
Specifically, C3Net first aligns the conditions from multi-modalities to the same semantic latent space using modality-specific encoders based on contrastive training.
Stable Segment Anything Model
Thus, our solution, termed Stable-SAM, offers several advantages: 1) improved SAM's segmentation stability across a wide range of prompt qualities, while 2) retaining SAM's powerful promptable segmentation efficiency and generality, with 3) minimal learnable parameters (0. 08 M) and fast adaptation (by 1 training epoch).
InceptionHuman: Controllable Prompt-to-NeRF for Photorealistic 3D Human Generation
This paper presents InceptionHuman, a prompt-to-NeRF framework that allows easy control via a combination of prompts in different modalities (e. g., text, poses, edge, segmentation map, etc) as inputs to generate photorealistic 3D humans.
EgoPCA: A New Framework for Egocentric Hand-Object Interaction Understanding
With the surge in attention to Egocentric Hand-Object Interaction (Ego-HOI), large-scale datasets such as Ego4D and EPIC-KITCHENS have been proposed.
Segment Anything Meets Point Tracking
The Segment Anything Model (SAM) has established itself as a powerful zero-shot image segmentation model, enabled by efficient point-centric annotation and prompt-based models.
UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks
Our thorough studies validate that models pre-trained as such can learn rich representations of both modalities, improving their ability to understand how images and text relate to each other.
Segment Anything in High Quality
HQ-SAM is only trained on the introduced detaset of 44k masks, which takes only 4 hours on 8 GPUs.
Ranked #1 on
Zero-Shot Instance Segmentation
on LVIS v1.0 val
FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models
The ability to create high-quality 3D faces from a single image has become increasingly important with wide applications in video conferencing, AR/VR, and advanced video editing in movie industries.
Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillation
We believe this paradigm will open up new avenues in the dynamics of distillation and pave the way for efficient dataset distillation.
Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstruction
Novel view synthesis via Neural Radiance Fields (NeRFs) or 3D Gaussian Splatting (3DGS) typically necessitates dense observations with hundreds of input images to circumvent artifacts.
Registering Neural Radiance Fields as 3D Density Images
No significant work has been done to directly merge two partially overlapping scenes using NeRF representations.
Instance Neural Radiance Field
This paper presents one of the first learning-based NeRF 3D instance segmentation pipelines, dubbed as {\bf \inerflong}, or \inerf.
Mask-Free Video Instance Segmentation
A consistency loss is then enforced on the found matches.
Clean-NeRF: Reformulating NeRF to account for View-Dependent Observations
This paper analyzes the NeRF's struggles in such settings and proposes Clean-NeRF for accurate 3D reconstruction and novel view rendering in complex scenes.
Ultrahigh Resolution Image/Video Matting With Spatio-Temporal Sparsity
Instead, our method resorts to spatial and temporal sparsity for solving general UHR matting.
ONeRF: Unsupervised 3D Object Segmentation from Multiple Views
We present ONeRF, a method that automatically segments and reconstructs object instances in 3D from multi-view RGB images without any additional manual annotations.
NeRF-RPN: A general framework for object detection in NeRFs
This paper presents the first significant object detection framework, NeRF-RPN, which directly operates on NeRF.
FLNeRF: 3D Facial Landmarks Estimation in Neural Radiance Fields
This paper presents the first significant work on directly predicting 3D face landmarks on neural radiance fields (NeRFs).
H-VFI: Hierarchical Frame Interpolation for Videos with Large Motions
The learnt deformable kernel is then utilized in convolving the input frames for predicting the interpolated frame.
Normalization Perturbation: A Simple Domain Generalization Method for Real-World Domain Shifts
Thus, we propose to perturb the channel statistics of source domain features to synthesize various latent styles, so that the trained deep model can perceive diverse potential domains and generalizes well even without observations of target domain data in training.
Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation
The core of our method is a novel propagation strategy for individual objects' radiance fields with a bidirectional photometric loss, enabling an unsupervised partitioning of a scene into salient or meaningful regions corresponding to different object instances.
Occlusion-Aware Instance Segmentation via BiLayer Network Architectures
Unlike previous instance segmentation methods, we model image formation as a composition of two overlapping layers, and propose Bilayer Convolutional Network (BCNet), where the top layer detects occluding objects (occluders) and the bottom layer infers partially occluded instances (occludees).
Video Mask Transfiner for High-Quality Video Instance Segmentation
While Video Instance Segmentation (VIS) has seen rapid progress, current approaches struggle to predict high-quality masks with accurate boundary details.
Ranked #1 on
Video Instance Segmentation
on HQ-YTVIS
Self-Support Few-Shot Semantic Segmentation
Motivated by the simple Gestalt principle that pixels belonging to the same object are more similar than those to different objects of same class, we propose a novel self-support matching strategy to alleviate this problem, which uses query prototypes to match query features, where the query prototypes are collected from high-confidence query predictions.
Ranked #13 on
Few-Shot Semantic Segmentation
on PASCAL-5i (5-Shot)
GCoNet+: A Stronger Group Collaborative Co-Salient Object Detector
In this paper, we present a novel end-to-end group collaborative learning network, termed GCoNet+, which can effectively and efficiently (250 fps) identify co-salient objects in natural scenes.
Ranked #1 on
Co-Salient Object Detection
on CoCA
Human Instance Matting via Mutual Guidance and Multi-Instance Refinement
A new instance matting metric called instance matting quality (IMQ) is proposed, which addresses the absence of a unified and fair means of evaluation emphasizing both instance recognition and matting quality.
Interactiveness Field in Human-Object Interactions
Human-Object Interaction (HOI) detection plays a core role in activity understanding.
HAA4D: Few-Shot Human Atomic Action Recognition via 3D Spatio-Temporal Skeletal Alignment
All training and testing 3D skeletons in HAA4D are globally aligned, using a deep alignment model to the same global space, making each skeleton face the negative z-direction.
Mask Transfiner for High-Quality Instance Segmentation
Instead of operating on regular dense tensors, our Mask Transfiner decomposes and represents the image regions as a quadtree.
Ranked #1 on
Instance Segmentation
on BDD100K val
Occlusion-Aware Video Object Inpainting
To facilitate this new research, we construct the first large-scale video object inpainting benchmark YouTube-VOI to provide realistic occlusion scenarios with both occluded and visible object masks available.
Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation
We propose Prototypical Cross-Attention Network (PCAN), capable of leveraging rich spatio-temporal information for online multiple object tracking and segmentation.
Ranked #1 on
Video Instance Segmentation
on BDD100K val
Multi-Object Tracking and Segmentation
Multiple Object Track and Segmentation
+3
Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation
This paper presents a simple yet effective approach to modeling space-time correspondences in the context of video object segmentation.
Ranked #7 on
Video Object Segmentation
on YouTube-VOS 2019
Semantic Segmentation
Semi-Supervised Video Object Segmentation
+1
Few-Shot Video Object Detection
We introduce Few-Shot Video Object Detection (FSVOD) with three contributions to real-world visual learning challenge in our highly diverse and dynamic world: 1) a large-scale video dataset FSVOD-500 comprising of 500 classes with class-balanced videos in each category for few-shot learning; 2) a novel Tube Proposal Network (TPN) to generate high-quality video tube proposals for aggregating feature representation for the target video object which can be highly dynamic; 3) a strategically improved Temporal Matching Network (TMN+) for matching representative query tube features with better discriminative ability thus achieving higher diversity.
Deep Video Matting via Spatio-Temporal Alignment and Aggregation
Despite the significant progress made by deep learning in natural image matting, there has been so far no representative work on deep learning for video matting due to the inherent technical challenges in reasoning temporal domain and lack of large-scale video matting datasets.
Semantic Image Matting
Specifically, we consider and learn 20 classes of matting patterns, and propose to extend the conventional trimap to semantic trimap.
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers
Segmenting highly-overlapping objects is challenging, because typically no distinction is made between real object contours and occlusion boundaries.
Ranked #1 on
Instance Segmentation
on KINS
Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion
We present Modular interactive VOS (MiVOS) framework which decouples interaction-to-mask and mask propagation, allowing for higher generalizability and better performance.
Ranked #1 on
Interactive Video Object Segmentation
on DAVIS 2017
(using extra training data)
Interactive Video Object Segmentation
Semantic Segmentation
+2
Semi-Supervised Few-Shot Atomic Action Recognition
Despite excellent progress has been made, the performance on action recognition still heavily relies on specific datasets, which are difficult to extend new action classes due to labor-intensive labeling.
HAA500: Human-Centric Atomic Action Dataset with Curated Videos
We contribute HAA500, a manually annotated human-centric atomic action dataset for action recognition on 500 classes with over 591K labeled frames.
Ranked #1 on
Action Recognition
on HAA500
Pose-Guided High-Resolution Appearance Transfer via Progressive Training
We propose a novel pose-guided appearance transfer network for transferring a given reference appearance to a target pose in unprecedented image resolution (1024 * 1024), given respectively an image of the reference and target person.
GSNet: Joint Vehicle Pose and Shape Reconstruction with Geometrical and Scene-aware Supervision
GSNet utilizes a unique four-way feature extraction and fusion scheme and directly regresses 6DoF poses and shapes in a single forward pass.
Ranked #1 on
Autonomous Driving
on ApolloCar3D
Commonality-Parsing Network across Shape and Appearance for Partially Supervised Instance Segmentation
We propose to learn the underlying class-agnostic commonalities that can be generalized from mask-annotated categories to novel categories.
Ranked #81 on
Instance Segmentation
on COCO test-dev
Fast Video Object Segmentation With Temporal Aggregation Network and Dynamic Template Matching
Significant progress has been made in Video Object Segmentation (VOS), the video object tracking task in its finest level.
Ranked #71 on
Semi-Supervised Video Object Segmentation
on DAVIS 2016
Cascaded deep monocular 3D human pose estimation with evolutionary training data
End-to-end deep representation learning has achieved remarkable accuracy for monocular 3D human pose estimation, yet these models may fail for unseen poses with limited and fixed training data.
Ranked #13 on
Weakly-supervised 3D Human Pose Estimation
on Human3.6M
One-Shot Object Detection without Fine-Tuning
Deep learning has revolutionized object detection thanks to large-scale datasets, but their object categories are still arguably very limited.
CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement
In this paper, we propose a novel approach to address the high-resolution segmentation problem without using any high-resolution training data.
Ranked #1 on
Semantic Segmentation
on BIG
(using extra training data)
Template-Instance Loss for Offline Handwritten Chinese Character Recognition
The long-standing challenges for offline handwritten Chinese character recognition (HCCR) are twofold: Chinese characters can be very diverse and complicated while similarly looking, and cursive handwriting (due to increased writing speed and infrequent pen lifting) makes strokes and even characters connected together in a flowing manner.
Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector
To train our network, we contribute a new dataset that contains 1000 categories of various objects with high-quality annotations.
Ranked #23 on
Few-Shot Object Detection
on MS-COCO (10-shot)
DAWN: Dual Augmented Memory Network for Unsupervised Video Object Tracking
Our Dual Augmented Memory Network (DAWN) is unique in remembering both target and background, and using an improved attention LSTM memory to guide the focus on memorized features.
FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation
In this paper, we are interested in few-shot object segmentation where the number of annotated training examples are limited to 5 only.
Ranked #21 on
Few-Shot Semantic Segmentation
on FSS-1000 (5-shot)
StableNet: Semi-Online, Multi-Scale Deep Video Stabilization
Video stabilization algorithms are of greater importance nowadays with the prevalence of hand-held devices which unavoidably produce videos with undesirable shaky motions.
LADN: Local Adversarial Disentangling Network for Facial Makeup and De-Makeup
Central to our method are multiple and overlapping local adversarial discriminators in a content-style disentangling network for achieving local detail transfer between facial images, with the use of asymmetric loss functions for dramatic makeup styles with high-frequency details.
Annotation-Free and One-Shot Learning for Instance Segmentation of Homogeneous Object Clusters
We propose a novel approach for instance segmen- tation given an image of homogeneous object clus- ter (HOC).
MAVOT: Memory-Augmented Video Object Tracking
We introduce a one-shot learning approach for video object tracking.
Deep High Dynamic Range Imaging with Large Foreground Motions
In state-of-the-art deep HDR imaging, input images are first aligned using optical flows before merging, which are still error-prone due to occlusion and large motions.
Image Generation from Sketch Constraint Using Contextual GAN
We train a generated adversarial network, i. e, contextual GAN to learn the joint distribution of sketch and the corresponding image by using joint images.
Deep Video Generation, Prediction and Completion of Human Action Sequences
In the second stage, a skeleton-to-image network is trained, which is used to generate a human action video given the complete human pose sequence generated in the first stage.
Ranked #5 on
Human action generation
on NTU RGB+D 2D
Online Video Object Detection Using Association LSTM
Video object detection is a fundamental tool for many applications.
Attribute-Guided Face Generation Using Conditional CycleGAN
We are interested in attribute-guided face generation: given a low-res face input image, an attribute vector that can be extracted from a high-res image (attribute image), our new method generates a high-res face image for the low-res input that satisfies the given attributes.
Beyond Holistic Object Recognition: Enriching Image Understanding with Part States
Important high-level vision tasks such as human-object interaction, image captioning and robotic manipulation require rich semantic descriptions of objects at part level.
Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
We alternate the pruning and retraining to further reduce zero activations in a network.
A Closed-Form Solution to Tensor Voting: Theory and Applications
We prove a closed-form solution to tensor voting (CFTV): given a point set in any dimensions, our closed-form solution provides an exact, continuous and efficient algorithm for computing a structure-aware tensor that simultaneously achieves salient structure detection and outlier attenuation.
Contour Box: Rejecting Object Proposals Without Explicit Closed Contours
Closed contour is an important objectness indicator.
Square Localization for Efficient and Accurate Object Detection
In the testing phase, sliding CNN models are applied which produces a set of response maps that can be effectively filtered by the learned co-presence prior to output the final bounding boxes for localizing an object.
Complexity-Adaptive Distance Metric for Object Proposals Generation
Distance metric plays a key role in grouping superpixels to produce object proposals for object detection.
1-HKUST: Object Detection in ILSVRC 2014
We participated in the object detection track of ILSVRC 2014 and received the fourth place among the 38 teams.
Two-Class Weather Classification
Given a single outdoor image, this paper proposes a collaborative learning approach for labeling it as either sunny or cloudy.
Shadow Removal from Single RGB-D Images
We present the first automatic method to remove shadows from single RGB-D images.
