Search Results for author:
Found 95 papers, 54 papers with code
MikuDance: Animating Character Art with Mixed Motion Dynamics
We propose MikuDance, a diffusion-based pipeline incorporating mixed motion dynamics to animate stylized character art.
MVPaint: Synchronized Multi-View Diffusion for Painting Anything 3D
Furthermore, MVPaint employs a UVR module to improve the texture quality in the UV space, which first performs a UV-space Super-Resolution, followed by a Spatial-aware Seam-Smoothing algorithm for revising spatial texturing discontinuities caused by UV unwrapping.
Polyp-E: Benchmarking the Robustness of Deep Segmentation Models via Polyp Editing
To benchmark the model robustness, we focus on evaluating the robustness of segmentation models on the polyps with various attributes (e. g. location and size) and healthy samples.
Scene123: One Prompt to 3D Scene Generation via Video-Assisted and Consistency-Enhanced MAE
As Artificial Intelligence Generated Content (AIGC) advances, a variety of methods have been developed to generate text, images, videos, and 3D objects from single or multimodal inputs, contributing efforts to emulate human-like cognitive content creation.
Lightweight Model Pre-training via Language Guided Knowledge Distillation
In this paper, for the first time, we introduce language guidance to the distillation process and propose a new method named Language-Guided Distillation (LGD) system, which uses category names of the target downstream task to help refine the knowledge transferred between the teacher and student.
MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers
Recently, 3D assets created via reconstruction and generation have matched the quality of manually crafted assets, highlighting their potential for replacement.
SCKansformer: Fine-Grained Classification of Bone Marrow Cells via Kansformer Backbone and Hierarchical Attention Mechanisms
The model integrates the Kansformer Encoder, SCConv Encoder, and Global-Local Attention Encoder.
Cross-Dimensional Medical Self-Supervised Representation Learning Based on a Pseudo-3D Transformation
This transformation enables seamless integration of 2D and 3D data, and facilitates cross-dimensional self-supervised learning for 3D medical image analysis.
MeshXL: Neural Coordinate Field for Generative 3D Foundation Models
The polygon mesh representation of 3D data exhibits great flexibility, fast rendering speed, and storage efficiency, which is widely preferred in various applications.
GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splatting
In this paper, we propose GaussianTalker, a novel method for audio-driven talking head synthesis based on 3D Gaussian Splatting.
MotionChain: Conversational Motion Controllers via Multimodal Prompts
However, this proficiency remains largely unexplored in other multimodal generative models, particularly in human motion models.
Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
For metric depth estimation, we show that the key to a zero-shot single-view model lies in resolving the metric ambiguity from various camera models and large-scale data training.
Ranked #1 on
Surface Normals Estimation
on NYU Depth v2
(using extra training data)
Generative Motion Stylization of Cross-structure Characters within Canonical Motion Space
Our key insight is to embed motion style into a cross-modality latent space and perceive the cross-structure skeleton topologies, allowing for motion stylization within a canonical motion space.
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
Diffusion models have demonstrated remarkable performance in the domain of text-to-image generation.
MovieLLM: Enhancing Long Video Understanding with AI-Generated Movies
Development of multimodal models has marked a significant step forward in how machines understand videos.
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding Reasoning and Planning
Recent progress in Large Multimodal Models (LMM) has opened up great possibilities for various applications in the field of human-machine interactions.
AppAgent: Multimodal Agents as Smartphone Users
Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks.
Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models
This paper presents Paint3D, a novel coarse-to-fine generative framework that is capable of producing high-resolution, lighting-less, and diverse 2K UV texture maps for untextured 3D meshes conditioned on text or image inputs.
M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts
Furthermore, we establish a new benchmark for assessing the performance of large models in understanding multi-modal 3D prompts.
FaceStudio: Put Your Face Everywhere in Seconds
This study investigates identity-preserving image synthesis, an intriguing task in image generation that seeks to maintain a subject's identity while adding a personalized, stylistic touch.
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
However, developing LMMs that can comprehend, reason, and plan in complex and diverse 3D environments remains a challenging topic, especially considering the demand for understanding permutation-invariant point cloud 3D representations of the 3D scene.
ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model
The advent of large language models, enabling flexibility through instruction-driven approaches, has revolutionized many traditional generative tasks, but large models for 3D data, particularly in comprehensively handling 3D shapes with other modalities, are still under-explored.
ChartLlama: A Multimodal LLM for Chart Understanding and Generation
Next, we introduce ChartLlama, a multi-modal large language model that we've trained using our created dataset.
VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations
Recent advancements in implicit neural representations have contributed to high-fidelity surface reconstruction and photorealistic novel view synthesis.
TapMo: Shape-aware Motion Generation of Skeleton-free Characters
In this work, we present TapMo, a Text-driven Animation Pipeline for synthesizing Motion in a broad spectrum of skeleton-free 3D characters.
Robust Geometry-Preserving Depth Estimation Using Differentiable Rendering
In this paper, we propose a learning framework that trains models to predict geometry-preserving depth without requiring extra data or annotations.
Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning
Moreover, we argue that object localization and description generation require different levels of scene understanding, which could be challenging for a shared set of queries to capture.
IT3D: Improved Text-to-3D Generation with Explicit View Synthesis
Recent strides in Text-to-3D techniques have been propelled by distilling knowledge from powerful large text-to-image diffusion models (LDMs).
StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data
However, these datasets often exhibit domain bias, potentially constraining the generative capabilities of the models.
Ranked #116 on
Visual Question Answering
on MM-Vet
Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image
State-of-the-art (SOTA) monocular metric depth estimation methods can only handle a single camera model and are unable to perform mixed-data training due to the metric ambiguity.
Ranked #25 on
Monocular Depth Estimation
on NYU-Depth V2
(using extra training data)
Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation
We present a novel alignment-before-generation approach to tackle the challenging task of generating general 3D shapes based on 2D images or texts.
MotionGPT: Human Motion as a Foreign Language
Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language.
Ranked #4 on
Motion Captioning
on HumanML3D
STAR Loss: Reducing Semantic Ambiguity in Facial Landmark Detection
To solve this problem, we propose a Self-adapTive Ambiguity Reduction (STAR) loss by exploiting the properties of semantic ambiguity.
Ranked #1 on
Face Alignment
on 300W
Synchro-Transient-Extracting Transform for the Analysis of Signals with Both Harmonic and Impulsive Components
This allows the technique to generate an energy-concentrated TF representation for both harmonics and impulses in the signal.
StyleAvatar3D: Leveraging Image-Text Diffusion Models for High-Fidelity 3D Avatar Generation
The recent advancements in image-text diffusion models have stimulated research interest in large-scale 3D generative models.
Disentangled Pre-training for Image Matting
The pre-training task is designed in a similar manner as image matting, where random trimap and alpha matte are generated to achieve an image disentanglement objective.
Capturing the motion of every joint: 3D human pose and shape estimation with independent tokens
In this paper we present a novel method to estimate 3D human pose and shape from monocular videos.
Ranked #42 on
3D Human Pose Estimation
on 3DPW
SeaFormer++: Squeeze-enhanced Axial Transformer for Mobile Visual Recognition
Since the introduction of Vision Transformers, the landscape of many computer vision tasks (e. g., semantic segmentation), which has been overwhelmingly dominated by CNNs, recently has significantly revolutionized.
A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction
Meanwhile, a new benchmark for several outdoor NeRF-based tasks is established, such as novel view synthesis, surface reconstruction, and multi-modal NeRF.
End-to-End 3D Dense Captioning with Vote2Cap-DETR
Compared with prior arts, our framework has several appealing advantages: 1) Without resorting to numerous hand-crafted components, our method is based on a full transformer encoder-decoder architecture with a learnable vote query driven object decoder, and a caption decoder that produces the dense captions in a set-prediction manner.
Executing your Commands via Motion Diffusion in Latent Space
We study a challenging task, conditional human motion generation, which produces plausible human motion sequences according to various conditional inputs, such as action classes or textual descriptors.
Ranked #2 on
Motion Synthesis
on HumanAct12
Efficient Single-Image Depth Estimation on Mobile Devices, Mobile AI & AIM 2022 Challenge: Report
Various depth estimation models are now widely used on many mobile and IoT devices for image segmentation, bokeh effect rendering, object tracking and many other mobile tasks.
Learning Variational Motion Prior for Video-based Motion Capture
To improve the generalization capacity of prior space, we propose a transformer-based variational autoencoder pretrained over marker-based 3D mocap data, with a novel style-mapping block to boost the generation quality.
Coordinates Are NOT Lonely -- Codebook Prior Helps Implicit Neural 3D Representations
Implicit neural 3D representation has achieved impressive results in surface or scene reconstruction and novel view synthesis, which typically uses the coordinate-based multi-layer perceptrons (MLPs) to learn a continuous scene representation.
Hierarchical Normalization for Robust Monocular Depth Estimation
In this paper, we address monocular depth estimation with deep neural networks.
Ranked #7 on
Monocular Depth Estimation
on ETH3D
D&D: Learning Human Dynamics from Dynamic Camera
In this work, we present D&D (Learning Human Dynamics from Dynamic Camera), which leverages the laws of physics to reconstruct 3D human motion from the in-the-wild videos with a moving camera.
Enhancing Quality of Pose-varied Face Restoration with Local Weak Feature Sensing and GAN Prior
In the asymmetric codec, we adopt a mixed multi-path residual block (MMRB) to gradually extract weak texture features of input images, which can better preserve the original facial features and avoid excessive fantasy.
Preparing data for pathological artificial intelligence with clinical-grade performance
The digital pathology is fundamental of clinical-grade PAI, and the techniques of data standardization and weakly supervised learning methods based on whole slide image (WSI) are effective ways to overcome obstacles of performance reproduction.
Designing thermal radiation metamaterials via hybrid adversarial autoencoder and Bayesian optimization
Designing thermal radiation metamaterials is challenging especially for problems with high degrees of freedom and complex objective.
NTIRE 2022 Challenge on Super-Resolution and Quality Enhancement of Compressed Video: Dataset, Methods and Results
This challenge includes three tracks.
TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation
Although vision transformers (ViTs) have achieved great success in computer vision, the heavy computational cost hampers their applications to dense prediction tasks such as semantic segmentation on mobile devices.
An Energy-concentrated Wavelet Transform for Time Frequency Analysis of Transient Signals
Transient signals are often composed of a series of modes that have multivalued time-dependent instantaneous frequency (IF), which brings challenges to the development of signal processing technology.
Attribute-specific Control Units in StyleGAN for Fine-grained Image Manipulation
Image manipulation with StyleGAN has been an increasing concern in recent years. Recent works have achieved tremendous success in analyzing several semantic latent spaces to edit the attributes of the generated images. However, due to the limited semantic and spatial manipulation precision in these latent spaces, the existing endeavors are defeated in fine-grained StyleGAN image manipulation, i. e., local attribute translation. To address this issue, we discover attribute-specific control units, which consist of multiple channels of feature maps and modulation styles.
Fine-grained Identity Preserving Landmark Synthesis for Face Reenactment
Recent face reenactment works are limited by the coarse reference landmarks, leading to unsatisfactory identity preserving performance due to the distribution gap between the manipulated landmarks and those sampled from a real person.
Sketch Me A Video
Video creation has been an attractive yet challenging task for artists to explore.
A multi-stage semi-supervised improved deep embedded clustering method for bearing fault diagnosis under the situation of insufficient labeled samples
The labeled dataset can be augmented by these pseudo-labeled data and then leveraged to train a bearing fault diagnosis model.
Object-aware Long-short-range Spatial Alignment for Few-Shot Fine-Grained Image Classification
The goal of few-shot fine-grained image classification is to recognize rarely seen fine-grained objects in the query set, given only a few samples of this class in the support set.
Identification of Pediatric Respiratory Diseases Using Fine-grained Diagnosis System
The first stage structuralizes test results by extracting relevant numerical values from clinical notes, and the disease identification stage provides a diagnosis based on text-form clinical notes and the structured data obtained from the first stage.
Shuffle Transformer with Feature Alignment for Video Face Parsing
This is a short technical report introducing the solution of the Team TCParser for Short-video Face Parsing Track of The 3rd Person in Context (PIC) Workshop and Challenge at CVPR 2021.
Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
In this work, we revisit the spatial shuffle as an efficient way to build connections among windows.
Ranked #46 on
Semantic Segmentation
on ADE20K val
Fast and Accurate Single-Image Depth Estimation on Mobile Devices, Mobile AI 2021 Challenge: Report
While many solutions have been proposed for this task, they are usually very computationally expensive and thus are not applicable for on-device inference.
Multi-scale super-resolution generation of low-resolution scanned pathological images
We design a strategy to scan slides with low resolution (5X) and a super-resolution method is proposed to restore the image details when in diagnosis.
BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation
We propose to treat these spatial details and categorical semantics separately to achieve high accuracy and high efficiency for realtime semantic segmentation.
Ranked #1 on
Real-Time Semantic Segmentation
on COCO-Stuff
Context Prior for Scene Segmentation
Given an input image and corresponding ground truth, Affinity Loss constructs an ideal affinity map to supervise the learning of Context Prior.
Ranked #1 on
Scene Understanding
on ADE20K val
High-Order Information Matters: Learning Relation and Topology for Occluded Person Re-Identification
When aligning two groups of local features from two images, we view it as a graph matching problem and propose a cross-graph embedded-alignment (CGEA) layer to jointly learn and embed topology information to local features, and straightly predict similarity score.
State-Aware Tracker for Real-Time Video Object Segmentation
For higher efficiency, SAT takes advantage of the inter-frame consistency and deals with each target object as a tracklet.
Real-Time Semantic Segmentation via Multiply Spatial Fusion Network
Specifically, it achieves 77. 1% Mean IOU on the Cityscapes test dataset with the speed of 41 FPS for a 1024*2048 input, and 75. 4% Mean IOU with the speed of 91 FPS on the Camvid test dataset.
SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines
Following these guidelines, we design our Fully Convolutional Siamese tracker++ (SiamFC++) by introducing both classification and target state estimation branch(G1), classification score without ambiguity(G2), tracking without prior knowledge(G3), and estimation quality score(G4).
Ranked #2 on
Visual Object Tracking
on VOT2017/18
(using extra training data)
Learnable Tree Filter for Structure-preserving Feature Transform
To this end, tree filtering modules are embedded to formulate a unified framework for semantic segmentation.
Double Anchor R-CNN for Human Detection in a Crowd
Double Anchor RPN is developed to capture body and head parts in pairs.
Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection
This report presents our method which wins the nuScenes3D Detection Challenge [17] held in Workshop on Autonomous Driving(WAD, CVPR 2019).
Ranked #6 on
3D Object Detection
on nuScenes LiDAR only
Efficient and Accurate Arbitrary-Shaped Text Detection with Pixel Aggregation Network
Recently, some methods have been proposed to tackle arbitrary-shaped text detection, but they rarely take the speed of the entire pipeline into consideration, which may fall short in practical applications. In this paper, we propose an efficient and accurate arbitrary-shaped text detector, termed Pixel Aggregation Network (PAN), which is equipped with a low computational-cost segmentation head and a learnable post-processing.
Ranked #8 on
Scene Text Detection
on SCUT-CTW1500
Attention-Based Multi-Context Guiding for Few-Shot Semantic Segmentation
Few-shot learning is a nascent research topic, motivated by the fact that traditional deep learning methods require tremen- dous amounts of data.
Ranked #1 on
Few-Shot Semantic Segmentation
on Pascal5i
TACNet: Transition-Aware Context Network for Spatio-Temporal Action Detection
In this paper, we define these ambiguous samples as "transitional states", and propose a Transition-Aware Context Network (TACNet) to distinguish transitional states.
Ranked #7 on
Action Detection
on J-HMDB
Shape Robust Text Detection with Progressive Scale Expansion Network
Due to the fact that there are large geometrical margins among the minimal scale kernels, our method is effective to split the close text instances, making it easier to use segmentation-based methods to detect arbitrary-shaped text instances.
Ranked #12 on
Scene Text Detection
on SCUT-CTW1500
ThunderNet: Towards Real-time Generic Object Detection
In this paper, we investigate the effectiveness of two-stage detectors in real-time generic detection and propose a lightweight two-stage detector named ThunderNet.
Ranked #15 on
Object Detection
on PASCAL VOC 2007
An End-to-End Network for Panoptic Segmentation
Panoptic segmentation, which needs to assign a category label to each pixel and segment each object instance simultaneously, is a challenging topic.
WIDER Face and Pedestrian Challenge 2018: Methods and Results
This paper presents a review of the 2018 WIDER Challenge on Face and Pedestrian.
Rethinking on Multi-Stage Networks for Human Pose Estimation
Existing pose estimation approaches fall into two categories: single-stage and multi-stage methods.
Ranked #1 on
Pose Estimation
on COCO minival
Scene Text Detection with Supervised Pyramid Context Network
We propose a supervised pyramid context network (SPCNET) to precisely locate text regions while suppressing false positives.
Ranked #2 on
Scene Text Detection
on ICDAR 2013
Modeling Local Geometric Structure of 3D Point Clouds using Geo-CNN
This encourages the network to preserve the geometric structure in Euclidean space throughout the feature extraction hierarchy.
Associating Inter-Image Salient Instances for Weakly Supervised Semantic Segmentation
We also combine our method with Mask R-CNN for instance segmentation, and demonstrated for the first time the ability of weakly supervised instance segmentation using only keyword annotations.
Ranked #6 on
Image-level Supervised Instance Segmentation
on COCO test-dev
(using extra training data)
DetNet: Design Backbone for Object Detection
(1) Recent object detectors like FPN and RetinaNet usually involve extra stages against the task of image classification to handle the objects with various scales.
BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation
Semantic segmentation requires both rich spatial information and sizeable receptive field.
Ranked #4 on
Semantic Segmentation
on SkyScapes-Dense
Dichotomous Image Segmentation
Real-Time Semantic Segmentation
+2
CrowdHuman: A Benchmark for Detecting Human in a Crowd
There are a total of $470K$ human instances from the train and validation subsets, and $~22. 6$ persons per image, with various kinds of occlusions in the dataset.
Ranked #7 on
Pedestrian Detection
on Caltech
(using extra training data)
Learning a Discriminative Feature Network for Semantic Segmentation
Most existing methods of semantic segmentation still suffer from two aspects of challenges: intra-class inconsistency and inter-class indistinction.
Ranked #5 on
Semantic Segmentation
on PASCAL VOC 2012 test
SFace: An Efficient Network for Face Detection in Large Scale Variations
A new dataset called 4K-Face is also introduced to evaluate the performance of face detection with extreme large scale variations.
DetNet: A Backbone network for Object Detection
Due to the gap between the image classification and object detection, we propose DetNet in this paper, which is a novel backbone network specifically designed for object detection.
SOT for MOT
Another contribution is that we show with a deep learning based appearance model, it is easy to associate detections of the same object efficiently and also with high accuracy.
Face Attention Network: An Effective Face Detector for the Occluded Faces
The performance of face detection has been largely improved with the development of convolutional neural network.
Ranked #1 on
Occluded Face Detection
on MAFA
Light-Head R-CNN: In Defense of Two-Stage Object Detector
More importantly, simply replacing the backbone with a tiny network (e. g, Xception), our Light-Head R-CNN gets 30. 7 mmAP at 102 FPS on COCO, significantly outperforming the single-stage, fast detectors like YOLO and SSD on both speed and accuracy.
Cascaded Pyramid Network for Multi-Person Pose Estimation
In this paper, we present a novel network structure called Cascaded Pyramid Network (CPN) which targets to relieve the problem from these "hard" keypoints.
Ranked #4 on
Keypoint Detection
on COCO test-challenge
MegDet: A Large Mini-Batch Object Detector
The improvements in recent CNN-based object detection works, from R-CNN [11], Fast/Faster R-CNN [10, 31] to recent Mask R-CNN [14] and RetinaNet [24], mainly come from new network, new framework, or novel loss design.
Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network
One of recent trends [30, 31, 14] in network architec- ture design is stacking small filters (e. g., 1x1 or 3x3) in the entire network because the stacked small filters is more ef- ficient than a large kernel, given the same computational complexity.
Ranked #8 on
Semantic Segmentation
on PASCAL VOC 2012 val
Fast Action Proposals for Human Action Detection and Search
Assuming each action is performed by a human with meaningful motion, both appearance and motion cues are utilized to measure the actionness of the video tubes.
