Search Results for author:
Found 62 papers, 36 papers with code
Personalize Anything for Free with Diffusion Transformer
Personalized image generation aims to produce images of user-specified concepts while enabling flexible editing.
T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation
Based on this taxonomy and prompt set, we build a large-scale T2I dataset with 68K manually annotated images and train an evaluator capable of detecting critical risks that previous work has failed to identify, including risks that even ultra-large proprietary models like GPTs cannot correctly detect.
Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection
The core of our method is to formulate both tasks as a unified set of spatio-temporal constraint satisfaction problems and use VLM-generated code to evaluate them for real-time monitoring.
MV-Adapter: Multi-view Consistent Image Generation Made Easy
To efficiently model the 3D geometric knowledge within the adapter, we introduce innovative designs that include duplicated self-attention layers and parallel attention architecture, enabling the adapter to inherit the powerful priors of the pre-trained models to model the novel 3D knowledge.
MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation
This paper introduces MIDI, a novel paradigm for compositional 3D scene generation from a single image.
Ranked #1 on
Speech Separation
on GRID corpus (mixed-speech)
A Topic-level Self-Correctional Approach to Mitigate Hallucinations in MLLMs
Through a deconfounded strategy that replaces each topic within the response with the best or worst alternatives generated by the model itself, TPO creates more contrasting pairwise preference feedback, enhancing the feedback quality without human or proprietary model intervention.
WorldSimBench: Towards Video Generation Models as World Simulators
WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation.
Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
However, training these two stages separately leads to significant data bias in the inference phase, thus affecting the quality of reconstructed results.
From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation
Addressing this, we introduce Parts2Whole, a novel framework designed for generating customized portraits from multiple reference images, including pose images and various aspects of human appearance.
Self-Supervised Monocular Depth Estimation in the Dark: Towards Data Distribution Compensation
In this paper, we propose a self-supervised nighttime monocular depth estimation method that does not use any night images during training.
RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents
Achieving generalizability in solving out-of-distribution tasks is one of the ultimate goals of learning robotic manipulation.
Assessment of Multimodal Large Language Models in Alignment with Human Values
Large Language Models (LLMs) aim to serve as versatile assistants aligned with human values, as defined by the principles of being helpful, honest, and harmless (hhh).
MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control
It is a long-lasting goal to design a generalist-embodied agent that can follow diverse instructions in human-like ways.
Data-Free Generalized Zero-Shot Learning
Firstly, to recover the virtual features of the base data, we model the CLIP features of base class images as samples from a von Mises-Fisher (vMF) distribution based on the pre-trained classifier.
From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities
Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents.
Multi-modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation
The point affinity proposed in this paper is characterized by features from multiple modalities (e. g., point cloud and RGB), and is further refined by normalizing the classifier weights to alleviate the detrimental effects of long-tailed distribution without the need of the prior of category distribution.
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception
It is a long-lasting goal to design an embodied system that can solve long-horizon open-world tasks in human-like ways.
EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion
Generating multiview images from a single view facilitates the rapid generation of a 3D mesh conditioned on a single image.
ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
We will publicly release all the detailed implementations for further analysis, as well as an easy-to-use modular toolkit for the integration of new recipes and models, so that ChEF can be a growing evaluation framework for the MLLM community.
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
While this phenomenon has been overlooked in previous work, we propose a novel and extensible framework, called Octavius, for comprehensive studies and experimentation on multimodal learning with Multimodal Large Language Models (MLLMs).
Stable Diffusion Reference Only: Image Prompt and Blueprint Jointly Guided Multi-Condition Diffusion Model for Secondary Painting
Therefore, we present a new method in this paper, Stable Diffusion Reference Only, a images-to-image self-supervised model that uses only two types of conditional images for precise control generation to accelerate secondary painting.
Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter
The pre-trained text-image discriminative models, such as CLIP, has been explored for open-vocabulary semantic segmentation with unsatisfactory results due to the loss of crucial localization information and awareness of object shapes.
LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark
To the best of our knowledge, we present one of the very first open-source endeavors in the field, LAMM, encompassing a Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark.
Siamese DETR
In this work, we present Siamese DETR, a Siamese self-supervised pretraining approach for the Transformer architecture in DETR.
VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud
Since 2D images provide rich semantics and scene graphs are in nature coped with languages, in this study, we propose Visual-Linguistic Semantics Assisted Training (VL-SAT) scheme that can significantly empower 3DSSG prediction models with discrimination about long-tailed and ambiguous semantic relations.
Ranked #1 on
3d scene graph generation
on 3DSSG
(using extra training data)
Fast-BEV: A Fast and Strong Bird's-Eye View Perception Baseline
Our Fast-BEV consists of five parts, We novelly propose (1) a lightweight deployment-friendly view transformation which fast transfers 2D image feature to 3D voxel space, (2) an multi-scale image encoder which leverages multi-scale information for better performance, (3) an efficient BEV encoder which is particularly designed to speed up on-vehicle inference.
Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baseline
We also propose a new 3D VQA framework to effectively predict the completely visually grounded and explainable answer.
Improving RGB-D Point Cloud Registration by Learning Multi-scale Local Linear Transformation
In addition to previous methods that seek correspondences by hand-crafted or learnt geometric features, recent point cloud registration methods have tried to apply RGB-D data to achieve more accurate correspondence.
SketchSampler: Sketch-based 3D Reconstruction via View-dependent Depth Sampling
Reconstructing a 3D shape based on a single sketch image is challenging due to the large domain gap between a sparse, irregular sketch and a regular, dense 3D shape.
X-Learner: Learning Cross Sources and Tasks for Universal Visual Representation
2) Squeeze Stage: X-Learner condenses the model to a reasonable size and learns the universal and generalizable representation for various tasks transferring.
Bamboo: Building Mega-Scale Vision Dataset Continually with Human-Machine Synergy
This work thus proposes a novel active learning framework for realistic dataset annotation.
Ranked #1 on
Image Classification
on Food-101
(using extra training data)
3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point Clouds
Observing that the 3D captioning task and the 3D grounding task contain both shared and complementary information in nature, in this work, we propose a unified framework to jointly solve these two distinct but closely related tasks in a synergistic fashion, which consists of both shared task-agnostic modules and lightweight task-specific modules.
ForgeryNet -- Face Forgery Analysis Challenge 2021: Methods and Results
The rapid progress of photorealistic synthesis techniques has reached a critical point where the boundary between real and manipulated images starts to blur.
VoteHMR: Occlusion-Aware Voting Network for Robust 3D Human Mesh Recovery from Partial Point Clouds
3D human mesh recovery from point clouds is essential for various tasks, including AR/VR and human behavior understanding.
Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds
Inspired by the back-tracing strategy in the conventional Hough voting methods, in this work, we introduce a new 3D object detection method, named as Back-tracing Representative Points Network (BRNet), which generatively back-traces the representative points from the vote centers and also revisits complementary seed points around these generated points, so as to better capture the fine local structural features surrounding the potential objects from the raw point clouds.
Ranked #19 on
3D Object Detection
on SUN-RGBD val
DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer
In this paper, we reformulate it by a two-stage process, ie, a key pose generation and then an in-between parametric motion curve prediction, where the key poses are easier to be synchronized with the music beats and the parametric curves can be efficiently regressed to render fluent rhythm-aligned movements.
ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis
To counter this emerging threat, we construct the ForgeryNet dataset, an extremely large face forgery dataset with unified annotations in image- and video-level data across four tasks: 1) Image Forgery Classification, including two-way (real / fake), three-way (real / fake with identity-replaced forgery approaches / fake with identity-remained forgery approaches), and n-way (real and 15 respective forgery approaches) classification.
3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds
Visual grounding on 3D point clouds is an emerging vision and language task that benefits various applications in understanding the 3D visual world.
StyleFormer: Real-Time Arbitrary Style Transfer via Parametric Style Composition
In this work, we propose a new feed-forward arbitrary style transfer method, referred to as StyleFormer, which can simultaneously fulfill fine-grained style diversity and semantic content coherency.
PV-NAS: Practical Neural Architecture Search for Video Recognition
Recently, deep learning has been utilized to solve video recognition problem due to its prominent representation ability.
Adaptive Gradient Method with Resilience and Momentum
Several variants of stochastic gradient descent (SGD) have been proposed to improve the learning effectiveness and efficiency when training deep neural networks, among which some recent influential attempts would like to adaptively control the parameter-wise learning rate (e. g., Adam and RMSProp).
Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues
As realistic facial manipulation technologies have achieved remarkable progress, social concerns about potential malicious abuse of these technologies bring out an emerging research topic of face forgery detection.
Unsupervised Domain Expansion from Multiple Sources
Given an existing system learned from previous source domains, it is desirable to adapt the system to new domains without accessing and forgetting all the previous domains in some applications.
Powering One-shot Topological NAS with Stabilized Share-parameter Proxy
Specifically, the difficulties for architecture searching in such a complex space has been eliminated by the proposed stabilized share-parameter proxy, which employs Stochastic Gradient Langevin Dynamics to enable fast shared parameter sampling, so as to achieve stabilized measurement of architecture performance even in search space with complex topological structures.
Morphing and Sampling Network for Dense Point Cloud Completion
3D point cloud completion, the task of inferring the complete geometric shape from a partial point cloud, has been attracting attention in the community.
Ranked #8 on
Point Cloud Completion
on ShapeNet
Improving Pedestrian Attribute Recognition With Weakly-Supervised Multi-Scale Attribute-Specific Localization
To predict the existence of a particular attribute, it is demanded to localize the regions related to the attribute.
Ranked #1 on
Pedestrian Attribute Recognition
on RAP
Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry Towards Monocular Deep SLAM
In this paper we tackle the joint learning problem of keyframe detection and visual odometry towards monocular visual SLAM systems.
CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
Text-image cross-modal retrieval is a challenging task in the field of language and vision.
Ranked #9 on
Image Retrieval
on Flickr30K 1K test
Visibility Constrained Generative Model for Depth-based 3D Facial Pose Tracking
In this paper, we propose a generative framework that unifies depth-based 3D facial pose tracking and face model adaptation on-the-fly, in the unconstrained scenarios with heavy occlusions and arbitrary facial expression variations.
Context and Attribute Grounded Dense Captioning
Dense captioning aims at simultaneously localizing semantic regions and describing these regions-of-interest (ROIs) with short phrases or sentences in natural language.
Ranked #3 on
Dense Captioning
on Visual Genome
Semantics Disentangling for Text-to-Image Generation
Synthesizing photo-realistic images from text descriptions is a challenging problem.
GS3D: An Efficient 3D Object Detection Framework for Autonomous Driving
We present an efficient 3D object detection framework based on a single RGB image in the scenario of autonomous driving.
Ranked #18 on
Vehicle Pose Estimation
on KITTI Cars Hard
Video Generation from Single Semantic Label Map
This paper proposes the novel task of video generation conditioned on a SINGLE semantic label map, which provides a good balance between flexibility and quality in the generation process.
Unsupervised Bi-directional Flow-based Video Generation from one Snapshot
Imagining multiple consecutive frames given one single snapshot is challenging, since it is difficult to simultaneously predict diverse motions from a single image and faithfully generate novel frames without visual distortions.
Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection
Specifically, given the image-level annotations, (1) we first develop a weakly-supervised detection (WSD) model, and then (2) construct an end-to-end multi-label image classification framework augmented by a knowledge distillation module that guides the classification model by the WSD model according to the class-level predictions for the whole image and the object-level visual features for object RoIs.
Ranked #9 on
Multi-Label Classification
on NUS-WIDE
Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
We show that by encouraging deep message propagation and interactions between local object features and global predicate features, one can achieve compelling performance in recognizing complex relationships without using any linguistic priors.
Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
Zero-shot artistic style transfer is an important image synthesis problem aiming at transferring arbitrary style into content images.
Exploring Disentangled Feature Representation Beyond Face Identification
This paper proposes learning disentangled but complementary face features with minimal supervision by face identification.
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition
In this study, we introduce a novel compact motion representation for video action recognition, named Optical Flow guided Feature (OFF), which enables the network to distill temporal information through a fast and robust approach.
Ranked #39 on
Action Recognition
on UCF101
HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis
Pedestrian analysis plays a vital role in intelligent video surveillance and is a key component for security-centric computer vision systems.
Ranked #2 on
Pedestrian Attribute Recognition
on RAP
A Generative Model for Depth-Based Robust 3D Facial Pose Tracking
We consider the problem of depth-based robust 3D facial pose tracking under unconstrained scenarios with heavy occlusions and arbitrary facial expression variations.
