Search Results for author:
Found 39 papers, 7 papers with code
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models.
UNIT: Unifying Image and Text Recognition in One Vision Encoder
Starting with a vision encoder pre-trained with image recognition tasks, UNIT introduces a lightweight language decoder for predicting text outputs and a lightweight vision decoder to prevent catastrophic forgetting of the original image encoding capabilities.
EasyControl: Transfer ControlNet to Video Diffusion for Controllable Generation and Interpolation
Specifically, for the sketch-to-video generation task, EasyControl achieves an improvement of 152. 0 on FVD and 19. 9 on IS, respectively, in UCF101 compared with VideoComposer.
HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models
High-resolution inputs enable Large Vision-Language Models (LVLMs) to discern finer visual details, enhancing their comprehension capabilities.
HumanRefiner: Benchmarking Abnormal Human Generation and Refining with Coarse-to-fine Pose-Reversible Guidance
To address this issue, we introduce AbHuman, the first large-scale synthesized human benchmark focusing on anatomical anomalies.
DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection
This is followed by a fine-tuning stage that leverages a small number of high-resolution samples to further enhance detection performance.
Ranked #2 on
Object Detection
on ODinW Full-Shot 13 Tasks
LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Model
Specifically, an inter-layer attention module is designed to encourage information exchange and learning between layers, while a text-guided intra-layer attention module incorporates layer-specific prompts to direct the specific-content generation for each layer.
NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning
Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions.
From Summary to Action: Enhancing Large Language Models for Complex Tasks with Open World APIs
In this work, we introduce a novel tool invocation pipeline designed to control massive real-world APIs.
Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Experts
It also obtains new state-of-the-art self-supervised learning results on detection and segmentation.
PanGu-Draw: Advancing Resource-Efficient Text-to-Image Synthesis with Time-Decoupled Training and Reusable Coop-Diffusion
Current large-scale diffusion models represent a giant leap forward in conditional image synthesis, capable of interpreting diverse cues like text, human poses, and edges.
G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model
We first analyze the limitations of current Multimodal Large Language Models (MLLMs) in this area: they struggle to accurately comprehending basic geometric elements and their relationships.
Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving
Large vision-language models (VLMs) have garnered increasing interest in autonomous driving areas, due to their advanced capabilities in complex reasoning tasks essential for highly autonomous vehicle behavior.
Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
The rapid development of large language models (LLMs) has not only provided numerous opportunities but also presented significant challenges.
Implicit Concept Removal of Diffusion Models
Subsequently, the model is optimized to identify and disentangle this information, which is then adopted as negative prompts during generation.
HiLM-D: Towards High-Resolution Understanding in Multimodal Large Language Models for Autonomous Driving
For the first time, we leverage singular multimodal large language models (MLLMs) to consolidate multiple autonomous driving tasks from videos, i. e., the Risk Object Localization and Intention and Suggestion Prediction (ROLISP) task.
Any-Size-Diffusion: Toward Efficient Text-Driven Synthesis for Any-Size HD Images
Stable diffusion, a generative model used in text-to-image synthesis, frequently encounters resolution-induced composition problems when generating images of varying sizes.
GrowCLIP: Data-aware Automatic Model Growing for Large-scale Contrastive Language-Image Pre-training
Compared with the existing methods, GrowCLIP improves 2. 3% average top-1 accuracy on zero-shot image classification of 9 downstream tasks.
DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability
DiffDis first formulates the image-text discriminative problem as a generative diffusion process of the text embedding from the text encoder conditioned on the image.
CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigation
Understanding and following natural language instructions while navigating through complex, real-world environments poses a significant challenge for general-purpose robots.
RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignment
Recent advances in text-to-image diffusion models have achieved remarkable success in generating high-quality, realistic images from textual descriptions.
DetGPT: Detect What You Need via Reasoning
Overall, our proposed paradigm and DetGPT demonstrate the potential for more sophisticated and intuitive interactions between humans and machines.
DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment
This paper presents DetCLIPv2, an efficient and scalable training framework that incorporates large-scale image-text pairs to achieve open-vocabulary object detection (OVD).
Ranked #5 on
Object Detection
on ODinW Full-Shot 13 Tasks
CLIP$^2$: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
Contrastive Language-Image Pre-training, benefiting from large-scale unlabeled text-image pairs, has demonstrated great performance in open-world vision understanding tasks.
Ranked #3 on
Zero-shot 3D Point Cloud Classification
on ScanNetV2
Towards Universal Vision-language Omni-supervised Segmentation
To improve the open-world segmentation ability, we leverage omni-supervised data (i. e., panoptic segmentation data, object detection data, and image-text pairs data) into training, thus enriching the open-world segmentation ability and achieving better segmentation accuracy.
CapDet: Unifying Dense Captioning and Open-World Detection Pretraining
Besides, our CapDet also achieves state-of-the-art performance on dense captioning tasks, e. g., 15. 44% mAP on VG V1. 2 and 13. 98% on the VG-COCO dataset.
Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Driving
Multi-task learning has emerged as a powerful paradigm to solve a range of tasks simultaneously with good efficiency in both computation resources and inference time.
CLIP2: Contrastive Language-Image-Point Pretraining From Real-World Point Cloud Data
Contrastive Language-Image Pre-training, benefiting from large-scale unlabeled text-image pairs, has demonstrated great performance in open-world vision understanding tasks.
NLIP: Noise-robust Language-Image Pre-training
Large-scale cross-modal pre-training paradigms have recently shown ubiquitous success on a wide range of downstream tasks, e. g., zero-shot classification, retrieval and image captioning.
Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection
Inspired by the success of vision-language methods (VLMs) in zero-shot classification, recent works attempt to extend this line of work into object detection by leveraging the localization ability of pre-trained VLMs and generating pseudo labels for unseen classes in a self-training manner.
Generative Negative Text Replay for Continual Vision-Language Pretraining
In this work, we focus on learning a VLP model with sequential chunks of image-text pair data.
DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection
We further design a concept dictionary~(with descriptions) from various online sources and detection datasets to provide prior knowledge for each concept.
Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving
Aiming towards a holistic understanding of multiple downstream tasks simultaneously, there is a need for extracting features with better transferability.
Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding
To bridge the gap between supervised semantic segmentation and real-world applications that acquires one model to recognize arbitrary new concepts, recent zero-shot segmentation attracts a lot of attention by exploring the relationships between unseen and seen object categories, yet requiring large amounts of densely-annotated data with diverse base classes.
Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routing
On the other hand, for existing SSL methods, it is burdensome and infeasible to use different downstream-task-customized datasets in pre-training for different tasks.
ONCE-3DLanes: Building Monocular 3D Lane Detection
We present ONCE-3DLanes, a real-world autonomous driving dataset with lane layout annotation in 3D space.
Laneformer: Object-aware Row-Column Transformers for Lane Detection
We present Laneformer, a conceptually simple yet powerful transformer-based architecture tailored for lane detection that is a long-standing research topic for visual perception in autonomous driving.
CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving
One main reason that impedes the development of truly reliably self-driving systems is the lack of public datasets for evaluating the performance of object detectors on corner cases.
SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving
Experiments show that SODA10M can serve as a promising pre-training dataset for different self-supervised learning methods, which gives superior performance when fine-tuning with different downstream tasks (i. e., detection, semantic/instance segmentation) in autonomous driving domain.
