Search Results for author:
Found 63 papers, 38 papers with code
Coarse Correspondence Elicit 3D Spacetime Understanding in Multimodal Language Model
We introduce Coarse Correspondence, a simple, training-free, effective, and general-purpose visual prompting method to elicit 3D and temporal understanding in multimodal LLMs.
Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arena
In this paper, we introduce Arena Learning, an innovative offline strategy designed to simulate these arena battles using AI-driven annotations to evaluate battle outcomes, thus facilitating the continuous improvement of the target model through both supervised fine-tuning and reinforcement learning.
RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Models
We present RodinHD, which can generate high-fidelity 3D avatars from a portrait image.
Hierarchical Memory for Long Video QA
This paper describes our champion solution to the LOVEU Challenge @ CVPR'24, Track 1 (Long Video VQA).
LIPE: Learning Personalized Identity Prior for Non-rigid Image Editing
In this paper, we explore a novel task: learning the personalized identity prior for text-based non-rigid image editing.
GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation
In this work, we introduce the Geometry-Aware Large Reconstruction Model (GeoLRM), an approach which can predict high-quality assets with 512k Gaussians and 21 input images in only 11 GB GPU memory.
VoCo-LLaMA: Towards Vision Compression with Large Language Models
Previous approaches compress vision tokens with external modules and force LLMs to understand the compressed ones, leading to visual information loss.
Localizing Events in Videos with Multimodal Queries
Our new benchmark aims to evaluate how well models can localize an event given a multimodal semantic query that consists of a reference image, which depicts the event, and a refinement text to adjust the images' semantics.
Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams
Our model is able to process extremely long video streams in real-time and respond to user queries simultaneously.
Ranked #1 on
Zero-Shot Video Question Answer
on MSRVTT-QA
ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulation
To model this process, we design a consistency distillation technique to predict the action sample directly instead of predicting the noise within the vision community for fast convergence in the low-dimensional action manifold.
FlowIE: Efficient Image Enhancement via Rectified Flow
Image enhancement holds extensive applications in real-world scenarios due to complex environments and limitations of imaging devices.
PTM-VQA: Efficient Video Quality Assessment Leveraging Diverse PreTrained Models from the Wild
In this paper, we propose a VQA method named PTM-VQA, which leverages PreTrained Models to transfer knowledge from models pretrained on various pre-tasks, enabling benefits for VQA from different aspects.
OpenCarbonEval: A Unified Carbon Emission Estimation Framework in Large-Scale AI Models
By promoting sustainable AI development and deployment, OpenCarbonEval can help reduce the environmental impact of large-scale models and contribute to a more environmentally responsible future for the AI community.
MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model
By employing one-step (or few-step) inference, we further improve the runtime efficiency of the motion latent diffusion model for motion generation.
Ranked #19 on
Motion Synthesis
on HumanML3D
Narrative Action Evaluation with Prompt-Guided Multimodal Interaction
NAE is a more challenging task because it requires both narrative flexibility and evaluation rigor.
LOGO: A Long-Form Video Dataset for Group Action Quality Assessment
Action quality assessment (AQA) has become an emerging topic since it can be extensively applied in numerous scenarios.
DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery
The recovery of occluded human meshes presents challenges for current methods due to the difficulty in extracting effective image features under severe occlusion.
GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling
We introduce a radiance representation that is both structured and fully explicit and thus greatly facilitates 3D generative modeling.
Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression
Recent Vision Transformer Compression (VTC) works mainly follow a two-stage scheme, where the importance score of each model unit is first evaluated or preset in each submodule, followed by the sparsity score evaluation according to the target sparsity constraint.
Learning Dual-Level Deformable Implicit Representation for Real-World Scale Arbitrary Super-Resolution
Scale arbitrary super-resolution based on implicit image function gains increasing popularity since it can better represent the visual world in a continuous manner.
ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation
Performing language-conditioned robotic manipulation tasks in unstructured environments is highly demanded for general intelligent robots.
MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer
To this end, we propose a novel framework named Multimodal Alignment-Guided Dynamic Token Pruning (MADTP) for accelerating various VLTs.
1st Place Solution for 5th LSVOS Challenge: Referring Video Object Segmentation
The recent transformer-based models have dominated the Referring Video Object Segmentation (RVOS) task due to the superior performance.
Plan, Posture and Go: Towards Open-World Text-to-Motion Generation
Some works use the CLIP model to align the motion space and the text space, aiming to enable motion generation from natural language motion descriptions.
OccNeRF: Advancing 3D Occupancy Prediction in LiDAR-Free Environments
Occupancy prediction reconstructs 3D structures of surrounding environments.
ThinkBot: Embodied Instruction Following with Thought Chain Reasoning
Embodied Instruction Following (EIF) requires agents to complete human instruction by interacting objects in complicated surrounding environments.
Open-Vocabulary Segmentation with Semantic-Assisted Calibration
We attribute this to the in-vocabulary embedding and domain-biased CLIP prediction.
Fine-tuning vision foundation model for crack segmentation in civil infrastructures
Two parameter-efficient fine-tuning methods, adapter and low-rank adaptation, are adopted to fine-tune the foundation model in semantic segmentation: the Segment Anything Model (SAM).
Universal Segmentation at Arbitrary Granularity with Language Instruction
This paper aims to achieve universal segmentation of arbitrary semantic level.
Ranked #1 on
Referring Expression Segmentation
on RefCOCOg-test
(using extra training data)
Segment and Caption Anything
We propose a method to efficiently equip the Segment Anything Model (SAM) with the ability to generate regional captions.
Post-training Quantization for Text-to-Image Diffusion Models with Progressive Calibration and Activation Relaxing
In this paper, we propose a novel post-training quantization method PCR (Progressive Calibration and Relaxing) for text-to-image diffusion models, which consists of a progressive calibration strategy that considers the accumulated quantization error across timesteps, and an activation relaxing strategy that improves the performance with negligible cost.
Lightweight Diffusion Models with Distillation-Based Block Neural Architecture Search
When retraining the searched architecture, we adopt a dynamic joint loss to maintain the consistency between supernet training and subnet retraining, which also provides informative objectives for each block and shortens the paths of gradient propagation.
MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory
Due to the high price and heavy energy consumption of GPUs, deploying deep models on IoT devices such as microcontrollers makes significant contributions for ecological AI.
Skip-Plan: Procedure Planning in Instructional Videos via Condensed Action Space Learning
By this means, our model explores all sorts of reliable sub-relations within an action sequence in the condensed action space.
Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer
To bridge these gaps, in this paper, we propose Tem-Adapter, which enables the learning of temporal dynamics and complex semantics by a visual Temporal Aligner and a textual Semantic Aligner.
Ranked #1 on
Video Question Answering
on SUTD-TrafficQA
Ada-DQA: Adaptive Diverse Quality-aware Feature Acquisition for Video Quality Assessment
Video quality assessment (VQA) has attracted growing attention in recent years.
Language-free Compositional Action Generation via Decoupling Refinement
Composing simple elements into complex concepts is crucial yet challenging, especially for 3D action generation.
Efficient Text-Guided 3D-Aware Portrait Generation with Score Distillation Sampling on Distribution
These elaborated designs enable our model to generate portraits with robust multi-view semantic consistency, eliminating the need for optimization-based methods.
Towards Accurate Post-training Quantization for Diffusion Models
On the contrary, we design group-wise quantization functions for activation discretization in different timesteps and sample the optimal timestep for informative calibration image generation, so that our quantized diffusion model can reduce the discretization errors with negligible computational overhead.
SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation
To address this issue, we propose Semantic-assisted Object Cluster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment.
Ranked #2 on
Referring Expression Segmentation
on A2D Sentences
(using extra training data)
Self-similarity-based super-resolution of photoacoustic angiography from hand-drawn doodles
To address this challenge, we propose a novel approach that employs a super-resolution PAA method trained with forged PAA images.
Efficient Meshy Neural Fields for Animatable Human Avatars
Moreover, only minutes of optimization is enough for plausible reconstruction results.
Global Knowledge Calibration for Fast Open-Vocabulary Segmentation
Recent advancements in pre-trained vision-language models, such as CLIP, have enabled the segmentation of arbitrary concepts solely from textual inputs, a process commonly referred to as open-vocabulary semantic segmentation (OVS).
Knowledge Distillation
Open Vocabulary Semantic Segmentation
+4
Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentation
Referring image segmentation segments an image from a language expression.
FLAG3D: A 3D Fitness Activity Dataset with Language Instruction
With the continuously thriving popularity around the world, fitness activity analytic has become an emerging research topic in computer vision.
FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generation
To address these problems, we propose FineDance, which contains 14. 6 hours of music-dance paired data, with fine-grained hand motions, fine-grained genres (22 dance genres), and accurate posture.
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions
In this paper, we show that the key ingredients behind the vision Transformers, namely input-adaptive, long-range and high-order spatial interactions, can also be efficiently implemented with a convolution-based framework.
Ranked #20 on
Semantic Segmentation
on ADE20K
Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition
Different from the conventional adversarial learning-based approaches for UDA, we utilize a self-supervision scheme to reduce the domain shift between two skeleton-based action datasets.
OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression
In this paper, we propose to learn the rank concepts from the rich semantic CLIP latent space.
Ranked #1 on
Few-shot Age Estimation
on MORPH Album2
BNV-Fusion: Dense 3D Reconstruction using Bi-level Neural Volume Fusion
Dense 3D reconstruction from a stream of depth images is the key to many mixed reality and robotic applications.
ScalableViT: Rethinking the Context-oriented Generalization of Vision Transformer
The vanilla self-attention mechanism inherently relies on pre-defined and steadfast computational dimensions.
YouMVOS: An Actor-Centric Multi-Shot Video Object Segmentation Dataset
Many video understanding tasks require analyzing multi-shot videos, but existing datasets for video object segmentation (VOS) only consider single-shot videos.
Semantic-Aware Auto-Encoders for Self-Supervised Representation Learning
Inspired by perceptual learning that could use cross-view learning to perceive concepts and semantics, we propose a novel AE that could learn semantic-aware representation via cross-view image reconstruction.
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
Referring image segmentation is a fundamental vision-language task that aims to segment out an object referred to by a natural language expression from an image.
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
In this work, we present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP.
Hierarchical interaction network for video object segmentation from referring expressions
In this paper, we investigate the problem of video object segmentation from referring expressions (VOSRE).
Ranked #1 on
Referring Expression Segmentation
on J-HMDB
(Precision@0.9 metric)
Optical Flow Estimation
Referring Expression Segmentation
+3
Unsupervised Embedding Learning from Uncertainty Momentum Modeling
We justify that the performance limitation is caused by the gradient vanishing on these sample outliers.
Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning
Previous cycle-consistency correspondence learning methods usually leverage image patches for training.
Uncertainty-aware Score Distribution Learning for Action Quality Assessment
Assessing action quality from videos has attracted growing attention in recent years.
Ranked #4 on
Action Quality Assessment
on AQA-7
Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation
We believe the introduction of the COIN dataset will promote the future in-depth research on instructional video analysis for the community.
COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis
There are substantial instructional videos on the Internet, which enables us to acquire knowledge for completing various tasks.
Deep Progressive Reinforcement Learning for Skeleton-Based Action Recognition
In this paper, we propose a deep progressive reinforcement learning (DPRL) method for action recognition in skeleton-based videos, which aims to distil the most informative frames and discard ambiguous frames in sequences for recognizing actions.
Ranked #3 on
Skeleton Based Action Recognition
on UT-Kinect
