Search Results for author:
Found 56 papers, 27 papers with code
ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RL
Although chain-of-thought reasoning and reinforcement learning (RL) have driven breakthroughs in NLP, their integration into generative vision models remains underexplored.
ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning
Video understanding is inherently intention-driven-humans naturally focus on relevant frames based on their goals.
Efficient RL Training for Reasoning Models via Length-Aware Optimization
Large reasoning models, such as OpenAI o1 or DeepSeek R1, have demonstrated remarkable performance on reasoning tasks but often incur a long reasoning path with significant memory and time costs.
JointDiT: Enhancing RGB-Depth Joint Modeling with Diffusion Transformers
We present JointDiT, a diffusion transformer that models the joint distribution of RGB and depth.
Subject-driven Video Generation via Disentangled Identity and Motion
We propose to train a subject-driven customized video generation model through decoupling the subject-specific learning from temporal dynamics in zero-shot without additional tuning.
HiTVideo: Hierarchical Tokenizers for Enhancing Text-to-Video Generation with Autoregressive Large Language Models
We explore the trade-offs between compression and reconstruction, while emphasizing the advantages of high-compressed semantic tokens in text-to-video tasks.
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
Inspired by the success of DeepSeek-R1, we explore the potential of rule-based reinforcement learning (RL) in large reasoning models.
FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis
To synthesize natural object motion while supporting detailed camera control, our framework adopts a two-stage video synthesis pipeline consisting of optical flow generation and flow-conditioned video synthesis.
HomoGen: Enhanced Video Inpainting via Homography Propagation and Diffusion
In this paper, we present HomoGen, an enhanced video inpainting method based on homography propagation and diffusion models.
MageBench: Bridging Large Multimodal Models to Agents
Such vision-in-the-chain reasoning paradigm is more aligned with the needs of multimodal agents, while being rarely evaluated.
StableAnimator: High-Quality Identity-Preserving Human Image Animation
During inference, we propose a novel Hamilton-Jacobi-Bellman (HJB) equation-based optimization to further enhance the face quality.
REDUCIO! Generating 1024$\times$1024 Video within 16 Seconds using Extremely Compressed Motion Latents
Commercial video generation models have exhibited realistic, high-fidelity results but are still restricted to limited access.
LLM2CLIP: Powerful Language Model Unlocks Richer Visual Representation
CLIP is a foundational multimodal model that aligns image and text features into a shared representation space via contrastive learning on large-scale image-text pairs.
PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting
We then introduce lightweight, learnable modules to refine depth and pose estimates from the coarse alignments, improving the quality of 3D reconstruction and novel view synthesis.
Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms
In this paper, we target the realm of visual aesthetics and aim to align vision models with human aesthetic standards in a retrieval system.
MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion
In this paper, we propose MotionFollower, a lightweight score-guided diffusion model for video motion editing.
Chameleon: A Data-Efficient Generalist for Dense Visual Prediction in the Wild
In this study, we explore a universal model that can flexibly adapt to unseen dense label structures with a few examples, enabling it to serve as a data-efficient vision generalist in diverse real-world scenarios.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
We introduce phi-3-mini, a 3. 8 billion parameter language model trained on 3. 3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3. 5 (e. g., phi-3-mini achieves 69% on MMLU and 8. 38 on MT-bench), despite being small enough to be deployed on a phone.
Ranked #5 on
MMR total
on MRR-Benchmark
(using extra training data)
OmniVid: A Generative Framework for Universal Video Understanding
The core of video understanding tasks, such as recognition, captioning, and tracking, is to automatically detect objects or actions in a video and analyze their temporal evolution.
Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering
Visual text rendering poses a fundamental challenge for contemporary text-to-image generation models, with the core problem lying in text encoder deficiencies.
Unifying Correspondence Pose and NeRF for Generalized Pose-Free Novel View Synthesis
This work delves into the task of pose-free novel view synthesis from stereo pairs a challenging and pioneering task in 3D vision.
Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs
This work delves into the task of pose-free novel view synthesis from stereo pairs, a challenging and pioneering task in 3D vision.
MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generation
We present MicroCinema, a straightforward yet effective framework for high-quality and coherent text-to-video generation.
ART$\boldsymbol{\cdot}$V: Auto-Regressive Text-to-Video Generation with Diffusion Models
Second, it preserves the high-fidelity generation ability of the pre-trained image diffusion models by making only minimal network modifications.
Panacea: Panoramic and Controllable Video Generation for Autonomous Driving
This work notably propels the field of autonomous driving by effectively augmenting the training dataset used for advanced BEV perception techniques.
CCEdit: Creative and Controllable Video Editing via Diffusion Models
The versatility of our framework is demonstrated through a diverse range of choices in both structure representations and personalized T2I models, as well as the option to provide the edited key frame.
ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System
Existing deep video models are limited by specific tasks, fixed input-output spaces, and poor generalization capabilities, making it difficult to deploy them in real-world scenarios.
LaMD: Latent Motion Diffusion for Image-Conditional Video Generation
This can be achieved by breaking down the video generation process into latent motion generation and video reconstruction.
Filler Word Detection with Hard Category Mining and Inter-Category Focal Loss
Filler words like ``um" or ``uh" are common in spontaneous speech.
Streaming Video Model
We believe that the concept of streaming video model and the implementation of S-ViT are solid steps towards a unified deep learning architecture for video understanding.
Universal Few-shot Learning of Dense Prediction Tasks with Visual Token Matching
We propose Visual Token Matching (VTM), a universal few-shot learner for arbitrary dense prediction tasks.
OmniTracker: Unifying Object Tracking by Tracking-with-Detection
Object tracking (OT) aims to estimate the positions of target objects in a video sequence.
Look Before You Match: Instance Understanding Matters in Video Object Segmentation
Towards this goal, we present a two-branch network for VOS, where the query-based instance segmentation (IS) branch delves into the instance details of the current frame and the VOS branch performs spatial-temporal matching with the memory bank.
Ranked #1 on
Semi-Supervised Video Object Segmentation
on Long Video Dataset
(using extra training data)
TridentSE: Guiding Speech Enhancement with 32 Global Tokens
In this paper, we present TridentSE, a novel architecture for speech enhancement, which is capable of efficiently capturing both global information and local details.
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
This paper presents OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture.
Ranked #4 on
Cross-Modal Retrieval
on Flickr30k
(using extra training data)
An Anchor-Free Detector for Continuous Speech Keyword Spotting
Continuous Speech Keyword Spotting (CSKWS) is a task to detect predefined keywords in a continuous speech.
RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion
In the proposed paradigm, global and local factors in speech are explicitly decomposed and separately manipulated to achieve high speaker similarity and continuous prosody.
Peripheral Vision Transformer
We propose to incorporate peripheral position encoding to the multi-head self-attention layers to let the network learn to partition the visual field into diverse peripheral regions given training data.
Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph
Last, an innovative link attention module serves as the decoder to reconstruct data from the decomposed content and style, with the help of the linking keys.
When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism
It can be even replaced by a zero-parameter operation.
Ranked #79 on
Object Detection
on COCO minival
(APM metric)
Make It Move: Controllable Image-to-Video Generation with Text Descriptions
With both controllable appearance and motion, TI2V aims at generating videos from a static image and a text description.
Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?
Specifically, we replace the MLP module in the token-mixing step with a novel sparse MLP (sMLP) module.
Ranked #425 on
Image Classification
on ImageNet
Zero-Shot Text-to-Speech for Text-Based Insertion in Audio Narration
Given a piece of speech and its transcript text, text-based speech editing aims to generate speech that can be seamlessly inserted into the given speech by editing the transcript.
A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP
Convolutional neural networks (CNN) are the dominant deep neural network (DNN) architecture for computer vision.
Self-Supervised Visual Representations Learning by Contrastive Mask Prediction
In this paper, we propose a novel contrastive mask prediction (CMP) task for visual representation learning and design a mask contrast (MaskCo) framework to implement the idea.
Unsupervised Visual Representation Learning by Tracking Patches in Video
The proxy task is to estimate the position and size of the image patch in a sequence of video frames, given only the target bounding box in the first frame.
General-Purpose Speech Representation Learning through a Self-Supervised Multi-Granularity Framework
This paper presents a self-supervised learning framework, named MGF, for general-purpose speech representation learning.
VAE^2: Preventing Posterior Collapse of Variational Video Predictions in the Wild
We believe that VAE$^2$ is also applicable to other stochastic sequence prediction problems where training data are lack of stochasticity.
Spatiotemporal Fusion in 3D CNNs: A Probabilistic View
Based on the probability space, we further generate new fusion strategies which achieve the state-of-the-art performance on four well-known action recognition datasets.
Tracking by Instance Detection: A Meta-Learning Approach
We propose a principled three-step approach to build a high-performance tracker.
PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network
We discover that the two streams should communicate with each other, and this is crucial to phase prediction.
Sound Audio and Speech Processing
Posterior-Guided Neural Architecture Search
Accordingly, a hybrid network representation is presented which enables us to leverage the Variational Dropout so that the approximation of the posterior distribution becomes fully gradient-based and highly efficient.
SPM-Tracker: Series-Parallel Matching for Real-Time Visual Object Tracking
The two stages are connected in series as the input proposals of the FM stage are generated by the CM stage.
Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale
Labeling training data is one of the most costly bottlenecks in developing machine learning-based applications.
Towards a Better Match in Siamese Network Based Visual Object Tracker
Recently, Siamese network based trackers have received tremendous interest for their fast tracking speed and high performance.
Ranked #10 on
Visual Object Tracking
on VOT2017/18
A Twofold Siamese Network for Real-Time Object Tracking
SA-Siam is composed of a semantic branch and an appearance branch.
Ranked #1 on
Visual Object Tracking
on OTB-50
