Search Results for author:
Found 59 papers, 34 papers with code
I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models
By this means, I2VGen-XL can simultaneously enhance the semantic accuracy, continuity of details and clarity of generated videos.
ModelScope Text-to-Video Technical Report
This paper introduces ModelScopeT2V, a text-to-video synthesis model that evolves from a text-to-image synthesis model (i. e., Stable Diffusion).
Ranked #8 on
Text-to-Video Generation
on MSR-VTT
Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation
At the structure level, we decompose the T2V task into two steps, including spatial reasoning and temporal reasoning, using a unified denoiser.
Ranked #5 on
Text-to-Video Generation
on MSR-VTT
DreamVideo: Composing Your Dream Videos with Customized Subject and Motion
In motion learning, we architect a motion adapter and fine-tune it on the given videos to effectively model the target motion pattern.
VideoLCM: Video Latent Consistency Model
Consistency models have demonstrated powerful capability in efficient image generation and allowed synthesis within a few sampling steps, alleviating the high computational cost in diffusion models.
VideoComposer: Compositional Video Synthesis with Motion Controllability
The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis.
Ranked #5 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
Self-supervised Motion Learning from Static Images
We furthermore introduce a static mask in pseudo motions to create local motion patterns, which forces the model to additionally locate notable motion areas for the correct classification. We demonstrate that MoSI can discover regions with large motion even without fine-tuning on the downstream datasets.
Towards Training Stronger Video Vision Transformers for EPIC-KITCHENS-100 Action Recognition
In this paper, we present empirical results for training a stronger video vision transformer on the EPIC-KITCHENS-100 Action Recognition dataset.
A Stronger Baseline for Ego-Centric Action Detection
This technical report analyzes an egocentric video action detection method we used in the 2021 EPIC-KITCHENS-100 competition hosted in CVPR2021 Workshop.
ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning
The visualizations show that ParamCrop adaptively controls the center distance and the IoU between two augmented views, and the learned change in the disparity along the training process is beneficial to learning a strong representation.
TAda! Temporally-Adaptive Convolutions for Video Understanding
This work presents Temporally-Adaptive Convolutions (TAdaConv) for video understanding, which shows that adaptive weight calibration along the temporal dimension is an efficient way to facilitate modelling complex temporal dynamics in videos.
Ranked #67 on
Action Recognition
on Something-Something V2
(using extra training data)
Temporally-Adaptive Models for Efficient Video Understanding
Spatial convolutions are extensively used in numerous deep video models.
Ranked #3 on
Action Recognition
on EPIC-KITCHENS-100
(using extra training data)
TCTrack: Temporal Contexts for Aerial Tracking
Temporal contexts among consecutive frames are far from being fully utilized in existing visual trackers.
Towards Real-World Visual Tracking with Temporal Contexts
To handle those problems, we propose a two-level framework (TCTrack) that can exploit temporal contexts efficiently.
End-to-end Temporal Action Detection with Transformer
Temporal action detection (TAD) aims to determine the semantic label and the temporal interval of every action instance in an untrimmed video.
Ranked #8 on
Temporal Action Localization
on HACS
RLIPv2: Fast Scaling of Relational Language-Image Pre-training
In this paper, we propose RLIPv2, a fast converging model that enables the scaling of relational pre-training to large-scale pseudo-labelled scene graph data.
Ranked #1 on
Zero-Shot Human-Object Interaction Detection
on HICO-DET
(using extra training data)
OadTR: Online Action Detection with Transformers
Most recent approaches for online action detection tend to apply Recurrent Neural Network (RNN) to capture long-range temporal structure.
Ranked #8 on
Online Action Detection
on THUMOS'14
CBR-Net: Cascade Boundary Refinement Network for Action Detection: Submission to ActivityNet Challenge 2020 (Task 1)
In this report, we present our solution for the task of temporal action localization (detection) (task 1) in ActivityNet Challenge 2020.
Self-Supervised Learning for Semi-Supervised Temporal Action Proposal
In this paper, we focus on applying the power of self-supervised methods to improve semi-supervised action proposal generation.
Ranked #2 on
Semi-Supervised Action Detection
on ActivityNet-1.3
Proposal Relation Network for Temporal Action Detection
We calculate the detection results by assigning the proposals with corresponding classification results.
Ranked #2 on
Temporal Action Localization
on ActivityNet-1.3
(using extra training data)
Open-world Semantic Segmentation for LIDAR Point Clouds
Current methods for LIDAR semantic segmentation are not robust enough for real-world applications, e. g., autonomous driving, since it is closed-set and static.
CLIP-guided Prototype Modulating for Few-shot Action Recognition
Learning from large-scale contrastive language-image pre-training like CLIP has shown remarkable success in a wide range of downstream tasks recently, but it is still under-explored on the challenging few-shot action recognition (FSAR) task.
MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognition
To address these issues, we develop a Motion-augmented Long-short Contrastive Learning (MoLo) method that contains two crucial components, including a long-short contrastive objective and a motion autodecoder.
MAR: Masked Autoencoders for Efficient Action Recognition
Inspired by this, we propose propose Masked Action Recognition (MAR), which reduces the redundant computation by discarding a proportion of patches and operating only on a part of the videos.
Ranked #12 on
Action Recognition
on Something-Something V2
Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning
When pre-training on the large-scale Kinetics-710, we achieve 89. 7% on Kinetics-400 with a frozen ViT-L model, which verifies the scalability of DiST.
Hybrid Relation Guided Set Matching for Few-shot Action Recognition
To overcome the two limitations, we propose a novel Hybrid Relation guided Set Matching (HyRSM) approach that incorporates two key components: hybrid relation module and set matching metric.
Enlarging Instance-specific and Class-specific Information for Open-set Action Recognition
In this paper, we begin with analyzing the feature representation behavior in the open-set action recognition (OSAR) problem based on the information bottleneck (IB) theory, and propose to enlarge the instance-specific (IS) and class-specific (CS) information contained in the feature for better performance.
The Devil is in the Wrongly-classified Samples: Towards Unified Open-set Recognition
Recently, Unified Open-set Recognition (UOSR) has been proposed to reject not only unknown samples but also known but wrongly classified samples, which tends to be more practical in real-world applications.
CMDFusion: Bidirectional Fusion Network with Cross-modality Knowledge Distillation for LIDAR Semantic Segmentation
In this way, RGB images are not required during inference anymore since the 2D knowledge branch provides 2D information according to the 3D LIDAR input.
HyRSM++: Hybrid Relation Guided Temporal Set Matching for Few-shot Action Recognition
To be specific, HyRSM++ consists of two key components, a hybrid relation module and a temporal set matching metric.
Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words
Prompt-based fine-tuning for pre-trained models has proven effective for many natural language processing tasks under few-shot settings in general domain.
InstructVideo: Instructing Video Diffusion Models with Human Feedback
To tackle this problem, we propose InstructVideo to instruct text-to-video diffusion models with human feedback by reward fine-tuning.
A Recipe for Scaling up Text-to-Video Generation with Text-free Videos
Following such a pipeline, we study the effect of doubling the scale of training set (i. e., video-only WebVid10M) with some randomly collected text-free videos and are encouraged to observe the performance improvement (FID from 9. 67 to 8. 19 and FVD from 484 to 441), demonstrating the scalability of our approach.
Ranked #6 on
Text-to-Video Generation
on MSR-VTT
Less is More: Rejecting Unreliable Reviews for Product Question Answering
In the literature, PQA is formulated as a retrieval problem with the goal to search for the most relevant reviews to answer a given product question.
TACNet: Transition-Aware Context Network for Spatio-Temporal Action Detection
In this paper, we define these ambiguous samples as "transitional states", and propose a Transition-Aware Context Network (TACNet) to distinguish transitional states.
Ranked #7 on
Action Detection
on J-HMDB
Temporal Fusion Network for Temporal Action Localization:Submission to ActivityNet Challenge 2020 (Task E)
This technical report analyzes a temporal action localization method we used in the HACS competition which is hosted in Activitynet Challenge 2020. The goal of our task is to locate the start time and end time of the action in the untrimmed video, and predict action category. Firstly, we utilize the video-level feature information to train multiple video-level action classification models.
Multi-Level Temporal Pyramid Network for Action Detection
By this means, the proposed MLTPN can learn rich and discriminative features for different action instances with different durations.
Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw
With the constrained jigsaw puzzles, instead of solving them directly, which could still be extremely hard, we carefully design four surrogate tasks that are more solvable but meanwhile still ensure that the learned representation is sensitive to spatiotemporal continuity at both the local and global levels.
Tuning the quantumness of simple Bose systems: A universal phase diagram
We present a comprehensive theoretical study of the phase diagram of a system of many Bose particles interacting with a two-body central potential of the so-called Lennard-Jones form.
Statistical Mechanics
Relation Modeling in Spatio-Temporal Action Localization
This paper presents our solution to the AVA-Kinetics Crossover Challenge of ActivityNet workshop at CVPR 2021.
Ranked #4 on
Spatio-Temporal Action Localization
on AVA-Kinetics
(using extra training data)
Weakly-Supervised Temporal Action Localization Through Local-Global Background Modeling
Then our proposed Local-Global Background Modeling Network (LGBM-Net) is trained to localize instances by using only video-level labels based on Multi-Instance Learning (MIL).
Weakly-supervised Learning
Weakly-supervised Temporal Action Localization
+1
Exploring Stronger Feature for Temporal Action Localization
Temporal action localization aims to localize starting and ending time with action category.
Support-Set Based Cross-Supervision for Video Grounding
The contrastive objective aims to learn effective representations by contrastive learning, while the caption objective can train a powerful video encoder supervised by texts.
Discovery-and-Selection: Towards Optimal Multiple Instance Learning for Weakly Supervised Object Detection
Weakly supervised object detection (WSOD) is a challenging task that requires simultaneously learn object classifiers and estimate object locations under the supervision of image category labels.
Does QA-based intermediate training help fine-tuning language models for text classification?
Fine-tuning pre-trained language models for downstream tasks has become a norm for NLP.
Learning from Untrimmed Videos: Self-Supervised Video Representation Learning with Hierarchical Consistency
In this work, we aim to learn representations by leveraging more abundant information in untrimmed videos.
G4: Grounding-guided Goal-oriented Dialogues Generation with Multiple Documents
Goal-oriented dialogues generation grounded in multiple documents(MultiDoc2Dial) is a challenging and realistic task.
Context-aware Proposal Network for Temporal Action Detection
This technical report presents our first place winning solution for temporal action detection task in CVPR-2022 AcitivityNet Challenge.
Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning
The former is to reduce the memory cost by preserving only one condensed frame instead of the whole video, while the latter aims to compensate the lost spatio-temporal details in the Frame Condensing stage.
Expediting Distributed DNN Training with Device Topology-Aware Graph Deployment
This paper presents TAG, an automatic system to derive optimized DNN training graph and its deployment onto any device topology, for expedited training in device- and topology- heterogeneous ML clusters.
Auto-Parallelizing Large Models with Rhino: A Systematic Approach on Production AI Platform
We present Rhino, a system for accelerating tensor programs with automatic parallelization on AI platform for real production environment.
Crowd Counting with Sparse Annotation
This paper presents a new annotation method called Sparse Annotation (SA) for crowd counting, which reduces human labeling efforts by sparsely labeling individuals in an image.
LipFormer: High-Fidelity and Generalizable Talking Face Generation With a Pre-Learned Facial Codebook
Generating a talking face video from the input audio sequence is a practical yet challenging task.
Unleashing Potential of Evidence in Knowledge-Intensive Dialogue Generation
To fully Unleash the potential of evidence, we propose a framework to effectively incorporate Evidence in knowledge-Intensive Dialogue Generation (u-EIDG).
Space-time Prompting for Video Class-incremental Learning
In this paper, we will fill this gap by learning multiple prompts based on a powerful image-language pre-trained model, i. e., CLIP, making it fit for video class-incremental learning (VCIL).
Few-shot Action Recognition with Captioning Foundation Models
In this paper, we develop an effective plug-and-play framework called CapFSAR to exploit the knowledge of multimodal models without manually annotating text.
Utilising a Large Language Model to Annotate Subject Metadata: A Case Study in an Australian National Research Data Catalogue
To the best of our knowledge, we are introducing, for the first time, an in-context learning method that harnesses large language models for automated subject metadata annotation.
Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation
To align the generated image with layout instructions, we present a training-free layout calibration system SimM that intervenes in the generative process on the fly during inference time.
DreamTalk: When Expressive Talking Head Generation Meets Diffusion Probabilistic Models
In this work, we propose a DreamTalk framework to fulfill this gap, which employs meticulous design to unlock the potential of diffusion models in generating expressive talking heads.
