Search Results for author:
Found 46 papers, 17 papers with code
Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models
A primary limitation is the substantial number of image tokens required for AR models, which constrains both training and inference efficiency, as well as image resolution.
MoCha: Towards Movie-Grade Talking Character Synthesis
We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text.
OceanSim: A GPU-Accelerated Underwater Robot Perception Simulation Framework
Underwater simulators offer support for building robust underwater perception solutions.
Movie Weaver: Tuning-Free Multi-Concept Video Personalization with Anchored Prompts
This challenge arises due to the lack of a mechanism to link each concept with its specific reference image.
HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models
We introduce HOIGPT, a token-based generative method that unifies 3D hand-object interactions (HOI) perception and generation, offering the first comprehensive solution for captioning and generating high-quality 3D HOI sequences from a diverse range of conditional signals (e. g. text, objects, partial sequences).
DirectorLLM for Human-Centric Video Generation
In this paper, we introduce DirectorLLM, a novel video generation model that employs a large language model (LLM) to orchestrate human poses within videos.
A Mamba Foundation Model for Time Series Forecasting
In the first stage, the forward and backward backbones are optimized via patch-wise autoregressive prediction; in the second stage, the model trains a prediction head and refines other components for long-term forecasting.
Movie Gen: A Cast of Media Foundation Models
Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation.
Imagine yourself: Tuning-Free Personalized Image Generation
Our study demonstrates that Imagine yourself surpasses the state-of-the-art personalization model, exhibiting superior capabilities in identity preservation, visual quality, and text alignment.
MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers
Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control.
Instance Tracking in 3D Scenes from Egocentric Videos
We explore this problem by first introducing a new benchmark dataset, consisting of RGB and depth videos, per-frame camera pose, and instance-level annotations in both 2D camera and 3D world coordinates.
CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer
Reconstructing personalized animatable head avatars has significant implications in the fields of AR/VR.
HarmonyDream: Task Harmonization Inside World Models
Model-based reinforcement learning (MBRL) holds the promise of sample-efficient learning by utilizing a world model, which models how the environment works and typically encompasses components for two tasks: observation modeling and reward modeling.
Ranked #4 on
Atari Games 100k
on Atari 100k
Light Field Diffusion for Single-View Novel View Synthesis
Starting from the camera pose matrices, LFD transforms them into light field encoding, with the same shape as the reference image, to describe the direction of each ray.
Hybrid-CSR: Coupling Explicit and Implicit Shape Representation for Cortical Surface Reconstruction
We present Hybrid-CSR, a geometric deep-learning model that combines explicit and implicit shape representations for cortical surface reconstruction.
Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning
To tackle this issue, we introduce Contextualized World Models (ContextWM) that explicitly separate context and dynamics modeling to overcome the complexity and diversity of in-the-wild videos and facilitate knowledge transfer between distinct scenes.
Hybrid Transformer and CNN Attention Network for Stereo Image Super-resolution
Multi-stage strategies are frequently employed in image restoration tasks.
OPDN: Omnidirectional Position-aware Deformable Network for Omnidirectional Image Super-Resolution
Model A aims to enhance the feature extraction ability of 360{\deg} image positional information, while Model B further focuses on the high-frequency information of 360{\deg} images.
Localized Region Contrast for Enhancing Self-Supervised Learning in Medical Image Segmentation
In this paper, we propose a novel contrastive learning framework that integrates Localized Region Contrast (LRC) to enhance existing self-supervised pre-training methods for medical image segmentation.
I2F: A Unified Image-to-Feature Approach for Domain Adaptive Semantic Segmentation
This paper proposes a novel UDA pipeline for semantic segmentation that unifies image-level and feature-level adaptation.
Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training
Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage at both training and inference time limit their generalization.
Identity-Aware Hand Mesh Estimation and Personalization from RGB Images
In this paper, we propose an identity-aware hand mesh estimation model, which can incorporate the identity information represented by the intrinsic shape parameters of the subject.
PPT: token-Pruned Pose Transformer for monocular and multi-view human pose estimation
In this paper, we propose the token-Pruned Pose Transformer (PPT) for 2D human pose estimation, which can locate a rough human mask and performs self-attention only within selected tokens.
Ranked #21 on
3D Human Pose Estimation
on Human3.6M
(using extra training data)
Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization
Compared with state-of-the-art ViT quantization work (algorithmic approach only without hardware acceleration), our quantization achieves 0. 47% to 1. 36% higher Top-1 accuracy under the same bit-width.
Training Your Sparse Neural Network Better with Any Mask
Pruning large neural networks to create high-quality, independently trainable sparse masks, which can maintain similar performance to their dense counterparts, is very desirable due to the reduced space and time complexity.
Medical Image Registration via Neural Fields
Traditional methods for image registration are primarily optimization-driven, finding the optimal deformations that maximize the similarity between two images.
NTIRE 2022 Challenge on Super-Resolution and Quality Enhancement of Compressed Video: Dataset, Methods and Results
This challenge includes three tracks.
Diffeomorphic Image Registration with Neural Velocity Field
Here we propose a new optimization-based method named DNVF (Diffeomorphic Image Registration with Neural Velocity Field) which utilizes deep neural network to model the space of admissible transformations.
Sparsity Winning Twice: Better Robust Generalization from More Efficient Training
We introduce two alternatives for sparse adversarial training: (i) static sparsity, by leveraging recent results from the lottery ticket hypothesis to identify critical sparse subnetworks arising from the early training; (ii) dynamic sparsity, by allowing the sparse subnetwork to adaptively adjust its connectivity pattern (while sticking to the same sparsity ratio) throughout training.
VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer
To the best of our knowledge, this is the first time quantization has been incorporated into ViT acceleration on FPGAs with the help of a fully automatic framework to guide the quantization strategy on the software side and the accelerator implementations on the hardware side given the target frame rate.
EI-CLIP: Entity-Aware Interventional Contrastive Learning for E-Commerce Cross-Modal Retrieval
recommendation, and marketing services.
Over-the-Air Aggregation for Federated Learning: Waveform Superposition and Prototype Validation
In this paper, we develop an orthogonal-frequency-division-multiplexing (OFDM)-based over-the-air (OTA) aggregation solution for wireless federated learning (FL).
AFTer-UNet: Axial Fusion Transformer UNet for Medical Image Segmentation
One has to either downsample the image or use cropped local patches to reduce GPU memory usage, which limits its performance.
TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation
The 3D position encoding guided by the epipolar field provides an efficient way of encoding correspondences between pixels of different views.
Ranked #25 on
3D Human Pose Estimation
on Human3.6M
(using extra training data)
Stingy Teacher: Sparse Logits Suffice to Fail Knowledge Distillation
However, it is unclear why the distorted distribution of the logits is catastrophic to the student model.
SGE net: Video object detection with squeezed GRU and information entropy map
Compared with object detection of static images, video object detection is more challenging due to the motion of objects, while providing rich temporal information.
Undistillable: Making A Nasty Teacher That CANNOT teach students
Knowledge Distillation (KD) is a widely used technique to transfer knowledge from pre-trained teacher models to (usually more lightweight) student models.
Coarse-to-Fine Domain Adaptive Semantic Segmentation with Photometric Alignment and Category-Center Regularization
Unsupervised domain adaptation (UDA) in semantic segmentation is a fundamental yet promising task relieving the need for laborious annotation works.
Ranked #25 on
Synthetic-to-Real Translation
on SYNTHIA-to-Cityscapes
Spending Your Winning Lottery Better After Drawing It
In this paper, we demonstrate that it is unnecessary for spare retraining to strictly inherit those properties from the dense network.
SIA-GCN: A Spatial Information Aware Graph Neural Network with 2D Convolutions for Hand Pose Estimation
In this paper, we extend GNNs along two directions: a) allowing features at each node to be represented by 2D spatial confidence maps instead of 1D vectors; and b) proposing an efficient operation to integrate information from neighboring nodes through 2D convolutions with different learnable kernels at each edge.
Real-MFF: A Large Realistic Multi-focus Image Dataset with Ground Truth
In this letter, we introduce a large and realistic multi-focus dataset called Real-MFF, which contains 710 pairs of source images with corresponding ground truth images.
Rotation-invariant Mixed Graphical Model Network for 2D Hand Pose Estimation
In this paper, we propose a new architecture named Rotation-invariant Mixed Graphical Model Network (R-MGMN) to solve the problem of 2D hand pose estimation from a monocular RGB image.
Nonparametric Structure Regularization Machine for 2D Hand Pose Estimation
We propose a novel Nonparametric Structure Regularization Machine (NSRM) for 2D hand pose estimation, adopting a cascade multi-task architecture to learn hand structure and keypoint representations jointly.
An α-Matte Boundary Defocus Model Based Cascaded Network for Multi-focus Image Fusion
Based on this {\alpha}-matte defocus model and the generated data, a cascaded boundary aware convolutional network termed MMF-Net is proposed and trained, aiming to achieve clearer fusion results around the FDB.
Adaptive Graphical Model Network for 2D Handpose Estimation
In this paper, we propose a new architecture called Adaptive Graphical Model Network (AGMN) to tackle the task of 2D hand pose estimation from a monocular RGB image.
Boundary Aware Multi-Focus Image Fusion Using Deep Neural Network
Since it is usually difficult to capture an all-in-focus image of a 3D scene directly, various multi-focus image fusion methods are employed to generate it from several images focusing at different depths.
