Search Results for author:
Found 21 papers, 11 papers with code
MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes
To be specific, (1) we first come up with a person-agnostic 3D TFG model as the base model and propose to adapt it into a specific identity; (2) we propose a static-dynamic-hybrid adaptation pipeline to help the model learn the personalized static appearance and facial dynamic features; (3) To generate the facial motion of the personalized talking style, we propose an in-context stylized audio-to-motion model that mimics the implicit talking style provided in the reference video without information loss by an explicit style representation.
OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
Given the foundational role of multimodal joint representation in understanding and generation pipelines, high-quality omni joint representations would be a step toward co-processing more diverse multimodal information.
Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT
Lumina-T2X is a nascent family of Flow-based Large Diffusion Transformers that establishes a unified framework for transforming noise into various modalities, such as images and videos, conditioned on text instructions.
Frieren: Efficient Video-to-Audio Generation with Rectified Flow Matching
By employing a non-autoregressive vector field estimator based on a feed-forward transformer and channel-level cross-modal feature fusion with strong temporal alignment, our model generates audio that is highly synchronized with the input video.
FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion
In this work, we propose FreeBind, an idea that treats multimodal representation spaces as basic units, and freely augments pre-trained unified space by integrating knowledge from extra expert spaces via "space bonds".
TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation
However, talking head translation, converting audio-visual speech (i. e., talking head video) from one language into another, still confronts several challenges compared to audio speech: (1) Existing methods invariably rely on cascading, synthesizing via both audio and text, resulting in delays and cascading errors.
Multi-Modal Domain Adaptation Across Video Scenes for Temporal Video Grounding
Thus, to address this issue and enhance model performance on new scenes, we explore the TVG task in an unsupervised domain adaptation (UDA) setting across scenes for the first time, where the video-query pairs in the source scene (domain) are labeled with temporal boundaries, while those in the target scene are not.
Extending Multi-modal Contrastive Representations
Inspired by recent C-MCR, this paper proposes Extending Multimodal Contrastive Representation (Ex-MCR), a training-efficient and paired-data-free method to flexibly learn unified contrastive representation space for more than three modalities by integrating the knowledge of existing MCR spaces.
Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes
This paper presents Chat-3D, which combines the 3D visual perceptual ability of pre-trained 3D representations and the impressive reasoning and conversation capabilities of advanced LLMs to achieve the first universal dialogue systems for 3D scenes.
3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding
3D visual grounding aims to localize the target object in a 3D point cloud by a free-form language description.
Distilling Coarse-to-Fine Semantic Matching Knowledge for Weakly Supervised 3D Visual Grounding
To accomplish this, we design a novel semantic matching model that analyzes the semantic similarity between object proposals and sentences in a coarse-to-fine manner.
Connecting Multi-modal Contrastive Representations
This paper proposes a novel training-efficient method for learning MCR without paired data called Connecting Multi-modal Contrastive Representations (C-MCR).
MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition
However, despite researchers exploring cross-lingual translation techniques such as machine translation and audio speech translation to overcome language barriers, there is still a shortage of cross-lingual studies on visual speech.
DsMCL: Dual-Level Stochastic Multiple Choice Learning for Multi-Modal Trajectory Prediction
One of the important and inherent factors is the multi-modality of vehicle motion.
Frame Interpolation with Multi-Scale Deep Loss Functions and Generative Adversarial Networks
To improve the quality of synthesised intermediate video frames, our network is jointly supervised at different levels with a perceptual loss function that consists of an adversarial and two content losses.
Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize
Compared to sub-pixel convolution initialized with schemes designed for standard convolution kernels, it is free from checkerboard artifacts immediately after initialization.
Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation
Convolutional neural networks have enabled accurate image super-resolution in real-time.
Ranked #11 on
Video Super-Resolution
on MSU Video Upscalers: Quality Enhancement
(VMAF metric)
Is the deconvolution layer the same as a convolutional layer?
In this note, we want to focus on aspects related to two questions most people asked us at CVPR about the network we presented.
Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
This means that the super-resolution (SR) operation is performed in HR space.
Ranked #1 on
Video Super-Resolution
on Xiph HD - 4x upscaling
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images.
Ranked #3 on
Image Super-Resolution
on VggFace2 - 8x upscaling
