Search Results for author:
Found 91 papers, 40 papers with code
BOOTPLACE: Bootstrapped Object Placement with Detection Transformers
In this paper, we tackle the copy-paste image-to-image composition problem with a focus on object placement learning.
AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformers
Despite the recent progress of audio-driven video generation, existing methods mostly focus on driving facial movements, leading to non-coherent head and body dynamics.
Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model
Current digital human studies focusing on lip-syncing and body movement are no longer sufficient to meet the growing industrial demand, while human video generation techniques that support interacting with real-world environments (e. g., objects) have not been well investigated.
Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers
Co-speech gesture video synthesis is a challenging task that requires both probabilistic modeling of human gestures and the synthesis of realistic images that align with the rhythmic nuances of speech.
MDE: Modality Discrimination Enhancement for Multi-modal Recommendation
Multi-modal recommendation systems aim to enhance performance by integrating an item's content features across various modalities with user behavior data.
Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries
Although deep models have been widely explored in solving partial differential equations (PDEs), previous works are primarily limited to data only with up to tens of thousands of mesh points, far from the million-point scale required by industrial simulations that involve complex geometries.
Video Anomaly Detection with Motion and Appearance Guided Patch Diffusion Model
To address this, we introduce innovative motion and appearance conditions that are seamlessly integrated into our patch diffusion model.
Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer
Existing methodologies for animating portrait images face significant challenges, particularly in handling non-frontal perspectives, rendering dynamic objects around the portrait, and generating immersive, realistic backgrounds.
Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning
Our empirical studies, including instruction tuning experiments with models such as Pythia and LLaMA, demonstrate the effectiveness of the proposed framework.
GhostRNN: Reducing State Redundancy in RNN with Cheap Operations
Recurrent neural network (RNNs) that are capable of modeling long-distance dependencies are widely used in various speech tasks, eg., keyword spotting (KWS) and speech enhancement (SE).
TALK-Act: Enhance Textural-Awareness for 2D Speaking Avatar Reenactment with Diffusion Model
We propose a Motion-Enhanced Textural Alignment module to enhance the bond between driving and target signals.
Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation
To the best of our knowledge, Hallo2, proposed in this paper, is the first method to achieve 4K resolution and generate hour-long, audio-driven portrait image animations enhanced with textual prompts.
ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer
Lip-syncing videos with given audio is the foundation for various applications including the creation of virtual presenters or performers.
Forest2Seq: Revitalizing Order Prior for Sequential Indoor Scene Synthesis
Synthesizing realistic 3D indoor scenes is a challenging task that traditionally relies on manual arrangement and annotation by expert designers.
Hybrid Alignment Training for Large Language Models
Alignment training is crucial for enabling large language models (LLMs) to cater to human intentions and preferences.
Sample-efficient Imitative Multi-token Decision Transformer for Real-world Driving
Recent advancements in autonomous driving technologies involve the capability to effectively process and learn from extensive real-world driving data.
DiffPop: Plausibility-Guided Object Placement Diffusion for Image Composition
In this paper, we address the problem of plausible object placement for the challenging task of realistic image composition.
Stock Movement Prediction with Multimodal Stable Fusion via Gated Cross-Attention Mechanism
The accurate prediction of stock movements is crucial for investment strategies.
Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Model
However, fusing multiple modalities is challenging for SSMs due to its hardware-aware parallelism designs.
Unisolver: PDE-Conditional Transformers Are Universal PDE Solvers
This limits the generalization of neural solvers to diverse PDEs, impeding them from being practical surrogate models for numerical solvers.
Prior Constraints-based Reward Model Training for Aligning Large Language Models
Reinforcement learning with human feedback for aligning large language models (LLMs) trains a reward model typically using ranking loss with comparison pairs. However, the training procedure suffers from an inherent problem: the uncontrolled scaling of reward scores during reinforcement learning due to the lack of constraints while training the reward model. This paper proposes a Prior Constraints-based Reward Model (namely PCRM) training method to mitigate this problem.
Attacking Transformers with Feature Diversity Adversarial Perturbation
Understanding the mechanisms behind Vision Transformer (ViT), particularly its vulnerability to adversarial perturba tions, is crucial for addressing challenges in its real-world applications.
AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation
In this paper, we propose AVI-Talking, an Audio-Visual Instruction system for expressive Talking face generation.
Fine-grained Appearance Transfer with Diffusion Models
A pivotal aspect of our approach is the strategic use of the predicted $x_0$ space by diffusion models within the latent space of diffusion processes.
DAE-Net: Deforming Auto-Encoder for fine-grained shape co-segmentation
To accommodate structural variations in the collection, our network composes each shape by a selected subset of template parts which are affine-transformed.
PGODE: Towards High-quality System Dynamics Modeling
In this paper, we propose a new approach named Prototypical Graph ODE (PGODE) to address the problem.
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation
In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks.
Progressive Text-to-Image Diffusion with Soft Latent Direction
In spite of the rapidly evolving landscape of text-to-image generation, the synthesis and manipulation of multiple entities while adhering to specific relational constraints pose enduring challenges.
ReliTalk: Relightable Talking Portrait Generation from a Single Video
Our key insight is to decompose the portrait's reflectance from implicitly learned audio-driven facial normals and images.
Learning Evaluation Models from Large Language Models for Sequence Generation
Automatic evaluation of sequence generation, traditionally reliant on metrics like BLEU and ROUGE, often fails to capture the semantic accuracy of generated text sequences due to their emphasis on n-gram overlap.
ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generation
Applying Reinforcement Learning (RL) to sequence generation models enables the direct optimization of long-term rewards (\textit{e. g.,} BLEU and human feedback), but typically requires large-scale sampling over a space of action sequences.
ShaDDR: Interactive Example-Based Geometry and Texture Generation via 3D Shape Detailization and Differentiable Rendering
Furthermore, we showcase the ability of our method to learn geometric details and textures from shapes reconstructed from real-world photos.
Detecting Errors in a Numerical Response via any Regression Model
Noise plagues many numerical datasets, where the recorded values in the data may fail to match the true underlying values due to reasons including: erroneous sensors, data entry/processing mistakes, or imperfect human estimates.
Building an Invisible Shield for Your Portrait against Deepfakes
Unlike authentic images, where the hidden messages can be extracted with precision, manipulating the facial attributes through deepfake techniques can disrupt the decoding process.
StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator
Despite recent advances in syncing lip movements with any audio waves, current methods still struggle to balance generation quality and the model's generalization ability.
Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement
Neural Radiance Fields (NeRF) have constituted a remarkable breakthrough in image-based 3D reconstruction.
Make Your Brief Stroke Real and Stereoscopic: 3D-Aware Simplified Sketch to Portrait Generation
Creating the photo-realistic version of people sketched portraits is useful to various entertainment purposes.
Dual Memory Units with Uncertainty Regulation for Weakly Supervised Video Anomaly Detection
To address this issue, we propose an Uncertainty Regulated Dual Memory Units (UR-DMU) model to learn both the representations of normal data and discriminative features of abnormal data.
Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformers
This requires masking a large percentage of the original image and seamlessly inpainting it with the aid of audio and reference frames.
Audio-Driven Co-Speech Gesture Video Generation
Our key insight is that the co-speech gestures can be decomposed into common motion patterns and subtle rhythmic dynamics.
Ada3Diff: Defending against 3D Adversarial Point Clouds via Adaptive Diffusion
Deep 3D point cloud models are sensitive to adversarial attacks, which poses threats to safety-critical applications such as autonomous driving.
Dynamic Feature Pruning and Consolidation for Occluded Person Re-Identification
The framework mainly consists of a sparse encoder, a multi-view feature mathcing module, and a feature consolidation decoder.
Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition
While dynamic Neural Radiance Fields (NeRF) have shown success in high-fidelity 3D modeling of talking portraits, the slow training and inference speed severely obstruct their potential usage.
Person Text-Image Matching via Text-Feature Interpretability Embedding and External Attack Node Implantation
Specifically, we improve the interpretability of text features by providing them with consistent semantic information with image features to achieve the alignment of text and describe image region features. To address the challenges posed by the diversity of text and the corresponding person images, we treat the variation caused by diversity to features as caused by perturbation information and propose a novel adversarial attack and defense method to solve it.
Learning to Immunize Images for Tamper Localization and Self-Recovery
Image immunization (Imuge) is a technology of protecting the images by introducing trivial perturbation, so that the protected images are immune to the viruses in that the tampered contents can be auto-recovered.
TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis
TimesBlock can discover the multi-periodicity adaptively and extract the complex temporal variations from transformed 2D tensors by a parameter-efficient inception block.
StyleSwap: Style-Based Generator Empowers Robust Face Swapping
Our core idea is to leverage a style-based generator to empower high-fidelity and robust face swapping, thus the generator's advantage can be adopted for optimizing identity similarity.
PointCAT: Contrastive Adversarial Training for Robust Point Cloud Recognition
Notwithstanding the prominent performance achieved in various applications, point cloud recognition models have often suffered from natural corruptions and adversarial perturbations.
StyleFaceV: Face Video Generation via Decomposing and Recomposing Pretrained StyleGAN3
Notably, StyleFaceV is capable of generating realistic $1024\times1024$ face videos even without high-resolution training videos.
Detecting Deepfake by Creating Spatio-Temporal Regularity Disruption
Specifically, by carefully examining the spatial and temporal properties, we propose to disrupt a real video through a Pseudo-fake Generator and create a wide range of pseudo-fake videos for training.
TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformers
In this paper, we propose a novel data augmentation technique TokenMix to improve the performance of vision transformers.
Delving into Sequential Patches for Deepfake Detection
Recent advances in face forgery techniques produce nearly visually untraceable deepfake videos, which could be leveraged with malicious intentions.
Agriculture-Vision Challenge 2022 -- The Runner-Up Solution for Agricultural Pattern Recognition via Transformer-based Models
The Agriculture-Vision Challenge in CVPR is one of the most famous and competitive challenges for global researchers to break the boundary between computer vision and agriculture sectors, aiming at agricultural pattern recognition from aerial images.
MultiEarth 2022 -- The Champion Solution for Image-to-Image Translation Challenge via Generation Models
The MultiEarth 2022 Image-to-Image Translation challenge provides a well-constrained test bed for generating the corresponding RGB Sentinel-2 imagery with the given Sentinel-1 VV & VH imagery.
MultiEarth 2022 -- The Champion Solution for the Matrix Completion Challenge via Multimodal Regression and Generation
Earth observation satellites have been continuously monitoring the earth environment for years at different locations and spectral bands with different modalities.
Image Protection for Robust Cropping Localization and Recovery
Existing image cropping detection schemes ignore that recovering the cropped-out contents can unveil the purpose of the behaved cropping attack.
EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model
Although significant progress has been made to audio-driven talking face generation, existing methods either neglect facial emotion or cannot be applied to arbitrary subjects.
Few-Shot Head Swapping in the Wild
Particularly, seamless blending is achieved with the help of a Semantic-Guided Color Reference Creation procedure and a Blending UNet.
Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing
This paper focuses on the weakly-supervised audio-visual video parsing task, which aims to recognize all events belonging to each modality and localize their temporal boundaries.
SeCo: Separating Unknown Musical Visual Sounds with Consistency Guidance
Recent years have witnessed the success of deep learning on the visual sound separation task.
Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation
To enhance the quality of synthesized gestures, we develop a contrastive learning strategy based on audio-text alignment for better audio representations.
Ranked #3 on
Gesture Generation
on TED Gesture Dataset
Shape-invariant 3D Adversarial Point Clouds
In this paper, we propose a novel Point-Cloud Sensitivity Map to boost both the efficiency and imperceptibility of point perturbations.
Visual Sound Localization in the Wild by Cross-Modal Interference Erasing
Specifically, we observe that the previous practice of learning only a single audio representation is insufficient due to the additive nature of audio signals.
STC: Spatio-Temporal Contrastive Learning for Video Instance Segmentation
In contrast, we present a simple and efficient single-stage VIS framework based on the instance segmentation method CondInst by adding an extra tracking head.
Ranked #27 on
Video Instance Segmentation
on YouTube-VIS validation
Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation
Moreover, to enable portrait rendering in one unified neural radiance field, a Torso Deformation module is designed to stabilize the large-scale non-rigid torso motions.
Expressive Talking Head Generation With Granular Audio-Visual Control
Generating expressive talking heads is essential for creating virtual humans.
SAC-GAN: Structure-Aware Image Composition
Specifically, our network takes the semantic layout features from the input scene image, features encoded from the edges and silhouette in the input object patch, as well as a latent code as inputs, and generates a 2D spatial affine transform defining the translation and scaling of the object patch.
SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation
Depth models trained with SUB-Depth outperform the same models trained in a standard single-task SDE framework.
Hyperspectral Mixed Noise Removal via Subspace Representation and Weighted Low-rank Tensor Regularization
Recently, the low-rank property of different components extracted from the image has been considered in man hyperspectral image denoising methods.
From Image to Imuge: Immunized Image Generation
At the recipient's side, the verifying network localizes the malicious modifications, and the original content can be approximately recovered by the decoder, despite the presence of the attacks.
Self-Supervised Monocular Depth Estimation with Internal Feature Fusion
Therefore, it is natural to exploit semantic segmentation networks for depth estimation.
Ranked #5 on
Unsupervised Monocular Depth Estimation
on KITTI-C
Hiding Images into Images with Real-world Robustness
The existing image embedding networks are basically vulnerable to malicious attacks such as JPEG compression and noise adding, not applicable for real-world copyright protection tasks.
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
While speech content information can be defined by learning the intrinsic synchronization between audio-visual modalities, we identify that a pose code will be complementarily learned in a modulated convolution-based reconstruction framework.
Audio-Driven Emotional Video Portraits
In this work, we present Emotional Video Portraits (EVP), a system for synthesizing high-quality video portraits with vivid emotional dynamics driven by audios.
Visually Informed Binaural Audio Generation without Binaural Audios
Moreover, combined with binaural recordings, our method is able to further boost the performance of binaural audio generation under supervised settings.
Reversible Watermarking in Deep Convolutional Neural Networks for Integrity Authentication
Specifically, we present the reversible watermarking problem of deep convolutional neural networks and utilize the pruning theory of model compression technology to construct a host sequence used for embedding watermarking information by histogram shift.
Adversarial Examples Detection beyond Image Space
To detect both few-perturbation attacks and large-perturbation attacks, we propose a method beyond image space by a two-stream architecture, in which the image stream focuses on the pixel artifacts and the gradient stream copes with the confidence artifacts.
LG-GAN: Label Guided Adversarial Network for Flexible Targeted Attack of Point Cloud-based Deep Networks
To overcome these shortcomings, this paper proposes a novel label guided adversarial network (LG-GAN) for real-time flexible targeted point cloud attack.
Discriminability Distillation in Group Representation Learning
We show the discrimiability knowledge has good properties that can be distilled by a light-weight distillation network and can be generalized on the unseen target set.
Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation
Stereophonic audio is an indispensable ingredient to enhance human auditory experience.
LG-GAN: Label Guided Adversarial Network for Flexible Targeted Attack of Point Cloud Based Deep Networks
To overcome these shortcomings, this paper proposes a novel label guided adversarial network (LG-GAN) for real-time flexible targeted point cloud attack.
Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images
Though face rotation has achieved rapid progress in recent years, the lack of high-quality paired training data remains a great hurdle for existing methods.
Powerful Speaker Embedding Training Framework by Adversarially Disentangled Identity Representation
The main challenge of speaker verification in the wild is the interference caused by irrelevant information in speech and the lack of speaker labels in speech datasets.
Visual Summarization of Scholarly Videos using Word Embeddings and Keyphrase Extraction
Effective learning with audiovisual content depends on many factors.
Self-supervised Adversarial Training
Recent work has demonstrated that neural networks are vulnerable to adversarial examples.
A Graph-Based Framework to Bridge Movies and Synopses
On top of this dataset, we develop a framework to perform matching between movie segments and synopsis paragraphs.
Vision-Infused Deep Audio Inpainting
Extensive experiments demonstrate that our framework is capable of inpainting realistic and varying audio segments with or without visual contexts.
ET-GAN: Cross-Language Emotion Transfer Based on Cycle-Consistent Generative Adversarial Networks
Despite the remarkable progress made in synthesizing emotional speech from text, it is still challenging to provide emotion information to existing speech segments.
DUP-Net: Denoiser and Upsampler Network for 3D Adversarial Point Clouds Defense
We propose a Denoiser and UPsampler Network (DUP-Net) structure as defenses for 3D adversarial point cloud classification, where the two modules reconstruct surface smoothness by dropping or adding points.
Hierarchical Neural Network Architecture In Keyword Spotting
Keyword Spotting (KWS) provides the start signal of ASR problem, and thus it is essential to ensure a high recall rate.
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation
Talking face generation aims to synthesize a sequence of face images that correspond to a clip of speech.
