Search Results for author:
Found 121 papers, 50 papers with code
Contextual Transformer Networks for Visual Recognition
Such design fully capitalizes on the contextual information among input keys to guide the learning of dynamic attention matrix and thus strengthens the capacity of visual representation.
Ranked #288 on
Image Classification
on ImageNet
Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning
Motivated by the wavelet theory, we construct a new Wavelet Vision Transformer (\textbf{Wave-ViT}) that formulates the invertible down-sampling with wavelet transforms and self-attention learning in a unified way.
Ranked #209 on
Image Classification
on ImageNet
X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics
Nevertheless, there has not been an open-source codebase in support of training and deploying numerous neural network models for cross-modal analytics in a unified and modular fashion.
Comprehending and Ordering Semantics for Image Captioning
In this paper, we propose a new recipe of Transformer-style structure, namely Comprehending and Ordering Semantics Networks (COS-Net), that novelly unifies an enriched semantic comprehending and a learnable semantic ordering processes into a single architecture.
Semantic-Conditional Diffusion Networks for Image Captioning
The rich semantics are further regarded as semantic prior to trigger the learning of Diffusion Transformer, which produces the output sentence in a diffusion process.
daBNN: A Super Fast Inference Framework for Binary Neural Networks on ARM devices
It is always well believed that Binary Neural Networks (BNNs) could drastically accelerate the inference efficiency by replacing the arithmetic operations in float-valued Deep Neural Networks (DNNs) with bit-wise operations.
Sogou Machine Reading Comprehension Toolkit
In this paper, we present a Sogou Machine Reading Comprehension (SMRC) toolkit that can be used to provide the fast and efficient development of modern machine comprehension models, including both published models and original prototypes.
Relation Distillation Networks for Video Object Detection
In this paper, we introduce a new design to capture the interactions across the objects in spatio-temporal context.
Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
In this paper, we devise multiple variants of bottleneck building blocks in a residual learning framework by simulating $3\times3\times3$ convolutions with $1\times3\times3$ convolutional filters on spatial domain (equivalent to 2D CNN) plus $3\times1\times1$ convolutions to construct temporal connections on adjacent feature maps in time.
Ranked #7 on
Action Recognition
on Sports-1M
X-Linear Attention Networks for Image Captioning
Recent progress on fine-grained visual recognition and visual question answering has featured Bilinear Pooling, which effectively models the 2$^{nd}$ order interactions across multi-modal inputs.
Ranked #21 on
Image Captioning
on COCO Captions
Dual Vision Transformer
Dual-ViT is henceforth able to reduce the computational complexity without compromising much accuracy.
SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning
In this paper, we compose a trilogy of exploring the basic and generic supervision in the sequence from spatial, spatiotemporal and sequential perspectives.
T2Ranking: A large-scale Chinese Benchmark for Passage Ranking
T2Ranking comprises more than 300K queries and over 2M unique passages from real-world search engines.
Learning a Unified Sample Weighting Network for Object Detection
To this end, we devise a general loss function to cover most region-based object detectors with various sampling strategies, and then based on it we propose a unified sample weighting network to predict a sample's task weights.
3D Human Pose Estimation With Spatio-Temporal Criss-Cross Attention
On this basis, we devise STCFormer by stacking multiple STC blocks and further integrate a new Structure-enhanced Positional Embedding (SPE) into STCFormer to take the structure of human body into consideration.
Ranked #6 on
3D Human Pose Estimation
on MPI-INF-3DHP
Generalized One-shot Domain Adaptation of Generative Adversarial Networks
While previous works mainly focus on style transfer, we propose a novel and concise framework to address the \textit{generalized one-shot adaptation} task for both style and entity transfer, in which a reference image and its binary entity mask are provided.
Multi-Source Domain Adaptation and Semi-Supervised Domain Adaptation with Focus on Visual Domain Adaptation Challenge 2019
Semi-Supervised Domain Adaptation: For this task, we adopt a standard self-learning framework to construct a classifier based on the labeled source and target data, and generate the pseudo labels for unlabeled target data.
Lightweight and Progressively-Scalable Networks for Semantic Segmentation
In this paper, we thoroughly analyze the design of convolutional blocks (the type of convolutions and the number of channels in convolutions), and the ways of interactions across multiple scales, all from lightweight standpoint for semantic segmentation.
Exploring Object Relation in Mean Teacher for Cross-Domain Detection
The whole architecture is then optimized with three consistency regularizations: 1) region-level consistency to align the region-level predictions between teacher and student, 2) inter-graph consistency for matching the graph structures between teacher and student, and 3) intra-graph consistency to enhance the similarity between regions of same class within the graph of student.
Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detection
Such design decomposes the process of HOI set prediction into two subsequent phases, i. e., an interaction proposal generation is first performed, and then followed by transforming the non-parametric interaction proposals into HOI predictions via a structure-aware Transformer.
Ranked #3 on
Human-Object Interaction Detection
on V-COCO
Joint Contrastive Learning with Infinite Possibilities
This paper explores useful modifications of the recent development in contrastive learning via novel probabilistic modeling.
A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction
Neural models have achieved great success on machine reading comprehension (MRC), many of which typically consist of two components: an evidence extractor and an answer predictor.
Stand-Alone Inter-Frame Attention in Video Models
In this paper, we present a new recipe of inter-frame attention block, namely Stand-alone Inter-Frame Attention (SIFA), that novelly delves into the deformation across frames to estimate local self-attention on each spatial location.
Ranked #13 on
Action Recognition
on Something-Something V1
ReCO: A Large Scale Chinese Reading Comprehension Dataset on Opinion
The release of ReCO consists of 300k questions that to our knowledge is the largest in Chinese reading comprehension.
3D Cascade RCNN: High Quality Object Detection in Point Clouds
Recent progress on 2D object detection has featured Cascade RCNN, which capitalizes on a sequence of cascade detectors to progressively improve proposal quality, towards high-quality object detection.
ChiQA: A Large Scale Image-based Real-World Question Answering Dataset for Multi-Modal Understanding
However, in most of the public visual question answering datasets such as VQA, CLEVR, the questions are human generated that specific to the given image, such as `What color are her eyes?'.
Learning To Generate Language-Supervised and Open-Vocabulary Scene Graph Using Pre-Trained Visual-Semantic Space
Specifically, cheap scene graph supervision data can be easily obtained by parsing image language descriptions into semantic graphs.
3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models
In this work, we propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models.
Single Shot Video Object Detector
Single shot detectors that are potentially faster and simpler than two-stage detectors tend to be more applicable to object detection in videos.
PointClustering: Unsupervised Point Cloud Pre-Training Using Transformation Invariance in Clustering
Feature invariance under different data transformations, i. e., transformation invariance, can be regarded as a type of self-supervision for representation learning.
Silver-Bullet-3D at ManiSkill 2021: Learning-from-Demonstrations and Heuristic Rule-based Methods for Object Manipulation
This paper presents an overview and comparative analysis of our systems designed for the following two tracks in SAPIEN ManiSkill Challenge 2021: No Interaction Track: The No Interaction track targets for learning policies from pre-collected demonstration trajectories.
CORE-Text: Improving Scene Text Detection with Contrastive Relational Reasoning
In this work, we quantitatively analyze the sub-text problem and present a simple yet effective design, COntrastive RElation (CORE) module, to mitigate that issue.
Learning Orthogonal Prototypes for Generalized Few-Shot Semantic Segmentation
POP builds a set of orthogonal prototypes, each of which represents a semantic class, and makes the prediction for each class separately based on the features projected onto its prototype.
Learning to Localize Actions from Moments
In this paper, we introduce a new design of transfer learning type to learn action localization for a large set of action categories, but only on action moments from the categories of interest and temporal annotations of untrimmed videos from a small set of action classes.
Dynamic Temporal Filtering in Video Models
The pre-determined kernel size severely limits the temporal receptive fields and the fixed weights treat each spatial location across frames equally, resulting in sub-optimal solution for long-range temporal modeling in natural scenes.
CARIS: Context-Augmented Referring Image Segmentation
Technically, CARIS develops a context-aware mask decoder with sequential bidirectional cross-modal attention to integrate the linguistic features with visual context, which are then aligned with pixel-wise visual features.
ComQA:Compositional Question Answering via Hierarchical Graph Neural Networks
In compositional question answering, the systems should assemble several supporting evidence from the document to generate the final answer, which is more difficult than sentence-level or phrase-level QA.
Motion-Focused Contrastive Learning of Video Representations
To this end, we compose a duet of exploiting the motion for data augmentation and feature learning in the regime of contrastive learning.
Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning
Moreover, the inherently recurrent dependency in RNN prevents parallelization within a sequence during training and therefore limits the computations.
SPE-Net: Boosting Point Cloud Analysis via Rotation Robustness Enhancement
In this paper, we propose a novel deep architecture tailored for 3D point cloud applications, named as SPE-Net.
Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
Despite having impressive vision-language (VL) pretraining with BERT-based encoder for VL understanding, the pretraining of a universal encoder-decoder for both VL understanding and generation remains challenging.
Out-of-Distribution Detection with Hilbert-Schmidt Independence Optimization
Existing works attempt to solve the problem by explicitly imposing uncertainty on classifiers when OOD inputs are exposed to the classifier during training.
Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration
Our work reveals a structured shortcoming of the existing mainstream self-supervised learning methods.
Gaussian Temporal Awareness Networks for Action Localization
Temporally localizing actions in a video is a fundamental challenge in video understanding.
DPTDR: Deep Prompt Tuning for Dense Passage Retrieval
We believe this work facilitates the industry, as it saves enormous efforts and costs of deployment and increases the utility of computing resources.
Optimization Planning for 3D ConvNets
In this paper, we decompose the path into a series of training "states" and specify the hyper-parameters, e. g., learning rate and the length of input clips, in each state.
Fully Convolutional Adaptation Networks for Semantic Segmentation
The recent advances in deep neural networks have convincingly demonstrated high capability in learning vision models on large datasets.
Memory Matching Networks for One-Shot Image Recognition
In this paper, we introduce the new ideas of augmenting Convolutional Neural Networks (CNNs) with Memory and learning to learn the network parameters for the unlabelled images on the fly in one-shot learning.
Deep Semantic Hashing with Generative Adversarial Networks
Specifically, a novel deep semantic hashing with GANs (DSH-GANs) is presented, which mainly consists of four components: a deep convolution neural networks (CNN) for learning image representations, an adversary stream to distinguish synthetic images from real ones, a hash stream for encoding image representations to hash codes and a classification stream.
Jointly Localizing and Describing Events for Dense Video Captioning
A valid question is how to temporally localize and then describe events, which is known as "dense video captioning."
To Create What You Tell: Generating Videos from Captions
In this paper, we present a novel Temporal GANs conditioning on Captions, namely TGANs-C, in which the input to the generator network is a concatenation of a latent noise vector and caption embedding, and then is transformed into a frame sequence with 3D spatio-temporal convolutions.
Deep Learning for Video Classification and Captioning
Accelerated by the tremendous increase in Internet bandwidth and storage space, video data has been generated, published and spread explosively, becoming an indispensable part of today's big data.
Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
Image captioning often requires a large set of training image-sentence pairs.
Deep Quantization: Encoding Convolutional Activations with Deep Generative Model
In this paper, we present Fisher Vector encoding with Variational Auto-Encoder (FV-VAE), a novel deep architecture that quantizes the local activations of convolutional layer in a deep generative model, by training them in an end-to-end manner.
Video Captioning with Transferred Semantic Attributes
Automatically generating natural language descriptions of videos plays a fundamental challenge for computer vision community.
Boosting Image Captioning with Attributes
Automatically describing an image with a natural language has been an emerging challenge in both fields of computer vision and natural language processing.
Jointly Modeling Embedding and Translation to Bridge Video and Language
Our proposed LSTM-E consists of three components: a 2-D and/or 3-D deep convolutional neural networks for learning powerful video representation, a deep RNN for generating sentences, and a joint embedding model for exploring the relationships between visual content and sentence semantics.
YH Technologies at ActivityNet Challenge 2018
This notebook paper presents an overview and comparative analysis of our systems designed for the following five tasks in ActivityNet Challenge 2018: temporal action proposals, temporal action localization, dense-captioning events in videos, trimmed action recognition, and spatio-temporal action localization.
Exploring Visual Relationship for Image Captioning
Technically, we build graphs over the detected objects in an image based on their spatial and semantic connections.
Recurrent Tubelet Proposal and Recognition Networks for Action Detection
The RTP initializes action proposals of the start frame through a Region Proposal Network and then estimates the movements of proposals in next frame in a recurrent manner.
Learning Query and Image Similarities With Ranking Canonical Correlation Analysis
One of the fundamental problems in image search is to learn the ranking functions, i. e., similarity between the query and image.
Semi-Supervised Domain Adaptation With Subspace Learning for Visual Recognition
In many real-world applications, we are often facing the problem of cross domain learning, i. e., to borrow the labeled data or transfer the already learnt knowledge from a source domain to a target domain.
You Lead, We Exceed: Labor-Free Video Concept Learning by Jointly Exploiting Web Videos and Images
The Web images are then filtered by the learnt network and the selected images are additionally fed into the network to enhance the architecture and further trim the videos.
Highlight Detection With Pairwise Deep Ranking for First-Person Video Summarization
The emergence of wearable devices such as portable cameras and smart glasses makes it possible to record life logging first-person videos.
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
In this paper we present MSR-VTT (standing for "ABC-Video to Text") which is a new large-scale video benchmark for video understanding, especially the emerging task of translating video to text.
Pointing Novel Objects in Image Captioning
Image captioning has received significant attention with remarkable improvements in recent advances.
Transferrable Prototypical Networks for Unsupervised Domain Adaptation
Specifically, we present Transferrable Prototypical Networks (TPN) for adaptation such that the prototypes for each class in source and target domains are close in the embedding space and the score distributions predicted by prototypes separately on source and target data are similar.
Learning Spatio-Temporal Representation with Local and Global Diffusion
Diffusions effectively interact two aspects of information, i. e., localized and holistic, for more powerful way of representation learning.
Ranked #8 on
Action Recognition
on UCF101
Trimmed Action Recognition, Dense-Captioning Events in Videos, and Spatio-temporal Action Localization with Focus on ActivityNet Challenge 2019
This notebook paper presents an overview and comparative analysis of our systems designed for the following three tasks in ActivityNet Challenge 2019: trimmed action recognition, dense-captioning events in videos, and spatio-temporal action localization.
vireoJD-MM at Activity Detection in Extended Videos
This notebook paper presents an overview and comparative analysis of our system designed for activity detection in extended videos (ActEV-PC) in ActivityNet Challenge 2019.
Convolutional Auto-encoding of Sentence Topics for Image Paragraph Generation
A valid question is how to encapsulate such gists/topics that are worthy of mention from an image, and then describe the image from one topic to another but holistically with a coherent structure.
Customizable Architecture Search for Semantic Segmentation
As a result, our CAS is able to search an optimized architecture with customized constraints.
Mocycle-GAN: Unpaired Video-to-Video Translation
Unsupervised image-to-image translation is the task of translating an image from one domain to another in the absence of any paired training examples and tends to be more applicable to practical applications.
Hierarchy Parsing for Image Captioning
It is always well believed that parsing an image into constituent visual patterns would be helpful for understanding and representing an image.
Deep Metric Learning with Density Adaptivity
The problem of distance metric learning is mostly considered from the perspective of learning an embedding space, where the distances between pairs of examples are in correspondence with a similarity metric.
Scheduled Differentiable Architecture Search for Visual Recognition
Moreover, we enlarge the search space of SDAS particularly for video recognition by devising several unique operations to encode spatio-temporal dynamics and demonstrate the impact in affecting the architecture search of SDAS.
Vision and Language: from Visual Perception to Content Creation
The real-world deployment or services of vision and language are elaborated as well.
Long Short-Term Relation Networks for Video Action Detection
It has been well recognized that modeling human-object or object-object relations would be helpful for detection task.
Transferring and Regularizing Prediction for Semantic Segmentation
In the view of extremely expensive expert labeling, recent research has shown that the models trained on photo-realistic synthetic data (e. g., computer games) with computer-generated annotations can be adapted to real images.
Ranked #17 on
Domain Adaptation
on SYNTHIA-to-Cityscapes
Exploring Category-Agnostic Clusters for Open-Set Domain Adaptation
A clustering branch is capitalized on to ensure that the learnt representation preserves such underlying structure by matching the estimated assignment distribution over clusters to the inherent cluster distribution for each target sample.
Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-training
In this work, we present Auto-captions on GIF, which is a new large-scale pre-training dataset for generic video understanding.
Pre-training for Video Captioning Challenge 2020 Summary
The Pre-training for Video Captioning Challenge 2020 Summary: results and challenge participants' technical reports.
Three-dimensional cell culture model for hepatocytes opens a new avenue of real world research on liver
3-demensional (3D) culture model is a valuable in vitro tool to study liver biology, metabolism, organogenesis, tissue morphology, drug discovery and cell-based assays.
A Low Rank Promoting Prior for Unsupervised Contrastive Learning
In this paper, we construct a novel probabilistic graphical model that effectively incorporates the low rank promoting prior into the framework of contrastive learning, referred to as LORAC.
A Style and Semantic Memory Mechanism for Domain Generalization
Correspondingly, we also propose a novel "jury" mechanism, which is particularly effective in learning useful semantic feature commonalities among domains.
Ranked #37 on
Domain Generalization
on PACS
Transferrable Contrastive Learning for Visual Domain Adaptation
From this point, we present a particular paradigm of self-supervised learning tailored for domain adaptation, i. e., Transferrable Contrastive Learning (TCL), which links the SSL and the desired cross-domain transferability congruently.
CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising
BERT-type structure has led to the revolution of vision-language pre-training and the achievement of state-of-the-art results on numerous vision-language downstream tasks.
Responsive Listening Head Generation: A Benchmark Dataset and Baseline
Automatically synthesizing listening behavior that actively responds to a talking head, is critical to applications such as digital human, virtual agents and social robots.
Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training
Vision-language pre-training has been an emerging and fast-developing research topic, which transfers multi-modal knowledge from rich-resource pre-training task to limited-resource downstream tasks.
Smart Director: An Event-Driven Directing System for Live Broadcasting
To our best knowledge, our system is the first end-to-end automated directing system for multi-camera sports broadcasting, completely driven by the semantic understanding of sports events.
Representing Videos as Discriminative Sub-graphs for Action Recognition
For each action category, we execute online clustering to decompose the graph into sub-graphs on each scale through learning Gaussian Mixture Layer and select the discriminative sub-graphs as action prototypes for recognition.
Boosting Video Representation Learning with Multi-Faceted Integration
Video content is multifaceted, consisting of objects, scenes, interactions or actions.
Condensing a Sequence to One Informative Frame for Video Recognition
This paper studies a two-step alternative that first condenses the video sequence to an informative "frame" and then exploits off-the-shelf image recognition system on the synthetic frame.
Visualizing and Understanding Patch Interactions in Vision Transformer
Vision Transformer (ViT) has become a leading tool in various computer vision tasks, owing to its unique self-attention mechanism that learns visual representations explicitly through cross-patch information interactions.
MLP-3D: A MLP-like 3D Architecture with Grouped Time Mixing
By deriving the novel grouped time mixing (GTM) operations, we equip the basic token-mixing MLP with the ability of temporal modeling.
Ranked #21 on
Action Recognition
on Something-Something V1
Bridging the Gap Between Training and Inference of Bayesian Controllable Language Models
Large-scale pre-trained language models have achieved great success on natural language generation tasks.
Bi-Calibration Networks for Weakly-Supervised Video Representation Learning
The video-to-text/video-to-query projections over text prototypes/query vocabulary then start the text-to-query or query-to-text calibration to estimate the amendment to query or text.
Scale Attention for Learning Deep Face Representation: A Study Against Visual Scale Variation
Such multi-shot scheme brings inference burden, and the predefined scales inevitably have gap from real data.
Explaining Cross-Domain Recognition with Interpretable Deep Classifier
In this paper, we ask the question: how much each sample in source domain contributes to the network's prediction on the samples from target domain.
Modality-Agnostic Debiasing for Single Domain Generalization
We validate the superiority of our MAD in a variety of single-DG scenarios with different modalities, including recognition on 1D texts, 2D images, 3D point clouds, and semantic segmentation on 2D images.
Transforming Radiance Field with Lipschitz Network for Photorealistic 3D Scene Stylization
Simply coupling NeRF with photorealistic style transfer (PST) will result in cross-view inconsistency and degradation of stylized view syntheses.
HGNet: Learning Hierarchical Geometry From Points, Edges, and Surfaces
Next, as every two neighbor edges compose a surface, we obtain the edge-level representation of each anchor edge via surface-to-edge aggregation over all neighbor surfaces.
Visual-Aware Text-to-Speech
Dynamically synthesizing talking speech that actively responds to a listening head is critical during the face-to-face interaction.
Deep Equilibrium Multimodal Fusion
Multimodal fusion integrates the complementary information present in multiple modalities and has gained much attention recently.
Interactive Conversational Head Generation
Based on ViCo and ViCo-X, we define three novel tasks targeting the interaction modeling during the face-to-face conversation: 1) responsive listening head generation making listeners respond actively to the speaker with non-verbal signals, 2) expressive talking head generation guiding speakers to be aware of listeners' behaviors, and 3) conversational head generation to integrate the talking/listening ability in one interlocutor.
Learning and Evaluating Human Preferences for Conversational Head Generation
In this paper, we propose a novel learning-based evaluation metric named Preference Score (PS) for fitting human preference according to the quantitative evaluations across different dimensions.
Selective Volume Mixup for Video Action Recognition
In this paper, we propose a novel video augmentation strategy named Selective Volume Mixup (SV-Mix) to improve the generalization ability of deep models with limited training videos.
ObjectFusion: Multi-modal 3D Object Detection with Object-Centric Fusion
Recent progress on multi-modal 3D object detection has featured BEV (Bird-Eye-View) based fusion, which effectively unifies both LiDAR point clouds and camera images in a shared BEV space.
Learning Neural Implicit Surfaces with Object-Aware Radiance Fields
Then, we build the geometric correspondence between 2D planes and 3D meshes by rasterization, and project the estimated object regions into 3D explicit object surfaces by aggregating the object information across multiple views.
Bidirectional Knowledge Reconfiguration for Lightweight Point Cloud Analysis
Directly employing small models may result in a significant drop in performance since it is difficult for a small model to adequately capture local structure and global shape information simultaneously, which are essential clues for point cloud analysis.
ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors
To achieve this, we present a new diffusion model (ControlStyle) via upgrading a pre-trained text-to-image model with a trainable modulation network enabling more conditions of text prompts and style images.
Control3D: Towards Controllable Text-to-3D Generation
In particular, a 2D conditioned diffusion model (ControlNet) is remoulded to guide the learning of 3D scene parameterized as NeRF, encouraging each view of 3D scene aligned with the given text prompt and hand-drawn sketch.
VideoDrafter: Content-Consistent Multi-Scene Video Generation with LLM
The diffusion model incorporates the reference images as the condition and alignment to strengthen the content consistency of multi-scene videos.
HIRI-ViT: Scaling Vision Transformer with High Resolution Inputs
Instead, we present a new hybrid backbone with HIgh-Resolution Inputs (namely HIRI-ViT), that upgrades prevalent four-stage ViT to five-stage ViT tailored for high-resolution inputs.
Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution
Technically, SATeCo freezes all the parameters of the pre-trained UNet and VAE, and only optimizes two deliberately-designed spatial feature adaptation (SFA) and temporal feature alignment (TFA) modules, in the decoder of UNet and VAE.
VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation
In this work, we introduce a novel Visual Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the visual appearance knowledge in 2D visual prompt to boost text-to-3D generation.
SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer
Despite this progress, mask strategy still suffers from two inherent limitations: (a) training-inference discrepancy and (b) fuzzy relations between mask reconstruction & generative diffusion process, resulting in sub-optimal training of DiT.
TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models
Next, TRIP executes a residual-like dual-path scheme for noise prediction: 1) a shortcut path that directly takes image noise prior as the reference noise of each frame to amplify the alignment between the first frame and subsequent frames; 2) a residual path that employs 3D-UNet over noised video and static image latent codes to enable inter-frame relational reasoning, thereby easing the learning of the residual noise for each frame.
Boosting Diffusion Models with Moving Average Sampling in Frequency Domain
Diffusion models have recently brought a powerful revolution in image generation.
