Search Results for author:
Found 90 papers, 45 papers with code
StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completion
3) The story visualization and continuation models are trained and inferred independently, which is not user-friendly.
CreativeSynth: Creative Blending and Synthesis of Visual Arts based on Multimodal Diffusion
Large-scale text-to-image generative models have made impressive strides, showcasing their ability to synthesize a vast array of high-quality images.
Hierarchical Prompts for Rehearsal-free Continual Learning
Nonetheless, these learnable prompts tend to concentrate on the discriminatory knowledge of the current task while ignoring past task knowledge, leading to that learnable prompts still suffering from catastrophic forgetting.
Hierarchical Augmentation and Distillation for Class Incremental Audio-Visual Video Recognition
We introduce Hierarchical Augmentation and Distillation (HAD), which comprises the Hierarchical Augmentation Module (HAM) and Hierarchical Distillation Module (HDM) to efficiently utilize the hierarchical structure of data and models, respectively.
Three Heads Are Better Than One: Complementary Experts for Long-Tailed Semi-supervised Learning
For instance, on CIFAR-10-LT, CPE improves test accuracy by over 2. 22% compared to baselines.
Erasing Self-Supervised Learning Backdoor by Cluster Activation Masking
After obtaining the threat model trained on the poisoned dataset, our method can precisely detect poisonous samples based on the assumption that masking the backdoor trigger can effectively change the activation of a downstream clustering model.
MotionCrafter: One-Shot Motion Customization of Diffusion Models
The essence of a video lies in its dynamic motions, including character actions, object movements, and camera movements.
TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model
However, those textual tokens have a limited generalization ability regarding unseen domains, as they cannot dynamically adjust to the distribution of testing classes.
Test-time Adaptive Vision-and-Language Navigation
Then, these components are adaptively accumulated to pinpoint a concordant direction for fast model adaptation.
Learning Transferable Conceptual Prototypes for Interpretable Unsupervised Domain Adaptation
Despite the great progress of unsupervised domain adaptation (UDA) with the deep neural networks, current UDA models are opaque and cannot provide promising explanations, limiting their applications in the scenarios that require safe and controllable model decisions.
A Survey on Interpretable Cross-modal Reasoning
In recent years, cross-modal reasoning (CMR), the process of understanding and reasoning across different modalities, has emerged as a pivotal area with applications spanning from multimedia analysis to healthcare diagnostics.
Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection
In this paper, we for the first time explore helpful multi-modal contextual knowledge to understand novel categories for open-vocabulary object detection (OVD).
Introducing Foundation Models as Surrogate Models: Advancing Towards More Practical Adversarial Attacks
Harnessing the concept of non-robust features, we elaborate on two guiding principles for surrogate model selection to explain why the foundational model is an optimal choice for this role.
Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing
In this paper, to balance the feature learning processes of different modalities, a dynamic gradient modulation (DGM) mechanism is explored, where a novel and effective metric function is designed to measure the imbalanced feature learning between audio and visual modalities.
Multi-modal Queried Object Detection in the Wild
To address the learning inertia problem brought by the frozen detector, a vision conditioned masked language prediction strategy is proposed.
Ranked #1 on
Few-Shot Object Detection
on ODinW-35
ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models
We apply ProSpect in various personalized attribute-aware image generation applications, such as image-guided or text-driven manipulations of materials, style, and layout, achieving previously unattainable results from a single image input without fine-tuning the diffusion models.
Camera-Incremental Object Re-Identification with Identity Knowledge Evolution
In this paper, we propose a novel Identity Knowledge Evolution (IKE) framework for CIOR, consisting of the Identity Knowledge Association (IKA), Identity Knowledge Distillation (IKD), and Identity Knowledge Update (IKU).
CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding
In order to utilize vision and language pre-trained models to address the grounding problem, and reasonably take advantage of pseudo-labels, we propose CLIP-VG, a novel method that can conduct self-paced curriculum adapting of CLIP with pseudo-language labels.
Open-World Social Event Classification
To tackle these challenges, we propose an Open-World Social Event Classifier (OWSEC) model in this paper.
Visual-Language Prompt Tuning with Knowledge-guided Context Optimization
Representative CoOp-based work combines the learnable textual tokens with the class tokens to obtain specific textual knowledge.
A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learning
Our framework consists of three key components, i. e., a parallel contrastive learning scheme for style representation and style transfer, a domain enhancement module for effective learning of style distribution, and a generative network for style transfer.
Backdoor for Debias: Mitigating Model Bias with Backdoor Attack-based Artificial Bias
In this work, we found that the backdoor attack can construct an artificial bias similar to the model bias derived in standard training.
Region-Aware Diffusion for Zero-shot Text-driven Image Editing
Different from current mask-based image editing methods, we propose a novel region-aware diffusion model (RDM) for entity-level image editing, which could automatically locate the region of interest and replace it following given text prompts.
GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis
The complex scene understanding ability of CLIP enables the discriminator to accurately assess the image quality.
Ranked #3 on
Text-to-Image Generation
on CUB
Variational Causal Inference Network for Explanatory Visual Question Answering
To address these issues, we propose a Variational Causal Inference Network (VCIN) that establishes the causal correlation between predicted answers and explanations, and captures cross-modal relationships to generate rational explanations.
Ranked #1 on
Explanatory Visual Question Answering
on GQA-REX
Explanation Generation
Explanatory Visual Question Answering
+2
Active Exploration of Multimodal Complementarity for Few-Shot Action Recognition
In meta-training, we design an Active Sample Selection (ASS) module to organize query samples with large differences in the reliability of modalities into different groups based on modality-specific posterior distributions.
Cascade Evidential Learning for Open-World Weakly-Supervised Temporal Action Localization
Targeting at recognizing and localizing action instances with only video-level labels during training, Weakly-supervised Temporal Action Localization (WTAL) has achieved significant progress in recent years.
Open Set Learning
Weakly-supervised Temporal Action Localization
+1
VQACL: A Novel Visual Question Answering Continual Learning Setting
Research on continual learning has recently led to a variety of work in unimodal community, however little attention has been paid to multimodal tasks like visual question answering (VQA).
Collecting Cross-Modal Presence-Absence Evidence for Weakly-Supervised Audio-Visual Event Perception
We argue that, for an event residing in one modality, the modality itself should provide ample presence evidence of this event, while the other complementary modality is encouraged to afford the absence evidence as a reference signal.
UTM: A Unified Multiple Object Tracking Model With Identity-Aware Feature Enhancement
Recently, Multiple Object Tracking has achieved great success, which consists of object detection, feature embedding, and identity association.
Unlearnable Clusters: Towards Label-agnostic Unlearnable Examples
Furthermore, we propose to leverage VisionandLanguage Pre-trained Models (VLPMs) like CLIP as the surrogate model to improve the transferability of the crafted UCs to diverse domains.
SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification
Although significant progress has been made in few-shot learning, most of existing few-shot image classification methods require supervised pre-training on a large amount of samples of base classes, which limits their generalization ability in real world application.
Inversion-Based Style Transfer with Diffusion Models
Our key idea is to learn artistic style directly from a single painting and then guide the synthesis without providing complex textual descriptions.
DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization
Despite the impressive results of arbitrary image-guided style transfer methods, text-driven image stylization has recently been proposed for transferring a natural image into a stylized one according to textual descriptions of the target style provided by the user.
Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models
Pretrained vision-language models (VLMs) such as CLIP have shown impressive generalization capability in downstream vision tasks with appropriate text prompts.
MMT: Image-guided Story Ending Generation with Multimodal Memory Transformer
Finally, a multimodal transformer decoder constructs attention among multimodal features to learn the story dependency and generates informative, reasonable, and coherent story endings.
Ranked #1 on
Image-guided Story Ending Generation
on LSMDC-E
Draw Your Art Dream: Diverse Digital Art Synthesis with Multimodal Guided Diffusion
Extensive experimental results on the quality and quantity of the generated digital art paintings confirm the effectiveness of the combination of the diffusion model and multimodal guidance.
Integrating multi-label contrastive learning with dual adversarial graph neural networks for cross-modal retrieval
To date, most of the existing techniques mainly convert multimodal data into a common representation space where similarities in semantics between samples can be easily measured across multiple modalities.
Learning Muti-expert Distribution Calibration for Long-tailed Video Classification
Most existing state-of-the-art video classification methods assume that the training data obey a uniform distribution.
Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning
Our framework consists of three key components, i. e., a multi-layer style projector for style code encoding, a domain enhancement module for effective learning of style distribution, and a generative network for image style transfer.
MGDCF: Distance Learning via Markov Graph Diffusion for Neural Collaborative Filtering
Based on a Markov process that trades off two types of distances, we present Markov Graph Diffusion Collaborative Filtering (MGDCF) to generalize some state-of-the-art GNN-based CF models.
Ranked #2 on
Recommendation Systems
on Gowalla
Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding
Then, a commonsense-aware interaction module is designed to obtain bridged visual and text features by utilizing the learned commonsense concepts.
Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization
We target at the task of weakly-supervised action localization (WSAL), where only video-level action labels are available during model training.
CrossRectify: Leveraging Disagreement for Semi-supervised Object Detection
Semi-supervised object detection has recently achieved substantial progress.
StyTr2: Image Style Transfer With Transformers
The goal of image style transfer is to render an image with artistic features guided by a style reference while maintaining the original content.
Dynamic Scene Graph Generation via Anticipatory Pre-Training
Humans can not only see the collection of objects in visual scenes, but also identify the relationship between objects.
Dual Cluster Contrastive learning for Object Re-Identification
Unlike the individual-based updating mechanism, the centroid-based updating mechanism that applies the mean feature of each cluster to update the cluster memory can reduce the impact of individual samples.
Ranked #50 on
Person Re-Identification
on Market-1501
SSAGCN: Social Soft Attention Graph Convolution Network for Pedestrian Trajectory Prediction
In detail, when modeling social interaction, we propose a new \emph{social soft attention function}, which fully considers various interaction factors among pedestrians.
Contrastive Adaptive Propagation Graph Neural Networks for Efficient Graph Learning
Recently the field has advanced from local propagation schemes that focus on local neighbors towards extended propagation schemes that can directly deal with extended neighbors consisting of both local and high-order neighbors.
Weakly-Supervised Video Object Grounding via Causal Intervention
With this in mind, we design a unified causal framework to learn the deconfounded object-relevant association for more accurate and robust video object grounding.
GRecX: An Efficient and Unified Benchmark for GNN-based Recommendation
In this paper, we present GRecX, an open-source TensorFlow framework for benchmarking GNN-based recommendation models in an efficient and unified way.
Learning to Learn a Cold-start Sequential Recommender
The cold-start recommendation is an urgent problem in contemporary online applications.
Towards Predictable Feature Attribution: Revisiting and Improving Guided BackPropagation
Recently, backpropagation(BP)-based feature attribution methods have been widely adopted to interpret the internal mechanisms of convolutional neural networks (CNNs), and expected to be human-understandable (lucidity) and faithful to decision-making processes (fidelity).
Adaptive label-aware graph convolutional networks for cross-modal retrieval
Firstly, we construct an instance representation learning branch to transform instances of different modalities into a common representation space.
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer
Vision transformers (ViTs) have recently received explosive popularity, but the huge computational cost is still a severe issue.
Ranked #11 on
Efficient ViTs
on ImageNet-1K (with DeiT-T)
DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering
However, existing graph-based methods fail to perform multi-step reasoning well, neglecting two properties of VideoQA: (1) Even for the same video, different questions may require different amount of video clips or objects to infer the answer with relational reasoning; (2) During reasoning, appearance and motion features have complicated interdependence which are correlated and complementary to each other.
Ranked #26 on
Visual Question Answering (VQA)
on MSRVTT-QA
ECKPN: Explicit Class Knowledge Propagation Network for Transductive Few-shot Learning
Specifically, we first employ the comparison module to explore the pairwise sample relations to learn rich sample representations in the instance-level graph.
User-Guided Personalized Image Aesthetic Assessment based on Deep Reinforcement Learning
This framework leverages user interactions to retouch and rank images for aesthetic assessment based on deep reinforcement learning (DRL), and generates personalized aesthetic distribution that is more in line with the aesthetic preferences of different users.
StyTr$^2$: Image Style Transfer with Transformers
The goal of image style transfer is to render an image with artistic features guided by a style reference while maintaining the original content.
Dual adversarial graph neural networks for multi-label cross-modal retrieval
To date, most existing methods transform multimodal data into a common representation space where semantic similarities between items can be directly measured across different modalities.
Towards Corruption-Agnostic Robust Domain Adaptation
However, due to unpredictable corruptions (e. g., noise and blur) in real data like web images, domain adaptation methods are increasingly required to be corruption robust on target domains.
Health Status Prediction with Local-Global Heterogeneous Behavior Graph
However, these data streams are multi-source and heterogeneous, containing complex temporal structures with local contextual and global temporal aspects, which makes the feature learning and data joint utilization challenging.
Unveiling the Potential of Structure Preserving for Weakly Supervised Object Localization
Weakly supervised object localization(WSOL) remains an open problem given the deficiency of finding object extent information using a classification network.
Efficient Graph Deep Learning in TensorFlow with tf_geometric
We introduce tf_geometric, an efficient and friendly library for graph deep learning, which is compatible with both TensorFlow 1. x and 2. x.
Active Universal Domain Adaptation
This paper proposes a new paradigm for unsupervised domain adaptation, termed as Active Universal Domain Adaptation (AUDA), which removes all label set assumptions and aims for not only recognizing target samples from source classes but also inferring those from target-private classes by using active learning to annotate a small budget of target data.
Fast Video Moment Retrieval
To tackle this issue, we replace the cross-modal interaction module with a cross-modal common space, in which moment-query alignment is learned and efficient moment search can be performed.
Effective Label Propagation for Discriminative Semi-Supervised Domain Adaptation
For intra-domain propagation, we propose an effective self-training strategy to mitigate the noises in pseudo-labeled target domain data and improve the feature discriminability in the target domain.
Arbitrary Video Style Transfer via Multi-Channel Correlation
Towards this end, we propose Multi-Channel Correction network (MCCNet), which can be trained to fuse the exemplar style features and input content features for efficient style transfer while naturally maintaining the coherence of input videos.
DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis
To these ends, we propose a simpler but more effective Deep Fusion Generative Adversarial Networks (DF-GAN).
Ranked #4 on
Text-to-Image Generation
on CUB
(Inception score metric)
MMCGAN: Generative Adversarial Network with Explicit Manifold Prior
Since the basic assumption of conventional manifold learning fails in case of sparse and uneven data distribution, we introduce a new target, Minimum Manifold Coding (MMC), for manifold learning to encourage simple and unfolded manifold.
Distribution Aligned Multimodal and Multi-Domain Image Stylization
Currently, there are few methods that can perform both multimodal and multi-domain stylization simultaneously.
Attribute-Induced Bias Eliminating for Transductive Zero-Shot Learning
Then, an attentional graph attribute embedding is proposed to reduce the semantic bias between seen and unseen categories, which utilizes the graph operation to capture the semantic relationship between categories.
Joint Person Objectness and Repulsion for Person Search
Based on this repulsion constraint, the repulsion term is proposed to reduce the similarity of distractor images that are not most similar to the probe person.
Arbitrary Style Transfer via Multi-Adaptation Network
Arbitrary style transfer is a significant topic with research value and application prospect.
Adaptive Adversarial Logits Pairing
Specifically, AALP consists of an adaptive feature optimization module with Guided Dropout to systematically pursue fewer high-contribution features, and an adaptive sample weighting module by setting sample-specific training weights to balance between logits pairing loss and classification loss.
Dynamic Refinement Network for Oriented and Densely Packed Object Detection
However, the detection of oriented and densely packed objects remains challenging because of following inherent reasons: (1) receptive fields of neurons are all axis-aligned and of the same shape, whereas objects are usually of diverse shapes and align along various directions; (2) detection models are typically trained with generic knowledge and may not generalize well to handle specific objects at test time; (3) the limited dataset hinders the development on this task.
Multi-Attribute Guided Painting Generation
Controllable painting generation plays a pivotal role in image stylization.
Time-Guided High-Order Attention Model of Longitudinal Heterogeneous Healthcare Data
(1) It can model longitudinal heterogeneous EHRs data via capturing the 3-order correlations of different modalities and the irregular temporal impact of historical events.
A Generalization Theory based on Independent and Task-Identically Distributed Assumption
The derived generalization bound based on the ITID assumption identifies the significance of hypothesis invariance in guaranteeing generalization performance.
I Know the Relationships: Zero-Shot Action Recognition via Two-Stream Graph Convolutional Networks and Knowledge Graphs
To effectively leverage the knowledge graph, we design a novel Two-Stream Graph Convolutional Network (TS-GCN) consisting of a classifier branch and an instance branch.
Ranked #5 on
Zero-Shot Action Recognition
on Olympics
Adversarial Multimodal Network for Movie Question Answering
In AMN, as inspired by generative adversarial networks, we propose to learn multimodal feature representations by finding a more coherent subspace for video clips and the corresponding texts (e. g., subtitles and questions).
Exploring Feature Representation and Training strategies in Temporal Action Localization
Temporal action localization has recently attracted significant interest in the Computer Vision community.
Joint Pose and Expression Modeling for Facial Expression Recognition
First, the encoder-decoder structure of the generator can learn a generative and discriminative identity representation for face images.
Facial Expression Recognition
Facial Expression Recognition (FER)
+2
Depth Information Guided Crowd Counting for Complex Crowd Scenes
DigCrowd first uses the depth information of an image to segment the scene into a far-view region and a near-view region.
Understanding Deep Learning Generalization by Maximum Entropy
DNN is then regarded as approximating the feature conditions with multilayer feature learning, and proved to be a recursive solution towards maximum entropy principle.
Multi-Task Correlation Particle Filter for Robust Object Tracking
In this paper, we propose a multi-task correlation particle filter (MCPF) for robust visual tracking.
Structural Correlation Filter for Robust Visual Tracking
In this paper, we propose a novel structural correlation filter (SCF) model for robust visual tracking.
Structural Sparse Tracking
Sparse representation has been applied to visual tracking by finding the best target candidate with minimal reconstruction error by use of target templates.
Matching-CNN Meets KNN: Quasi-Parametric Human Parsing
Under the classic K Nearest Neighbor (KNN)-based nonparametric framework, the parametric Matching Convolutional Neural Network (M-CNN) is proposed to predict the matching confidence and displacements of the best matched region in the testing image for a particular semantic region in one KNN image.
Partial Occlusion Handling for Visual Tracking via Robust Part Matching
The proposed part matching tracker (PMT) has a number of attractive properties.
