Search Results for author:
Found 94 papers, 52 papers with code
Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves
Prompt tuning (PT) has long been recognized as an effective and efficient paradigm for transferring large pre-trained vision-language models (VLMs) to downstream tasks by learning a tiny set of context vectors.
GT23D-Bench: A Comprehensive General Text-to-3D Generation Benchmark
Our metrics are dissected into a) Textual-3D Alignment measures textual alignment with multi-granularity visual 3D representations; and b) 3D Visual Quality which considers texture fidelity, multi-view consistency, and geometry correctness.
Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models
Recent advances in Large Vision-Language Models (LVLMs) have showcased strong reasoning abilities across multiple modalities, achieving significant breakthroughs in various real-world applications.
SeMv-3D: Towards Semantic and Mutil-view Consistency simultaneously for General Text-to-3D Generation with Triplane Priors
Recent advancements in generic 3D content generation from text prompts have been remarkable by fine-tuning text-to-image diffusion (T2I) models or employing these T2I models as priors to learn a general text-to-3D model.
One-step Noisy Label Mitigation
Mitigating the detrimental effects of noisy labels on the training process has become increasingly critical, as obtaining entirely clean or human-annotated samples for large-scale pre-training tasks is often impractical.
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
This framework iteratively improve data quality through a refined combination of fine-grained perception, cognitive reasoning, and interaction evolution, generating a more complex and diverse image-text instruction dataset that empowers MLLMs with enhanced capabilities.
Any Target Can be Offense: Adversarial Example Generation via Generalized Latent Infection
Notably, compared with other generative methods, our method achieves an approximately $14. 13\%$ higher attack success rate for unknown classes and an approximately $4. 23\%$ higher success rate for known classes.
Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization
To tackle this issue, we conducted the theoretical analysis to promote the effectiveness of contrast decoding.
Text-Video Retrieval with Global-Local Semantic Consistent Learning
Then, we devise a shared local interaction module that employs several learnable queries to capture latent semantic concepts for learning fine-grained alignment.
RoScenes: A Large-scale Multi-view 3D Dataset for Roadside Perception
We introduce RoScenes, the largest multi-view roadside perception dataset, which aims to shed light on the development of vision-centric Bird's Eye View (BEV) approaches for more challenging traffic scenes.
AICL: Action In-Context Learning for Video Diffusion Model
The open-domain video generation models are constrained by the scale of the training video datasets, and some less common actions still cannot be generated.
CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model
To this end, we introduce MoELoRA to MLLMs which is effective to retain the previous instruction alignment.
Training-Free Semantic Video Composition via Pre-trained Diffusion Model
Therefore, we propose a training-free pipeline employing a pre-trained diffusion model imbued with semantic prior knowledge, which can process composite videos with broader semantic disparities.
ALF: Adaptive Label Finetuning for Scene Graph Generation
Scene Graph Generation (SGG) endeavors to predict the relationships between subjects and objects in a given image.
ProS: Prompting-to-simulate Generalized knowledge for Universal Cross-Domain Retrieval
Then, in Context-aware Simulator Learning stage, we train a Content-aware Prompt Simulator under a simulated test scenarios to produce the corresponding CaDP.
Make-A-Storyboard: A General Framework for Storyboard with Disentangled and Merged Control
Story Visualization aims to generate images aligned with story prompts, reflecting the coherence of storybooks through visual consistency among characters and scenes. Whereas current approaches exclusively concentrate on characters and neglect the visual consistency among contextually correlated scenes, resulting in independent character images without inter-image coherence. To tackle this issue, we propose a new presentation form for Story Visualization called Storyboard, inspired by film-making, as illustrated in Fig. 1. Specifically, a Storyboard unfolds a story into visual representations scene by scene.
F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis
Recently Text-to-Video (T2V) synthesis has undergone a breakthrough by training transformers or diffusion models on large-scale datasets.
Towards Redundancy-Free Sub-networks in Continual Learning
Catastrophic Forgetting (CF) is a prominent issue in continual learning.
MotionZero:Exploiting Motion Priors for Zero-shot Text-to-Video Generation
To tackle the two issues, we propose a prompt-adaptive and disentangled motion control strategy coined as MotionZero, which derives motion priors from prompts of different objects by Large-Language-Models and accordingly applies motion control of different objects to corresponding regions in disentanglement.
CUCL: Codebook for Unsupervised Continual Learning
To address this issue, after analyzing the phenomenon and identifying the lack of diversity as a vital factor, we propose a method named Codebook for Unsupervised Continual Learning (CUCL) which promotes the model to learn discriminative features to complete the class boundary.
Continual Referring Expression Comprehension via Dual Modular Memorization
In this paper, we propose Continual Referring Expression Comprehension (CREC), a new setting for REC, where a model is learning on a stream of incoming tasks.
Class Gradient Projection For Continual Learning
Catastrophic forgetting is one of the most critical challenges in Continual Learning (CL).
Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection
In light of this, we introduce SG2HOI+, a unified one-step model based on the Transformer architecture.
Less is More: On the Feature Redundancy of Pretrained Models When Transferring to Few-shot Tasks
Transferring a pretrained model to a downstream task can be as easy as conducting linear probing with target data, that is, training a linear classifier upon frozen features extracted from the pretrained model.
Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval
In this paper, we propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
Ranked #16 on
Video Retrieval
on MSVD
DePT: Decoupled Prompt Tuning
Specifically, through an in-depth analysis of the learned features of the base and new tasks, we observe that the BNT stems from a channel bias issue, i. e., the vast majority of feature channels are occupied by base-specific knowledge, resulting in the collapse of taskshared knowledge important to new tasks.
Ranked #2 on
Prompt Engineering
on Stanford Cars
CIParsing: Unifying Causality Properties into Multiple Human Parsing
Existing methods of multiple human parsing (MHP) apply statistical models to acquire underlying associations between images and labeled body parts.
From Global to Local: Multi-scale Out-of-distribution Detection
Out-of-distribution (OOD) detection aims to detect "unknown" data whose labels have not been seen during the in-distribution (ID) training process.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
+1
Informative Scene Graph Generation via Debiasing
It integrates two components: Semantic Debiasing (SD) and Balanced Predicate Learning (BPL), for these imbalances.
Local-Global Information Interaction Debiasing for Dynamic Scene Graph Generation
Long-temporal human actions supervise the model to generate multiple scene graphs that conform to the global constraints and avoid the model being unable to learn the tail predicates.
Generalized Unbiased Scene Graph Generation
To the end, we propose the Multi-Concept Learning (MCL) framework, which ensures a balanced learning process across rare/ uncommon/ common concepts.
Part-Aware Transformer for Generalizable Person Re-identification
Based on the local similarity obtained in CSL, a Part-guided Self-Distillation (PSD) is proposed to further improve the generalization of global features.
Domain Generalization
Generalizable Person Re-identification
MobileVidFactory: Automatic Diffusion-Based Social Media Video Generation for Mobile Devices from Text
In the basic generation, we take advantage of the pretrained image diffusion model, and adapt it to a high-quality open-domain vertical video generator for mobile devices.
MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images
To generate videos, we extend the capabilities of a pretrained text-to-image diffusion model through a two-stage process.
Prototype-based Embedding Network for Scene Graph Generation
Current Scene Graph Generation (SGG) methods explore contextual information to predict relationships among entity pairs.
DETA: Denoised Task Adaptation for Few-Shot Learning
Test-time task adaptation in few-shot learning aims to adapt a pre-trained task-agnostic model for capturing taskspecific knowledge of the test task, rely only on few-labeled support samples.
Boosting Adversarial Attacks by Leveraging Decision Boundary Information
However, our method still outperforms existing methods when attacking transformers.
A Closer Look at Few-shot Classification Again
Few-shot classification consists of a training phase where a model is learned on a relatively large dataset and an adaptation phase where the learned model is adapted to previously-unseen tasks with limited labeled samples.
A Differentiable Semantic Metric Approximation in Probabilistic Embedding for Cross-Modal Retrieval
To verify the effectiveness of our approach, extensive experiments are conducted on MS-COCO, CUB Captions, and Flickr30K, which are commonly used in cross-modal retrieval.
Progressive Tree-Structured Prototype Network for End-to-End Image Captioning
Specifically, we design a novel embedding method called tree-structured prototype, producing a set of hierarchical representative embeddings which capture the hierarchical semantic structure in textual space.
Visual Commonsense-aware Representation Network for Video Captioning
Generating consecutive descriptions for videos, i. e., Video Captioning, requires taking full advantage of visual representation along with the generation process.
A Lower Bound of Hash Codes' Performance
By testing on a series of hash-models, we obtain performance improvements among all of them, with an up to $26. 5\%$ increase in mean Average Precision and an up to $20. 5\%$ increase in accuracy.
Natural Color Fool: Towards Boosting Black-box Unrestricted Attacks
Unrestricted color attacks, which manipulate semantically meaningful color of an image, have shown their stealthiness and success in fooling both human eyes and deep neural networks.
RepParser: End-to-End Multiple Human Parsing with Representative Parts
Different from mainstream methods, RepParser solves the multiple human parsing in a new single-stage manner without resorting to person detection or post-grouping. To this end, RepParser decouples the parsing pipeline into instance-aware kernel generation and part-aware human parsing, which are responsible for instance separation and instance-specific part segmentation, respectively.
Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning
In this paper, we introduce open-vocabulary scene graph generation, a novel, realistic and challenging setting in which a model is trained on a set of base object classes but is required to infer relations for unseen target object classes.
Dual-branch Hybrid Learning Network for Unbiased Scene Graph Generation
Experiments show that our approach achieves a new state-of-the-art performance on VG and GQA datasets and makes a trade-off between the performance of tail predicates and head ones.
Frequency Domain Model Augmentation for Adversarial Attack
Specifically, we apply a spectrum transformation to the input and thus perform the model augmentation in the frequency domain.
Adaptive Fine-Grained Predicates Learning for Scene Graph Generation
The performance of current Scene Graph Generation (SGG) models is severely hampered by hard-to-distinguish predicates, e. g., woman-on/standing on/walking on-beach.
Skeleton-based Action Recognition via Adaptive Cross-Form Learning
Specifically, each GCN model in ACFL not only learns action representation from the single-form skeletons, but also adaptively mimics useful representations derived from other forms of skeletons.
Learning To Generate Scene Graph from Head to Tail
SCM is proposed to relieve semantic deviation by ensuring the semantic consistency between the generated scene graph and the ground truth in global and local representations.
KTN: Knowledge Transfer Network for Learning Multi-person 2D-3D Correspondences
Human densepose estimation, aiming at establishing dense correspondences between 2D pixels of human body and 3D human body template, is a key technique in enabling machines to have an understanding of people in images.
KE-RCNN: Unifying Knowledge based Reasoning into Part-level Attribute Parsing
In this article, we propose a Knowledge Embedded RCNN (KE-RCNN) to identify attributes by leveraging rich knowledges, including implicit knowledge (e. g., the attribute ``above-the-hip'' for a shirt requires visual/geometry relations of shirt-hip) and explicit knowledge (e. g., the part of ``shorts'' cannot have the attribute of ``hoodie'' or ``lining'').
From Pixels to Objects: Cubic Visual Attention for Visual Question Answering
Existing visual attention models are generally planar, i. e., different channels of the last conv-layer feature map of an image share the same weight.
Structured Two-stream Attention Network for Video Question Answering
To date, visual question answering (VQA) (i. e., image QA and video QA) is still a holy grail in vision and language understanding, especially for video QA.
Support-set based Multi-modal Representation Enhancement for Video Captioning
Video captioning is a challenging task that necessitates a thorough comprehension of visual scenes.
Fine-Grained Predicates Learning for Scene Graph Generation
The performance of current Scene Graph Generation models is severely hampered by some hard-to-distinguish predicates, e. g., "woman-on/standing on/walking on-beach" or "woman-near/looking at/in front of-child".
Unified Multivariate Gaussian Mixture for Efficient Neural Image Compression
Modeling latent variables with priors and hyperpriors is an essential problem in variational image compression.
Practical Evaluation of Adversarial Robustness via Adaptive Auto Attack
Specifically, by observing that adversarial examples to a specific defense model follow some regularities in their starting points, we design an Adaptive Direction Initialization strategy to speed up the evaluation.
Practical No-box Adversarial Attacks with Training-free Hybrid Image Transformation
In this paper, we move a step forward and show the existence of a \textbf{training-free} adversarial perturbation under the no-box threat model, which can be successfully used to attack different DNNs in real-time.
Exploiting long-term temporal dynamics for video captioning
In TS-LSTM, a temporal pooling LSTM (TP-LSTM) is designed to incorporate both spatial and temporal information to extract long-term temporal dynamics within video sub-shots; and a stacked LSTM is introduced to generate a list of words to describe the video.
One-shot Scene Graph Generation
Specifically, the Relational Knowledge represents the prior knowledge of relationships between entities extracted from the visual content, e. g., the visual relationships "standing in", "sitting in", and "lying in" may exist between "dog" and "yard", while the Commonsense Knowledge encodes "sense-making" knowledge like "dog can guard yard".
Relation Regularized Scene Graph Generation
Inspired by this observation, in this article, we propose a relation regularized network (R2-Net), which can predict whether there is a relationship between two objects and encode this relation into object feature refinement and better SGG.
Beyond ImageNet Attack: Towards Crafting Adversarial Examples for Black-box Domains
Notably, our methods outperform state-of-the-art approaches by up to 7. 71\% (towards coarse-grained domains) and 25. 91\% (towards fine-grained domains) on average.
Wnet: Audio-Guided Video Object Segmentation via Wavelet-Based Cross-Modal Denoising Networks
Audio-Guided video semantic segmentation is a challenging problem in visual analysis and editing, which automatically separates foreground objects from background in a video sequence according to the referring audio expressions.
Technical Report: Disentangled Action Parsing Networks for Accurate Part-level Action Parsing
Despite of dramatic progresses in the area of video classification research, a severe problem faced by the community is that the detailed understanding of human actions is ignored.
Fast Gradient Non-sign Methods
Based on the linearity hypothesis, under $\ell_\infty$ constraint, $sign$ operation applied to the gradients is a good choice for generating perturbations.
Adversarial Attacks on ML Defense Models Competition
Due to the vulnerability of deep neural networks (DNNs) to adversarial examples, a large number of defense techniques have been proposed to alleviate this problem in recent years.
From General to Specific: Informative Scene Graph Generation via Balance Adjustment
The scene graph generation (SGG) task aims to detect visual relationship triplets, i. e., subject, predicate, object, in an image, providing a structural vision layout for scene understanding.
Semi-supervised Network Embedding with Differentiable Deep Quantisation
Learning accurate low-dimensional embeddings for a network is a crucial task as it facilitates many downstream network analytics tasks.
Unsupervised Domain-adaptive Hash for Networks
Abundant real-world data can be naturally represented by large-scale networks, which demands efficient and effective learning algorithms.
Exploiting Scene Graphs for Human-Object Interaction Detection
Human-Object Interaction (HOI) detection is a fundamental visual task aiming at localizing and recognizing interactions between humans and objects.
Semantic Compositional Learning for Low-shot Scene Graph Generation
Scene graphs provide valuable information to many downstream tasks.
Feature Space Targeted Attacks by Statistic Alignment
However, the current choice of pixel-wise Euclidean Distance to measure the discrepancy is questionable because it unreasonably imposes a spatial-consistency constraint on the source and target features.
Revisiting Multi-Codebook Quantization
However, finding the optimal solution to MCQ is proved to be NP-hard due to its encoding process, \textit{i. e.}, converting an input vector to a binary code.
Staircase Sign Method for Boosting Adversarial Attacks
Crafting adversarial examples for the transfer-based attack is challenging and remains a research hot spot.
Patch-wise++ Perturbation for Adversarial Targeted Attacks
Specifically, we introduce an amplification factor to the step size in each iteration, and one pixel's overall gradient overflowing the $\epsilon$-constraint is properly assigned to its surrounding regions by a project kernel.
Patch-wise Attack for Fooling Deep Neural Network
By adding human-imperceptible noise to clean images, the resultant adversarial examples can fool other unknown models.
Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation
Despite the huge progress in scene graph generation in recent years, its long-tail distribution in object relationships remains a challenging and pestering issue.
A Foreground-background Parallel Compression with Residual Encoding for Surveillance Video
In this paper, we propose a video compression method that extracts and compresses the foreground and background of the video separately.
Template-based math word problem solvers with recursive neural networks
Then, we design a recursive neural network to encode the quantity with Bi-LSTM and self attention, and infer the unknown operator nodes in a bottom-up manner.
One Network for Multi-Domains: Domain Adaptive Hashing with Intersectant Generative Adversarial Network
We evaluate our framework on {four} public benchmark datasets, all of which show that our method is superior to the other state-of-the-art methods on the tasks of object recognition and image retrieval.
Beyond Product Quantization: Deep Progressive Quantization for Image Retrieval
In this work, we propose a deep progressive quantization (DPQ) model, as an alternative to PQ, for large scale image retrieval.
Deep Recurrent Quantization for Generating Sequential Binary Codes
To the end, when the model is trained, a sequence of binary codes can be generated and the code length can be easily controlled by adjusting the number of recurrent iterations.
Hierarchical LSTMs with Adaptive Attention for Visual Captioning
Also, a hierarchical LSTMs is designed to simultaneously consider both low-level visual information and high-level language context information to support the caption generation.
Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks
Being composed of node attention component and edge attention component, the proposed graph attention mechanism explicitly represents inter-object relationships, and properties with a flexibility and power impossible with competing approaches.
Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder
Existing video hash functions are built on three isolated stages: frame pooling, relaxed learning, and binarization, which have not adequately explored the temporal order of video frames in a joint binary optimization model, resulting in severe information loss.
From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning
In this paper, we propose a generative approach, referred to as multi-modal stochastic RNNs networks (MS-RNN), which models the uncertainty observed in the data using latent stochastic variables.
Learning in High-Dimensional Multimedia Data: The State of the Art
Given the high dimensionality and the high complexity of multimedia data, it is important to investigate new machine learning algorithms to facilitate multimedia data analysis.
Deep Discrete Hashing with Self-supervised Pairwise Labels
2) how to equip the binary representation with the ability of accurate image retrieval and classification in an unsupervised way?
Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
Specifically, the proposed framework utilizes the temporal attention for selecting specific frames to predict the related words, while the adjusted temporal attention is for deciding whether to depend on the visual information or the language context information.
Deep Region Hashing for Efficient Large-scale Instance Search from Images
Specifically, DRH is an end-to-end deep neural network which consists of object proposal, feature extraction, and hash code generation.
Optimal Graph Learning With Partial Tags and Multiple Features for Image and Video Annotation
In multimedia annotation, due to the time constraints and the tediousness of manual tagging, it is quite common to utilize both tagged and untagged data to improve the performance of supervised learning when only limited tagged training data are available.
Estimating Fire Weather Indices via Semantic Reasoning over Wireless Sensor Network Data Streams
Wildfires are frequent, devastating events in Australia that regularly cause significant loss of life and widespread property damage.
