Search Results for author:
Found 115 papers, 50 papers with code
SeMv-3D: Towards Semantic and Mutil-view Consistency simultaneously for General Text-to-3D Generation with Triplane Priors
Recent advancements in generic 3D content generation from text prompts have been remarkable by fine-tuning text-to-image diffusion (T2I) models or employing these T2I models as priors to learn a general text-to-3D model.
On Efficient Variants of Segment Anything Model: A Survey
The Segment Anything Model (SAM) is a foundational model for image segmentation tasks, known for its strong generalization across diverse applications.
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
This framework iteratively improve data quality through a refined combination of fine-grained perception, cognitive reasoning, and interaction evolution, generating a more complex and diverse image-text instruction dataset that empowers MLLMs with enhanced capabilities.
VQ-Flow: Taming Normalizing Flows for Multi-Class Anomaly Detection via Hierarchical Vector Quantization
Through the integration of vector quantization (VQ), we empower the flow models to distinguish different concepts of multi-class normal data in an unsupervised manner, resulting in a novel flow-based unified method, named VQ-Flow.
DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion
Moreover, we introduce a novel identity-oriented LoRA weights construction pipeline to facilitate the training process of DiffLoRA.
Leveraging Weak Cross-Modal Guidance for Coherence Modelling via Iterative Learning
The iterative cross-modal boosting also functions in inference to further enhance coherence prediction in each modality.
GalleryGPT: Analyzing Paintings with Large Multimodal Models
Specifically, we first propose a task of composing paragraph analysis for artworks, i. e., painting in this paper, only focusing on visual characteristics to formulate more comprehensive understanding of artworks.
Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization
To tackle this issue, we conducted the theoretical analysis to promote the effectiveness of contrast decoding.
AICL: Action In-Context Learning for Video Diffusion Model
The open-domain video generation models are constrained by the scale of the training video datasets, and some less common actions still cannot be generated.
Region-aware Distribution Contrast: A Novel Approach to Multi-Task Partially Supervised Learning
Aligning these distributions between corresponding regions from different tasks imparts higher flexibility and capacity to capture intra-region structures, accommodating a broader range of tasks.
Interpreting and Improving Attention From the Perspective of Large Kernel Convolution
These findings highlight the effectiveness of LKCA in bridging local and global feature modeling, offering a practical and robust solution for real-world applications with limited data and resources.
Ensemble Diversity Facilitates Adversarial Transferability
With the advent of ensemble-based attacks the transferability of generated adversarial examples is elevated by a noticeable margin despite many methods only employing superficial integration yet ignoring the diversity between ensemble models.
Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion
However in this paper we show that the potential noise in unimodal data could be well quantified and further employed to enhance more stable unimodal embeddings via contrastive learning.
ALF: Adaptive Label Finetuning for Scene Graph Generation
Scene Graph Generation (SGG) endeavors to predict the relationships between subjects and objects in a given image.
JoReS-Diff: Joint Retinex and Semantic Priors in Diffusion Model for Low-light Image Enhancement
By treating Retinex- and semantic-based priors as the condition, JoReS-Diff presents a unique perspective for establishing an diffusion model for LLIE and similar image enhancement tasks.
ProS: Prompting-to-simulate Generalized knowledge for Universal Cross-Domain Retrieval
Then, in Context-aware Simulator Learning stage, we train a Content-aware Prompt Simulator under a simulated test scenarios to produce the corresponding CaDP.
Make-A-Storyboard: A General Framework for Storyboard with Disentangled and Merged Control
Story Visualization aims to generate images aligned with story prompts, reflecting the coherence of storybooks through visual consistency among characters and scenes. Whereas current approaches exclusively concentrate on characters and neglect the visual consistency among contextually correlated scenes, resulting in independent character images without inter-image coherence. To tackle this issue, we propose a new presentation form for Story Visualization called Storyboard, inspired by film-making, as illustrated in Fig. 1. Specifically, a Storyboard unfolds a story into visual representations scene by scene.
Towards Redundancy-Free Sub-networks in Continual Learning
Catastrophic Forgetting (CF) is a prominent issue in continual learning.
Continual Referring Expression Comprehension via Dual Modular Memorization
In this paper, we propose Continual Referring Expression Comprehension (CREC), a new setting for REC, where a model is learning on a stream of incoming tasks.
SAM Meets UAP: Attacking Segment Anything Model With Universal Adversarial Perturbation
On top of the ablation study to understand various components in our proposed method, we shed light on the roles of positive and negative samples in making the generated UAP effective for attacking SAM.
Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval
In this paper, we propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
Ranked #16 on
Video Retrieval
on MSVD
DePT: Decoupled Prompt Tuning
Specifically, through an in-depth analysis of the learned features of the base and new tasks, we observe that the BNT stems from a channel bias issue, i. e., the vast majority of feature channels are occupied by base-specific knowledge, resulting in the collapse of taskshared knowledge important to new tasks.
Ranked #2 on
Prompt Engineering
on Stanford Cars
MSFlow: Multi-Scale Flow-based Framework for Unsupervised Anomaly Detection
Unsupervised anomaly detection (UAD) attracts a lot of research interest and drives widespread applications, where only anomaly-free samples are available for training.
Ranked #8 on
Anomaly Detection
on MVTec AD
Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions
With the exponential surge in diverse multi-modal data, traditional uni-modal retrieval methods struggle to meet the needs of users seeking access to data across various modalities.
Informative Scene Graph Generation via Debiasing
It integrates two components: Semantic Debiasing (SD) and Balanced Predicate Learning (BPL), for these imbalances.
Generalized Unbiased Scene Graph Generation
To the end, we propose the Multi-Concept Learning (MCL) framework, which ensures a balanced learning process across rare/ uncommon/ common concepts.
Unifying Two-Stream Encoders with Transformers for Cross-Modal Retrieval
Specifically, on two key tasks, \textit{i. e.}, image-to-text and text-to-image retrieval, HAT achieves 7. 6\% and 16. 7\% relative score improvement of Recall@1 on MSCOCO, and 4. 4\% and 11. 6\% on Flickr30k respectively.
Your Negative May not Be True Negative: Boosting Image-Text Matching with False Negative Elimination
Most existing image-text matching methods adopt triplet loss as the optimization objective, and choosing a proper negative sample for the triplet of <anchor, positive, negative> is important for effectively training the model, e. g., hard negatives make the model learn efficiently and effectively.
Part-Aware Transformer for Generalizable Person Re-identification
Based on the local similarity obtained in CSL, a Part-guided Self-Distillation (PSD) is proposed to further improve the generalization of global features.
Domain Generalization
Generalizable Person Re-identification
Feature Noise Boosts DNN Generalization under Label Noise
In this study, we introduce and theoretically demonstrate a simple feature noise method, which directly adds noise to the features of training data, can enhance the generalization of DNNs under label noise.
A Universal Unbiased Method for Classification from Aggregate Observations
This motivates the study on classification from aggregate observations (CFAO), where the supervision is provided to groups of instances, instead of individual instances.
Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Models
Results from the experiments demonstrate that there is a significant performance gap between the in-distribution (ID) and OOD settings for document images, and that fine-grained analysis of distribution shifts can reveal the brittle nature of existing pre-trained VDU models and OOD generalization algorithms.
Caterpillar: A Pure-MLP Architecture with Shifted-Pillars-Concatenation
Then, we build a pure-MLP architecture called Caterpillar by replacing the convolutional layer with the SPC module in a hybrid model of sMLPNet.
Faster Video Moment Retrieval with Point-Level Supervision
Existing VMR methods suffer from two defects: (1) massive expensive temporal annotations are required to obtain satisfying performance; (2) complicated cross-modal interaction modules are deployed, which lead to high computational cost and low efficiency for the retrieval process.
Non-Autoregressive Math Word Problem Solver with Unified Tree Structure
For evaluating the possible expression variants, we design a path-based metric to evaluate the partial accuracy of expressions of a unified tree.
Instance-Variant Loss with Gaussian RBF Kernel for 3D Cross-modal Retriveal
Hence, the performance of cross-modal retrieval methods heavily depends on the representational capacity of this embedding space.
T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering
To address these issues, we propose a novel method termed T-SciQ that aims at teaching science question answering with LLM signals.
Learning Semantic-Aware Knowledge Guidance for Low-Light Image Enhancement
To address this issue, we propose a novel semantic-aware knowledge-guided framework (SKF) that can assist a low-light enhancement model in learning rich and diverse priors encapsulated in a semantic segmentation model.
Ranked #4 on
Low-Light Image Enhancement
on LOLv2
ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understanding
Despite their effectiveness, existing methods suffer from the difficulty of low recognition accuracy in cases of multiple adjacent objects with similar appearances. To address this issue, this work intuitively introduces the human-robot interaction as a cue to facilitate the development of 3D visual grounding.
A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?
As ChatGPT goes viral, generative AI (AIGC, a. k. a AI-generated content) has made headlines everywhere because of its ability to analyze and create text, images, and beyond.
ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction
To this end, we propose a simple but effective in-context learning framework called ICL-D3IE, which enables LLMs to perform DIE with different types of demonstration examples.
Imbalanced Open Set Domain Adaptation via Moving-threshold Estimation and Gradual Alignment
To alleviate the negative effects raised by label shift in OSDA, we propose Open-set Moving-threshold Estimation and Gradual Alignment (OMEGA) - a novel architecture that improves existing OSDA methods on class-imbalanced data.
A Comprehensive Survey on Source-free Domain Adaptation
Over the past decade, domain adaptation has become a widely studied branch of transfer learning that aims to improve performance on target domains by leveraging knowledge from the source domain.
Multivariate, Multi-Frequency and Multimodal: Rethinking Graph Neural Networks for Emotion Recognition in Conversation
Yet, previous works tend to encode multimodal and contextual relationships in a loosely-coupled manner, which may harm relationship modelling.
Multilateral Semantic Relations Modeling for Image Text Retrieval
Specifically, a given query is first mapped as a probabilistic embedding to learn its true semantic distribution based on Mahalanobis distance.
Semantic Enhanced Knowledge Graph for Large-Scale Zero-Shot Learning
In this paper, we provide a new semantic enhanced knowledge graph that contains both expert knowledge and categories semantic correlation.
Visual Commonsense-aware Representation Network for Video Captioning
Generating consecutive descriptions for videos, i. e., Video Captioning, requires taking full advantage of visual representation along with the generation process.
A Lower Bound of Hash Codes' Performance
By testing on a series of hash-models, we obtain performance improvements among all of them, with an up to $26. 5\%$ increase in mean Average Precision and an up to $20. 5\%$ increase in accuracy.
Dual-branch Hybrid Learning Network for Unbiased Scene Graph Generation
Experiments show that our approach achieves a new state-of-the-art performance on VG and GQA datasets and makes a trade-off between the performance of tail predicates and head ones.
Adaptive Fine-Grained Predicates Learning for Scene Graph Generation
The performance of current Scene Graph Generation (SGG) models is severely hampered by hard-to-distinguish predicates, e. g., woman-on/standing on/walking on-beach.
KE-RCNN: Unifying Knowledge based Reasoning into Part-level Attribute Parsing
In this article, we propose a Knowledge Embedded RCNN (KE-RCNN) to identify attributes by leveraging rich knowledges, including implicit knowledge (e. g., the attribute ``above-the-hip'' for a shirt requires visual/geometry relations of shirt-hip) and explicit knowledge (e. g., the part of ``shorts'' cannot have the attribute of ``hoodie'' or ``lining'').
From Pixels to Objects: Cubic Visual Attention for Visual Question Answering
Existing visual attention models are generally planar, i. e., different channels of the last conv-layer feature map of an image share the same weight.
Structured Two-stream Attention Network for Video Question Answering
To date, visual question answering (VQA) (i. e., image QA and video QA) is still a holy grail in vision and language understanding, especially for video QA.
Thunder: Thumbnail based Fast Lightweight Image Denoising Network
To achieve promising results on removing noise from real-world images, most of existing denoising networks are formulated with complex network structure, making them impractical for deployment.
Support-set based Multi-modal Representation Enhancement for Video Captioning
Video captioning is a challenging task that necessitates a thorough comprehension of visual scenes.
Fine-Grained Predicates Learning for Scene Graph Generation
The performance of current Scene Graph Generation models is severely hampered by some hard-to-distinguish predicates, e. g., "woman-on/standing on/walking on-beach" or "woman-near/looking at/in front of-child".
Unified Multivariate Gaussian Mixture for Efficient Neural Image Compression
Modeling latent variables with priors and hyperpriors is an essential problem in variational image compression.
Relation Regularized Scene Graph Generation
Inspired by this observation, in this article, we propose a relation regularized network (R2-Net), which can predict whether there is a relationship between two objects and encode this relation into object feature refinement and better SGG.
One-shot Scene Graph Generation
Specifically, the Relational Knowledge represents the prior knowledge of relationships between entities extracted from the visual content, e. g., the visual relationships "standing in", "sitting in", and "lying in" may exist between "dog" and "yard", while the Commonsense Knowledge encodes "sense-making" knowledge like "dog can guard yard".
Semi-Supervised Video Paragraph Grounding With Contrastive Encoder
Video events grounding aims at retrieving the most relevant moments from an untrimmed video in terms of a given natural language query.
Meta Distribution Alignment for Generalizable Person Re-Identification
Domain Generalizable (DG) person ReID is a challenging task which trains a model on source domains yet generalizes well on target domains.
Fast Gradient Non-sign Methods
Based on the linearity hypothesis, under $\ell_\infty$ constraint, $sign$ operation applied to the gradients is a good choice for generating perturbations.
Salience-Guided Iterative Asymmetric Mutual Hashing for Fast Person Re-identification
Moreover, a novel iterative asymmetric mutual training strategy (IAMT) is proposed to alleviate drawbacks of common mutual learning, which can continuously refine the discriminative regions for SSB and extract regularized dark knowledge for two models as well.
From General to Specific: Informative Scene Graph Generation via Balance Adjustment
The scene graph generation (SGG) task aims to detect visual relationship triplets, i. e., subject, predicate, object, in an image, providing a structural vision layout for scene understanding.
Adversarial Energy Disaggregation for Non-intrusive Load Monitoring
Energy disaggregation, also known as non-intrusive load monitoring (NILM), challenges the problem of separating the whole-home electricity usage into appliance-specific individual consumptions, which is a typical application of data analysis.
Multi-Stage Aggregated Transformer Network for Temporal Language Localization in Videos
Then for a moment candidate, we concatenate the starting/middle/ending representations of its starting/middle/ending elements respectively to form the final moment representation.
Staircase Sign Method for Boosting Adversarial Attacks
Crafting adversarial examples for the transfer-based attack is challenging and remains a research hot spot.
Patch-wise++ Perturbation for Adversarial Targeted Attacks
Specifically, we introduce an amplification factor to the step size in each iteration, and one pixel's overall gradient overflowing the $\epsilon$-constraint is properly assigned to its surrounding regions by a project kernel.
Dual ResGCN for Balanced Scene GraphGeneration
We propose to incorporate the prior about the co-occurrence of relation pairs into the graph to further help alleviate the class imbalance issue.
Universal Weighting Metric Learning for Cross-Modal Matching
Furthermore, we introduce a new polynomial loss under the universal weighting framework, which defines a weight function for the positive and negative informative pairs respectively.
Patch-wise Attack for Fooling Deep Neural Network
By adding human-imperceptible noise to clean images, the resultant adversarial examples can fool other unknown models.
Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships
On the other hand, TSE module is used to generate sub-targets which allow agent to learn from failures.
MetaMixUp: Learning Adaptive Interpolation Policy of MixUp with Meta-Learning
Since deep networks are capable of memorizing the entire dataset, the corrupted samples generated by vanilla MixUp with a badly chosen interpolation policy will degrade the performance of networks.
Cooperative Cross-Stream Network for Discriminative Action Representation
It extracts this complementary information of different modality from a connection block, which aims at exploring correlations of different stream features.
Ranked #15 on
Action Recognition
on HMDB-51
(using extra training data)
Temporal Reasoning Graph for Activity Recognition
In this paper, we propose an efficient temporal reasoning graph (TRG) to simultaneously capture the appearance features and temporal relation between video sequences at multiple time scales.
Ranked #54 on
Action Recognition
on Something-Something V1
Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking
We propose a novel framework that achieves remarkable matching performance with acceptable model complexity.
Template-based math word problem solvers with recursive neural networks
Then, we design a recursive neural network to encode the quantity with Bi-LSTM and self attention, and infer the unknown operator nodes in a bottom-up manner.
Deep Recurrent Quantization for Generating Sequential Binary Codes
To the end, when the model is trained, a sequence of binary codes can be generated and the code length can be easily controlled by adjusting the number of recurrent iterations.
Beyond Product Quantization: Deep Progressive Quantization for Image Retrieval
In this work, we propose a deep progressive quantization (DPQ) model, as an alternative to PQ, for large scale image retrieval.
One-Shot Image-to-Image Translation via Part-Global Learning with a Multi-adversarial Framework
It is well known that humans can learn and recognize objects effectively from several limited image samples.
Exact Adversarial Attack to Image Captioning via Structured Output Learning with Latent Variables
Due to the sequential dependencies among words in a caption, we formulate the generation of adversarial noises for targeted partial captions as a structured output learning problem with latent variables.
Exploring Auxiliary Context: Discrete Semantic Transfer Hashing for Scalable Image Retrieval
To address the problem, in this paper, we propose a novel hashing approach, dubbed as \emph{Discrete Semantic Transfer Hashing} (DSTH).
A Large-scale Varying-view RGB-D Action Dataset for Arbitrary-view Human Action Recognition
Besides, we propose a View-guided Skeleton CNN (VS-CNN) to tackle the problem of arbitrary-view action recognition.
Hierarchical LSTMs with Adaptive Attention for Visual Captioning
Also, a hierarchical LSTMs is designed to simultaneously consider both low-level visual information and high-level language context information to support the caption generation.
Highly-Economized Multi-View Binary Compression for Scalable Image Clustering
How to economically cluster large-scale multi-view images is a long-standing problem in computer vision.
Generative Domain-Migration Hashing for Sketch-to-Image Retrieval
The generative model learns a mapping that the distributions of sketches can be indistinguishable from the distribution of natural images using an adversarial loss, and simultaneously learns an inverse mapping based on the cycle consistency loss in order to enhance the indistinguishability.
The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers
Solving mathematical word problems (MWPs) automatically is challenging, primarily due to the semantic gap between human-readable words and machine-understandable logics.
Leveraging Weak Semantic Relevance for Complex Video Event Classification
In this paper, we propose a novel approach to automatically maximize the utility of weak semantic annotations (formalized as the semantic relevance of video shots to the target event) to facilitate video event classification.
From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning
In this paper, we propose a generative approach, referred to as multi-modal stochastic RNNs networks (MS-RNN), which models the uncertainty observed in the data using latent stochastic variables.
Multi-Attention Network for One Shot Learning
One-shot learning is a challenging problem where the aim is to recognize a class identified by a single training image.
Matrix Tri-Factorization With Manifold Regularizations for Zero-Shot Learning
By additionally introducing manifold regularizations on visual data and semantic embeddings, the learned projection can effectively captures the geometrical manifold structure residing in both visual and semantic spaces.
Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
Specifically, the proposed framework utilizes the temporal attention for selecting specific frames to predict the related words, while the adjusted temporal attention is for deciding whether to depend on the visual information or the language context information.
Deep Region Hashing for Efficient Large-scale Instance Search from Images
Specifically, DRH is an end-to-end deep neural network which consists of object proposal, feature extraction, and hash code generation.
Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering
Along with the prosperity of recurrent neural network in modelling sequential data and the power of attention mechanism in automatically identify salient information, image captioning, a. k. a., image description, has been remarkably advanced in recent years.
Binary Subspace Coding for Query-by-Image Video Retrieval
The first approach, termed Inner-product Binary Coding (IBC), preserves the inner relationships of images and videos in a common Hamming space.
Exploiting Depth from Single Monocular Images for Object Detection and Semantic Segmentation
Augmenting RGB data with measured depth has been shown to improve the performance of a range of tasks in computer vision including object detection and semantic segmentation.
Where to Focus: Query Adaptive Matching for Instance Retrieval Using Convolutional Feature Maps
Instance retrieval requires one to search for images that contain a particular object within a large corpus.
Zero-Shot Hashing via Transferring Supervised Knowledge
Supervised knowledge e. g. semantic labels or pair-wise relationship) associated to data is capable of significantly improving the quality of hash codes and hash functions.
Bidirectional Long-Short Term Memory for Video Description
Video captioning has been attracting broad research attention in multimedia community.
What's Wrong With That Object? Identifying Images of Unusual Objects by Modelling the Detection Score Distribution
The key observation motivating our approach is that "regular object" images, "unusual object" images and "other objects" images exhibit different region-level scores in terms of both the score values and the spatial distributions.
A Survey on Learning to Hash
In this paper, we present a comprehensive survey of the learning to hash algorithms, categorize them according to the manners of preserving the similarities into: pairwise similarity preserving, multiwise similarity preserving, implicit similarity preserving, as well as quantization, and discuss their relations.
Learning Binary Codes and Binary Weights for Efficient Classification
The optimization alternatively proceeds over the binary classifiers and image hash codes.
Structured Learning of Binary Codes with Column Generation
Our column generation based method can be further generalized from the triplet loss to a general structured learning based framework that allows one to directly optimize multivariate performance measures.
Hi Detector, What's Wrong with that Object? Identifying Irregular Object From Images by Modelling the Detection Score Distribution
To address this problem, we propose a novel approach by inspecting the distribution of the detection scores at multiple image regions based on the detector trained from the "regular object" and "other objects".
Order-aware Convolutional Pooling for Video Based Action Recognition
Most video based action recognition approaches create the video-level representation by temporally pooling the features extracted at each frame.
Compositional Model based Fisher Vector Coding for Image Classification
To handle this limitation, in this paper we break the convention which assumes that a local feature is drawn from one of few Gaussian distributions.
Learning Binary Codes for Maximum Inner Product Search
Inspired by the latest advance in asymmetric hashing schemes, we propose an asymmetric binary code learning framework based on inner product fitting.
Optimal Graph Learning With Partial Tags and Multiple Features for Image and Video Annotation
In multimedia annotation, due to the time constraints and the tediousness of manual tagging, it is quite common to utilize both tagged and untagged data to improve the performance of supervised learning when only limited tagged training data are available.
Temporal Pyramid Pooling Based Convolutional Neural Networks for Action Recognition
One challenge is that video contains a varying number of frames which is incompatible to the standard input format of CNNs.
Hashing on Nonlinear Manifolds
In addition, a supervised inductive manifold hashing framework is developed by incorporating the label information, which is shown to greatly advance the semantic retrieval performance.
Hashing for Similarity Search: A Survey
Similarity search (nearest neighbor search) is a problem of pursuing the data items whose distances to a query item are the smallest from a large database.
Face Identification with Second-Order Pooling
Spatial pyramid pooling of features encoded by an over-complete dictionary has been the key component of many state-of-the-art image classification systems.
Face Image Classification by Pooling Raw Features
We propose a very simple, efficient yet surprisingly effective feature extraction method for face recognition (about 20 lines of Matlab code), which is mainly inspired by spatial pyramid pooling in generic image classification.
Optimized Cartesian $K$-Means
In OCKM, multiple sub codewords are used to encode the subvector of a data point in a subspace.
