Search Results for author:
Found 22 papers, 3 papers with code
The 1st EReL@MIR Workshop on Efficient Representation Learning for Multimodal Information Retrieval
Multimodal representation learning has garnered significant attention in the AI community, largely due to the success of large pre-trained multimodal foundation models like LLaMA, GPT, Mistral, and CLIP.
CROSSAN: Towards Efficient and Effective Adaptation of Multiple Multimodal Foundation Models for Sequential Recommendation
Leveraging the fully decoupled side adapter-based paradigm, CROSSAN achieves high efficiency while enabling cross-modal learning across diverse modalities.
Multimodal Representation Learning Techniques for Comprehensive Facial State Analysis
First, we compile a new Multimodal Face Dataset (MFA) by generating detailed multilevel language descriptions of face, incorporating Action Unit (AU) and emotion descriptions, by leveraging GPT-4o.
LLMPopcorn: An Empirical Study of LLMs as Assistants for Popular Micro-video Generation
(iii) How well do various LLMs and video generators perform in the popular micro-video generation task?
Multimodal Sentiment Analysis Based on Causal Reasoning
However, multimodal sentiment analysis is affected by unimodal data bias, e. g., text sentiment is misleading due to explicit sentiment semantic, leading to low accuracy in the final sentiment classification.
Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation
To overcome this, we developed IISAN-Versa, a versatile plug-and-play architecture compatible with both symmetrical and asymmetrical MFMs.
R^3AG: First Workshop on Refined and Reliable Retrieval Augmented Generation
At the end of the workshop, we aim to have a clearer understanding of how to improve the reliability and applicability of RAG with more robust information retrieval and language generation.
HpEIS: Learning Hand Pose Embeddings for Multimedia Interactive Systems
We present a novel Hand-pose Embedding Interactive System (HpEIS) as a virtual sensor, which maps users' flexible hand poses to a two-dimensional visual space using a Variational Autoencoder (VAE) trained on a variety of hand poses.
Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning
Facial action units (AUs), as defined in the Facial Action Coding System (FACS), have received significant research interest owing to their diverse range of applications in facial state analysis.
Hire: Hybrid-modal Interaction with Multiple Relational Enhancements for Image-Text Matching
In this paper, we propose a Hybrid-modal Interaction with multiple Relational Enhancements (termed \textit{Hire}) for image-text matching, which correlates the intra- and inter-modal semantics between objects and words with implicit and explicit relationship modelling.
Detail-Enhanced Intra- and Inter-modal Interaction for Audio-Visual Emotion Recognition
Capturing complex temporal relationships between video and audio modalities is vital for Audio-Visual Emotion Recognition (AVER).
3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting
In this paper, we propose a novel visual Semantic-Spatial Self-Highlighting Network (termed 3SHNet) for high-precision, high-efficiency and high-generalization image-sentence retrieval.
Ranked #1 on
Cross-Modal Retrieval
on MSCOCO
IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT
This is also a notable improvement over the Adapter and LoRA, which require 37-39 GB GPU memory and 350-380 seconds per epoch for training.
CFIR: Fast and Effective Long-Text To Image Retrieval for Large Corpora
Text-to-image retrieval aims to find the relevant images based on a text query, which is important in various use-cases, such as digital libraries, e-commerce, and multimedia databases.
The Relationship Between Speech Features Changes When You Get Depressed: Feature Correlations for Improving Speed and Performance of Depression Detection
The results show that the models used in the experiments improve in terms of training speed and performance when fed with feature correlation matrices rather than with feature vectors.
Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrieval
To correlate the context of objects with the textual context, we further refine the visual semantic representation via the cross-level object-sentence and word-image based interactive attention.
MGRR-Net: Multi-level Graph Relational Reasoning Network for Facial Action Units Detection
While the region-level feature learning from local face patches features via graph neural network can encode the correlation across different AUs, the pixel-wise and channel-wise feature learning via graph attention network can enhance the discrimination ability of AU features from global face features.
Factored Attention and Embedding for Unstructured-view Topic-related Ultrasound Report Generation
Echocardiography is widely used to clinical practice for diagnosis and treatment, e. g., on the common congenital heart defects.
Automatic Facial Paralysis Estimation with Facial Action Units
In this paper, we propose a novel Adaptive Local-Global Relational Network (ALGRNet) for facial AU detection and use it to classify facial paralysis severity.
Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval
In this work, we propose to address the above issue from two aspects: (i) constructing intrinsic structure (along with relations) among the fragments of respective modalities, e. g., "dog $\to$ play $\to$ ball" in semantic structure for an image, and (ii) seeking explicit inter-modal structural and semantic correspondence between the visual and textual modalities.
Variational Structured Semantic Inference for Diverse Image Captioning
To model these two inherent diversities in image captioning, we propose a Variational Structured Semantic Inferring model (termed VSSI-cap) executed in a novel structured encoder-inferer-decoder schema.
