Search Results for author:
Found 35 papers, 15 papers with code
Prior Knowledge and Memory Enriched Transformer for Sign Language Translation
This paper attacks the challenging problem of sign language translation (SLT), which involves not only visual and textual understanding but also additional prior knowledge learning (i. e. performing style, syntax).
ConceptGuard: Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation
In this paper, we investigate concept forgetting and concept confusion in the continual customization.
Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises
To address this challenging problem, we propose two novel strategies: sample identification with interquartile range Smoothing and unimodal assistance, and Mutual information sharing (SuMi).
Efficient Prompting for Continual Adaptation to Missing Modalities
In this paper, we formulate the dynamic missing modality problem as a continual learning task and introduce the continual multimodal missing modality task.
Low-rank Prompt Interaction for Continual Vision-Language Retrieval
Considering that the training parameters scale to the number of layers and tasks, we propose low-rank interaction-augmented decomposition to avoid memory explosion while enhancing the cross-modal association through sharing and separating common-specific low-rank factors.
OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios
We introduce ShareChatX, the first comprehensive, large-scale dataset for spoken dialogue that spans diverse scenarios.
Speech Watermarking with Discrete Intermediate Representations
In this paper, we propose DiscreteWM, a novel speech watermarking framework that injects watermarks into the discrete intermediate representations of speech.
A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter
With these designs, Wander enables token-level interactions between sequences of different modalities in a parameter-efficient way.
Bridging the Gap for Test-Time Multimodal Sentiment Analysis
However, in real-world dynamic scenarios, the distribution of target data is always changing and different from the source data used to train the model, which leads to performance degradation.
Classifier-guided Gradient Modulation for Enhanced Multimodal Learning
Existing methods to balance the training process always have some limitations on the loss functions, optimizers and the number of modalities and only consider modulating the magnitude of the gradients while ignoring the directions of the gradients.
OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup
The scaling up has brought tremendous success in the fields of vision and language in recent years.
Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition
The development of multimodal models has significantly advanced multimodal sentiment analysis and emotion recognition.
EAGER: Two-Stream Generative Recommender with Behavior-Semantic Collaboration
Generative retrieval has recently emerged as a promising approach to sequential recommendation, framing candidate item retrieval as an autoregressive sequence generation problem.
Non-confusing Generation of Customized Concepts in Diffusion Models
We tackle the common challenge of inter-concept visual confusion in compositional concept generation using text-guided diffusion models (TGDMs).
FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion
In this work, we propose FreeBind, an idea that treats multimodal representation spaces as basic units, and freely augments pre-trained unified space by integrating knowledge from extra expert spaces via "space bonds".
Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt
Recent singing-voice-synthesis (SVS) methods have achieved remarkable audio quality and naturalness, yet they lack the capability to control the style attributes of the synthesized singing explicitly.
DART: Implicit Doppler Tomography for Radar Novel View Synthesis
Simulation is an invaluable tool for radio-frequency system designers that enables rapid prototyping of various algorithms for imaging, target detection, classification, and tracking.
MPOD123: One Image to 3D Content Generation Using Mask-enhanced Progressive Outline-to-Detail Optimization
Specifically in the first stage MPOD123 utilizes the pretrained view-conditioned diffusion model to guide the outline shape optimization of the 3D content.
TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation
However, talking head translation, converting audio-visual speech (i. e., talking head video) from one language into another, still confronts several challenges compared to audio speech: (1) Existing methods invariably rely on cascading, synthesizing via both audio and text, resulting in delays and cascading errors.
Extending Multi-modal Contrastive Representations
Inspired by recent C-MCR, this paper proposes Extending Multimodal Contrastive Representation (Ex-MCR), a training-efficient and paired-data-free method to flexibly learn unified contrastive representation space for more than three modalities by integrating the knowledge of existing MCR spaces.
Variance-Aware Regret Bounds for Stochastic Contextual Dueling Bandits
Dueling bandits is a prominent framework for decision-making involving preferential feedback, a valuable feature that fits various applications involving human interaction, such as ranking, information retrieval, and recommendation systems.
Gloss Attention for Gloss-free Sign Language Translation
We find that it can provide two aspects of information for the model, 1) it can help the model implicitly learn the location of semantic boundaries in continuous sign language videos, 2) it can help the model understand the sign language video globally.
Ranked #7 on
Gloss-free Sign Language Translation
on PHOENIX14T
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
We demonstrate that OpenSR enables modality transfer from one to any in three different settings (zero-, few- and full-shot), and achieves highly competitive zero-shot performance compared to the existing few-shot and full-shot lip-reading methods.
DATE: Domain Adaptive Product Seeker for E-commerce
Then, we present two cooperative seekers to simultaneously search the image for PR and localize the product for PG.
Borda Regret Minimization for Generalized Linear Dueling Bandits
To attain this lower bound, we propose an explore-then-commit type algorithm for the stochastic setting, which has a nearly matching regret upper bound $\tilde{O}(d^{2/3} T^{2/3})$.
MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition
However, despite researchers exploring cross-lingual translation techniques such as machine translation and audio speech translation to overcome language barriers, there is still a shortage of cross-lingual studies on visual speech.
Exploring Group Video Captioning with Efficient Relational Approximation
In this study, we propose a new task, group video captioning, which aims to infer the desired content among a group of target videos and describe it with another group of related reference videos.
Generalizable Multi-linear Attention Network
The majority of existing multimodal sequential learning methods focus on how to obtain effective representations and ignore the importance of multimodal fusion.
Adaptive Sampling for Heterogeneous Rank Aggregation from Noisy Pairwise Comparisons
In heterogeneous rank aggregation problems, users often exhibit various accuracy levels when comparing pairs of items.
Dual Low-Rank Multimodal Fusion
Tensor-based fusion methods have been proven effective in multimodal fusion tasks.
SBAT: Video Captioning with Sparse Boundary-Aware Transformer
However, video captioning is a multimodal learning problem, and the video features have much redundancy between different time steps.
Rank Aggregation via Heterogeneous Thurstone Preference Models
By allowing different noise distributions, the proposed HTM model maintains the generality of Thurstone's original framework, and as such, also extends the Bradley-Terry-Luce (BTL) model for pairwise comparisons to heterogeneous populations of users.
Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning
This paper addresses the challenging task of video captioning which aims to generate descriptions for video data.
Inpatient2Vec: Medical Representation Learning for Inpatients
In addition, for unordered medical activity set, existing medical RL methods utilize a simple pooling strategy, which would result in indistinguishable contributions among the activities for learning.
Online Bayesian Collaborative Topic Regression
In this paper, we propose a novel scheme of Online Bayesian Collaborative Topic Regression (OBCTR) which is efficient and scalable for learning from data streams.
