Search Results for author:
Found 72 papers, 26 papers with code
SACA Net: Cybersickness Assessment of Individual Viewers for VR Content via Graph-based Symptom Relation Embedding
In this paper, we propose a novel symptom-aware cybersickness assessment network (SACA Net) that quantifies physical symptom levels for assessing cybersickness of individual viewers.
MSCoTDet: Language-driven Multi-modal Fusion for Improved Multispectral Pedestrian Detection
Specifically, we generate text descriptions of the pedestrian in each RGB and thermal modality and design a Multispectral Chain-of-Thought (MSCoT) prompting, which models a step-by-step process to facilitate cross-modal reasoning at the semantic level and perform accurate detection.
What if...?: Counterfactual Inception to Mitigate Hallucination Effects in Large Multimodal Models
This paper presents a way of enhancing the reliability of Large Multimodal Models (LMMs) in addressing hallucination effects, where models generate incorrect or unrelated responses.
MoAI: Mixture of All Intelligence for Large Language and Vision Models
Therefore, we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages auxiliary visual information obtained from the outputs of external segmentation, detection, SGG, and OCR models.
Ranked #24 on
Visual Question Answering
on MM-Vet
Persona Extraction Through Semantic Similarity for Emotional Support Conversation Generation
We devise completeness loss and consistency loss based on semantic similarity scores.
Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detection
As a result, multispectral pedestrian detectors show poor generalization ability on examples beyond this statistical correlation, such as ROTX data.
TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages
We propose a novel Tri-Modal Translation (TMT) model that translates between arbitrary modalities spanning speech, image, and text.
Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing
In visual speech processing, context modeling capability is one of the most important requirements due to the ambiguous nature of lip movements.
Ranked #4 on
Lipreading
on LRS3-TED
(using extra training data)
CoLLaVO: Crayon Large Language and Vision mOdel
Our findings reveal that the image understanding capabilities of current VLMs are strongly correlated with their zero-shot performance on vision language (VL) tasks.
Ranked #32 on
Visual Question Answering
on MM-Vet
Multilingual Visual Speech Recognition with a Single Model by Learning with Discrete Visual Speech Units
By using the visual speech units as the inputs of our system, we pre-train the model to predict corresponding text outputs on massive multilingual data constructed by merging several VSR databases.
AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation
To mitigate the problem of the absence of a parallel AV2AV translation dataset, we propose to train our spoken language translation system with the audio-only dataset of A2A.
Integrating Language-Derived Appearance Elements with Visual Cues in Pedestrian Detection
The obtained knowledge elements are adaptable to various detection frameworks, so that we can provide plentiful appearance information by integrating the language-derived appearance elements with visual cues within a detector.
Causal Unsupervised Semantic Segmentation
Unsupervised semantic segmentation aims to achieve high-quality semantic grouping without human-labeled annotations.
Ranked #1 on
Unsupervised Semantic Segmentation
on COCO-Stuff-81
Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens
To this end, we start with importing the rich knowledge related to image comprehension and language modeling from a large-scale pre-trained vision-language model into Im2Sp.
Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper
Different from previous methods that tried to improve the VSR performance for the target language by using knowledge learned from other languages, we explore whether we can increase the amount of training data itself for the different languages without human intervention.
DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion
We contribute a new large-scale 3D facial mesh dataset, 3D-HDTF to enable the synthesis of variations in identities, poses, and facial motions of 3D face mesh.
Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge
In order to mitigate the challenge, we try to learn general speech knowledge, the ability to model lip movements, from a high-resource language through the prediction of speech units.
DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embedding
In doing so, the rich speaker embedding information can be produced solely from input visual information, and the extra audio information is not necessary during the inference time.
AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model
Visual Speech Recognition (VSR) is the task of predicting spoken words from silent lip movements.
Many-to-Many Spoken Language Translation via Unified Speech and Text Representation Learning with Unit-to-Unit Translation
A single pre-trained model with UTUT can be employed for diverse multilingual speech- and text-related tasks, such as Speech-to-Speech Translation (STS), multilingual Text-to-Speech Synthesis (TTS), and Text-to-Speech Translation (TTST).
Mitigating Adversarial Vulnerability through Causal Parameter Estimation by Adversarial Double Machine Learning
Adversarial examples derived from deliberately crafted perturbations on visual inputs can easily harm decision process of deep neural networks.
Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models
The visual speaker embedding is derived from a single target face image and enables improved mapping of input text to the learned audio latent space by incorporating the speaker characteristics inherent in the audio.
Advancing Adversarial Training by Injecting Booster Signal
Different from the previous approaches, in this paper, we propose a new approach to improve the adversarial robustness by using an external signal rather than model parameters.
Robust Proxy: Improving Adversarial Robustness by Robust Proxy Learning
With the class-wise robust features, the model explicitly learns adversarially robust features through the proposed robust proxy learning framework.
Exploring Phonetic Context-Aware Lip-Sync For Talking Face Generation
The contextualized lip motion unit then guides the latter in synthesizing a target identity with context-aware lip motion.
Intelligible Lip-to-Speech Synthesis with Speech Units
Therefore, the proposed L2S model is trained to generate multiple targets, mel-spectrogram and speech units.
Multi-Temporal Lip-Audio Memory for Visual Speech Recognition
Visual Speech Recognition (VSR) is a task to predict a sentence or word from lip movements.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring
Thus, we firstly analyze that the previous AVSR models are not indeed robust to the corruption of multimodal input streams, the audio and the visual inputs, compared to uni-modal models.
Demystifying Causal Features on Adversarial Examples and Causal Inoculation for Robust Network by Adversarial Instrumental Variable Regression
The origin of adversarial examples is still inexplicable in research fields, and it arouses arguments from various viewpoints, albeit comprehensive investigations.
Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video
The proposed DVFA can align the input transcription (i. e., sentence) with the talking face video without accessing the speech audio.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Lip-to-Speech Synthesis in the Wild with Multi-task Learning
To this end, we design multi-task learning that guides the model using multimodal supervision, i. e., text and audio, to complement the insufficient word representations of acoustic feature reconstruction loss.
Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognition
As it focuses on visual information to model the speech, its performance is inherently sensitive to personal lip appearances and movements, and this makes the VSR models show degraded performance when they are applied to unseen speakers.
SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory
It stores lip motion features from sequential ground truth images in the value memory and aligns them with corresponding audio features so that they can be retrieved using audio input at inference time.
Meta Input: How to Leverage Off-the-Shelf Deep Neural Networks
The proposed meta input can be optimized with a small number of testing data only by considering the relation between testing input data and its output prediction.
Face Shape-Guided Deep Feature Alignment for Face Recognition Robust to Face Misalignment
This is mainly attributed to the mismatch between training and testing sets.
Speaker-adaptive Lip Reading with User-dependent Padding
In this paper, to remedy the performance degradation of lip reading model on unseen speakers, we propose a speaker-adaptive lip reading method, namely user-dependent padding.
Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition
The enhanced audio features are fused with the visual features and taken to an encoder-decoder model composed of Conformer and Transformer for speech recognition.
VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection
Thus, the proposed framework brings the advantage of synthesizing the speech containing the right content even with the silent talking face video of an unseen subject.
Defending Person Detection Against Adversarial Patch Attack by using Universal Defensive Frame
Person detection has attracted great attention in the computer vision area and is an imperative element in human-centric computer vision.
Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck
Adversarial examples, generated by carefully crafted perturbation, have attracted considerable attention in research fields.
Masking Adversarial Damage: Finding Adversarial Saliency for Robust and Sparse Network
Adversarial examples provoke weak reliability and potential security issues in deep neural networks.
Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video
By learning the interrelationship through the associative bridge, the proposed bridging framework is able to obtain the target modal representations inside the memory network, even with the source modal input only, and it provides rich information for its downstream tasks.
Ranked #3 on
Lipreading
on CAS-VSR-W1k (LRW-1000)
Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading
With the multi-head key memories, MVM extracts possible candidate audio features from the memory, which allows the lip reading model to consider the possibility of which pronunciations can be represented from the input lip movement.
Ranked #2 on
Lipreading
on CAS-VSR-W1k (LRW-1000)
Lip to Speech Synthesis with Visual Context Attentional GAN
In this paper, we propose a novel lip-to-speech generative adversarial network, Visual Context Attentional GAN (VCA-GAN), which can jointly model local and global lip movements during speech synthesis.
Weakly Paired Associative Learning for Sound and Image Representations via Bimodal Associative Memory
Existing sound and image representation learning methods necessarily require a large number of sound and image with corresponding pairs.
Speech Reconstruction with Reminiscent Sound via Visual Voice Memory
Our key contributions are: (1) proposing the Visual Voice memory that brings rich information of audio that complements the visual features, thus producing high-quality speech from silent video, and (2) enabling multi-speaker and unseen speaker training by memorizing auditory features and the corresponding visual features.
Towards a Better Understanding of VR Sickness: Physical Symptom Prediction for VR Contents
To make better understanding of VR sickness, it is required to predict and provide the level of major symptoms of VR sickness rather than overall degree of VR sickness.
Visual Comfort Aware-Reinforcement Learning for Depth Adjustment of Stereoscopic 3D Images
For a human expert, the depth adjustment procedure is a sequence of iterative decision making.
Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning
Our work addresses long-term motion context issues for predicting future frames.
Ranked #1 on
Video Prediction
on KTH
(Cond metric)
Robust Small-Scale Pedestrian Detection With Cued Recall via Memory Learning
The purpose of the proposed large-scale embedding learning is to memorize and recall the large-scale pedestrian appearance via the LPR Memory.
Towards Adversarial Robustness of Bayesian Neural Network through Hierarchical Variational Inference
Recent works have applied Bayesian Neural Network (BNN) to adversarial training, and shown the improvement of adversarial robustness via the BNN's strength of stochastic gradient defense.
Comprehensive Facial Expression Synthesis using Human-Interpretable Language
Recent advances in facial expression synthesis have shown promising results using diverse expression representations including facial action units.
Investigating Vulnerability to Adversarial Examples on Multimodal Data Fusion in Deep Learning
The success of multimodal data fusion in deep learning appears to be attributed to the use of complementary in-formation between multiple input data.
Robust Ensemble Model Training via Random Layer Sampling Against Adversarial Attack
After the ensemble models are trained, it can hide the gradient efficiently and avoid the gradient-based attack by the random layer sampling method.
Revisiting Role of Autoencoders in Adversarial Settings
Through the comprehensive experimental results and analysis, this paper presents the inherent property of adversarial robustness in the autoencoders.
Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation
Experimental results show that the proposed method could provide useful uncertainty information by Bayesian approximation with the efficient ensemble model generation and improve the predictive performance.
Generative Guiding Block: Synthesizing Realistic Looking Variants Capable of Even Large Change Demands
Generating realistic looking images with large variations (e. g., large spatial deformations and large pose change), however, is very challenging.
Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis
The ambiguity of the decision-making process has been pointed out as the main obstacle to applying the deep learning-based method in a practical way in spite of its outstanding performance.
Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition
The effectiveness of the proposed mode variational LSTM is verified using the facial expression recognition task.
Facial Expression Recognition
Facial Expression Recognition (FER)
Feature2Mass: Visual Feature Processing in Latent Space for Realistic Labeled Mass Generation
In order to learn deep network model to be well-behaved in bio-image computing fields, a lot of labeled data is required.
ICADx: Interpretable computer aided diagnosis of breast masses
In this paper, we investigate interpretability in CADx with the proposed interpretable CADx (ICADx) framework.
STAN: Spatio-Temporal Adversarial Networks for Abnormal Event Detection
In this paper, we propose a novel abnormal event detection method with spatio-temporal adversarial networks (STAN).
Ranked #17 on
Anomaly Detection
on ShanghaiTech
VR IQA NET: Deep Virtual Reality Image Quality Assessment using Adversarial Learning
The proposed human perception guider criticizes the predicted quality score of the predictor with the human perceptual score using adversarial learning.
Measurement of exceptional motion in VR video contents for VR sickness assessment using deep convolutional autoencoder
This paper proposes a new objective metric of exceptional motion in VR video contents for VR sickness assessment.
Dynamics Transfer GAN: Generating Video by Transferring Arbitrary Temporal Dynamics from a Source Video to a Single Target Image
The spatial constructs of a generated video sequence are acquired from the target image.
Facial Dynamics Interpreter Network: What are the Important Relations between Local Dynamics for Facial Trait Estimation?
In this paper, a novel deep learning approach, named facial dynamics interpreter network, has been proposed to interpret the important relations between local dynamics for estimating facial traits from expression sequence.
Learning Spatio-temporal Features with Partial Expression Sequences for on-the-Fly Prediction
At test time, most spatio-temporal encoding methods assume that a temporally segmented sequence is fed to a learned model, which could require the prediction to wait until the full sequence is available to an auxiliary task that performs the temporal segmentation.
Differential Generative Adversarial Networks: Synthesizing Non-linear Facial Variations with Limited Number of Training Data
through a generative model is helpful in addressing the lack of training data.
Iterative Deep Convolutional Encoder-Decoder Network for Medical Image Segmentation
In this paper, we propose a novel medical image segmentation using iterative deep learning framework.
Modality-bridge Transfer Learning for Medical Image Classification
This paper presents a new approach of transfer learning-based medical image classification to mitigate insufficient labeled data problem in medical domain.
Convolution with Logarithmic Filter Groups for Efficient Shallow CNN
Our classification results on Multi-PIE dataset for facial expression recognition and CIFAR-10 dataset for object classification reveal that the compact CNN with the proposed logarithmic filter grouping scheme outperforms the same network with the uniform filter grouping in terms of accuracy and parameter efficiency.
EvaluationNet: Can Human Skill be Evaluated by Deep Networks?
In order to improve the effectiveness of the learning with instructional video, observation and evaluation of the activity are required.
