Search Results for author:
Found 49 papers, 18 papers with code
End-To-End Visual Speech Recognition With LSTMs
Recently, several deep learning approaches have been presented which automatically extract features from the mouth images and aim to replace the feature extraction stage.
Local Deep Neural Networks for Age and Gender Classification
We tested the proposed modified local deep neural networks approach on the LFW and Adience databases for the task of gender and age classification.
End-to-End Multi-View Lipreading
To the best of our knowledge, this is the first model which simultaneously learns to extract features directly from the pixels and performs visual speech classification from multiple views and also achieves state-of-the-art performance.
End-to-End Audiovisual Fusion with LSTMs
To the best of our knowledge, this is the first audiovisual fusion model which simultaneously learns to extract features directly from the pixels and spectrograms and perform classification of speech and nonlinguistic vocalisations.
End-to-end Audiovisual Speech Recognition
In presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models.
Ranked #17 on
Lipreading
on Lip Reading in the Wild
Visual-Only Recognition of Normal, Whispered and Silent Speech
We show that an absolute decrease in classification rate of up to 3. 7% is observed when training and testing on normal and whispered, respectively, and vice versa.
A real-time and unsupervised face Re-Identification system for Human-Robot Interaction
In this paper, we present an effective and unsupervised face Re-ID system which simultaneously re-identifies multiple faces for HRI.
End-to-End Speech-Driven Facial Animation with Temporal GANs
To the best of our knowledge, this is the first method capable of generating subject independent realistic videos directly from raw audio.
Transfer Learning for Action Unit Recognition
This paper presents a classifier ensemble for Facial Expression Recognition (FER) based on models derived from transfer learning.
Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture
Therefore, we could use a CTC loss in combination with an attention-based model in order to force monotonic alignments and at the same time get rid of the conditional independence assumption.
Ranked #5 on
Audio-Visual Speech Recognition
on LRS2
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+3
End-to-End Visual Speech Recognition for Small-Scale Datasets
In this work, we present an end-to-end visual speech recognition system based on fully-connected layers and Long-Short Memory (LSTM) networks which is suitable for small-scale datasets.
Investigating the Lombard Effect Influence on End-to-End Audio-Visual Speech Recognition
Several audio-visual speech recognition models have been recently proposed which aim to improve the robustness over audio-only models in the presence of noise.
Video-Driven Speech Reconstruction using Generative Adversarial Networks
Speech is a means of communication which relies on both audio and visual information.
Realistic Speech-Driven Facial Animation with GANs
We present an end-to-end system that generates videos of a talking head, using only a still image of a person and an audio clip containing speech, without relying on handcrafted intermediate features.
Towards Pose-invariant Lip-Reading
The proposed model significantly outperforms previous approaches on non-frontal views while retaining the superior performance on frontal and near frontal mouth views.
Speech-driven facial animation using polynomial fusion of features
Speech-driven facial animation involves using a speech signal to generate realistic videos of talking faces.
Detecting Adversarial Attacks On Audiovisual Speech Recognition
In this work, we propose an efficient and straightforward detection method based on the temporal correlation between audio and video streams.
Visually Guided Self Supervised Learning of Speech Representations
Self supervised representation learning has recently attracted a lot of research interest for both the audio and visual modalities.
Ranked #8 on
Speech Emotion Recognition
on CREMA-D
Lipreading using Temporal Convolutional Networks
We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively.
Ranked #7 on
Lipreading
on CAS-VSR-W1k (LRW-1000)
Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?
Our results demonstrate the potential of visual self-supervision for audio feature learning and suggest that joint visual and audio self-supervision leads to more informative audio representations for speech and emotion recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision
This enriches the audio encoder with visual information and the encoder can be used for evaluation without the visual modality.
Towards Practical Lipreading with Distilled and Efficient Models
However, our most promising lightweight models are on par with the current state-of-the-art while showing a reduction of 8. 2x and 3. 9x in terms of computational cost and number of parameters, respectively, which we hope will enable the deployment of lipreading models in practical applications.
Ranked #4 on
Lipreading
on Lip Reading in the Wild
Lip-reading with Densely Connected Temporal Convolutional Networks
In this work, we present the Densely Connected Temporal Convolutional Network (DC-TCN) for lip-reading of isolated words.
Domain Adversarial Neural Networks for Dysarthric Speech Recognition
The results are compared to a speaker-adaptive (SA) model as well as speaker-dependent (SD) and multi-task learning models (MTL).
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection
Extensive experiments show that this simple approach significantly surpasses the state-of-the-art in terms of generalisation to unseen manipulations and robustness to perturbations, as well as shed light on the factors responsible for its performance.
Ranked #5 on
DeepFake Detection
on FakeAVCeleb
End-to-end Audio-visual Speech Recognition with Conformers
In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner.
Ranked #3 on
Audio-Visual Speech Recognition
on LRS2
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+6
DINO: A Conditional Energy-Based GAN for Domain Translation
Domain translation is the process of transforming data from one domain to another while preserving the common semantics.
End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks
In this work, we propose a new end-to-end video-to-speech model based on Generative Adversarial Networks (GANs) which translates spoken video to waveform end-to-end without using any intermediate representation or separate waveform synthesis algorithm.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
The large amount of audiovisual content being shared online today has drawn substantial attention to the prospect of audiovisual self-supervised learning.
Domain Generalisation for Apparent Emotional Facial Expression Recognition across Age-Groups
We then study the effect of variety and number of age-groups used during training on generalisation to unseen age-groups and observe that an increase in the number of training age-groups tends to increase the apparent emotional facial expression recognition performance on unseen age-groups.
Facial Expression Recognition
Facial Expression Recognition (FER)
Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection
One of the most pressing challenges for the detection of face-manipulated videos is generalising to forgery methods not seen during training while remaining effective under common corruptions such as compression.
Ranked #2 on
DeepFake Detection
on FakeAVCeleb
Visual Speech Recognition for Multiple Languages in the Wild
However, these advances are usually due to the larger training sets rather than the model design.
Ranked #1 on
Lipreading
on GRID corpus (mixed-speech)
(using extra training data)
Self-supervised Video-centralised Transformer for Video Face Clustering
We also investigate face clustering in egocentric videos, a fast-emerging field that has not been studied yet in works related to face clustering.
Ranked #1 on
Face Clustering
on EasyCom
SVTS: Scalable Video-to-Speech Synthesis
Video-to-speech synthesis (also known as lip-to-speech) refers to the translation of silent lip movements into the corresponding audio.
Training Strategies for Improved Lip-reading
In this paper, we systematically investigate the performance of state-of-the-art data augmentation approaches, temporal models and other training strategies, like self-distillation and using word boundary indicators.
Ranked #1 on
Lipreading
on Lip Reading in the Wild
(using extra training data)
SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Video
This work focuses on the apparent emotional reaction recognition (AERR) from the video-only input, conducted in a self-supervised fashion.
Streaming Audio-Visual Speech Recognition with Alignment Regularization
In this work, we propose a streaming AV-ASR system based on a hybrid connectionist temporal classification (CTC)/attention neural network architecture.
Audio-Visual Speech Recognition
Automatic Speech Recognition
+5
LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoders
Audio-visual speech enhancement aims to extract clean speech from a noisy environment by leveraging not only the audio itself but also the target speaker's lip movements.
Jointly Learning Visual and Auditory Speech Representations from Raw Data
We observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained.
Ranked #1 on
Speech Recognition
on LRS2
(using extra training data)
Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
Talking face generation has historically struggled to produce head movements and natural facial expressions without guidance from additional reference videos.
Learning Cross-lingual Visual Speech Representations
Cross-lingual self-supervised learning has been a growing research topic in the last few years.
Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets.
Ranked #1 on
Automatic Speech Recognition (ASR)
on LRS3-TED
Audio-Visual Speech Recognition
Automatic Speech Recognition
+4
SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision
Furthermore, when combined with large-scale pseudo-labeled audio-visual data SynthVSR yields a new state-of-the-art VSR WER of 16. 9% using publicly available data only, surpassing the recent state-of-the-art approaches trained with 29 times more non-public machine-transcribed video data (90, 000 hours).
Medical records condensation: a roadmap towards healthcare data democratisation
The prevalence of artificial intelligence (AI) has envisioned an era of healthcare democratisation that promises every stakeholder a new and better way of life.
Laughing Matters: Introducing Laughing-Face Generation using Diffusion Models
Speech-driven animation has gained significant traction in recent years, with current methods achieving near-photorealistic results.
SparseVSR: Lightweight and Noise Robust Visual Speech Recognition
We evaluate our 50% sparse model on 7 different visual noise types and achieve an overall absolute improvement of more than 2% WER compared to the dense equivalent.
TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorch
TorchAudio is an open-source audio and speech processing library built for PyTorch.
Hearing Loss Detection from Facial Expressions in One-on-one Conversations
In this work, we build on this idea and introduce the problem of detecting hearing loss from an individual's facial expressions during a conversation.
BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition
In this work, we propose BRAVEn, an extension to the recent RAVEn method, which learns speech representations entirely from raw audio-visual data.
