Search Results for author:
Found 108 papers, 28 papers with code
Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs
We demonstrate that training a single model for all three tasks enhances VSR and AVSR performance, overcoming typical optimisation challenges when training from scratch.
Ranked #2 on
Lipreading
on LRS2
(using extra training data)
Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
This paper investigates the under-explored area of low-rank weight training for large-scale Conformer-based speech recognition models from scratch.
Large Language Models Are Strong Audio-Visual Speech Recognition Learners
For example, in the audio and speech domains, an LLM can be equipped with (automatic) speech recognition (ASR) abilities by just concatenating the audio tokens, computed with an audio encoder, and the text tokens to achieve state-of-the-art results.
Ranked #2 on
Speech Recognition
on LRS3-TED
(using extra training data)
Audio-Visual Speech Recognition
Automatic Speech Recognition
+3
RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement
In this paper, we aim to generate clean speech frame by frame from a live video stream and a noisy audio stream without relying on future inputs.
Dynamic Data Pruning for Automatic Speech Recognition
The recent success of Automatic Speech Recognition (ASR) is largely attributed to the ever-growing amount of training data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization
In this study, we propose a regularization technique that facilitates the training of visual and audio-visual speech recognition models (VSR and AVSR) from scratch.
EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars
Head avatars animated by visual signals have gained popularity, particularly in cross-driving synthesis where the driver differs from the animated character, a challenging but highly practical approach.
BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition
In this work, we propose BRAVEn, an extension to the recent RAVEn method, which learns speech representations entirely from raw audio-visual data.
Audio-visual video-to-speech synthesis with synthesized input audio
The implicit assumption of this task is that the sound signal is either missing or contains a high amount of noise/corruption such that it is not useful for processing.
SparseVSR: Lightweight and Noise Robust Visual Speech Recognition
We evaluate our 50% sparse model on 7 different visual noise types and achieve an overall absolute improvement of more than 2% WER compared to the dense equivalent.
Large-scale unsupervised audio pre-training for video-to-speech synthesis
Most established approaches to date involve a two-step process, whereby an intermediate representation from the video, such as a spectrogram, is extracted first and then passed to a vocoder to produce the raw audio.
Laughing Matters: Introducing Laughing-Face Generation using Diffusion Models
Speech-driven animation has gained significant traction in recent years, with current methods achieving near-photorealistic results.
SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision
Furthermore, when combined with large-scale pseudo-labeled audio-visual data SynthVSR yields a new state-of-the-art VSR WER of 16. 9% using publicly available data only, surpassing the recent state-of-the-art approaches trained with 29 times more non-public machine-transcribed video data (90, 000 hours).
Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets.
Ranked #1 on
Lipreading
on LRS2
(using extra training data)
Audio-Visual Speech Recognition
Automatic Speech Recognition
+4
Learning Cross-lingual Visual Speech Representations
Cross-lingual self-supervised learning has been a growing research topic in the last few years.
Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
Talking face generation has historically struggled to produce head movements and natural facial expressions without guidance from additional reference videos.
Jointly Learning Visual and Auditory Speech Representations from Raw Data
We observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained.
Ranked #1 on
Speech Recognition
on LRS2
(using extra training data)
LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoders
Audio-visual speech enhancement aims to extract clean speech from a noisy environment by leveraging not only the audio itself but also the target speaker's lip movements.
FAN-Trans: Online Knowledge Distillation for Facial Action Unit Detection
Our model consists of a hybrid network of convolution and transformer blocks to learn per-AU features and to model AU co-occurrences.
Streaming Audio-Visual Speech Recognition with Alignment Regularization
In this work, we propose a streaming AV-ASR system based on a hybrid connectionist temporal classification (CTC)/attention neural network architecture.
Audio-Visual Speech Recognition
Automatic Speech Recognition
+6
SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Video
This work focuses on the apparent emotional reaction recognition (AERR) from the video-only input, conducted in a self-supervised fashion.
Training Strategies for Improved Lip-reading
In this paper, we systematically investigate the performance of state-of-the-art data augmentation approaches, temporal models and other training strategies, like self-distillation and using word boundary indicators.
Ranked #2 on
Lipreading
on Lip Reading in the Wild
(using extra training data)
SVTS: Scalable Video-to-Speech Synthesis
Video-to-speech synthesis (also known as lip-to-speech) refers to the translation of silent lip movements into the corresponding audio.
Self-supervised Video-centralised Transformer for Video Face Clustering
We also investigate face clustering in egocentric videos, a fast-emerging field that has not been studied yet in works related to face clustering.
Ranked #1 on
Face Clustering
on EasyCom
Visual Speech Recognition for Multiple Languages in the Wild
However, these advances are usually due to the larger training sets rather than the model design.
Ranked #1 on
Lipreading
on GRID corpus (mixed-speech)
(using extra training data)
Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection
One of the most pressing challenges for the detection of face-manipulated videos is generalising to forgery methods not seen during training while remaining effective under common corruptions such as compression.
Ranked #2 on
DeepFake Detection
on FakeAVCeleb
Defensive Tensorization
We propose defensive tensorization, an adversarial defence technique that leverages a latent high-order factorization of the network.
Domain Generalisation for Apparent Emotional Facial Expression Recognition across Age-Groups
We then study the effect of variety and number of age-groups used during training on generalisation to unseen age-groups and observe that an increase in the number of training age-groups tends to increase the apparent emotional facial expression recognition performance on unseen age-groups.
Facial Expression Recognition
Facial Expression Recognition (FER)
EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments
In this work, we describe, evaluate and release a dataset that contains over 5 hours of multi-modal data useful for training and testing algorithms for the application of improving conversations for an AR glasses wearer.
Ranked #1 on
Speech Enhancement
on EasyCom
FP-Age: Leveraging Face Parsing Attention for Facial Age Estimation in the Wild
To evaluate our method on in-the-wild data, we also introduce a new challenging large-scale benchmark called IMDB-Clean.
Ranked #1 on
Age Estimation
on KANFace
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
The large amount of audiovisual content being shared online today has drawn substantial attention to the prospect of audiovisual self-supervised learning.
End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks
In this work, we propose a new end-to-end video-to-speech model based on Generative Adversarial Networks (GANs) which translates spoken video to waveform end-to-end without using any intermediate representation or separate waveform synthesis algorithm.
DINO: A Conditional Energy-Based GAN for Domain Translation
Domain translation is the process of transforming data from one domain to another while preserving the common semantics.
End-to-end Audio-visual Speech Recognition with Conformers
In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner.
Ranked #4 on
Audio-Visual Speech Recognition
on LRS2
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+7
RoI Tanh-polar Transformer Network for Face Parsing in the Wild
Face parsing aims to predict pixel-wise labels for facial components of a target face in an image.
Ranked #1 on
Face Parsing
on iBugMask
Cauchy-Schwarz Regularized Autoencoder
To perform efficient inference for GMM priors, we introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs.
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection
Extensive experiments show that this simple approach significantly surpasses the state-of-the-art in terms of generalisation to unseen manipulations and robustness to perturbations, as well as shed light on the factors responsible for its performance.
Ranked #5 on
DeepFake Detection
on FakeAVCeleb
Lip-reading with Densely Connected Temporal Convolutional Networks
In this work, we present the Densely Connected Temporal Convolutional Network (DC-TCN) for lip-reading of isolated words.
Multilinear Latent Conditioning for Generating Unseen Attribute Combinations
Deep generative models rely on their inductive bias to facilitate generalization, especially for problems with high dimensional data, like images.
Towards Practical Lipreading with Distilled and Efficient Models
However, our most promising lightweight models are on par with the current state-of-the-art while showing a reduction of 8. 2x and 3. 9x in terms of computational cost and number of parameters, respectively, which we hope will enable the deployment of lipreading models in practical applications.
Ranked #7 on
Lipreading
on Lip Reading in the Wild
Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision
This enriches the audio encoder with visual information and the encoder can be used for evaluation without the visual modality.
Enhancing Facial Data Diversity with Style-based Face Aging
By evaluating on several age-annotated datasets in both single- and cross-database experiments, we show that the proposed method outperforms state-of-the-art algorithms for age transfer, especially in the case of age groups that lie in the tails of the label distribution.
Dilated Convolutions with Lateral Inhibitions for Semantic Image Segmentation
Introducing LI mechanisms improves the convolutional filter's sensitivity to semantic object boundaries.
Investigating Bias in Deep Face Analysis: The KANFace Dataset and Empirical Study
In this work, we investigate the demographic bias of deep learning models in face recognition, age estimation, gender recognition and kinship verification.
Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?
Our results demonstrate the potential of visual self-supervision for audio feature learning and suggest that joint visual and audio self-supervision leads to more informative audio representations for speech and emotion recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Toward fast and accurate human pose estimation via soft-gated skip connections
In addition, with a reduction of 3x in model size and complexity, we show no decrease in performance when compared to the original HourGlass network.
Ranked #2 on
Pose Estimation
on Leeds Sports Poses
Lipreading using Temporal Convolutional Networks
We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively.
Ranked #1 on
Lipreading
on LRW-1000
Visually Guided Self Supervised Learning of Speech Representations
Self supervised representation learning has recently attracted a lot of research interest for both the audio and visual modalities.
Ranked #9 on
Speech Emotion Recognition
on CREMA-D
Detecting Adversarial Attacks On Audiovisual Speech Recognition
In this work, we propose an efficient and straightforward detection method based on the temporal correlation between audio and video streams.
Speech-driven facial animation using polynomial fusion of features
Speech-driven facial animation involves using a speech signal to generate realistic videos of talking faces.
Towards Pose-invariant Lip-Reading
The proposed model significantly outperforms previous approaches on non-frontal views while retaining the superior performance on frontal and near frontal mouth views.
Shape Constrained Network for Eye Segmentation in the Wild
Specifically, we learn the shape prior from our dataset using VAE-GAN, and leverage the pre-trained encoder and discriminator to regularise the training of SegNet.
Fast and Effective Adaptation of Facial Action Unit Detection Deep Model
In this paper, we propose a deep learning approach for facial AU detection that can easily and in a fast manner adapt to a new AU or target subject by leveraging only a few labeled samples from the new task (either an AU or subject).
Defensive Tensorization: Randomized Tensor Parametrization for Robust Neural Networks
As deep neural networks become widely adopted for solving most problems in computer vision and audio-understanding, there are rising concerns about their potential vulnerability.
AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition
The Audio/Visual Emotion Challenge and Workshop (AVEC 2019) "State-of-Mind, Detecting Depression with AI, and Cross-cultural Affect Recognition" is the ninth competition event aimed at the comparison of multimedia processing and machine learning methods for automatic audiovisual health and emotion analysis, with all participants competing strictly under the same conditions.
Dynamic Face Video Segmentation via Reinforcement Learning
To the best of our knowledge, this is the first work to use reinforcement learning for online key-frame decision in dynamic video segmentation, and also the first work on its application on face videos.
Realistic Speech-Driven Facial Animation with GANs
We present an end-to-end system that generates videos of a talking head, using only a still image of a person and an audio clip containing speech, without relying on handcrafted intermediate features.
Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation
To alleviate this, one approach is to apply low-rank tensor decompositions to convolution kernels in order to compress the network and reduce its number of parameters.
Video-Driven Speech Reconstruction using Generative Adversarial Networks
Speech is a means of communication which relies on both audio and visual information.
Investigating the Lombard Effect Influence on End-to-End Audio-Visual Speech Recognition
Several audio-visual speech recognition models have been recently proposed which aim to improve the robustness over audio-only models in the presence of noise.
Matrix and tensor decompositions for training binary neural networks
This paper is on improving the training of binary neural networks in which both activations and weights are binary.
Incremental multi-domain learning with network latent tensor factorization
Adapting the learned classification to new domains is a hard problem due to at least three reasons: (1) the new domains and the tasks might be drastically different; (2) there might be very limited amount of annotated data on the new domain and (3) full training of a new model for each new task is prohibitive in terms of computation and memory, due to the sheer number of parameters of deep CNNs.
Improved training of binary networks for human pose estimation and image recognition
Big neural networks trained on large datasets have advanced the state-of-the-art for a large variety of challenging problems, improving performance by a large margin.
T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor
In this paper, we propose to fully parametrize Convolutional Neural Networks (CNNs) with a single high-order, low-rank tensor.
Ranked #35 on
Pose Estimation
on MPII Human Pose
End-to-End Visual Speech Recognition for Small-Scale Datasets
In this work, we present an end-to-end visual speech recognition system based on fully-connected layers and Long-Short Memory (LSTM) networks which is suitable for small-scale datasets.
MeshGAN: Non-linear 3D Morphable Models of Faces
Generative Adversarial Networks (GANs) are currently the method of choice for generating visual data.
SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild
Natural human-computer interaction and audio-visual human behaviour sensing systems, which would achieve robust performance in-the-wild are more needed than ever as digital devices are increasingly becoming an indispensable part of our life.
Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture
Therefore, we could use a CTC loss in combination with an attention-based model in order to force monotonic alignments and at the same time get rid of the conditional independence assumption.
Ranked #6 on
Audio-Visual Speech Recognition
on LRS2
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+3
Face Mask Extraction in Video Sequence
Inspired by the recent development of deep network-based methods in semantic image segmentation, we introduce an end-to-end trainable model for face mask extraction in video sequence.
Transfer Learning for Action Unit Recognition
This paper presents a classifier ensemble for Facial Expression Recognition (FER) based on models derived from transfer learning.
4DFAB: A Large Scale 4D Database for Facial Expression Analysis and Biometric Applications
The progress we are currently witnessing in many computer vision applications, including automatic face analysis, would not be made possible without tremendous efforts in collecting and annotating large scale visual databases.
Facial Expression Recognition
Facial Expression Recognition (FER)
MobiFace: A Novel Dataset for Mobile Face Tracking in the Wild
36 state-of-the-art trackers, including facial landmark trackers, generic object trackers and trackers that we have fine-tuned or improved, are evaluated.
End-to-End Speech-Driven Facial Animation with Temporal GANs
To the best of our knowledge, this is the first method capable of generating subject independent realistic videos directly from raw audio.
A real-time and unsupervised face Re-Identification system for Human-Robot Interaction
In this paper, we present an effective and unsupervised face Re-ID system which simultaneously re-identifies multiple faces for HRI.
Multi-Instance Dynamic Ordinal Random Fields for Weakly-supervised Facial Behavior Analysis
In this framework, we treat instance-labels as temporally-dependent latent variables in an Undirected Graphical Model.
End-to-end Audiovisual Speech Recognition
In presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models.
Ranked #20 on
Lipreading
on Lip Reading in the Wild
Visual-Only Recognition of Normal, Whispered and Silent Speech
We show that an absolute decrease in classification rate of up to 3. 7% is observed when training and testing on normal and whispered, respectively, and vice versa.
Modeling of Facial Aging and Kinship: A Survey
Computational facial models that capture properties of facial cues related to aging and kinship increasingly attract the attention of the research community, enabling the development of reliable methods for age progression, age estimation, age-invariant facial characterization, and kinship verification from visual data.
4DFAB: A Large Scale 4D Facial Expression Database for Biometric Applications
4DFAB contains recordings of 180 subjects captured in four different sessions spanning over a five-year period.
Facial Expression Recognition
Facial Expression Recognition (FER)
GAGAN: Geometry-Aware Generative Adversarial Networks
Deep generative models learned through adversarial training have become increasingly popular for their ability to generate naturalistic image textures.
End-to-End Audiovisual Fusion with LSTMs
To the best of our knowledge, this is the first audiovisual fusion model which simultaneously learns to extract features directly from the pixels and spectrograms and perform classification of speech and nonlinguistic vocalisations.
End-to-End Multi-View Lipreading
To the best of our knowledge, this is the first model which simultaneously learns to extract features directly from the pixels and performs visual speech classification from multiple views and also achieves state-of-the-art performance.
Deep Structured Learning for Facial Action Unit Intensity Estimation
The goal of this paper is to model these structures and estimate complex feature representations simultaneously by combining conditional random field (CRF) encoded AU dependencies with deep learning.
DeepCoder: Semi-parametric Variational Autoencoders for Automatic Facial Action Coding
Potentially, this makes VAEs a suitable approach for learning facial features for AU intensity estimation.
Local Deep Neural Networks for Age and Gender Classification
We tested the proposed modified local deep neural networks approach on the LFW and Adience databases for the task of gender and age classification.
FERA 2017 - Addressing Head Pose in the Third Facial Expression Recognition and Analysis Challenge
The FG 2017 Facial Expression Recognition and Analysis challenge (FERA 2017) extends FERA 2015 to the estimation of Action Units occurrence and intensity under different camera views.
End-To-End Visual Speech Recognition With LSTMs
Recently, several deep learning approaches have been presented which automatically extract features from the mouth images and aim to replace the feature extraction stage.
TensorLy: Tensor Learning in Python
In addition, using the deep-learning frameworks as backend allows users to easily design and train deep tensorized neural networks.
Multi-instance Dynamic Ordinal Random Fields for Weakly-Supervised Pain Intensity Estimation
In this paper, we address the Multi-Instance-Learning (MIL) problem when bag labels are naturally represented as ordinal variables (Multi--Instance--Ordinal Regression).
Variational Gaussian Process Auto-Encoder for Ordinal Prediction of Facial Action Units
In particular, we introduce GP encoders to project multiple observed features onto a latent space, while GP decoders are responsible for reconstructing the original features.
Copula Ordinal Regression for Joint Estimation of Facial Action Unit Intensity
Joint modeling of the intensity of facial action units (AUs) from face images is challenging due to the large number of AUs (30+) and their intensity levels (6).
Joint Unsupervised Deformable Spatio-Temporal Alignment of Sequences
Typically, the problems of spatial and temporal alignment of sequences are considered disjoint.
AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge
The Audio/Visual Emotion Challenge and Workshop (AVEC 2016) "Depression, Mood and Emotion" will be the sixth competition event aimed at comparison of multimedia processing and machine learning methods for automatic audio, visual and physiological depression and emotion analysis, with all participants competing under strictly the same conditions.
Gaussian Process Domain Experts for Model Adaptation in Facial Behavior Analysis
The adaptation of the classifier is facilitated in probabilistic fashion by conditioning the target expert on multiple source experts.
Multi-Conditional Latent Variable Model for Joint Facial Action Unit Detection
We propose a novel multi-conditional latent variable model for simultaneous facial feature fusion and detection of facial action units.
Robust Statistical Face Frontalization
The proposed method is assessed in frontal face reconstruction, face landmark localization, pose-invariant face recognition, and face verification in unconstrained conditions.
Variable-state Latent Conditional Random Fields for Facial Expression Recognition and Action Unit Detection
For instance, in the case of AU detection, the goal is to discriminate between the segments of an image sequence in which this AU is active or inactive.
Latent Trees for Estimating Intensity of Facial Action Units
Our model is a latent tree (LT) that represents input features of facial landmark points and FAU intensities as leaf nodes, and encodes their higher-order dependencies with latent nodes at tree levels closer to the root.
Face frontalization for Alignment and Recognition
The proposed method is assessed in frontal face reconstruction (pose correction), face landmark localization, and pose-invariant face recognition and verification by conducting experiments on $6$ facial images databases.
Full-Angle Quaternions for Robustly Matching Vectors of 3D Rotations
In this paper we introduce a new distance for robustly matching vectors of 3D rotations.
Merging SVMs with Linear Discriminant Analysis: A Combined Model
A key problem often encountered by many learning algorithms in computer vision dealing with high dimensional data is the so called "curse of dimensionality" which arises when the available training samples are less than the input feature space dimensionality.
RAPS: Robust and Efficient Automatic Construction of Person-Specific Deformable Models
Next, to correct the fittings of a generic model, image congealing (i. e., batch image aliment) is performed by employing only the learnt orthonormal subspace.
Incremental Face Alignment in the Wild
We propose very efficient strategies to update the model and we show that is possible to automatically construct robust discriminative person and imaging condition specific models 'in-the-wild' that outperform state-of-the-art generic face alignment strategies.
Gauss-Newton Deformable Part Models for Face Alignment in-the-Wild
To address this limitation, in this paper, we propose to jointly optimize a part-based, trained in-the-wild, flexible appearance model along with a global shape model which results in a joint translational motion model for the model parts via Gauss-Newton (GN) optimization.
Robust Canonical Time Warping for the Alignment of Grossly Corrupted Sequences
The superiority of the proposed method against the state-of-the-art time alignment methods, namely the canonical time warping and the generalized time warping, is indicated by the experimental results on both synthetic and real datasets.
Robust Discriminative Response Map Fitting with Constrained Local Models
We present a novel discriminative regression based approach for the Constrained Local Models (CLMs) framework, referred to as the Discriminative Response Map Fitting (DRMF) method, which shows impressive performance in the generic face fitting scenario.
A Unified Framework for Probabilistic Component Analysis
We present a unifying framework which reduces the construction of probabilistic component analysis techniques to a mere selection of the latent neighbourhood, thus providing an elegant and principled framework for creating novel component analysis models as well as constructing probabilistic equivalents of deterministic component analysis methods.
Heteroscedastic Conditional Ordinal Random Fields for Pain Intensity Estimation from Facial Images
We propose a novel method for automatic pain intensity estimation from facial images based on the framework of kernel Conditional Ordinal Random Fields (KCORF).
