Lipreading
30 papers with code • 7 benchmarks • 6 datasets
Lipreading is a process of extracting speech by watching lip movements of a speaker in the absence of sound. Humans lipread all the time without even noticing. It is a big part in communication albeit not as dominant as audio. It is a very helpful skill to learn especially for those who are hard of hearing.
Deep Lipreading is the process of extracting speech from a video of a silent talking face using deep neural networks. It is also known by few other names: Visual Speech Recognition (VSR), Machine Lipreading, Automatic Lipreading etc.
The primary methodology involves two stages: i) Extracting visual and temporal features from a sequence of image frames from a silent talking video ii) Processing the sequence of features into units of speech e.g. characters, words, phrases etc. We can find several implementations of this methodology either done in two separate stages or trained end-to-end in one go.
Benchmarks
These leaderboards are used to track progress in Lipreading
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
Libraries
Use these libraries to find Lipreading models and implementations
LRW
LRS2
LRS3-TED
CAS-VSR-W1k (LRW-1000)
GRID Dataset
GLips
Most implemented papers
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection
ahaliassos/lipforensics
•
•
CVPR 2021
Extensive experiments show that this simple approach significantly surpasses the state-of-the-art in terms of generalisation to unseen manipulations and robustness to perturbations, as well as shed light on the factors responsible for its performance.
Robust Self-Supervised Audio-Visual Speech Recognition
facebookresearch/av_hubert
•
•
Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine which speaker to transcribe.
Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition
lumia-group/leveraging-self-supervised-learning-for-avsr
•
•
ACL 2022
In particular, audio and visual front-ends are trained on large-scale unimodal datasets, then we integrate components of both front-ends into a larger multimodal framework which learns to recognize parallel audio-visual data into characters through a combination of CTC and seq2seq decoding.
Bayesian Neural Network Language Modeling for Speech Recognition
State-of-the-art neural network language models (NNLMs) represented by long short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming highly complex.
Training Strategies for Improved Lip-reading
mpc001/Lipreading_using_Temporal_Convolutional_Networks
•
•
In this paper, we systematically investigate the performance of state-of-the-art data augmentation approaches, temporal models and other training strategies, like self-distillation and using word boundary indicators.
Relaxed Attention for Transformer Models
The powerful modeling capabilities of all-attention-based transformer architectures often cause overfitting and - for natural language processing tasks - lead to an implicitly learned internal language model in the autoregressive transformer decoder complicating the integration of external language models.
Jointly Learning Visual and Auditory Speech Representations from Raw Data
ahaliassos/raven
•
•
We observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained.
LipLearner: Customizable Silent Speech Interactions on Mobile Devices
Silent speech interface is a promising technology that enables private communications in natural language.
Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
mpc001/auto_avsr
•
•
Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets.
Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing
sally-sh/vsp-llm
•
•
In visual speech processing, context modeling capability is one of the most important requirements due to the ambiguous nature of lip movements.
