Browse SoTA > Computer Vision > Talking Head Generation > Unconstrained Lip-synchronization

Unconstrained Lip-synchronization

3 papers with code · Computer Vision

Given a video of an arbitrary person, and an arbitrary driving speech, the task is to generate a lip-synced video that matches the given speech.

This task requires the approach to not be constrained by identity, voice, or language.

Benchmarks

Greatest papers with code

A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild

23 Aug 2020Rudrabha/Wav2Lip

However, they fail to accurately morph the lip movements of arbitrary identities in dynamic, unconstrained talking face videos, resulting in significant parts of the video being out-of-sync with the new audio.

UNCONSTRAINED LIP-SYNCHRONIZATION

Towards Automatic Face-to-Face Translation

ACM Multimedia, 2019 2019 Rudrabha/LipGAN

As today's digital communication becomes increasingly visual, we argue that there is a need for systems that can automatically translate a video of a person speaking in language A into a target language B with realistic lip synchronization.

 Ranked #1 on Talking Face Generation on LRW (using extra training data)

FACE TO FACE TRANSLATION MACHINE TRANSLATION UNCONSTRAINED LIP-SYNCHRONIZATION

You said that?

8 May 2017joonson/yousaidthat

To achieve this we propose an encoder-decoder CNN model that uses a joint embedding of the face and audio to generate synthesised talking face video frames.

UNCONSTRAINED LIP-SYNCHRONIZATION