Search Results for author:
Found 60 papers, 17 papers with code
Towards Automated Movie Trailer Generation
Movie trailers are an essential tool for promoting films and attracting audiences.
Scaling Up Video Summarization Pretraining with Large Language Models
Long-form video content constitutes a significant portion of internet traffic, making automated video summarization an essential research problem.
EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning
Recent advancements in self-supervised audio-visual representation learning have demonstrated its potential to capture rich and comprehensive representations.
FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocoder
The goal of this paper is to generate realistic audio with a lightweight and fast diffusion-based vocoder, named FreGrad.
From Coarse to Fine: Efficient Training for Audio Spectrogram Transformers
We introduce multi-phase training of audio spectrogram transformers by connecting the seminal idea of coarse-to-fine with transformer models.
Can CLIP Help Sound Source Localization?
Large-scale pre-trained image-text models demonstrate remarkable versatility across diverse tasks, benefiting from their robust representational capabilities and effective multimodal alignment.
Seeing Through the Conversation: Audio-Visual Speech Separation based on Diffusion Model
For an effective fusion of the two modalities for diffusion, we also propose a cross-attention-based feature fusion mechanism.
Rethinking Session Variability: Leveraging Session Embeddings for Session Robustness in Speaker Verification
In the field of speaker verification, session or channel variability poses a significant challenge.
VoiceLDM: Text-to-Speech with Environmental Context
This paper presents VoiceLDM, a model designed to produce audio that accurately follows two distinct natural language text prompts: the description prompt and the content prompt.
SlowFast Network for Continuous Sign Language Recognition
The objective of this work is the effective extraction of spatial and dynamic features for Continuous Sign Language Recognition (CSLR).
TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning
The goal of this work is Active Speaker Detection (ASD), a task to determine whether a person is speaking or not in a series of video frames.
Sound Source Localization is All about Cross-Modal Alignment
However, prior arts and existing benchmarks do not account for a more important aspect of the problem, cross-modal semantic understanding, which is essential for genuine sound source localization.
Let There Be Sound: Reconstructing High Quality Speech from Silent Videos
In this paper, we propose a novel lip-to-speech system that significantly improves the generation quality by alleviating the one-to-many mapping problem from multiple perspectives.
FlexiAST: Flexibility is What AST Needs
To overcome this limitation, this paper proposes a training procedure to provide flexibility to standard AST models without architectural changes, allowing them to work with various patch sizes at the inference stage - FlexiAST.
That's What I Said: Fully-Controllable Talking Face Generation
The goal of this paper is to synthesise talking faces with controllable facial motions.
Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples
The objective of this work is to explore the learning of visually grounded speech models (VGS) from multilingual perspective.
Self-Sufficient Framework for Continuous Sign Language Recognition
The goal of this work is to develop self-sufficient framework for Continuous Sign Language Recognition (CSLR) that addresses key issues of sign language recognition.
Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech
This is the first time that face images are used as a condition to train a TTS model.
VoxSRC 2022: The Fourth VoxCeleb Speaker Recognition Challenge
This paper summarises the findings from the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22), which was held in conjunction with INTERSPEECH 2022.
MarginNCE: Robust Sound Localization with a Negative Margin
Furthermore, we demonstrate that the introduction of a negative margin to existing methods results in a consistent improvement in performance.
Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition
Most existing Continuous Sign Language Recognition (CSLR) benchmarks have fixed backgrounds and are filmed in studios with a static monochromatic background.
Metric Learning for User-defined Keyword Spotting
In particular, we make the following contributions: (1) we construct a large-scale keyword dataset with an existing speech corpus and propose a filtering method to remove data that degrade model training; (2) we propose a metric learning-based two-stage training strategy, and demonstrate that the proposed method improves the performance on the user-defined keyword spotting task by enriching their representations; (3) to facilitate the fair comparison in the user-defined KWS field, we propose unified evaluation protocol and metrics.
Disentangled representation learning for multilingual speaker recognition
The goal of this paper is to learn robust speaker representation for bilingual speaking scenario.
In search of strong embedding extractors for speaker diarisation
First, the evaluation is not straightforward because the features required for better performance differ between speaker verification and diarisation.
Large-scale learning of generalised representations for speaker recognition
We also show that training with proposed large data configurations gives better performance.
Curriculum learning for self-supervised speaker verification
The goal of this paper is to train effective self-supervised speaker representations without identity labels.
Pushing the limits of raw waveform speaker recognition
Our best model achieves an equal error rate of 0. 89%, which is competitive with the state-of-the-art models based on handcrafted features, and outperforms the best model based on raw waveform inputs by a large margin.
Multi-scale speaker embedding-based graph attention networks for speaker diarisation
The objective of this work is effective speaker diarisation using multi-scale speaker embeddings.
Advancing the dimensionality reduction of speaker embeddings for speaker diarisation: disentangling noise and informing speech activity
The objective of this work is to train noise-robust speaker embeddings adapted for speaker diarisation.
Spell my name: keyword boosted speech recognition
Recognition of uncommon words such as names and technical terminology is important to understanding conversations in context.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks
Artefacts that differentiate spoofed from bona-fide utterances can reside in spectral or temporal domains.
Ranked #1 on
Voice Anti-spoofing
on ASVspoof 2019 - LA
Look Who's Talking: Active Speaker Detection in the Wild
Face tracks are extracted from the videos and active segments are annotated based on the timestamps of VoxConverse in a semi-automatic way.
Three-class Overlapped Speech Detection using a Convolutional Recurrent Neural Network
In this work, we propose an overlapped speech detection system trained as a three-class classifier.
Adapting Speaker Embeddings for Speaker Diarisation
The goal of this paper is to adapt speaker embeddings for solving the problem of speaker diarisation.
VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challenge
We held the second installment of the VoxCeleb Speaker Recognition Challenge in conjunction with Interspeech 2020.
Graph Attention Networks for Speaker Verification
The proposed framework inputs segment-wise speaker embeddings from an enrollment and a test utterance and directly outputs a similarity score.
Self-Supervised Learning of Audio-Visual Objects from Video
Our objective is to transform a video into a set of discrete audio-visual objects using self-supervised learning.
Augmentation adversarial training for self-supervised speaker recognition
Since the augmentation simulates the acoustic characteristics, training the network to be invariant to augmentation also encourages the network to be invariant to the channel information in general.
BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues
Recent progress in fine-grained gesture and action classification, and machine translation, point to the possibility of automated sign language recognition becoming a reality.
Ranked #4 on
Sign Language Recognition
on WLASL-2000
Spot the conversation: speaker diarisation in the wild
Finally, we use this pipeline to create a large-scale diarisation dataset called VoxConverse, collected from 'in the wild' videos, which we will release publicly to the research community.
FaceFilter: Audio-visual speech separation using still images
The objective of this paper is to separate a target speaker's speech from a mixture of two speakers using a deep audio-visual speech separation network.
Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision
We build on earlier work to train embeddings that are more discriminative for uni-modal downstream tasks.
Disentangled Speech Embeddings using Cross-modal Self-supervision
The objective of this paper is to learn representations of speaker identity without access to manually annotated data.
VoxSRC 2019: The first VoxCeleb Speaker Recognition Challenge
The VoxCeleb Speaker Recognition Challenge 2019 aimed to assess how well current speaker recognition technology is able to identify speakers in unconstrained or `in the wild' data.
ASR is all you need: cross-modal distillation for lip reading
The goal of this work is to train strong models for visual speech recognition without requiring human annotated ground truth data.
Ranked #14 on
Lipreading
on LRS3-TED
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Delving into VoxCeleb: environment invariant speaker recognition
Research in speaker recognition has recently seen significant progress due to the application of neural network models and the availability of new large-scale datasets.
My lips are concealed: Audio-visual speech enhancement through obstructions
To this end we introduce a deep audio-visual speech enhancement network that is able to separate a speaker's voice by conditioning on both the speaker's lip movements and/or a representation of their voice.
Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)
This report describes our submission to the ActivityNet Challenge at CVPR 2019.
Ranked #14 on
Audio-Visual Active Speaker Detection
on AVA-ActiveSpeaker
(using extra training data)
Who said that?: Audio-visual speaker diarisation of real-world meetings
The goal of this work is to determine 'who spoke when' in real-world meetings.
Utterance-level Aggregation For Speaker Recognition In The Wild
The objective of this paper is speaker recognition "in the wild"-where utterances may be of variable length and also contain irrelevant signals.
Perfect match: Improved cross-modal embeddings for audio-visual synchronisation
This paper proposes a new strategy for learning powerful cross-modal embeddings for audio-to-video synchronization.
Deep Audio-Visual Speech Recognition
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
Ranked #6 on
Audio-Visual Speech Recognition
on LRS2
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+4
LRS3-TED: a large-scale dataset for visual speech recognition
This paper introduces a new multi-modal dataset for visual and audio-visual speech recognition.
Deep Lip Reading: a comparison of models and an online application
The goal of this paper is to develop state-of-the-art models for lip reading -- visual speech recognition.
VoxCeleb2: Deep Speaker Recognition
The objective of this paper is speaker recognition under noisy and unconstrained conditions.
Ranked #1 on
Speaker Verification
on VoxCeleb2
(using extra training data)
The Conversation: Deep Audio-Visual Speech Enhancement
Our goal is to isolate individual speakers from multi-talker simultaneous speech in videos.
VoxCeleb: a large-scale speaker identification dataset
Our second contribution is to apply and compare various state of the art speaker identification techniques on our dataset to establish baseline performance.
Sound
You said that?
To achieve this we propose an encoder-decoder CNN model that uses a joint embedding of the face and audio to generate synthesised talking face video frames.
Lip Reading Sentences in the Wild
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
Ranked #4 on
Lipreading
on GRID corpus (mixed-speech)
(using extra training data)
Signs in time: Encoding human motion as a temporal image
The goal of this work is to recognise and localise short temporal signals in image time series, where strong supervision is not available for training.
