Search Results for author:
Found 54 papers, 24 papers with code
Attention Bottlenecks for Multimodal Fusion
Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio.
Ranked #2 on
Action Classification
on Kinetics-Sounds
UnLoc: A Unified Framework for Video Localization Tasks
While large-scale image-text pretrained models such as CLIP have been used for multiple video-level tasks on trimmed videos, their use for temporal localization in untrimmed videos is still a relatively unexplored task.
Ranked #1 on
Action Segmentation
on COIN
Streaming Dense Video Captioning
An ideal model for dense video captioning -- predicting captions localized temporally in a video -- should be able to handle long input videos, predict rich, detailed textual descriptions, and be able to produce outputs before processing the entire video.
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale.
Ranked #1 on
Dense Video Captioning
on ActivityNet Captions
(using extra training data)
Verbs in Action: Improving verb understanding in video-language models
Understanding verbs is crucial to modelling how people and objects interact with each other and the environment through space and time.
Ranked #11 on
Zero-Shot Video Question Answer
on NExT-QA
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Our objective in this work is video-text retrieval - in particular a joint embedding that enables efficient text-to-video retrieval.
Ranked #4 on
Video Retrieval
on QuerYD
(using extra training data)
VoxCeleb: a large-scale speaker identification dataset
Our second contribution is to apply and compare various state of the art speaker identification techniques on our dataset to establish baseline performance.
Sound
Utterance-level Aggregation For Speaker Recognition In The Wild
The objective of this paper is speaker recognition "in the wild"-where utterances may be of variable length and also contain irrelevant signals.
VoxCeleb2: Deep Speaker Recognition
The objective of this paper is speaker recognition under noisy and unconstrained conditions.
Ranked #1 on
Speaker Verification
on VoxCeleb2
(using extra training data)
Use What You Have: Video Retrieval Using Representations From Collaborative Experts
The rapid growth of video on the internet has made searching for video content using natural language queries a significant challenge.
Ranked #24 on
Video Retrieval
on MSVD
The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
This report summarizes the results of the first edition of the challenge together with the findings of the participants.
Condensed Movies: Story Based Retrieval with Contextual Embeddings
Our objective in this work is long range understanding of the narrative structure of movies.
AutoAD: Movie Description in Context
The objective of this paper is an automatic Audio Description (AD) model that ingests movies and outputs AD in text form.
AVATAR: Unconstrained Audiovisual Speech Recognition
Audio-visual automatic speech recognition (AV-ASR) is an extension of ASR that incorporates visual cues, often from the movements of a speaker's mouth.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition
We focus on multi-modal fusion for egocentric action recognition, and propose a novel architecture for multi-modal temporal-binding, i. e. the combination of modalities within a range of temporal offsets.
Ranked #2 on
Egocentric Activity Recognition
on EPIC-KITCHENS-55
Modular Visual Question Answering via Code Generation
We present a framework that formulates visual question answering as modular code generation.
Localizing Visual Sounds the Hard Way
We show that our algorithm achieves state-of-the-art performance on the popular Flickr SoundNet dataset.
PaLI-X: On Scaling up a Multilingual Vision and Language Model
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture.
Ranked #1 on
Fine-Grained Image Recognition
on OVEN
Slow-Fast Auditory Streams For Audio Recognition
We propose a two-stream convolutional network for audio recognition, that operates on time-frequency spectrogram inputs.
Ranked #1 on
Human Interaction Recognition
on EPIC-SOUNDS
Cough Against COVID: Evidence of COVID-19 Signature in Cough Sounds
Testing capacity for COVID-19 remains a challenge globally due to the lack of adequate supplies, trained personnel, and sample-processing equipment.
With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition
We capitalise on the action's temporal context and propose a method that learns to attend to surrounding actions in order to improve recognition performance.
VoxSRC 2022: The Fourth VoxCeleb Speaker Recognition Challenge
This paper summarises the findings from the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22), which was held in conjunction with INTERSPEECH 2022.
A CLIP-Hitchhiker's Guide to Long Video Retrieval
Our goal in this paper is the adaptation of image-text models for long video retrieval.
Ranked #4 on
Zero-Shot Action Recognition
on Charades
Learnable PINs: Cross-Modal Embeddings for Person Identity
We propose and investigate an identity sensitive joint embedding of face and voice.
Seeing Voices and Hearing Faces: Cross-modal biometric matching
We make the following contributions: (i) we introduce CNN architectures for both binary and multi-way cross-modal face and audio matching, (ii) we compare dynamic testing (where video information is available, but the audio is not from the same video) with static testing (where only a single still image is available), and (iii) we use human testing as a baseline to calibrate the difficulty of the task.
From Benedict Cumberbatch to Sherlock Holmes: Character Identification in TV series without a Script
The goal of this paper is the automatic identification of characters in TV and feature film material.
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
We make the following contributions: (i) we develop a strong teacher network for facial emotion recognition that achieves the state of the art on a standard benchmark; (ii) we use the teacher to train a student, tabula rasa, to learn representations (embeddings) for speech emotion recognition without access to labelled audio data; and (iii) we show that the speech emotion embedding can be used for speech emotion recognition on external benchmark datasets.
Ranked #3 on
Facial Expression Recognition (FER)
on FERPlus
Facial Emotion Recognition
Facial Expression Recognition (FER)
+1
Count, Crop and Recognise: Fine-Grained Recognition in the Wild
The goal of this paper is to label all the animal individuals present in every frame of a video.
WiCV 2019: The Sixth Women In Computer Vision Workshop
In this paper we present the Women in Computer Vision Workshop - WiCV 2019, organized in conjunction with CVPR 2019.
VoxSRC 2019: The first VoxCeleb Speaker Recognition Challenge
The VoxCeleb Speaker Recognition Challenge 2019 aimed to assess how well current speaker recognition technology is able to identify speakers in unconstrained or `in the wild' data.
Disentangled Speech Embeddings using Cross-modal Self-supervision
The objective of this paper is to learn representations of speaker identity without access to manually annotated data.
Speech2Action: Cross-modal Supervision for Action Recognition
We train a BERT-based Speech2Action classifier on over a thousand movie screenplays, to predict action labels from transcribed speech segments.
Spot the conversation: speaker diarisation in the wild
Finally, we use this pipeline to create a large-scale diarisation dataset called VoxConverse, collected from 'in the wild' videos, which we will release publicly to the research community.
Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos
Despite the recent advances in video classification, progress in spatio-temporal action recognition has lagged behind.
Look Before you Speak: Visually Contextualized Utterances
Leveraging recent advances in multimodal learning, our model consists of a novel co-attentional multimodal video transformer, and when trained on both textual and visual context, outperforms baselines that use textual inputs alone.
VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challenge
We held the second installment of the VoxCeleb Speaker Recognition Challenge in conjunction with Interspeech 2020.
WiCV 2020: The Seventh Women In Computer Vision Workshop
It provides a voice to a minority (female) group in computer vision community and focuses on increasingly the visibility of these researchers, both in academia and industry.
Composable Augmentation Encoding for Video Representation Learning
We focus on contrastive methods for self-supervised video representation learning.
Recognizing Multimodal Entailment
How information is created, shared and consumed has changed rapidly in recent decades, in part thanks to new social platforms and technologies on the web.
Masking Modalities for Cross-modal Video Retrieval
Our proposal is to pre-train a video encoder using all the available video modalities as supervision, namely, appearance, sound, and transcribed speech.
Audio-Visual Synchronisation in the wild
Finally, we set the first benchmark for general audio-visual synchronisation with over 160 diverse classes in the new VGG-Sound Sync video dataset.
End-to-end Generative Pretraining for Multimodal Video Captioning
Recent video and language pretraining frameworks lack the ability to generate sentences.
Ranked #13 on
Video Captioning
on MSR-VTT
(using extra training data)
Learning Audio-Video Modalities from Image Captions
To close this gap we propose a new video mining pipeline which involves transferring captions from image captioning datasets to video clips with no additional manual effort.
Ranked #6 on
Zero-shot Text to Audio Retrieval
on AudioCaps
M&M Mix: A Multimodal Multiview Transformer Ensemble
This report describes the approach behind our winning solution to the 2022 Epic-Kitchens Action Recognition Challenge.
Ranked #2 on
Action Recognition
on EPIC-KITCHENS-100
(using extra training data)
TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency
In this work, we focus on summarizing instructional videos, an under-explored area of video summarization.
AVATAR submission to the Ego4D AV Transcription Challenge
In this report, we describe our submission to the Ego4D AudioVisual (AV) Speech Transcription Challenge 2022.
AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR
(ii) We also introduce a simple curriculum scheme during training which we show is crucial to enable the model to jointly process audio and visual information effectively; and finally (iii) we show that our model achieves state of the art zero-shot results on three different AV-ASR benchmarks (How2, VisSpeech and Ego4D), while also crucially preserving decent performance on traditional audio-only speech recognition benchmarks (LibriSpeech).
VicTR: Video-conditioned Text Representations for Activity Recognition
In this paper, we argue the contrary, that better video-VLMs can be designed by focusing more on augmenting text, rather than visual information.
Ranked #8 on
Action Classification
on Charades
LanSER: Language-Model Supported Speech Emotion Recognition
Speech emotion recognition (SER) models typically rely on costly human-labeled data for training, making scaling methods to large speech datasets and nuanced emotion taxonomies difficult.
AutoAD II: The Sequel - Who, When, and What in Movie Audio Description
Audio Description (AD) is the task of generating descriptions of visual content, at suitable time intervals, for the benefit of visually impaired audiences.
AutoAD II: The Sequel -- Who, When, and What in Movie Audio Description
Audio Description (AD) is the task of generating descriptions of visual content, at suitable time intervals, for the benefit of visually impaired audiences.
VidChapters-7M: Video Chapters at Scale
To address this issue, we present VidChapters-7M, a dataset of 817K user-chaptered videos including 7M chapters in total.
Video Summarization: Towards Entity-Aware Captions
We also release a large-scale dataset, VIEWS (VIdeo NEWS), to support research on this task.
MoReVQA: Exploring Modular Reasoning Models for Video Question Answering
This paper addresses the task of video question answering (videoQA) via a decomposed multi-stage, modular reasoning framework.
