Search Results for author:
Found 219 papers, 102 papers with code
Amplifying Key Cues for Human-Object-Interaction Detection
In this paper we introduce two methods to amplify key cues in the image, and also a method to combine these and other cues when considering the interaction between a human and an object.
The Making and Breaking of Camouflage
We also incorporate the proposed camouflage score into a generative model as an auxiliary loss and show that effective camouflage images or videos can be synthesised in a scalable manner.
The Change You Want to See (Now in 3D)
The goal of this paper is to detect what has changed, if anything, between two "in the wild" images of the same 3D scene acquired from different camera positions and at different temporal instances.
Helping Hands: An Object-Aware Ego-Centric Video Recognition Model
We demonstrate the performance of the object-aware representations learnt by our model, by: (i) evaluating it for strong transfer, i. e. through zero-shot testing, on a number of downstream video-text retrieval and classification benchmarks; and (ii) by using the representations learned as input for long-term video understanding tasks (e. g. Episodic Memory in Ego4D).
OxfordVGG Submission to the EGO4D AV Transcription Challenge
This report presents the technical details of our submission on the EGO4D Audio-Visual (AV) Automatic Speech Recognition Challenge 2023 from the OxfordVGG team.
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement
We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence.
Ranked #1 on
Visual Tracking
on RGB-Stacking
Multi-Modal Classifiers for Open-Vocabulary Object Detection
The goal of this paper is open-vocabulary object detection (OVOD) $\unicode{x2013}$ building a model that can detect objects beyond the set of categories seen at training, thus enabling the user to specify categories of interest at inference without the need for model retraining.
Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion
Our approach outperforms the state-of-the-art on challenging scenes from the ScanNet, Hypersim, and Replica datasets, as well as on our newly created Messy Rooms dataset, demonstrating the effectiveness and scalability of our slow-fast clustering method.
Open-world Text-specified Object Counting
Our objective is open-world object counting in images, where the target object class is specified by a text description.
Ranked #6 on
Object Counting
on CARPK
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e. g. Flamingo, BEiT-3, or GPT-4).
Verbs in Action: Improving verb understanding in video-language models
Understanding verbs is crucial to modelling how people and objects interact with each other and the environment through space and time.
Ranked #8 on
Video Question Answering
on NExT-QA
(using extra training data)
Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime
This paper explores training medical vision-language models (VLMs) -- where the visual and language inputs are embedded into a common space -- with a particular focus on scenarios where training data is limited, as is often the case in clinical datasets.
AutoAD: Movie Description in Context
The objective of this paper is an automatic Audio Description (AD) model that ingests movies and outputs AD in text form.
Three ways to improve feature alignment for open vocabulary detection
The core problem in zero-shot open vocabulary detection is how to align visual and text features, so that the detector performs well on unseen classes.
VoxSRC 2022: The Fourth VoxCeleb Speaker Recognition Challenge
This paper summarises the findings from the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22), which was held in conjunction with INTERSPEECH 2022.
Epic-Sounds: A Large-scale Dataset of Actions That Sound
We introduce EPIC-SOUNDS, a large-scale dataset of audio annotations capturing temporal extents and class labels within the audio stream of the egocentric videos.
Zorro: the masked multimodal transformer
Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring very little fusion engineering.
AutoAD II: The Sequel - Who, When, and What in Movie Audio Description
Audio Description (AD) is the task of generating descriptions of visual content, at suitable time intervals, for the benefit of visually impaired audiences.
A Light Touch Approach to Teaching Transformers Multi-view Geometry
Transformers are powerful visual learners, in large part due to their conspicuous lack of manually-specified priors.
Weakly-supervised Fingerspelling Recognition in British Sign Language Videos
Through extensive evaluations, we verify our method for automatic annotation and our model architecture.
TAP-Vid: A Benchmark for Tracking Any Point in a Video
Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move.
End-to-end Tracking with a Multi-query Transformer
Multiple-object tracking (MOT) is a challenging task that requires simultaneous reasoning about location, appearance, and identity of the objects in the scene over time.
A Tri-Layer Plugin to Improve Occluded Detection
To this end we make the following four contributions: (1) We propose a simple 'plugin' module for the detection head of two-stage object detectors to improve the recall of partially occluded objects.
Ranked #1 on
Instance Segmentation
on Separated COCO
Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors
This contrasts with the case of synchronising videos of talking heads, where audio-visual correspondence is dense in both time and space.
Turbo Training with Token Dropout
The objective of this paper is an efficient training method for video tasks.
Compressed Vision for Efficient Video Understanding
In this work, we propose a framework enabling research on hour-long videos with the same hardware that can now process second-long videos.
CounTR: Transformer-based Generalised Visual Counting
In this paper, we consider the problem of generalised visual object counting, with the goal of developing a computational model for counting the number of objects from arbitrary semantic categories, using arbitrary number of "exemplars", i. e. zero-shot or few-shot counting.
Ranked #2 on
Object Counting
on FSC147
Automatic dense annotation of large-vocabulary sign language videos
Recently, sign language researchers have turned to sign language interpreted TV broadcasts, comprising (i) a video of continuous signing and (ii) subtitles corresponding to the audio content, as a readily available and large-scale source of training data.
Is an Object-Centric Video Representation Beneficial for Transfer?
The model learns a set of object-centric summary vectors for the video, and uses these vectors to fuse the visual and spatio-temporal trajectory 'modalities' of the video clip.
Segmenting Moving Objects via an Object-Centric Layered Representation
The objective of this paper is a model that is able to discover, track and segment multiple moving objects in a video.
Ranked #3 on
Unsupervised Object Segmentation
on FBMS-59
Context-Aware Transformers For Spinal Cancer Detection and Radiological Grading
(iii) We also apply SCT to an existing problem: radiological grading of inter-vertebral discs (IVDs) in lumbar MR scans for common degenerative changes. We show that by considering the context of vertebral bodies in the image, SCT improves the accuracy for several gradings compared to previously published model.
A CLIP-Hitchhiker's Guide to Long Video Retrieval
Our goal in this paper is the adaptation of image-text models for long video retrieval.
Ranked #4 on
Zero-Shot Action Recognition
on Charades
Scaling up sign spotting through sign language dictionaries
The focus of this work is $\textit{sign spotting}$ - given a video of an isolated sign, our task is to identify $\textit{whether}$ and $\textit{where}$ it has been signed in a continuous, co-articulated sign language video.
SpineNetV2: Automated Detection, Labelling and Radiological Grading Of Clinical MR Scans
This technical report presents SpineNetV2, an automated tool which: (i) detects and labels vertebral bodies in clinical spinal magnetic resonance (MR) scans across a range of commonly used sequences; and (ii) performs radiological grading of lumbar intervertebral discs in T2-weighted scans for a range of common degenerative changes.
Flamingo: a Visual Language Model for Few-Shot Learning
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research.
Ranked #1 on
Action Recognition
on RareAct
Temporal Alignment Networks for Long-term Video
The objective of this paper is a temporal alignment network that ingests long term video sequences, and associated text sentences, in order to: (1) determine if a sentence is alignable with the video; and (2) if it is alignable, then determine its alignment.
Object discovery and representation networks
The promise of self-supervised learning (SSL) is to leverage large amounts of unlabeled data to solve complex tasks.
HiP: Hierarchical Perceiver
This however hinders them from scaling up to the inputs sizes required to process raw high-resolution images or video.
Generalized Category Discovery
Here, the unlabelled images may come from labelled classes or from novel ones.
Ranked #1 on
Open-World Semi-Supervised Learning
on CIFAR-10
(Seen accuracy (50% Labeled) metric)
Fine-Grained Visual Recognition
Open-World Semi-Supervised Learning
+1
Reading To Listen at the Cocktail Party: Multi-Modal Speech Separation
The goal of this paper is speech separation and enhancement in multi-speaker and noisy environments using a combination of different modalities.
Persistent Animal Identification Leveraging Non-Visual Markers
Our objective is to locate and provide a unique identifier for each mouse in a cluttered home-cage environment through time, as a precursor to automated behaviour recognition for biological research.
Label, Verify, Correct: A Simple Few Shot Object Detection Method
The objective of this paper is few-shot object detection (FSOD) -- the task of expanding an object detector for a new category given only a few instances for training.
Audio-Visual Synchronisation in the wild
Finally, we set the first benchmark for general audio-visual synchronisation with over 160 diverse classes in the new VGG-Sound Sync video dataset.
Input-level Inductive Biases for 3D Reconstruction
Much of the recent progress in 3D vision has been driven by the development of specialized architectures that incorporate geometrical inductive biases.
It's About Time: Analog Clock Reading in the Wild
In this paper, we present a framework for reading analog clocks in natural images or videos.
BBC-Oxford British Sign Language Dataset
In this work, we introduce the BBC-Oxford British Sign Language (BOBSL) dataset, a large-scale video collection of British Sign Language (BSL).
With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition
We capitalise on the action's temporal context and propose a method that learns to attend to surrounding actions in order to improve recognition performance.
Visual Keyword Spotting with Attention
In this paper, we consider the task of spotting spoken keywords in silent video sequences -- also known as visual keyword spotting.
Ranked #1 on
Visual Keyword Spotting
on LRS2
Sub-word Level Lip Reading With Visual Attention
To this end, we make the following contributions: (1) we propose an attention-based pooling mechanism to aggregate visual speech representations; (2) we use sub-word units for lip reading for the first time and show that this allows us to better model the ambiguities of the task; (3) we propose a model for Visual Speech Detection (VSD), trained on top of the lip reading network.
Ranked #1 on
Visual Speech Recognition
on LRS2
(using extra training data)
Audio-Visual Active Speaker Detection
Automatic Speech Recognition
+5
Open-Set Recognition: a Good Closed-Set Classifier is All You Need?
In this paper, we first demonstrate that the ability of a classifier to make the 'none-of-above' decision is highly correlated with its accuracy on the closed-set classes.
Ranked #10 on
Out-of-Distribution Detection
on CIFAR-100 vs CIFAR-10
PASS: An ImageNet replacement for self-supervised pretraining without humans
On the other hand, state-of-the-art pretraining is nowadays obtained with unsupervised methods, meaning that labelled datasets such as ImageNet may not be necessary, or perhaps not even optimal, for model pretraining.
Perceiver IO: A General Architecture for Structured Inputs & Outputs
A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible.
Ranked #1 on
Optical Flow Estimation
on KITTI 2015
(Average End-Point Error metric)
Self-Supervised Multi-Modal Alignment for Whole Body Medical Imaging
This paper explores the use of self-supervised deep learning in medical imaging in cases where two scan modalities are available for the same subject.
AutoNovel: Automatically Discovering and Learning Novel Visual Categories
We present a new approach called AutoNovel to address this problem by combining three ideas: (1) we suggest that the common approach of bootstrapping an image representation using the labelled data only introduces an unwanted bias, and that this can be avoided by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) we use ranking statistics to transfer the model's knowledge of the labelled classes to the problem of clustering the unlabelled images; and, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classification of the labelled data, and the clustering of the unlabelled data.
Ranked #1 on
Novel Class Discovery
on SVHN
NeRF in detail: Learning to sample for view synthesis
In this work we address a clear limitation of the vanilla coarse-to-fine approach -- that it is based on a heuristic and not trained end-to-end for the task at hand.
Comment on Stochastic Polyak Step-Size: Performance of ALI-G
This is a short note on the performance of the ALI-G algorithm (Berrada et al., 2020) as reported in (Loizou et al., 2021).
Face, Body, Voice: Video Person-Clustering with Multiple Modalities
In this paper we make contributions to address both these deficiencies: first, we introduce a Multi-Modal High-Precision Clustering algorithm for person-clustering in videos using cues from several modalities (face, body, and voice).
Omnimatte: Associating Objects and Their Effects in Video
We show results on real-world videos containing interactions between different types of subjects (cars, animals, people) and complex effects, ranging from semi-transparent elements such as smoke and reflections, to fully opaque effects such as objects attached to the subject.
Aligning Subtitles in Sign Language Videos
The goal of this work is to temporally align asynchronous subtitles in sign language videos.
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
On semi-supervised learning benchmarks we improve performance significantly when only 1% ImageNet labels are available, from 53. 8% to 56. 5%.
Ranked #1 on
Image Classification
on PASCAL VOC 2007
Temporal Query Networks for Fine-grained Video Understanding
It attends to relevant segments for each query with a temporal attention mechanism, and can be trained using only the labels for each query.
TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval
In recent years, considerable progress on the task of text-video retrieval has been achieved by leveraging large-scale pretraining on visual and audio datasets to construct powerful video encoders.
Self-supervised Video Object Segmentation by Motion Grouping
We additionally evaluate on a challenging camouflage dataset (MoCA), significantly outperforming the other self-supervised approaches, and comparing favourably to the top supervised approach, highlighting the importance of motion cues, and the potential bias towards visual appearance in existing video segmentation models.
Ranked #7 on
Unsupervised Object Segmentation
on DAVIS 2016
Localizing Visual Sounds the Hard Way
We show that our algorithm achieves state-of-the-art performance on the popular Flickr SoundNet dataset.
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Our objective in this work is video-text retrieval - in particular a joint embedding that enables efficient text-to-video retrieval.
Ranked #13 on
Zero-Shot Video Retrieval
on DiDeMo
(using extra training data)
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
We also extend our method to the video domain, improving the state of the art on the VATEX dataset.
Read and Attend: Temporal Localisation in Sign Language Videos
Our contributions are as follows: (1) we demonstrate the ability to leverage large quantities of continuous signing videos with weakly-aligned subtitles to localise signs in continuous sign language; (2) we employ the learned attention to automatically generate hundreds of thousands of annotations for a large sign vocabulary; (3) we collect a set of 37K manually verified sign instances across a vocabulary of 950 sign classes to support our study of sign language recognition; (4) by training on the newly annotated data from our method, we outperform the prior state of the art on the BSL-1K sign language recognition benchmark.
Broaden Your Views for Self-Supervised Video Learning
Most successful self-supervised learning methods are trained to align the representations of two independent views from the data.
Ranked #1 on
Self-Supervised Audio Classification
on ESC-50
Slow-Fast Auditory Streams For Audio Recognition
We propose a two-stream convolutional network for audio recognition, that operates on time-frequency spectrogram inputs.
Perceiver: General Perception with Iterative Attention
The perception models used in deep learning on the other hand are designed for individual modalities, often relying on domain-specific assumptions such as the local grid structures exploited by virtually all existing vision models.
Ranked #25 on
Audio Classification
on AudioSet
Automated Video Labelling: Identifying Faces by Corroborative Evidence
We present a method for automatically labelling all faces in video archives, such as TV broadcasts, by combining multiple evidence sources and multiple modalities (visual and audio).
LAEO-Net++: revisiting people Looking At Each Other in videos
For this purpose, we propose LAEO-Net++, a new deep CNN for determining LAEO in videos.
VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challenge
We held the second installment of the VoxCeleb Speaker Recognition Challenge in conjunction with Interspeech 2020.
Betrayed by Motion: Camouflaged Object Discovery via Motion Segmentation
We make the following three contributions: (i) We propose a novel architecture that consists of two essential components for breaking camouflage, namely, a differentiable registration module to align consecutive frames based on the background, which effectively emphasises the object boundary in the difference image, and a motion segmentation module with memory that discovers the moving objects, while maintaining the object permanence even when motion is absent at some point.
QuerYD: A video dataset with high-quality text and audio narrations
We introduce QuerYD, a new large-scale dataset for retrieval and event localisation in video.
A Short Note on the Kinetics-700-2020 Human Action Dataset
We describe the 2020 edition of the DeepMind Kinetics human action dataset, which replenishes and extends the Kinetics-700 dataset.
Self-supervised Co-training for Video Representation Learning
The objective of this paper is visual-only self-supervised video representation learning.
Ranked #12 on
Self-Supervised Action Recognition
on HMDB51 (finetuned)
Watch, read and lookup: learning to spot signs from multiple supervisors
The focus of this work is sign spotting - given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video.
Layered Neural Rendering for Retiming People in Video
We present a method for retiming people in an ordinary, natural video -- manipulating and editing the time in which different motions of individuals in the video occur.
Adaptive Text Recognition through Visual Matching
In this work, our objective is to address the problems of generalization and flexibility for text recognition in documents.
Seeing wake words: Audio-visual Keyword Spotting
The goal of this work is to automatically determine whether and when a word of interest is spoken by a talking face, with or without the audio.
Inducing Predictive Uncertainty Estimation for Face Recognition
We describe three use cases on the public IJB-C face verification benchmark: (i) to improve 1:1 image-based verification error rates by rejecting low-quality face images; (ii) to improve quality score based fusion performance on the 1:1 set-based verification benchmark; and (iii) its use as a quality measure for selecting high quality (unblurred, good lighting, more frontal) faces from a collection, e. g. for automatic enrolment or display.
Self-Supervised Learning of Audio-Visual Objects from Video
Our objective is to transform a video into a set of discrete audio-visual objects using self-supervised learning.
The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
This report summarizes the results of the first edition of the challenge together with the findings of the participants.
RareAct: A video dataset of unusual interactions
This paper introduces a manually annotated video dataset of unusual actions, namely RareAct, including actions such as "blend phone", "cut keyboard" and "microwave shoes".
Memory-augmented Dense Predictive Coding for Video Representation Learning
The objective of this paper is self-supervised learning from video, in particular for representations for action recognition.
BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues
Recent progress in fine-grained gesture and action classification, and machine translation, point to the possibility of automated sign language recognition becoming a reality.
Ranked #4 on
Sign Language Recognition
on WLASL-2000
Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval
Optimising a ranking-based metric, such as Average Precision (AP), is notoriously challenging due to the fact that it is non-differentiable, and hence cannot be optimised directly using gradient-descent methods.
Ranked #4 on
Vehicle Re-Identification
on VehicleID Medium
CrossTransformers: spatially-aware few-shot transfer
In this work, we illustrate how the neural network representations which underpin modern vision systems are subject to supervision collapse, whereby they lose any information that is not necessary for performing the training task, including information that may be necessary for transfer to new tasks or domains.
Co-Attention for Conditioned Image Matching
We propose a new approach to determine correspondences between image pairs in the wild under large changes in illumination, viewpoint, context, and material.
A Convolutional Approach to Vertebrae Detection and Labelling in Whole Spine MRI
We propose a novel convolutional method for the detection and identification of vertebrae in whole spine MRIs.
Spot the conversation: speaker diarisation in the wild
Finally, we use this pipeline to create a large-scale diarisation dataset called VoxConverse, collected from 'in the wild' videos, which we will release publicly to the research community.
Self-Supervised MultiModal Versatile Networks
In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a common embedding.
Counting Out Time: Class Agnostic Video Repetition Counting in the Wild
We present an approach for estimating the period with which an action is repeated in a video.
LSD-C: Linearly Separable Deep Clusters
We present LSD-C, a novel method to identify clusters in an unlabeled dataset.
Condensed Movies: Story Based Retrieval with Contextual Embeddings
Our objective in this work is long range understanding of the narrative structure of movies.
The AVA-Kinetics Localized Human Actions Video Dataset
The dataset is collected by annotating videos from the Kinetics-700 dataset using the AVA annotation protocol, and extending the original AVA dataset with these new AVA annotated Kinetics clips.
VGGSound: A Large-scale Audio-Visual Dataset
Our goal is to collect a large-scale audio-visual dataset with low label noise from videos in the wild using computer vision techniques.
Monocular Depth Estimation with Self-supervised Instance Adaptation
Recent advances in self-supervised learning havedemonstrated that it is possible to learn accurate monoculardepth reconstruction from raw video data, without using any 3Dground truth for supervision.
Speech2Action: Cross-modal Supervision for Action Recognition
We train a BERT-based Speech2Action classifier on over a thousand movie screenplays, to predict action labels from transcribed speech segments.
Compact Deep Aggregation for Set Retrieval
The objective of this work is to learn a compact embedding of a set of descriptors that is suitable for efficient retrieval and ranking, whilst maintaining discriminability of the individual descriptors.
Visual Grounding in Video for Unsupervised Word Translation
Given this shared embedding we demonstrate that (i) we can map words between the languages, particularly the 'visual' words; (ii) that the shared embedding provides a good initialization for existing unsupervised text-based word translation techniques, forming the basis for our proposed hybrid visual-text mapping algorithm, MUVE; and (iii) our approach achieves superior performance by addressing the shortcomings of text-based methods -- it is more robust, handles datasets with less commonality, and is applicable to low-resource languages.
Disentangled Speech Embeddings using Cross-modal Self-supervision
The objective of this paper is to learn representations of speaker identity without access to manually annotated data.
Automatically Discovering and Learning New Visual Categories with Ranking Statistics
In this work we address this problem by combining three ideas: (1) we suggest that the common approach of bootstrapping an image representation using the labeled data only introduces an unwanted bias, and that this can be avoided by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) we use rank statistics to transfer the model's knowledge of the labelled classes to the problem of clustering the unlabelled images; and, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classification of the labelled data, and the clustering of the unlabelled data.
End-to-End Learning of Visual Representations from Uncurated Instructional Videos
Annotating videos is cumbersome, expensive and not scalable.
Ranked #3 on
Action Recognition
on RareAct
Synthetic Humans for Action Recognition from Unseen Viewpoints
Although synthetic training data has been shown to be beneficial for tasks such as human pose estimation, its use for RGB human action recognition is relatively unexplored.
VoxSRC 2019: The first VoxCeleb Speaker Recognition Challenge
The VoxCeleb Speaker Recognition Challenge 2019 aimed to assess how well current speaker recognition technology is able to identify speakers in unconstrained or `in the wild' data.
ASR is all you need: cross-modal distillation for lip reading
The goal of this work is to train strong models for visual speech recognition without requiring human annotated ground truth data.
Ranked #13 on
Lipreading
on LRS3-TED
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Self-supervised learning of class embeddings from video
This work explores how to use self-supervised learning on videos to learn a class-specific image embedding that encodes pose and shape information.
Controllable Attention for Structured Layered Video Decomposition
The objective of this paper is to be able to separate a video into its natural layers, and to control which of the separated layers to attend to.
Count, Crop and Recognise: Fine-Grained Recognition in the Wild
The goal of this paper is to label all the animal individuals present in every frame of a video.
Video Representation Learning by Dense Predictive Coding
The objective of this paper is self-supervised learning of spatio-temporal embeddings from video, suitable for human action recognition.
Ranked #32 on
Self-Supervised Action Recognition
on UCF101
Representation Learning
Self-Supervised Action Recognition
+2
Geometry-Aware Video Object Detection for Static Cameras
In this paper we propose a geometry-aware model for video object detection.
Learning to Discover Novel Visual Categories via Deep Transfer Clustering
The second contribution is a method to estimate the number of classes in the unlabelled data.
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition
We focus on multi-modal fusion for egocentric action recognition, and propose a novel architecture for multi-modal temporal-binding, i. e. the combination of modalities within a range of temporal offsets.
Ranked #2 on
Egocentric Activity Recognition
on EPIC-KITCHENS-55
AutoCorrect: Deep Inductive Alignment of Noisy Geometric Annotations
We propose AutoCorrect, a method to automatically learn object-annotation alignments from a dataset with annotations affected by geometric noise.
Use What You Have: Video Retrieval Using Representations From Collaborative Experts
The rapid growth of video on the internet has made searching for video content using natural language queries a significant challenge.
Ranked #20 on
Video Retrieval
on MSVD
A Short Note on the Kinetics-700 Human Action Dataset
We describe an extension of the DeepMind Kinetics human action dataset from 600 classes to 700 classes, where for each class there are at least 600 video clips from different YouTube videos.
My lips are concealed: Audio-visual speech enhancement through obstructions
To this end we introduce a deep audio-visual speech enhancement network that is able to separate a speaker's voice by conditioning on both the speaker's lip movements and/or a representation of their voice.
Sim2real transfer learning for 3D human pose estimation: motion to the rescue
In this paper, we show that standard neural-network approaches, which perform poorly when trained on synthetic RGB images, can perform well when the data is pre-processed to extract cues about the person's motion, notably as optical flow and the motion of 2D keypoints.
Unsupervised Learning of Object Keypoints for Perception and Control
In this work we aim to learn object representations that are useful for control and reinforcement learning (RL).
Training Neural Networks for and by Interpolation
In modern supervised learning, many deep neural networks are able to interpolate the data: the empirical loss can be driven to near zero on all samples simultaneously.
LAEO-Net: revisiting people Looking At Each Other in videos
For this purpose, we propose LAEO-Net, a new deep CNN for determining LAEO in videos.
A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities
Medical imaging only indirectly measures the molecular identity of the tissue within each voxel, which often produces only ambiguous image evidence for target measures of interest, like semantic segmentation.
Object Discovery with a Copy-Pasting GAN
We tackle the problem of object discovery, where objects are segmented for a given input image, and the system is trained without using any direct supervision whatsoever.
Semi-Supervised Learning with Scarce Annotations
The first is a simple but effective one: we leverage the power of transfer learning among different tasks and self-supervision to initialize a good representation of the data without making use of any label.
Exploiting temporal context for 3D human pose estimation in the wild
We present a bundle-adjustment-based algorithm for recovering accurate 3D human pose and meshes from monocular videos.
Ranked #1 on
Monocular 3D Human Pose Estimation
on Human3.6M
(Use Video Sequence metric)
A Geometric Approach to Obtain a Bird's Eye View from an Image
The objective of this paper is to rectify any monocular image by computing a homography matrix that transforms it to a bird's eye (overhead) view.
The VIA Annotation Software for Images, Audio and Video
In this paper, we introduce a simple and standalone manual annotation tool for images, audio and video: the VGG Image Annotator (VIA).
Temporal Cycle-Consistency Learning
We introduce a self-supervised representation learning method based on the task of temporal alignment between videos.
Ranked #1 on
Video Alignment
on UPenn Action
The StreetLearn Environment and Dataset
These datasets cannot be used for decision-making and reinforcement learning, however, and in general the perspective of navigation as an interactive learning task, where the actions and behaviours of a learning agent are learned simultaneously with the perception and planning, is relatively unsupported.
Utterance-level Aggregation For Speaker Recognition In The Wild
The objective of this paper is speaker recognition "in the wild"-where utterances may be of variable length and also contain irrelevant signals.
Video Action Transformer Network
We introduce the Action Transformer model for recognizing and localizing human actions in video clips.
Ranked #6 on
Action Recognition
on AVA v2.1
The Visual Centrifuge: Model-Free Layered Video Representations
True video understanding requires making sense of non-lambertian scenes where the color of light arriving at the camera sensor encodes information about not just the last object it collided with, but about multiple mediums -- colored windows, dirty mirrors, smoke or rain.
Deep Frank-Wolfe For Neural Network Optimization
Furthermore, we compare our algorithm to SGD with a hand-designed learning rate schedule, and show that it provides similar generalization while converging faster.
Class-Agnostic Counting
The model achieves competitive performance on cell and crowd counting datasets, and surpasses the state-of-the-art on the car dataset using only three training images.
GhostVLAD for set-based face recognition
The objective of this paper is to learn a compact representation of image sets for template-based face recognition.
Ranked #3 on
Face Verification
on IJB-A
Learning to Read by Spelling: Towards Unsupervised Text Recognition
This work presents a method for visual text recognition without using any paired supervisory data.
From Same Photo: Cheating on Visual Kinship Challenges
In the instance of data sets for visual kinship verification, one such unintended signal could be that the faces are cropped from the same photograph, since faces from the same photograph are more likely to be from the same family.
Deep Audio-Visual Speech Recognition
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
Ranked #6 on
Audio-Visual Speech Recognition
on LRS2
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+4
Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings
We demonstrate on this dataset, for a number of facial attribute classification tasks, that we are able to remove racial biases from the network feature representation.
3D Surface Reconstruction by Pointillism
Finally, we demonstrate that we can indeed obtain a depth map of a novel object from a single image for a variety of sculptures with varying shape/texture, and that the network generalises at test time to new domains (e. g. synthetic images).
LRS3-TED: a large-scale dataset for visual speech recognition
This paper introduces a new multi-modal dataset for visual and audio-visual speech recognition.
X2Face: A network for controlling face generation using images, audio, and pose codes
The objective of this paper is a neural network model that controls the pose and expression of a given face, using another face or modality (e. g. audio).
Self-supervised learning of a facial attribute embedding from video
We propose a self-supervised framework for learning facial attributes by simply watching videos of a human face speaking, laughing, and moving over time.
Ranked #2 on
Unsupervised Facial Landmark Detection
on 300W
Self-Supervised Learning
Unsupervised Facial Landmark Detection
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
We make the following contributions: (i) we develop a strong teacher network for facial emotion recognition that achieves the state of the art on a standard benchmark; (ii) we use the teacher to train a student, tabula rasa, to learn representations (embeddings) for speech emotion recognition without access to labelled audio data; and (iii) we show that the speech emotion embedding can be used for speech emotion recognition on external benchmark datasets.
Ranked #3 on
Facial Expression Recognition (FER)
on FERPlus
Facial Emotion Recognition
Facial Expression Recognition (FER)
+1
A Short Note about Kinetics-600
We describe an extension of the DeepMind Kinetics human action dataset from 400 classes, each with at least 400 video clips, to 600 classes, each with at least 600 video clips.
Ranked #59 on
Action Classification
on Kinetics-600
Comparator Networks
Our contributions are: (i) We propose a Deep Comparator Network (DCN) that can ingest a pair of sets (each may contain a variable number of images) as inputs, and compute a similarity between the pair--this involves attending to multiple discriminative local regions (landmarks), and comparing local descriptors between pairs of faces; (ii) To encourage high-quality representations for each set, internal competition is introduced for recalibration based on the landmark score; (iii) Inspired by image retrieval, a novel hard sample mining regime is proposed to control the sampling process, such that the DCN is complementary to the standard image classification models.
X2Face: A network for controlling face generation by using images, audio, and pose codes
The objective of this paper is a neural network model that controls the pose and expression of a given face, using another face or modality (e. g. audio).
A Better Baseline for AVA
We introduce a simple baseline for action localization on the AVA dataset.
Ranked #12 on
Action Recognition
on AVA v2.1
Multicolumn Networks for Face Recognition
In this paper, we design a neural network architecture that learns to aggregate based on both "visual" quality (resolution, illumination), and "content" quality (relative importance for discriminative classification).
Ranked #5 on
Face Verification
on IJB-C
(TAR @ FAR=1e-2 metric)
Inductive Visual Localisation: Factorised Training for Superior Generalisation
End-to-end trained Recurrent Neural Networks (RNNs) have been successfully applied to numerous problems that require processing sequences, such as image captioning, machine translation, and text recognition.
Deep Lip Reading: a comparison of models and an online application
The goal of this paper is to develop state-of-the-art models for lip reading -- visual speech recognition.
VoxCeleb2: Deep Speaker Recognition
The objective of this paper is speaker recognition under noisy and unconstrained conditions.
Ranked #1 on
Speaker Verification
on VoxCeleb2
(using extra training data)
Massively Parallel Video Networks
We introduce a class of causal video understanding models that aims to improve efficiency of video processing by maximising throughput, minimising latency, and reducing the number of clock cycles.
Learning and Using the Arrow of Time
We seek to understand the arrow of time in videos -- what makes videos look like they are playing forwards or backwards?
Ranked #49 on
Self-Supervised Action Recognition
on UCF101
Self-Supervised Action Recognition
Temporal Action Localization
+1
Learnable PINs: Cross-Modal Embeddings for Person Identity
We propose and investigate an identity sensitive joint embedding of face and voice.
The Conversation: Deep Audio-Visual Speech Enhancement
Our goal is to isolate individual speakers from multi-talker simultaneous speech in videos.
Seeing Voices and Hearing Faces: Cross-modal biometric matching
We make the following contributions: (i) we introduce CNN architectures for both binary and multi-way cross-modal face and audio matching, (ii) we compare dynamic testing (where video information is available, but the audio is not from the same video) with static testing (where only a single still image is available), and (iii) we use human testing as a baseline to calibrate the difficulty of the task.
Learning to Navigate in Cities Without a Map
We present an interactive navigation environment that uses Google StreetView for its photographic content and worldwide coverage, and demonstrate that our learning method allows agents to learn to navigate multiple cities and to traverse to target destinations that may be kilometres away.
Kickstarting Deep Reinforcement Learning
Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance.