Search Results for author:
Found 68 papers, 40 papers with code
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Egocentric Video-Language Pretraining
Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention.
Egocentric Video-Language Pretraining @ Ego4D Challenge 2022
In this report, we propose a video-language pretraining (VLP) based solution \cite{kevin2022egovlp} for four Ego4D challenge tasks, including Natural Language Query (NLQ), Moment Query (MQ), Object State Change Classification (OSCC), and PNR Localization (PNR).
Perception Test: A Diagnostic Benchmark for Multimodal Models
We propose a novel multimodal benchmark – the Perception Test – that aims to extensively evaluate perception and reasoning skills of multimodal models.
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e. g. Flamingo, SeViLA, or GPT-4).
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
First-person vision is gaining interest as it offers a unique viewpoint on people's interaction with objects, their attention, and even intention.
TIM: A Time Interval Machine for Audio-Visual Action Recognition
We address the interplay between the two modalities in long videos by explicitly modelling the temporal extents of audio and visual events.
Rescaling Egocentric Vision
This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS.
Ranked #6 on
Action Anticipation
on EPIC-KITCHENS-100
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition
We focus on multi-modal fusion for egocentric action recognition, and propose a novel architecture for multi-modal temporal-binding, i. e. the combination of modalities within a range of temporal offsets.
Ranked #2 on
Egocentric Activity Recognition
on EPIC-KITCHENS-55
Temporal-Relational CrossTransformers for Few-Shot Action Recognition
We propose a novel approach to few-shot action recognition, finding temporally-corresponding frame tuples between the query and videos in the support set.
An Evaluation of Action Recognition Models on EPIC-Kitchens
We benchmark contemporary action recognition models (TSN, TRN, and TSM) on the recently introduced EPIC-Kitchens dataset and release pretrained models on GitHub (https://github. com/epic-kitchens/action-models) for others to build upon.
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines
Our dataset features 55 hours of video consisting of 11. 5M frames, which we densely labelled for a total of 39. 6K action segments and 454. 2K object bounding boxes.
Slow-Fast Auditory Streams For Audio Recognition
We propose a two-stream convolutional network for audio recognition, that operates on time-frequency spectrogram inputs.
Ranked #1 on
Human Interaction Recognition
on EPIC-SOUNDS
Epic-Sounds: A Large-scale Dataset of Actions That Sound
We introduce EPIC-SOUNDS, a large-scale dataset of audio annotations capturing temporal extents and class labels within the audio stream of the egocentric videos.
Multi-Modal Domain Adaptation for Fine-Grained Action Recognition
We then combine adversarial training with multi-modal self-supervision, showing that our approach outperforms other UDA methods by 3%.
On Semantic Similarity in Video Retrieval
Current video retrieval efforts all found their evaluation on an instance-based assumption, that only a single caption is relevant to a query video and vice versa.
GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos
We address the task of generating temporally consistent and physically plausible images of actions and object state transformations.
The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videos
In addition to attending to task relevant video parts, our proposed loss jointly trains two attention modules to separately attend to video parts which are indicative of higher (pros) and lower (cons) skill.
EPIC Fields: Marrying 3D Geometry and Video Understanding
Compared to other neural rendering datasets, EPIC Fields is better tailored to video understanding because it is paired with labelled action segments and the recent VISOR segment annotations.
Action Modifiers: Learning from Adverbs in Instructional Videos
We present a method to learn a representation for adverbs from instructional videos using weak supervision from the accompanying narrations.
The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction
We propose a bottleneck-based attention model that captures the evolution of the action, through progressive sampling over fine-to-coarse scales.
Ranked #1 on
Early Action Prediction
on UCF101
With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition
We capitalise on the action's temporal context and propose a method that learns to attend to surrounding actions in order to improve recognition performance.
Play Fair: Frame Attributions in Video Models
We offer detailed analysis of supporting/distracting frames, and the relationships of ESVs to the frame's position, class prediction, and sequence length.
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations
VISOR annotates videos from EPIC-KITCHENS, which comes with a new set of challenges not encountered in current video segmentation datasets.
Dual-Domain Image Synthesis using Segmentation-Guided GAN
We build on the successes of few-shot StyleGAN and single-shot semantic segmentation to minimise the amount of training required in utilising two domains.
Every Shot Counts: Using Exemplars for Repetition Counting in Videos
We propose an exemplar-based approach that discovers visual correspondence of video exemplars across repetitions within target videos.
Play It Back: Iterative Attention for Audio Recognition
A key function of auditory cognition is the association of characteristic sounds with their corresponding semantics over time.
Ranked #4 on
Audio Classification
on EPIC-KITCHENS-100
TVNet: Temporal Voting Network for Action Localization
This incorporates a novel Voting Evidence Module to locate temporal boundaries, more accurately, where temporal contextual evidence is accumulated to predict frame-level probabilities of start and end action boundaries.
Refining Action Boundaries for One-stage Detection
We obtain state-of-the-art performance on the challenging EPIC-KITCHENS-100 action detection as well as the standard THUMOS14 action detection benchmarks, and achieve improvement on the ActivityNet-1. 3 benchmark.
Use Your Head: Improving Long-Tail Video Recognition
We demonstrate that, unlike naturally-collected video datasets and existing long-tail image benchmarks, current video benchmarks fall short on multiple long-tailed properties.
Learning Temporal Sentence Grounding From Narrated EgoVideos
Compared to traditional benchmarks on which this task is evaluated, these datasets offer finer-grained sentences to ground in notably longer videos.
Learning Visual Actions Using Multiple Verb-Only Labels
We collect multi-verb annotations for three action video datasets and evaluate the verb-only labelling representations for action recognition and cross-modal retrieval (video-to-text and text-to-video).
An Evaluation of OCR on Egocentric Data
In this paper, we evaluate state-of-the-art OCR methods on Egocentric data.
Action Completion: A Temporal Model for Moment Detection
The paper proposes a joint classification-regression recurrent model that predicts completion from a given frame, and then integrates frame-level contributions to detect sequence-level completion moment.
UnweaveNet: Unweaving Activity Stories
Our lives can be seen as a complex weaving of activities; we switch from one activity to another, to maximise our achievements or in reaction to demands placed upon us.
Centre Stage: Centricity-based Audio-Visual Temporal Action Detection
Previous one-stage action detection approaches have modelled temporal dependencies using only the visual modality.
Meta-Learning with Context-Agnostic Initialisations
This produces an initialisation for fine-tuning to target which is both context-agnostic and task-generalised.
Supervision Levels Scale (SLS)
We propose a three-dimensional discrete and incremental scale to encode a method's level of supervision - i. e. the data and labels used when training a model to achieve a given performance.
Semantically Selective Augmentation for Deep Compact Person Re-Identification
We present a deep person re-identification approach that combines semantically selective, deep data augmentation with clustering-based network compression to generate high performance, light and fast inference networks.
Towards an Unequivocal Representation of Actions
This work introduces verb-only representations for actions and interactions; the problem of describing similar motions (e. g. 'open door', 'open cupboard'), and distinguish differing ones (e. g. 'open door' vs 'open bottle') using verb-only labels.
Who's Better? Who's Best? Pairwise Deep Ranking for Skill Determination
We present a method for assessing skill from video, applicable to a variety of tasks, ranging from surgery to drawing and rolling pizza dough.
Detecting the Moment of Completion: Temporal Models for Localising Action Completion
Action completion detection is the problem of modelling the action's progression towards localising the moment of completion - when the action's goal is confidently considered achieved.
Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Video
Manual annotations of temporal bounds for object interactions (i. e. start and end times) are typical training input to recognition, localization and detection algorithms.
Improving Classification by Improving Labelling: Introducing Probabilistic Multi-Label Object Interaction Recognition
This work deviates from easy-to-define class boundaries for object interactions.
SEMBED: Semantic Embedding of Egocentric Action Videos
We present SEMBED, an approach for embedding an egocentric object interaction video in a semantic-visual graph to estimate the probability distribution over its potential semantic labels.
Calorie Counter: RGB-Depth Visual Estimation of Energy Expenditure at Home
We present a new framework for vision-based estimation of calorific expenditure from RGB-D data - the first that is validated on physical gas exchange measurements and applied to daily living scenarios.
Multiple Human Tracking in RGB-D Data: A Survey
Finally, we present a brief comparative evaluation of the performance of those works that have applied their methods to these datasets.
You-Do, I-Learn: Unsupervised Multi-User egocentric Approach Towards Video-Based Guidance
This paper presents an unsupervised approach towards automatically extracting video-based guidance on object usage, from egocentric video and wearable gaze tracking, collected from multiple users while performing tasks.
CaloriNet: From silhouettes to calorie estimation in private environments
We propose a novel deep fusion architecture, CaloriNet, for the online estimation of energy expenditure for free living monitoring in private environments, where RGB data is discarded and replaced by silhouettes.
Action Recognition from Single Timestamp Supervision in Untrimmed Videos
We propose a method that is supervised by single timestamps located around each action instance, in untrimmed videos.
DDLSTM: Dual-Domain LSTM for Cross-Dataset Action Recognition
Domain alignment in convolutional networks aims to learn the degree of layer-specific feature alignment beneficial to the joint learning of source and target datasets.
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings
We report the first retrieval results on fine-grained actions for the large-scale EPIC dataset, in a generalised zero-shot setting.
Retro-Actions: Learning 'Close' by Time-Reversing 'Open' Videos
We investigate video transforms that result in class-homogeneous label-transforms.
Sit-to-Stand Analysis in the Wild using Silhouettes for Longitudinal Health Monitoring
We present the first fully automated Sit-to-Stand or Stand-to-Sit (StS) analysis framework for long-term monitoring of patients in free-living environments using video silhouettes.
Weakly-Supervised Completion Moment Detection using Temporal Attention
In this work, we target detecting the completion moment of actions, that is the moment when the action's goal has been successfully accomplished.
Domain Adaptation in Multi-View Embedding for Cross-Modal Video Retrieval
Given a gallery of uncaptioned video sequences, this paper considers the task of retrieving videos based on their relevance to an unseen text query.
Hand-Object Interaction Reasoning
This paper proposes an interaction reasoning network for modelling spatio-temporal relationships between hands and objects in video.
Inertial Hallucinations -- When Wearable Inertial Devices Start Seeing Things
We propose a novel approach to multimodal sensor fusion for Ambient Assisted Living (AAL) which takes advantage of learning using privileged information (LUPI).
What can a cook in Italy teach a mechanic in India? Action Recognition Generalisation Over Scenarios and Locations
We propose and address a new generalisation problem: can a model trained for action recognition successfully classify actions when they are performed within a previously unseen scenario and in a previously unseen location?
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
Learning from One Continuous Video Stream
We introduce a framework for online learning from a single continuous video stream -- the way people and animals learn, without mini-batches, data augmentation or shuffling.
Perception Test 2023: A Summary of the First Challenge And Outcome
The First Perception Test challenge was held as a half-day workshop alongside the IEEE/CVF International Conference on Computer Vision (ICCV) 2023, with the goal of benchmarking state-of-the-art video models on the recently proposed Perception Test benchmark.
Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videos
Our proposed EPIC-Grasps dataset includes 390 object instances of 9 categories, featuring stable grasps from videos of daily interactions in 141 environments.
Video Editing for Video Retrieval
The teacher model is employed to edit the clips in the training set whereas the student model trains on the edited clips.
Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mind
As humans move around, performing their daily tasks, they are able to recall where they have positioned objects in their environment, even if these objects are currently out of sight.
HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision
Our results demonstrate that VLMs trained for referral on third person images fail to recognise and refer hands and objects in egocentric images.
