Search Results for author:
Found 55 papers, 7 papers with code
Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for VideoQA
We propose the Video Language Co-Attention Network (VLCN) – a novel memory-enhanced model for Video Question Answering (VideoQA).
CLAD: Constrained Latent Action Diffusion for Vision-Language Procedure Planning
By evaluating ablated versions of our method, we further show that the proposed integration of the action and observation representations learnt in the VAE latent space is key to these performance improvements.
HAIFAI: Human-AI Collaboration for Mental Face Reconstruction
We evaluate HAIFAI and HAIFAI-X in a 12-participant user study and show that HAIFAI outperforms the previous state of the art regarding reconstruction quality, usability, perceived workload, and reconstruction speed.
HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality
Human hand and head movements are the most pervasive input modalities in extended reality (XR) and are significant for a wide range of applications.
Amortized Bayesian Multilevel Models
Multilevel models (MLMs) are a central building block of the Bayesian workflow.
Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactions
We propose MToMnet - a Theory of Mind (ToM) neural network for predicting beliefs and their dynamics during human social interactions from multimodal input.
Multi-Modal Video Dialog State Tracking in the Wild
Then, it predicts the missing underlying structure of the selected constituents of each modality by learning local latent graphs using a novel multi-modal graph structure learning method.
HOIMotion: Forecasting Human Motion During Human-Object Interactions Using Egocentric 3D Object Bounding Boxes
We present HOIMotion - a novel approach for human motion forecasting during human-object interactions that integrates information about past body poses and egocentric 3D object bounding boxes.
The Overcooked Generalisation Challenge
We introduce the Overcooked Generalisation Challenge (OGC) - the first benchmark to study agents' zero-shot cooperation abilities when faced with novel partners and levels in the Overcooked-AI environment.
Benchmarking Mental State Representations in Language Models
We are the first to study how prompt variations impact probing performance on theory of mind tasks.
Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition
This finding calls for a deeper understanding of the role of ToM in CPA and beyond, as well as new methods for modelling and evaluating mental states in computational collaborative agents.
VSA4VQA: Scaling a Vector Symbolic Architecture to Visual Question Answering on Natural Images
Our method is based on the Semantic Pointer Architecture (SPA) to encode objects in a hyperdimensional vector space.
DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images
We present DiffGaze, a novel method for generating realistic and diverse continuous human gaze sequences on 360{\deg} images based on a conditional score-based denoising diffusion model.
UP-FacE: User-predictable Fine-grained Face Shape Editing
We present User-predictable Face Editing (UP-FacE) -- a novel method for predictable face shape editing.
Learning User Embeddings from Human Gaze for Personalised Saliency Prediction
At the core of our method is a Siamese convolutional neural encoder that learns the user embeddings by contrasting the image and personal saliency map pairs of different users.
GazeMotion: Gaze-guided Human Motion Forecasting
We present GazeMotion, a novel method for human motion forecasting that combines information on past human poses with human eye gaze.
ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videos
We show that by adding action embeddings into the noise mask the diffusion model can better learn action temporal dependencies and increase the performances on procedure planning.
PrivatEyes: Appearance-based Gaze Estimation Using Federated Secure Multi-Party Computation
Latest gaze estimation methods require large-scale training data but their collection and exchange pose significant privacy risks.
OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog
OLViT addresses these challenges by maintaining a global dialog state based on the output of an Object State Tracker (OST) and a Language State Tracker (LST): while the OST attends to the most important objects within the video, the LST keeps track of the most important linguistic co-references to previous dialog turns.
Mindful Explanations: Prevalence and Impact of Mind Attribution in XAI Research
When users perceive AI systems as mindful, independent agents, they hold them responsible instead of the AI experts who created and designed these systems.
Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses
We compare our method with state-of-the-art methods that predict eye gaze only from head movements and show that Pose2Gaze outperforms these baselines with an average improvement of 24. 0% on MoGaze, 10. 1% on ADT, 21. 3% on GIMO, and 28. 6% on EgoBody in mean angular error, respectively.
GazeMoDiff: Gaze-guided Diffusion Model for Stochastic Human Motion Prediction
Human motion prediction is important for many virtual and augmented reality (VR/AR) applications such as collision avoidance and realistic avatar generation.
Neural Reasoning About Agents' Goals, Preferences, and Actions
We propose the Intuitive Reasoning Network (IRENE) - a novel neural model for intuitive psychological reasoning about agents' goals, preferences, and actions that can generalise previous experiences to new situations.
$\mathbb{VD}$-$\mathbb{GR}$: Boosting $\mathbb{V}$isual $\mathbb{D}$ialog with Cascaded Spatial-Temporal Multi-Modal $\mathbb{GR}$aphs
We propose $\mathbb{VD}$-$\mathbb{GR}$ - a novel visual dialog model that combines pre-trained language models (LMs) with graph neural networks (GNNs).
MultiMediate'23: Engagement Estimation and Bodily Behaviour Recognition in Social Interactions
This paper describes the MultiMediate'23 challenge and presents novel sets of annotations for both tasks.
Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning
We show that intentions of human players, i. e. the precursor of goal-oriented decisions, can be robustly predicted from eye gaze even for the long-horizon sparse rewards task of Montezuma's Revenge - one of the most challenging RL tasks in the Atari2600 game suite.
Deep Reinforcement Learning
Hierarchical Reinforcement Learning
+3
Neuro-Symbolic Visual Dialog
We propose Neuro-Symbolic Visual Dialog (NSVD) -the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning.
Gaze-enhanced Crossmodal Embeddings for Emotion Recognition
Emotional expressions are inherently multimodal -- integrating facial behavior, speech, and gaze -- but their automatic recognition is often limited to a single modality, e. g. speech during a phone call.
Scanpath Prediction on Information Visualisations
We propose Unified Model of Saliency and Scanpaths (UMSS) -- a model that learns to predict visual saliency and scanpaths (i. e. sequences of eye fixations) on information visualisations.
Multimodal Integration of Human-Like Attention in Visual Question Answering
We present the Multimodal Human-like Attention Network (MULAN) - the first method for multimodal integration of human-like attention on image and text during training of VQA models.
VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
We present VQA-MHUG - a novel 49-participant dataset of multimodal human gaze on both images and questions during visual question answering (VQA) collected using a high-speed eye tracker.
Neural Photofit: Gaze-based Mental Image Reconstruction
The encoder extracts image features and predicts a neural activation map for each face looked at by a human observer.
Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention
A lack of corpora has so far limited advances in integrating human gaze data as a supervisory signal in neural attention mechanisms for natural language processing(NLP).
Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension
We compare state of the art networks based on long short-term memory (LSTM), convolutional neural models (CNN) and XLNet Transformer architectures.
Accurate and Robust Eye Contact Detection During Everyday Mobile Device Interactions
Moreover, we discuss how our method enables the calculation of additional attention metrics that, for the first time, enable researchers from different domains to study and quantify attention allocation during mobile interactions in the wild.
How far are we from quantifying visual attention in mobile HCI?
With an ever-increasing number of mobile devices competing for our attention, quantifying when, how often, or for how long users visually attend to their devices has emerged as a core challenge in mobile human-computer interaction.
Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings
Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras.
A Multimodal Corpus of Expert Gaze and Behavior during Phonetic Segmentation Tasks
Phonetic segmentation is the process of splitting speech into distinct phonetic units.
MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation
Second, we present an extensive evaluation of state-of-the-art gaze estimation methods on three current datasets, including MPIIGaze.
Visual Decoding of Targets During Visual Search From Human Eye Fixations
Such visual decoding is challenging for two reasons: 1) the search target only resides in the user's mind as a subjective visual pattern, and can most often not even be described verbally by the person, and 2) it is, as of yet, unclear if gaze fixations contain sufficient information for this task at all.
GazeDirector: Fully Articulated Eye Gaze Redirection in Video
We present GazeDirector, a new approach for eye gaze redirection that uses model-fitting.
Gaze Embeddings for Zero-Shot Image Classification
Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts.
Predicting the Category and Attributes of Visual Search Targets Using Deep Gaze Pooling
Predicting the target of visual search from eye fixation (gaze) data is a challenging problem with many applications in human-computer interaction.
It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation
Eye gaze is an important non-verbal cue for human affect analysis.
End-to-End Eye Movement Detection Using Convolutional Neural Networks
Common computational methods for automated eye movement detection - i. e. the task of detecting different types of eye movement in a continuous stream of gaze data - are limited in that they either involve thresholding on hand-crafted signal features, require individual detectors each only detecting a single movement, or require pre-segmented data.
Seeing with Humans: Gaze-Assisted Neural Image Captioning
Gaze reflects how humans process visual scenes and is therefore increasingly used in computer vision systems.
Contextual Media Retrieval Using Natural Language Queries
We show that our retrieval system can cope with this variability using personalisation through an online learning-based retrieval formulation.
3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers
3D gaze information is important for scene-centric attention analysis but accurate estimation and analysis of 3D gaze in real-world environments remains challenging.
Labeled pupils in the wild: A dataset for studying pupil detection in unconstrained environments
We further study the influence of image resolution, vision aids, as well as recording location (indoor, outdoor) on pupil detection performance.
Rendering of Eyes for Eye-Shape Registration and Gaze Estimation
Images of the eye are key in several computer vision problems, such as shape registration and gaze estimation.
GazeDPM: Early Integration of Gaze Information in Deformable Part Models
An increasing number of works explore collaborative human-computer systems in which human gaze is used to enhance computer vision systems.
Appearance-Based Gaze Estimation in the Wild
Appearance-based gaze estimation is believed to work well in real-world settings, but existing datasets have been collected under controlled laboratory conditions and methods have been not evaluated across multiple datasets.
Prediction of Search Targets From Fixations in Open-World Settings
Previous work on predicting the target of visual search from human fixations only considered closed-world settings in which training labels are available and predictions are performed for a known set of potential targets.
Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction
Commercial head-mounted eye trackers provide useful features to customers in industry and research but are expensive and rely on closed source hardware and software.
Ubic: Bridging the gap between digital cryptography and the physical world
We describe Ubic, a framework that allows users to bridge the gap between digital cryptography and the physical world.
