Search Results for author:
Found 33 papers, 7 papers with code
Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for VideoQA
We propose the Video Language Co-Attention Network (VLCN) – a novel memory-enhanced model for Video Question Answering (VideoQA).
$\mathbb{VD}$-$\mathbb{GR}$: Boosting $\mathbb{V}$isual $\mathbb{D}$ialog with Cascaded Spatial-Temporal Multi-Modal $\mathbb{GR}$aphs
We propose $\mathbb{VD}$-$\mathbb{GR}$ - a novel visual dialog model that combines pre-trained language models (LMs) with graph neural networks (GNNs).
MultiMediate'23: Engagement Estimation and Bodily Behaviour Recognition in Social Interactions
This paper describes the MultiMediate'23 challenge and presents novel sets of annotations for both tasks.
Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning
We show that intentions of human players, i. e. the precursor of goal-oriented decisions, can be robustly predicted from eye gaze even for the long-horizon sparse rewards task of Montezuma's Revenge - one of the most challenging RL tasks in the Atari2600 game suite.
Neuro-Symbolic Visual Dialog
We propose Neuro-Symbolic Visual Dialog (NSVD) -the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning.
Gaze-enhanced Crossmodal Embeddings for Emotion Recognition
Emotional expressions are inherently multimodal -- integrating facial behavior, speech, and gaze -- but their automatic recognition is often limited to a single modality, e. g. speech during a phone call.
Scanpath Prediction on Information Visualisations
We propose Unified Model of Saliency and Scanpaths (UMSS) -- a model that learns to predict visual saliency and scanpaths (i. e. sequences of eye fixations) on information visualisations.
Multimodal Integration of Human-Like Attention in Visual Question Answering
We present the Multimodal Human-like Attention Network (MULAN) - the first method for multimodal integration of human-like attention on image and text during training of VQA models.
VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
We present VQA-MHUG - a novel 49-participant dataset of multimodal human gaze on both images and questions during visual question answering (VQA) collected using a high-speed eye tracker.
Neural Photofit: Gaze-based Mental Image Reconstruction
The encoder extracts image features and predicts a neural activation map for each face looked at by a human observer.
Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention
A lack of corpora has so far limited advances in integrating human gaze data as a supervisory signal in neural attention mechanisms for natural language processing(NLP).
Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension
We compare state of the art networks based on long short-term memory (LSTM), convolutional neural models (CNN) and XLNet Transformer architectures.
How far are we from quantifying visual attention in mobile HCI?
With an ever-increasing number of mobile devices competing for our attention, quantifying when, how often, or for how long users visually attend to their devices has emerged as a core challenge in mobile human-computer interaction.
Accurate and Robust Eye Contact Detection During Everyday Mobile Device Interactions
Moreover, we discuss how our method enables the calculation of additional attention metrics that, for the first time, enable researchers from different domains to study and quantify attention allocation during mobile interactions in the wild.
Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings
Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras.
A Multimodal Corpus of Expert Gaze and Behavior during Phonetic Segmentation Tasks
Phonetic segmentation is the process of splitting speech into distinct phonetic units.
MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation
Second, we present an extensive evaluation of state-of-the-art gaze estimation methods on three current datasets, including MPIIGaze.
Visual Decoding of Targets During Visual Search From Human Eye Fixations
Such visual decoding is challenging for two reasons: 1) the search target only resides in the user's mind as a subjective visual pattern, and can most often not even be described verbally by the person, and 2) it is, as of yet, unclear if gaze fixations contain sufficient information for this task at all.
GazeDirector: Fully Articulated Eye Gaze Redirection in Video
We present GazeDirector, a new approach for eye gaze redirection that uses model-fitting.
Gaze Embeddings for Zero-Shot Image Classification
Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts.
It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation
Eye gaze is an important non-verbal cue for human affect analysis.
Predicting the Category and Attributes of Visual Search Targets Using Deep Gaze Pooling
Predicting the target of visual search from eye fixation (gaze) data is a challenging problem with many applications in human-computer interaction.
End-to-End Eye Movement Detection Using Convolutional Neural Networks
Common computational methods for automated eye movement detection - i. e. the task of detecting different types of eye movement in a continuous stream of gaze data - are limited in that they either involve thresholding on hand-crafted signal features, require individual detectors each only detecting a single movement, or require pre-segmented data.
Seeing with Humans: Gaze-Assisted Neural Image Captioning
Gaze reflects how humans process visual scenes and is therefore increasingly used in computer vision systems.
Contextual Media Retrieval Using Natural Language Queries
We show that our retrieval system can cope with this variability using personalisation through an online learning-based retrieval formulation.
3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers
3D gaze information is important for scene-centric attention analysis but accurate estimation and analysis of 3D gaze in real-world environments remains challenging.
Labeled pupils in the wild: A dataset for studying pupil detection in unconstrained environments
We further study the influence of image resolution, vision aids, as well as recording location (indoor, outdoor) on pupil detection performance.
GazeDPM: Early Integration of Gaze Information in Deformable Part Models
An increasing number of works explore collaborative human-computer systems in which human gaze is used to enhance computer vision systems.
Rendering of Eyes for Eye-Shape Registration and Gaze Estimation
Images of the eye are key in several computer vision problems, such as shape registration and gaze estimation.
Appearance-Based Gaze Estimation in the Wild
Appearance-based gaze estimation is believed to work well in real-world settings, but existing datasets have been collected under controlled laboratory conditions and methods have been not evaluated across multiple datasets.
Prediction of Search Targets From Fixations in Open-World Settings
Previous work on predicting the target of visual search from human fixations only considered closed-world settings in which training labels are available and predictions are performed for a known set of potential targets.
Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction
Commercial head-mounted eye trackers provide useful features to customers in industry and research but are expensive and rely on closed source hardware and software.
Ubic: Bridging the gap between digital cryptography and the physical world
We describe Ubic, a framework that allows users to bridge the gap between digital cryptography and the physical world.
