Search Results for author: Andreas Bulling

Found 33 papers, 7 papers with code

$\mathbb{VD}$-$\mathbb{GR}$: Boosting $\mathbb{V}$isual $\mathbb{D}$ialog with Cascaded Spatial-Temporal Multi-Modal $\mathbb{GR}$aphs

no code implementations25 Oct 2023 Adnen Abdessaied, Lei Shi, Andreas Bulling

We propose $\mathbb{VD}$-$\mathbb{GR}$ - a novel visual dialog model that combines pre-trained language models (LMs) with graph neural networks (GNNs).

Visual Dialog

Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning

no code implementations20 Jun 2023 Anna Penzkofer, Simon Schaefer, Florian Strohm, Mihai Bâce, Stefan Leutenegger, Andreas Bulling

We show that intentions of human players, i. e. the precursor of goal-oriented decisions, can be robustly predicted from eye gaze even for the long-horizon sparse rewards task of Montezuma's Revenge - one of the most challenging RL tasks in the Atari2600 game suite.

Hierarchical Reinforcement Learning Montezuma's Revenge +2

Neuro-Symbolic Visual Dialog

1 code implementation COLING 2022 Adnen Abdessaied, Mihai Bâce, Andreas Bulling

We propose Neuro-Symbolic Visual Dialog (NSVD) -the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning.

Question Answering

Gaze-enhanced Crossmodal Embeddings for Emotion Recognition

no code implementations30 Apr 2022 Ahmed Abdou, Ekta Sood, Philipp Müller, Andreas Bulling

Emotional expressions are inherently multimodal -- integrating facial behavior, speech, and gaze -- but their automatic recognition is often limited to a single modality, e. g. speech during a phone call.

Emotion Classification Emotion Recognition

Scanpath Prediction on Information Visualisations

no code implementations4 Dec 2021 Yao Wang, Mihai Bâce, Andreas Bulling

We propose Unified Model of Saliency and Scanpaths (UMSS) -- a model that learns to predict visual saliency and scanpaths (i. e. sequences of eye fixations) on information visualisations.

Saliency Prediction Scanpath prediction

Multimodal Integration of Human-Like Attention in Visual Question Answering

no code implementations27 Sep 2021 Ekta Sood, Fabian Kögel, Philipp Müller, Dominike Thomas, Mihai Bace, Andreas Bulling

We present the Multimodal Human-like Attention Network (MULAN) - the first method for multimodal integration of human-like attention on image and text during training of VQA models.

Question Answering Test +1

VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering

no code implementations CoNLL (EMNLP) 2021 Ekta Sood, Fabian Kögel, Florian Strohm, Prajit Dhar, Andreas Bulling

We present VQA-MHUG - a novel 49-participant dataset of multimodal human gaze on both images and questions during visual question answering (VQA) collected using a high-speed eye tracker.

Question Answering Visual Question Answering

Neural Photofit: Gaze-based Mental Image Reconstruction

no code implementations ICCV 2021 Florian Strohm, Ekta Sood, Sven Mayer, Philipp Müller, Mihai Bâce, Andreas Bulling

The encoder extracts image features and predicts a neural activation map for each face looked at by a human observer.

Image Reconstruction

Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention

no code implementations NeurIPS 2020 Ekta Sood, Simon Tannert, Philipp Mueller, Andreas Bulling

A lack of corpora has so far limited advances in integrating human gaze data as a supervisory signal in neural attention mechanisms for natural language processing(NLP).

Paraphrase Generation Sentence Compression

Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension

no code implementations CONLL 2020 Ekta Sood, Simon Tannert, Diego Frassinelli, Andreas Bulling, Ngoc Thang Vu

We compare state of the art networks based on long short-term memory (LSTM), convolutional neural models (CNN) and XLNet Transformer architectures.

Machine Reading Comprehension

How far are we from quantifying visual attention in mobile HCI?

no code implementations25 Jul 2019 Mihai Bâce, Sander Staal, Andreas Bulling

With an ever-increasing number of mobile devices competing for our attention, quantifying when, how often, or for how long users visually attend to their devices has emerged as a core challenge in mobile human-computer interaction.

Contact Detection Gaze Estimation +1

Accurate and Robust Eye Contact Detection During Everyday Mobile Device Interactions

no code implementations25 Jul 2019 Mihai Bâce, Sander Staal, Andreas Bulling

Moreover, we discuss how our method enables the calculation of additional attention metrics that, for the first time, enable researchers from different domains to study and quantify attention allocation during mobile interactions in the wild.

Contact Detection

Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings

2 code implementations12 May 2018 Seonwook Park, Xucong Zhang, Andreas Bulling, Otmar Hilliges

Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras.

Gaze Estimation

MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation

6 code implementations24 Nov 2017 Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling

Second, we present an extensive evaluation of state-of-the-art gaze estimation methods on three current datasets, including MPIIGaze.

Gaze Estimation

Visual Decoding of Targets During Visual Search From Human Eye Fixations

no code implementations19 Jun 2017 Hosnieh Sattar, Mario Fritz, Andreas Bulling

Such visual decoding is challenging for two reasons: 1) the search target only resides in the user's mind as a subjective visual pattern, and can most often not even be described verbally by the person, and 2) it is, as of yet, unclear if gaze fixations contain sufficient information for this task at all.

Gaze Embeddings for Zero-Shot Image Classification

no code implementations CVPR 2017 Nour Karessli, Zeynep Akata, Bernt Schiele, Andreas Bulling

Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts.

Classification Fine-Grained Image Classification +2

Predicting the Category and Attributes of Visual Search Targets Using Deep Gaze Pooling

no code implementations27 Nov 2016 Hosnieh Sattar, Andreas Bulling, Mario Fritz

Predicting the target of visual search from eye fixation (gaze) data is a challenging problem with many applications in human-computer interaction.

End-to-End Eye Movement Detection Using Convolutional Neural Networks

no code implementations8 Sep 2016 Sabrina Hoppe, Andreas Bulling

Common computational methods for automated eye movement detection - i. e. the task of detecting different types of eye movement in a continuous stream of gaze data - are limited in that they either involve thresholding on hand-crafted signal features, require individual detectors each only detecting a single movement, or require pre-segmented data.

Seeing with Humans: Gaze-Assisted Neural Image Captioning

no code implementations18 Aug 2016 Yusuke Sugano, Andreas Bulling

Gaze reflects how humans process visual scenes and is therefore increasingly used in computer vision systems.

Image Captioning Object Localization +2

Contextual Media Retrieval Using Natural Language Queries

no code implementations16 Feb 2016 Sreyasi Nag Chowdhury, Mateusz Malinowski, Andreas Bulling, Mario Fritz

We show that our retrieval system can cope with this variability using personalisation through an online learning-based retrieval formulation.

Natural Language Queries Retrieval

3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers

no code implementations11 Jan 2016 Mohsen Mansouryar, Julian Steil, Yusuke Sugano, Andreas Bulling

3D gaze information is important for scene-centric attention analysis but accurate estimation and analysis of 3D gaze in real-world environments remains challenging.

Gaze Estimation

Labeled pupils in the wild: A dataset for studying pupil detection in unconstrained environments

no code implementations18 Nov 2015 Marc Tonsen, Xucong Zhang, Yusuke Sugano, Andreas Bulling

We further study the influence of image resolution, vision aids, as well as recording location (indoor, outdoor) on pupil detection performance.

GazeDPM: Early Integration of Gaze Information in Deformable Part Models

no code implementations21 May 2015 Iaroslav Shcherbatyi, Andreas Bulling, Mario Fritz

An increasing number of works explore collaborative human-computer systems in which human gaze is used to enhance computer vision systems.

Gaze Estimation object-detection +1

Appearance-Based Gaze Estimation in the Wild

6 code implementations CVPR 2015 Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling

Appearance-based gaze estimation is believed to work well in real-world settings, but existing datasets have been collected under controlled laboratory conditions and methods have been not evaluated across multiple datasets.

Gaze Estimation

Prediction of Search Targets From Fixations in Open-World Settings

no code implementations CVPR 2015 Hosnieh Sattar, Sabine Müller, Mario Fritz, Andreas Bulling

Previous work on predicting the target of visual search from human fixations only considered closed-world settings in which training labels are available and predictions are performed for a known set of potential targets.

Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction

1 code implementation30 Apr 2014 Moritz Kassner, William Patera, Andreas Bulling

Commercial head-mounted eye trackers provide useful features to customers in industry and research but are expensive and rely on closed source hardware and software.

Gaze Estimation

Ubic: Bridging the gap between digital cryptography and the physical world

no code implementations6 Mar 2014 Mark Simkin, Dominique Schroeder, Andreas Bulling, Mario Fritz

We describe Ubic, a framework that allows users to bridge the gap between digital cryptography and the physical world.

Cannot find the paper you are looking for? You can Submit a new open access paper.