Search Results for author:
Found 109 papers, 34 papers with code
Partially Adaptive Multichannel Joint Reduction of Ego-noise and Environmental Noise
In this paper, we propose a multichannel partially adaptive scheme to jointly model ego-noise and environmental noise utilizing the VAE-NMF framework, where we take advantage of spatially and spectrally structured characteristics of ego-noise by pre-training the ego-noise model, while retaining the ability to adapt to unknown environmental noise.
Chat with the Environment: Interactive Multimodal Perception using Large Language Models
Programming robot behaviour in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning.
Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learning
Model-based reinforcement learning (MBRL) with real-time planning has shown great potential in locomotion and manipulation control tasks.
Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition
Furthermore, our proposed combined loss rescaling and weight consolidation methods can support continual learning of an ASR system.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Internally Rewarded Reinforcement Learning
We study a class of reinforcement learning problems where the reward signals for policy learning are generated by a discriminator that is dependent on and jointly optimized with the policy.
Learning Bidirectional Action-Language Translation with Limited Supervision and Incongruent Input
Only occasionally, a learning infant would receive a matching verbal description of an action it is committing, which is similar to supervised learning.
Disentangling Prosody Representations with Unsupervised Speech Reconstruction
Human speech can be characterized by different components, including semantic content, speaker identity and prosodic information.
Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanations
Natural language explanations promise to offer intuitively understandable explanations of a neural network's decision process in complex vision-language tasks, as pursued in recent VL-NLE models.
Ranked #1 on
Explanation Generation
on VQA-X
Neuro-Symbolic Spatio-Temporal Reasoning
In this chapter, we view this integration problem from the perspective of Neuro-Symbolic AI.
Introspection-based Explainable Reinforcement Learning in Episodic and Non-episodic Scenarios
Results obtained show the viability of introspection for episodic robotics tasks and, additionally, that the introspection-based approach can be used to generate explanations for the actions taken in a non-episodic robotics environment as well.
Whose Emotion Matters? Speaking Activity Localisation without Prior Knowledge
Finally, we devise a model for emotion recognition in conversations trained on the realigned MELD-FAIR videos, which outperforms state-of-the-art models for ERC based on vision alone.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Visually Grounded Commonsense Knowledge Acquisition
In this work, we present CLEVER, which formulates CKE as a distantly supervised multi-instance learning problem, where models learn to summarize commonsense relations from a bag of images about an entity pair without any human annotation on image instances.
Data Augmentation with Unsupervised Speaking Style Transfer for Speech Emotion Recognition
Once training is completed, EmoAug enriches expressions of emotional speech in different prosodic attributes, such as stress, rhythm and intensity, by feeding different styles into the paralinguistic encoder.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Learning to Autonomously Reach Objects with NICO and Grow-When-Required Networks
The act of reaching for an object is a fundamental yet complex skill for a robotic agent, requiring a high degree of visuomotor control and coordination.
Intelligent problem-solving as integrated hierarchical reinforcement learning
According to cognitive psychology and related disciplines, the development of complex problem-solving behaviour in biological agents depends on hierarchical cognitive mechanisms.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Impact Makes a Sound and Sound Makes an Impact: Sound Guides Representations and Explorations
Sound is one of the most informative and abundant modalities in the real world while being robust to sense without contacts by small and cheap sensors that can be placed on mobile devices.
Learning Flexible Translation between Robot Actions and Language Descriptions
In this work, we propose the paired gated autoencoders (PGAE) for flexible translation between robot actions and language descriptions in a tabletop object manipulation scenario.
Knowing Earlier what Right Means to You: A Comprehensive VQA Dataset for Grounding Relative Directions via Multi-Task Learning
Spatial reasoning poses a particular challenge for intelligent agents and is at the same time a prerequisite for their successful interaction and communication in the physical world.
GASP: Gated Attention For Saliency Prediction
We show that gaze direction and affective representations contribute a prediction to ground-truth correspondence improvement of at least 5% compared to dynamic saliency models without social cues.
Snapture -- A Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition
Our architecture enables learning both static and dynamic gestures: by capturing a so-called "snapshot" of the gesture performance at its peak, we integrate the hand pose along with the dynamic movement.
Conversational Analysis of Daily Dialog Data using Polite Emotional Dialogue Acts
One of the fundamental cues is politeness, which linguistically possesses properties such as social manners useful in conversational analysis.
What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning
Grounding relative directions is more difficult than grounding absolute directions because it not only requires a model to detect objects in the image and to identify spatial relation based on this information, but it also needs to recognize the orientation of objects and integrate this information into the reasoning process.
Explain yourself! Effects of Explanations in Human-Robot Interaction
However, the results show that the robot that explains its moves is perceived as more lively and human-like.
Grounding Hindsight Instructions in Multi-Goal Reinforcement Learning for Robotics
We show that hindsight instructions improve the learning performance, as expected.
Integrating Statistical Uncertainty into Neural Network-Based Speech Enhancement
Speech enhancement in the time-frequency domain is often performed by estimating a multiplicative mask to extract clean speech.
A Multimodal German Dataset for Automatic Lip Reading Systems and Transfer Learning
Large datasets as required for deep learning of lip reading do not exist in many languages.
Language Model-Based Paired Variational Autoencoders for Robotic Language Learning
Human infants learn language while interacting with their environment in which their caregivers may describe the objects and actions they perform.
LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading
The aim of this work is to investigate the impact of crossmodal self-supervised pre-training for speech reconstruction (video-to-audio) by leveraging the natural co-occurrence of audio and visual streams in videos.
Lifelong Learning from Event-based Data
Lifelong learning is a long-standing aim for artificial agents that act in dynamic environments, in which an agent needs to accumulate knowledge incrementally without forgetting previously learned representations.
A trained humanoid robot can perform human-like crossmodal social attention and conflict resolution
Our saliency prediction model was trained to detect social cues, predict audio-visual saliency, and attend selectively for the robot study.
Can AI detect pain and express pain empathy? A review from emotion recognition and a human-centered AI perspective
Furthermore, we discuss challenges for responsible evaluation of cognitive methods and computational techniques and show approaches to future work to contribute to affective assistants capable of empathy.
FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection
The strong relation between face and voice can aid active speaker detection systems when faces are visible, even in difficult settings, when the face of a speaker is not clear or when there are several people in the same scene.
Generalization in Multimodal Language Learning from Simulation
Neural networks can be powerful function approximators, which are able to model high-dimensional feature distributions from a subset of examples drawn from the target distribution.
Behavior Self-Organization Supports Task Inference for Continual Robot Learning
Task inference is made by finding the nearest behavior embedding to a demonstrated behavior, which is used together with the environment state as input to a multi-task policy trained with reinforcement learning to optimize performance over tasks.
Model Mediated Teleoperation with a Hand-Arm Exoskeleton in Long Time Delays Using Reinforcement Learning
Telerobotic systems must adapt to new environmental conditions and deal with high uncertainty caused by long-time delays.
DRILL: Dynamic Representations for Imbalanced Lifelong Learning
Continual or lifelong learning has been a long-standing challenge in machine learning to date, especially in natural language processing (NLP).
Survey on reinforcement learning for language processing
In recent years some researchers have explored the use of reinforcement learning (RL) algorithms as key components in the solution of various natural language processing tasks.
Visual Distant Supervision for Scene Graph Generation
In this work, we propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data.
A Sub-Layered Hierarchical Pyramidal Neural Architecture for Facial Expression Recognition
In domains where computational resources and labeled data are limited, such as in robotics, deep networks with millions of weights might not be the optimal solution.
An Ensemble with Shared Representations Based on Convolutional Networks for Continually Learning Facial Expressions
Social robots able to continually learn facial expressions could progressively improve their emotion recognition capability towards people interacting with them.
Disambiguating Affective Stimulus Associations for Robot Perception and Dialogue
In this paper, we utilize the NICO robot's appearance and capabilities to give the NICO the ability to model a coherent affective association between a perceived auditory stimulus and a temporally asynchronous emotion expression.
Continual Learning from Synthetic Data for a Humanoid Exercise Robot
In a practical scenario, a physical exercise is performed by an expert like a physiotherapist and then used as a reference for a humanoid robot like Pepper to give feedback on a patient's execution of the same exercise.
Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder
Recently, a generative variational autoencoder (VAE) has been proposed for speech enhancement to model speech statistics.
Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision
Using a model of the environment, reinforcement learning agents can plan their future moves and achieve superhuman performance in board games like Chess, Shogi, and Go, while remaining relatively sample-efficient.
CharacterGAN: Few-Shot Keypoint Character Animation and Reposing
Our model generates novel poses based on keypoint locations, which can be modified in real time while providing interactive feedback, allowing for intuitive reposing and animation.
Hierarchical principles of embodied reinforcement learning: A review
We then relate these insights with contemporary hierarchical reinforcement learning methods, and identify the key machine intelligence approaches that realise these mechanisms.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Sensorimotor representation learning for an "active self" in robots: A model survey
Safe human-robot interactions require robots to be able to learn how to behave appropriately in \sout{humans' world} \rev{spaces populated by people} and thus to cope with the challenges posed by our dynamic and unstructured environment, rather than being provided a rigid set of rules for operations.
Robotic self-representation improves manipulation skills and transfer learning
Cognitive science suggests that the self-representation is critical for learning and problem-solving.
Reinforcement Learning with Time-dependent Goals for Robotic Musicians
Reinforcement learning is a promising method to accomplish robotic control tasks.
Affect-Driven Modelling of Robot Personality for Collaborative Human-Robot Interactions
Collaborative interactions require social robots to adapt to the dynamics of human affective behaviour.
Integrating Intrinsic and Extrinsic Explainability: The Relevance of Understanding Neural Networks for Human-Robot Interaction
However, not only can humans benefit from a robot's explanation but the robot itself can also benefit from explanations given to him.
Enhancing a Neurocognitive Shared Visuomotor Model for Object Identification, Localization, and Grasping With Learning From Auxiliary Tasks
In this follow-up study, we expand the task and the model to reaching for objects in a three-dimensional space with a novel dataset based on augmented reality and a simulation environment.
Crossmodal Language Grounding in an Embodied Neurocognitive Model
From a neuroscientific perspective, natural language is embodied, grounded in most, if not all, sensory and sensorimotor modalities, and acquired by means of crossmodal integration.
Facial Expression Editing with Continuous Emotion Labels
Recently deep generative models have achieved impressive results in the field of automated facial expression editing.
Towards a self-organizing pre-symbolic neural model representing sensorimotor primitives
During the learning process of observing sensorimotor primitives, i. e. observing a set of trajectories of arm movements and its oriented object features, the pre-symbolic representation is self-organized in the parametric units.
Multimodal Target Speech Separation with Voice and Face References
Target speech separation refers to isolating target speech from a multi-speaker mixture signal by conditioning on auxiliary information about the target speaker.
Audio and Speech Processing Sound
Curious Hierarchical Actor-Critic Reinforcement Learning
Hierarchical abstraction and curiosity-driven exploration are two common paradigms in current reinforcement learning approaches to break down difficult problems into a sequence of simpler ones and to overcome reward sparsity.
Explainable Goal-Driven Agents and Robots -- A Comprehensive Review
The recent stance on the explainability of AI systems has witnessed several approaches on eXplainable Artificial Intelligence (XAI); however, most of the studies have focused on data-driven XAI systems applied in computational sciences.
Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imagination
In this paper, we present a novel dual-system motor learning approach where a meta-controller arbitrates online between model-based and model-free decisions based on an estimate of the local reliability of the learned model.
Improved Techniques for Training Single-Image GANs
Recently there has been an interest in the potential of learning generative models from a single image, as opposed to from a large dataset.
Efficient Facial Feature Learning with Wide Ensemble-based Convolutional Neural Networks
Experiments on large-scale datasets suggest that ESRs reduce the remaining residual generalization error on the AffectNet and FER+ datasets, reach human-level performance, and outperform state-of-the-art methods on facial expression recognition in the wild using emotion and affect concepts.
Ranked #9 on
Facial Expression Recognition (FER)
on FER+
(using extra training data)
Solving Visual Object Ambiguities when Pointing: An Unsupervised Learning Approach
The implementation of an intuitive gesture scenario is still challenging because both the pointing intention and the corresponding object have to be correctly recognized in real-time.
EDA: Enriching Emotional Dialogue Acts using an Ensemble of Neural Annotators
These neural models annotate the emotion corpora with dialogue act labels, and an ensemble annotator extracts the final dialogue act label.
Semantic Object Accuracy for Generative Text-to-Image Synthesis
To address these challenges we introduce a new model that explicitly models individual objects within an image and a new evaluation metric called Semantic Object Accuracy (SOA) that specifically evaluates images given an image caption.
Ranked #48 on
Text-to-Image Generation
on COCO
Efficient Intrinsically Motivated Robotic Grasping with Learning-Adaptive Imagination in Latent Space
The learned models are used to generate imagined experiences, augmenting the training set of real experiences.
What can computational models learn from human selective attention? A review from an audiovisual crossmodal perspective
Selective attention plays an essential role in information acquisition and utilization from the environment.
Hierarchical Control for Bipedal Locomotion using Central Pattern Generators and Neural Networks
Artificial central pattern generators (CPGs) can produce synchronized joint movements and have been used in the past for bipedal locomotion.
The OMG-Empathy Dataset: Evaluating the Impact of Affective Behavior in Storytelling
In this paper, we propose a novel dataset composed of dyadic interactions designed, collected and annotated with a focus on measuring the affective impact that eight different stories have on the listener.
Evaluating Defensive Distillation For Defending Text Processing Neural Networks Against Adversarial Examples
Adversarial examples are artificially modified input samples which lead to misclassifications, while not being detectable by humans.
Towards Learning How to Properly Play UNO with the iCub Robot
While interacting with another person, our reactions and behavior are much affected by the emotional changes within the temporal context of the interaction.
MoonGrad at SemEval-2019 Task 3: Ensemble BiRNNs for Contextual Emotion Detection in Dialogues
When reading {``}I don{'}t want to talk to you any more{''}, we might interpret this as either an angry or a sad emotion in the absence of context.
From semantics to execution: Integrating action planning with reinforcement learning for robotic causal problem-solving
In this article, we build on these novel methods to facilitate the integration of action planning with reinforcement learning by exploiting the reward-sparsity as a bridge between the high-level and low-level state- and control spaces.
Curious Meta-Controller: Adaptive Alternation between Model-Based and Model-Free Control in Deep Reinforcement Learning
Recent success in deep reinforcement learning for continuous control has been dominated by model-free approaches which, unlike model-based approaches, do not suffer from representational limitations in making assumptions about the world dynamics and model errors inevitable in complex domains.
A Personalized Affective Memory Neural Model for Improving Emotion Recognition
Recent models of emotion recognition strongly rely on supervised deep learning solutions for the distinction of general emotion expressions.
Improving interactive reinforcement learning: What makes a good teacher?
Interactive reinforcement learning has become an important apprenticeship approach to speed up convergence in classic reinforcement learning problems.
KT-Speech-Crawler: Automatic Dataset Construction for Speech Recognition from YouTube Videos
In this paper, we describe KT-Speech-Crawler: an approach for automatic dataset construction for speech recognition by crawling YouTube videos.
Incorporating End-to-End Speech Recognition Models for Sentiment Analysis
We argue that using ground-truth transcriptions during training and evaluation phases leads to a significant discrepancy in performance compared to real-world conditions, as the spoken text has to be recognized on the fly and can contain speech recognition mistakes.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Generating Multiple Objects at Spatially Distinct Locations
Our experiments show that through the use of the object pathway we can control object locations within images and can model complex scenes with multiple objects at various locations.
Ranked #60 on
Text-to-Image Generation
on COCO
On the role of neurogenesis in overcoming catastrophic forgetting
Lifelong learning capabilities are crucial for artificial autonomous agents operating on real-world data, which is typically non-stationary and temporally correlated.
Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learning
In this paper, we present a new intrinsically motivated actor-critic algorithm for learning continuous motor skills directly from raw visual input.
Assessing the Contribution of Semantic Congruency to Multisensory Integration and Conflict Resolution
The efficient integration of multisensory observations is a key property of the brain that yields the robust interaction with the environment.
Unsupervised Expectation Learning for Multisensory Binding
The model adapts concepts of expectation learning to enhance the unisensory representation based on the learned bindings.
Towards Dialogue-based Navigation with Multivariate Adaptation driven by Intention and Politeness for Social Robots
Service robots need to show appropriate social behaviour in order to be deployed in social environments such as healthcare, education, retail, etc.
Curriculum goal masking for continuous deep reinforcement learning
Deep reinforcement learning has recently gained a focus on problems where policy or value functions are independent of goals.
A Deep Neural Model Of Emotion Appraisal
We evaluate the performance of the proposed model with different challenging corpora and compare it with state-of-the-art models for external emotion appraisal.
Multi-modal Feedback for Affordance-driven Interactive Reinforcement Learning
Additionally, we modulate the influence of sensory-driven feedback in the IRL task using goal-oriented knowledge in terms of contextual affordances.
Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks
This is independent of the underlying optimization procedure, making the approach promising for many existing hyperparameter optimization algorithms.
Towards Modeling the Interaction of Spatial-Associative Neural Network Representations for Multisensory Perception
In the associative layer, congruent audiovisual representations are obtained via the experience-driven development of feature-based associations.
Deep Neural Object Analysis by Interactive Auditory Exploration with a Humanoid Robot
We present a novel approach for interactive auditory object analysis with a humanoid robot.
Discourse-Wizard: Discovering Deep Discourse Structure in your Conversation with RNNs
Spoken language understanding is one of the key factors in a dialogue system, and a context in a conversation plays an important role to understand the current utterance.
Lifelong Learning of Spatiotemporal Representations with Dual-Memory Recurrent Self-Organization
Both growing networks can expand in response to novel sensory experience: the episodic memory learns fine-grained spatiotemporal representations of object instances in an unsupervised fashion while the semantic memory uses task-relevant signals to regulate structural plasticity levels and develop more compact representations from episodic experience.
Conversational Analysis using Utterance-level Attention-based Bidirectional Recurrent Neural Networks
Recent approaches for dialogue act recognition have shown that context from preceding utterances is important to classify the subsequent one.
Ranked #9 on
Dialogue Act Classification
on Switchboard corpus
A Context-based Approach for Dialogue Act Recognition using Simple Recurrent Neural Networks
Dialogue act recognition is an important part of natural language understanding.
Ranked #10 on
Dialogue Act Classification
on Switchboard corpus
On the Robustness of Speech Emotion Recognition for Human-Robot Interaction with Deep Neural Networks
Speech emotion recognition (SER) is an important aspect of effective human-robot collaboration and received a lot of attention from the research community.
EmoRL: Continuous Acoustic Emotion Classification using Deep Reinforcement Learning
Acoustically expressed emotions can make communication with a robot more efficient.
Reusing Neural Speech Representations for Auditory Emotion Recognition
Acoustic emotion recognition aims to categorize the affective state of the speaker and is still a difficult task for machine learning models.
Automatically augmenting an emotion dataset improves classification using audio
In this work, we tackle a problem of speech emotion classification.
GradAscent at EmoInt-2017: Character- and Word-Level Recurrent Neural Network Models for Tweet Emotion Intensity Detection
Given the text of a tweet and its emotion category (anger, joy, fear, and sadness), the participants were asked to build a system that assigns emotion intensity values.
Image Generation and Translation with Disentangled Representations
We train an encoder to encode images into these representations and use a small amount of labeled data to specify what kind of information should be encoded in the disentangled part.
The OMG-Emotion Behavior Dataset
This paper is the basis paper for the accepted IJCNN challenge One-Minute Gradual-Emotion Recognition (OMG-Emotion) by which we hope to foster long-emotion classification using neural models for the benefit of the IJCNN community.
Human-Computer Interaction
Inferencing Based on Unsupervised Learning of Disentangled Representations
Combining Generative Adversarial Networks (GANs) with encoders that learn to encode data points has shown promising results in learning data representations in an unsupervised way.
Ranked #4 on
Unsupervised Image Classification
on MNIST
Representation Learning
Unsupervised Image Classification
+1
Continual Lifelong Learning with Neural Networks: A Review
Humans and animals have the ability to continually acquire, fine-tune, and transfer knowledge and skills throughout their lifespan.
Closing the loop on multisensory interactions: A neural architecture for multisensory causal inference and recalibration
Furthermore, we propose that these unisensory and multisensory neurons play dual roles in i) encoding spatial location as separate or integrated estimates and ii) accumulating evidence for the independence or relatedness of multisensory stimuli.
Expectation Learning for Adaptive Crossmodal Stimuli Association
The human brain is able to learn, generalize, and predict crossmodal stimuli.
An Incremental Self-Organizing Architecture for Sensorimotor Learning and Prediction
During visuomotor tasks, robots must compensate for temporal delays inherent in their sensorimotor processing systems.
A self-organizing neural network architecture for learning human-object interactions
We show that our unsupervised model shows competitive classification results on the benchmark dataset with respect to strictly supervised approaches.
GradAscent at EmoInt-2017: Character and Word Level Recurrent Neural Network Models for Tweet Emotion Intensity Detection
Given the text of a tweet and its emotion category (anger, joy, fear, and sadness), the participants were asked to build a system that assigns emotion intensity values.
Semi-Supervised Phoneme Recognition with Recurrent Ladder Networks
Ladder networks are a notable new concept in the field of semi-supervised learning by showing state-of-the-art results in image recognition tasks while being compatible with many existing neural architectures.
Interactive Natural Language Acquisition in a Multi-modal Recurrent Neural Architecture
For the complex human brain that enables us to communicate in natural language, we gathered good understandings of principles underlying language acquisition and processing, knowledge about socio-cultural conditions, and insights about activity patterns in the brain.
