Search Results for author:
Found 43 papers, 18 papers with code
Chat with the Environment: Interactive Multimodal Perception using Large Language Models
Programming robot behaviour in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning.
Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learning
Model-based reinforcement learning (MBRL) with real-time planning has shown great potential in locomotion and manipulation control tasks.
Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition
Furthermore, our proposed combined loss rescaling and weight consolidation methods can support continual learning of an ASR system.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Internally Rewarded Reinforcement Learning
We study a class of reinforcement learning problems where the reward signals for policy learning are generated by a discriminator that is dependent on and jointly optimized with the policy.
Learning Bidirectional Action-Language Translation with Limited Supervision and Incongruent Input
Only occasionally, a learning infant would receive a matching verbal description of an action it is committing, which is similar to supervised learning.
Disentangling Prosody Representations with Unsupervised Speech Reconstruction
Human speech can be characterized by different components, including semantic content, speaker identity and prosodic information.
Whose Emotion Matters? Speaking Activity Localisation without Prior Knowledge
Finally, we devise a model for emotion recognition in conversations trained on the realigned MELD-FAIR videos, which outperforms state-of-the-art models for ERC based on vision alone.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Visually Grounded Commonsense Knowledge Acquisition
In this work, we present CLEVER, which formulates CKE as a distantly supervised multi-instance learning problem, where models learn to summarize commonsense relations from a bag of images about an entity pair without any human annotation on image instances.
Data Augmentation with Unsupervised Speaking Style Transfer for Speech Emotion Recognition
Once training is completed, EmoAug enriches expressions of emotional speech in different prosodic attributes, such as stress, rhythm and intensity, by feeding different styles into the paralinguistic encoder.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Impact Makes a Sound and Sound Makes an Impact: Sound Guides Representations and Explorations
Sound is one of the most informative and abundant modalities in the real world while being robust to sense without contacts by small and cheap sensors that can be placed on mobile devices.
Learning Flexible Translation between Robot Actions and Language Descriptions
In this work, we propose the paired gated autoencoders (PGAE) for flexible translation between robot actions and language descriptions in a tabletop object manipulation scenario.
Knowing Earlier what Right Means to You: A Comprehensive VQA Dataset for Grounding Relative Directions via Multi-Task Learning
Spatial reasoning poses a particular challenge for intelligent agents and is at the same time a prerequisite for their successful interaction and communication in the physical world.
What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning
Grounding relative directions is more difficult than grounding absolute directions because it not only requires a model to detect objects in the image and to identify spatial relation based on this information, but it also needs to recognize the orientation of objects and integrate this information into the reasoning process.
A Multimodal German Dataset for Automatic Lip Reading Systems and Transfer Learning
Large datasets as required for deep learning of lip reading do not exist in many languages.
Language Model-Based Paired Variational Autoencoders for Robotic Language Learning
Human infants learn language while interacting with their environment in which their caregivers may describe the objects and actions they perform.
LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading
The aim of this work is to investigate the impact of crossmodal self-supervised pre-training for speech reconstruction (video-to-audio) by leveraging the natural co-occurrence of audio and visual streams in videos.
Lifelong Learning from Event-based Data
Lifelong learning is a long-standing aim for artificial agents that act in dynamic environments, in which an agent needs to accumulate knowledge incrementally without forgetting previously learned representations.
FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection
The strong relation between face and voice can aid active speaker detection systems when faces are visible, even in difficult settings, when the face of a speaker is not clear or when there are several people in the same scene.
Generalization in Multimodal Language Learning from Simulation
Neural networks can be powerful function approximators, which are able to model high-dimensional feature distributions from a subset of examples drawn from the target distribution.
Survey on reinforcement learning for language processing
In recent years some researchers have explored the use of reinforcement learning (RL) algorithms as key components in the solution of various natural language processing tasks.
Visual Distant Supervision for Scene Graph Generation
In this work, we propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data.
A Sub-Layered Hierarchical Pyramidal Neural Architecture for Facial Expression Recognition
In domains where computational resources and labeled data are limited, such as in robotics, deep networks with millions of weights might not be the optimal solution.
Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision
Using a model of the environment, reinforcement learning agents can plan their future moves and achieve superhuman performance in board games like Chess, Shogi, and Go, while remaining relatively sample-efficient.
Crossmodal Language Grounding in an Embodied Neurocognitive Model
From a neuroscientific perspective, natural language is embodied, grounded in most, if not all, sensory and sensorimotor modalities, and acquired by means of crossmodal integration.
Multimodal Target Speech Separation with Voice and Face References
Target speech separation refers to isolating target speech from a multi-speaker mixture signal by conditioning on auxiliary information about the target speaker.
Audio and Speech Processing Sound
Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imagination
In this paper, we present a novel dual-system motor learning approach where a meta-controller arbitrates online between model-based and model-free decisions based on an estimate of the local reliability of the learned model.
EDA: Enriching Emotional Dialogue Acts using an Ensemble of Neural Annotators
These neural models annotate the emotion corpora with dialogue act labels, and an ensemble annotator extracts the final dialogue act label.
Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search
Establishing associations between the structure and the generalisation ability of deep neural networks (DNNs) is a challenging task in modern machine learning.
Efficient Intrinsically Motivated Robotic Grasping with Learning-Adaptive Imagination in Latent Space
The learned models are used to generate imagined experiences, augmenting the training set of real experiences.
What can computational models learn from human selective attention? A review from an audiovisual crossmodal perspective
Selective attention plays an essential role in information acquisition and utilization from the environment.
Curious Meta-Controller: Adaptive Alternation between Model-Based and Model-Free Control in Deep Reinforcement Learning
Recent success in deep reinforcement learning for continuous control has been dominated by model-free approaches which, unlike model-based approaches, do not suffer from representational limitations in making assumptions about the world dynamics and model errors inevitable in complex domains.
KT-Speech-Crawler: Automatic Dataset Construction for Speech Recognition from YouTube Videos
In this paper, we describe KT-Speech-Crawler: an approach for automatic dataset construction for speech recognition by crawling YouTube videos.
Incorporating End-to-End Speech Recognition Models for Sentiment Analysis
We argue that using ground-truth transcriptions during training and evaluation phases leads to a significant discrepancy in performance compared to real-world conditions, as the spoken text has to be recognized on the fly and can contain speech recognition mistakes.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learning
In this paper, we present a new intrinsically motivated actor-critic algorithm for learning continuous motor skills directly from raw visual input.
Discourse-Wizard: Discovering Deep Discourse Structure in your Conversation with RNNs
Spoken language understanding is one of the key factors in a dialogue system, and a context in a conversation plays an important role to understand the current utterance.
Lifelong Learning of Spatiotemporal Representations with Dual-Memory Recurrent Self-Organization
Both growing networks can expand in response to novel sensory experience: the episodic memory learns fine-grained spatiotemporal representations of object instances in an unsupervised fashion while the semantic memory uses task-relevant signals to regulate structural plasticity levels and develop more compact representations from episodic experience.
Conversational Analysis using Utterance-level Attention-based Bidirectional Recurrent Neural Networks
Recent approaches for dialogue act recognition have shown that context from preceding utterances is important to classify the subsequent one.
Ranked #9 on
Dialogue Act Classification
on Switchboard corpus
A Context-based Approach for Dialogue Act Recognition using Simple Recurrent Neural Networks
Dialogue act recognition is an important part of natural language understanding.
Ranked #10 on
Dialogue Act Classification
on Switchboard corpus
On the Robustness of Speech Emotion Recognition for Human-Robot Interaction with Deep Neural Networks
Speech emotion recognition (SER) is an important aspect of effective human-robot collaboration and received a lot of attention from the research community.
EmoRL: Continuous Acoustic Emotion Classification using Deep Reinforcement Learning
Acoustically expressed emotions can make communication with a robot more efficient.
Automatically augmenting an emotion dataset improves classification using audio
In this work, we tackle a problem of speech emotion classification.
Reusing Neural Speech Representations for Auditory Emotion Recognition
Acoustic emotion recognition aims to categorize the affective state of the speaker and is still a difficult task for machine learning models.
Spectral Ergodicity in Deep Learning Architectures via Surrogate Random Matrices
It is observed that as the matrix size increases the level of spectral ergodicity of the ensemble rises, i. e., the eigenvalue spectra obtained for a single realisation at random from the ensemble is closer to the spectra obtained averaging over the whole ensemble.
