Search Results for author:
Found 88 papers, 27 papers with code
Dawn of the transformer era in speech emotion recognition: closing the valence gap
Recent advances in transformer-based architectures which are pre-trained in self-supervised manner have shown great promise in several machine learning tasks.
MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
The first Multimodal Emotion Recognition Challenge (MER 2023) was successfully held at ACM Multimedia.
openXBOW - Introducing the Passau Open-Source Crossmodal Bag-of-Words Toolkit
We introduce openXBOW, an open-source toolkit for the generation of bag-of-words (BoW) representations from multimodal input.
SVTS: Scalable Video-to-Speech Synthesis
Video-to-speech synthesis (also known as lip-to-speech) refers to the translation of silent lip movements into the corresponding audio.
HEAR: Holistic Evaluation of Audio Representations
The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios.
The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress
Multimodal Sentiment Analysis (MuSe) 2021 is a challenge focusing on the tasks of sentiment and emotion, as well as physiological-emotion and emotion-based stress recognition through more comprehensively integrating the audio-visual, language, and biological signal modalities.
The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress
For this year's challenge, we feature three datasets: (i) the Passau Spontaneous Football Coach Humor (Passau-SFCH) dataset that contains audio-visual recordings of German football coaches, labelled for the presence of humour; (ii) the Hume-Reaction dataset in which reactions of individuals to emotional stimuli have been annotated with respect to seven emotional expression intensities, and (iii) the Ulm-Trier Social Stress Test (Ulm-TSST) dataset comprising of audio-visual data labelled with continuous emotion values (arousal and valence) of people in stressful dispositions.
Speech Emotion Recognition using Semantic Information
In this paper, we propose a novel framework that can capture both the semantic and the paralinguistic information in the signal.
Speech Emotion Recognition
Sound
Audio and Speech Processing
audb -- Sharing and Versioning of Audio and Annotation Data in Python
Driven by the need for larger and more diverse datasets to pre-train and fine-tune increasingly complex machine learning models, the number of datasets is rapidly growing.
The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation
Participants predict the presence of spontaneous humour in a cross-cultural setting.
MuSe 2020 -- The First International Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop
Multimodal Sentiment Analysis in Real-life Media (MuSe) 2020 is a Challenge-based Workshop focusing on the tasks of sentiment recognition, as well as emotion-target engagement and trustworthiness detection by means of more comprehensively integrating the audio-visual and language modalities.
DeepSpectrumLite: A Power-Efficient Transfer Learning Framework for Embedded Speech and Audio Processing from Decentralised Data
By obtaining state-of-the-art results on a set of paralinguistics tasks, we demonstrate the suitability of the proposed transfer learning approach for embedded audio signal processing, even when data is scarce.
End-2-End COVID-19 Detection from Breath & Cough Audio
Our main contributions are as follows: (I) We demonstrate the first attempt to diagnose COVID-19 using end-to-end deep learning from a crowd-sourced dataset of audio samples, achieving ROC-AUC of 0. 846; (II) Our model, the COVID-19 Identification ResNet, (CIdeR), has potential for rapid scalability, minimal cost and improving performance as more data becomes available.
MuSe-Toolbox: The Multimodal Sentiment Analysis Continuous Annotation Fusion and Discrete Class Transformation Toolbox
With this in mind, the MuSe-Toolbox provides the functionality to run exhaustive searches for meaningful class clusters in the continuous gold standards.
An Improved StarGAN for Emotional Voice Conversion: Enhancing Voice Quality and Data Augmentation
Emotional Voice Conversion (EVC) aims to convert the emotional style of a source speech signal to a target style while preserving its content and speaker identity information.
A large-scale and PCR-referenced vocal audio dataset for COVID-19
The UK COVID-19 Vocal Audio Dataset represents the largest collection of SARS-CoV-2 PCR-referenced audio recordings to date.
Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers
Recent work has reported that AI classifiers trained on audio recordings can accurately predict severe acute respiratory syndrome coronavirus 2 (SARSCoV2) infection status.
A Novel Fusion of Attention and Sequence to Sequence Autoencoders to Predict Sleepiness From Speech
On the development partition of the data, we achieve Spearman's correlation coefficients of . 324, . 283, and . 320 with the targets on the Karolinska Sleepiness Scale by utilising attention and non-attention autoencoders, and the fusion of both autoencoders' representations, respectively.
GraphTMT: Unsupervised Graph-based Topic Modeling from Video Transcripts
To unfold the tremendous amount of multimedia data uploaded daily to social media platforms, effective topic modeling techniques are needed.
A novel policy for pre-trained Deep Reinforcement Learning for Speech Emotion Recognition
In addition, extended learning period is a general challenge for deep RL which can impact the speed of learning for SER.
Example-based Explanations with Adversarial Attacks for Respiratory Sound Analysis
Respiratory sound classification is an important tool for remote screening of respiratory-related diseases such as pneumonia, asthma, and COVID-19.
Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results
Our findings suggest that for the automatic analysis of humour and its sentiment, facial expressions are most promising, while humour direction can be best modelled via text-based features.
Exploring speaker enrolment for few-shot personalisation in emotional vocalisation prediction
In this work, we explore a novel few-shot personalisation architecture for emotional vocalisation prediction.
Knowledge Transfer For On-Device Speech Emotion Recognition with Neural Structured Learning
Our experiments demonstrate that training a lightweight SER model on the target dataset with speech samples and graphs can not only produce small SER models, but also enhance the model performance compared to models with speech samples only and those using classic transfer learning strategies.
Synthia's Melody: A Benchmark Framework for Unsupervised Domain Adaptation in Audio
Despite significant advancements in deep learning for vision and natural language, unsupervised domain adaptation in audio remains relatively unexplored.
Automatic Emotion Modelling in Written Stories
Telling stories is an integral part of human communication which can evoke emotions and influence the affective states of the audience.
Pre-training in Deep Reinforcement Learning for Automatic Speech Recognition
Deep reinforcement learning (deep RL) is a combination of deep learning with reinforcement learning principles to create efficient methods that can learn by interacting with its environment.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
Research on speech processing has traditionally considered the task of designing hand-engineered acoustic features (feature engineering) as a separate distinct problem from the task of designing efficient machine learning (ML) models to make prediction and classification decisions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
COVID-19 and Computer Audition: An Overview on What Speech & Sound Analysis Could Contribute in the SARS-CoV-2 Corona Crisis
We come to the conclusion that CA appears ready for implementation of (pre-)diagnosis and monitoring tools, and more generally provides rich and significant, yet so far untapped potential in the fight against COVID-19 spread.
An Early Study on Intelligent Analysis of Speech under COVID-19: Severity, Sleep Quality, Fatigue, and Anxiety
In particular, by analysing speech recordings from these patients, we construct audio-only-based models to automatically categorise the health state of patients from four aspects, including the severity of illness, sleep quality, fatigue, and anxiety.
deepSELF: An Open Source Deep Self End-to-End Learning Framework
To the best of our knowledge, it is the first public toolkit assembling a series of state-of-the-art deep learning technologies.
On Deep Speech Packet Loss Concealment: A Mini-Survey
In this mini-survey, we review all the literature we found to date, that attempt to solve the packet-loss in speech using deep learning methods.
ConcealNet: An End-to-end Neural Network for Packet Loss Concealment in Deep Speech Emotion Recognition
Additionally, extending this with an end-to-end emotion prediction neural network provides a network that performs SER from audio with lost frames, end-to-end.
"I have vxxx bxx connexxxn!": Facing Packet Loss in Deep Speech Emotion Recognition
We explore matched, mismatched, and multi-condition training settings.
High-Fidelity Audio Generation and Representation Learning with Guided Adversarial Autoencoder
Hence, with the extensive experimental results, we have demonstrated that by harnessing the power of the high-fidelity audio generation, the proposed GAAE model can learn powerful representation from unlabelled dataset leveraging a fewer percentage of labelled data as supervision/guidance.
Domain Adaptation with Joint Learning for Generic, Optical Car Part Recognition and Detection Systems (Go-CaRD)
Systems for the automatic recognition and detection of automotive parts are crucial in several emerging research areas in the development of intelligent vehicles.
MeDaS: An open-source platform as service to help break the walls between medicine and informatics
To the best of our knowledge, MeDaS is the first open-source platform proving a collaborative and interactive service for researchers from a medical background easily using DL related toolkits, and at the same time for scientists or engineers from information sciences to understand the medical knowledge side.
Audio, Speech, Language, & Signal Processing for COVID-19: A Comprehensive Overview
This drives the research focus towards identifying the markers of COVID-19 in speech and other human generated audio signals.
The voice of COVID-19: Acoustic correlates of infection
Group differences in the front vowels /i:/ and /e:/ are additionally reflected in the variation of the fundamental frequency and the harmonics-to-noise ratio, group differences in back vowels /o:/ and /u:/ in statistics of the Mel-frequency cepstral coefficients and the spectral slope.
Detecting COVID-19 from Breathing and Coughing Sounds using Deep Neural Networks
The COVID-19 pandemic has affected the world unevenly; while industrial economies have been able to produce the tests necessary to track the spread of the virus and mostly avoided complete lockdowns, developing countries have faced issues with testing capacity.
The INTERSPEECH 2021 Computational Paralinguistics Challenge: COVID-19 Cough, COVID-19 Speech, Escalation & Primates
The INTERSPEECH 2021 Computational Paralinguistics Challenge addresses four different problems for the first time in a research competition under well-defined conditions: In the COVID-19 Cough and COVID-19 Speech Sub-Challenges, a binary classification on COVID-19 infection has to be made based on coughing sounds and speech; in the Escalation SubChallenge, a three-way assessment of the level of escalation in a dialogue is featured; and in the Primates Sub-Challenge, four species vs background need to be classified.
Computational Emotion Analysis From Images: Recent Advances and Future Directions
Recently, extensive research efforts have been dedicated to understanding the emotions of images.
On the Impact of Word Error Rate on Acoustic-Linguistic Speech Emotion Recognition: An Update for the Deep Learning Era
Text encodings from automatic speech recognition (ASR) transcripts and audio representations have shown promise in speech emotion recognition (SER) ever since.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks
In this work, we propose a new end-to-end video-to-speech model based on Generative Adversarial Networks (GANs) which translates spoken video to waveform end-to-end without using any intermediate representation or separate waveform synthesis algorithm.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
The large amount of audiovisual content being shared online today has drawn substantial attention to the prospect of audiovisual self-supervised learning.
Affective Image Content Analysis: Two Decades Review and New Perspectives
Images can convey rich semantics and induce various emotions in viewers.
A Physiologically-Adapted Gold Standard for Arousal during Stress
We utilise a Long Short-Term Memory, Recurrent Neural Network to explore the benefit of fusing these physiological signals with arousal as the target, learning from various audio, video, and textual based features.
Evaluating the COVID-19 Identification ResNet (CIdeR) on the INTERSPEECH COVID-19 from Audio Challenges
We report on cross-running the recent COVID-19 Identification ResNet (CIdeR) on the two Interspeech 2021 COVID-19 diagnosis from cough and speech audio challenges: ComParE and DiCOVA.
Fairness and underspecification in acoustic scene classification: The case for disaggregated evaluations
Although these factors play a well-understood role in the performance of ASC models, most works report single evaluation metrics taking into account all different strata of a particular dataset.
Multistage linguistic conditioning of convolutional layers for speech emotion recognition
In this contribution, we investigate the effectiveness of deep fusion of text and audio features for categorical and dimensional speech emotion recognition (SER).
EIHW-MTG DiCOVA 2021 Challenge System Report
This paper aims to automatically detect COVID-19 patients by analysing the acoustic information embedded in coughs.
Facial Emotion Recognition using Deep Residual Networks in Real-World Environments
To show the favourable properties of our pre-trained model on modelling facial affect, we use the RECOLA database, and compare with the current state-of-the-art approach.
Emotion Intensity and its Control for Emotional Voice Conversion
As desired, the proposed network controls the fine-grained emotion intensity in the output speech.
Normalise for Fairness: A Simple Normalisation Technique for Fairness in Regression Machine Learning Problems
We present a theoretical analysis of the method, in addition to an empirical comparison against two standard methods for fairness, namely data balancing and adversarial training.
A Summary of the ComParE COVID-19 Challenges
The COVID-19 pandemic has caused massive humanitarian and economic damage.
Robust Federated Learning Against Adversarial Attacks for Speech Emotion Recognition
The proposed framework consists of i) federated learning for data privacy, and ii) adversarial training at the training stage and randomisation at the testing stage for model robustness.
Climate Change & Computer Audition: A Call to Action and Overview on Audio Intelligence to Help Save the Planet
We categorise potential computer audition applications according to the five elements of earth, water, air, fire, and aether, proposed by the ancient Greeks in their five element theory; this categorisation serves as a framework to discuss computer audition in relation to different ecological aspects.
Audiovisual Affect Assessment and Autonomous Automobiles: Applications
Emotion and a broader range of affective driver states can be a life decisive factor on the road.
Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition
To clearly understand the performance differences between recurrent and attention models in audiovisual affect recognition, we present a comprehensive evaluation of fusion models based on LSTM-RNNs, self-attention and cross-modal attention, trained for valence and arousal estimation.
An Overview & Analysis of Sequence-to-Sequence Emotional Voice Conversion
Emotional voice conversion (EVC) focuses on converting a speech utterance from a source to a target emotion; it can thus be a key enabling technology for human-computer interaction applications and beyond.
A Temporal-oriented Broadcast ResNet for COVID-19 Detection
Detecting COVID-19 from audio signals, such as breathing and coughing, can be used as a fast and efficient pre-testing method to reduce the virus transmission.
Probing Speech Emotion Recognition Transformers for Linguistic Knowledge
Large, pre-trained neural networks consisting of self-attention layers (transformers) have recently achieved state-of-the-art results on several speech emotion recognition (SER) datasets.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Journaling Data for Daily PHQ-2 Depression Prediction and Forecasting
Digital health applications are becoming increasingly important for assessing and monitoring the wellbeing of people suffering from mental health conditions like depression.
Fatigue Prediction in Outdoor Running Conditions using Audio Data
Although running is a common leisure activity and a core training regiment for several athletes, between $29\%$ and $79\%$ of runners sustain an overuse injury each year.
Insights on Modelling Physiological, Appraisal, and Affective Indicators of Stress using Audio Features
Stress is a major threat to well-being that manifests in a variety of physiological and mental symptoms.
The ACM Multimedia 2022 Computational Paralinguistics Challenge: Vocalisations, Stuttering, Activity, & Mosquitoes
The ACM Multimedia 2022 Computational Paralinguistics Challenge addresses four different problems for the first time in a research competition under well-defined conditions: In the Vocalisations and Stuttering Sub-Challenges, a classification on human non-verbal vocalisations and speech has to be made; the Activity Sub-Challenge aims at beyond-audio human activity recognition from smartwatch sensor data; and in the Mosquitoes Sub-Challenge, mosquitoes need to be detected.
Depression Diagnosis and Forecast based on Mobile Phone Sensor Data
Previous studies have shown the correlation between sensor data collected from mobile phones and human depression states.
COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition
In particular, we impose Calibration and Ordinal Ranking constraints on the variance vectors of audiovisual latent distributions.
COVYT: Introducing the Coronavirus YouTube and TikTok speech dataset featuring the same speakers with and without infection
As compared to other existing COVID-19 sound datasets, the unique feature of the COVYT dataset is that it comprises both COVID-19 positive and negative samples from all 65 speakers.
Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?
We also study how 3D face shapes performed on AU intensity estimation on BP4D and DISFA datasets, and report that 3D face features were on par with 2D appearance features in AUs 4, 6, 10, 12, and 25, but not the entire set of AUs.
Self-Supervised Attention Networks and Uncertainty Loss Weighting for Multi-Task Emotion Recognition on Vocal Bursts
Vocal bursts play an important role in communicating affect, making them valuable for improving speech emotion recognition.
An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era
Speech is the fundamental mode of human communication, and its synthesis has long been a core priority in human-computer interaction research.
Statistical Design and Analysis for Robust Machine Learning: A Case Study from COVID-19
Since early in the coronavirus disease 2019 (COVID-19) pandemic, there has been interest in using artificial intelligence methods to predict COVID-19 infection status based on vocal audio signals, for example cough recordings.
Computational Charisma -- A Brick by Brick Blueprint for Building Charismatic Artificial Intelligence
We then name exemplary use cases of computational charismatic skills before switching to ethical aspects and concluding this overview and perspective on building charisma-enabled AI.
A Comprehensive Survey on Heart Sound Analysis in the Deep Learning Era
Deep learning has been successfully applied to heart sound analysis in the past years.
Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPT
We utilise three baselines, a robust language model (RoBERTa-base), a legacy word model with pretrained embeddings (Word2Vec), and a simple bag-of-words baseline (BoW).
The ACM Multimedia 2023 Computational Paralinguistics Challenge: Emotion Share & Requests
The ACM Multimedia 2023 Computational Paralinguistics Challenge addresses two different problems for the first time in a research competition under well-defined conditions: In the Emotion Share Sub-Challenge, a regression on speech has to be made; and in the Requests Sub-Challenges, requests and complaints need to be detected.
Integrating Generative Artificial Intelligence in Intelligent Vehicle Systems
This paper aims to serve as a comprehensive guide for researchers and practitioners, offering insights into the current state, potential applications, and future research directions for generative artificial intelligence and foundation models within the context of intelligent vehicles.
Can ChatGPT's Responses Boost Traditional Natural Language Processing?
In this work, we extend this by exploring if ChatGPT has novel knowledge that would enhance existing specialised models when they are fused together.
Identifying depression-related topics in smartphone-collected free-response speech recordings using an automatic speech recognition system and a deep learning topic model
Our study identified 29 topics in 3919 smartphone-collected speech recordings from 265 participants using the Whisper tool and BERTopic model.
A Wide Evaluation of ChatGPT on Affective Computing Tasks
In this work, we widely study the capabilities of the ChatGPT models, namely GPT-4 and GPT-3. 5, on 13 affective computing problems, namely aspect extraction, aspect polarity classification, opinion extraction, sentiment analysis, sentiment intensity ranking, emotions intensity ranking, suicide tendency detection, toxicity detection, well-being assessment, engagement measurement, personality assessment, sarcasm detection, and subjectivity detection.
Exploring Meta Information for Audio-based Zero-shot Bird Classification
Advances in passive acoustic monitoring and machine learning have led to the procurement of vast datasets for computational bioacoustic research.
Task Selection and Assignment for Multi-modal Multi-task Dialogue Act Classification with Non-stationary Multi-armed Bandits
Our proposed method is significantly superior in terms of UAR and F1 to the single-task and multi-task baselines with p-values < 0. 05.
Customising General Large Language Models for Specialised Emotion Recognition Tasks
This indicates the strong transferability and feasibility of LLMs in the field of emotion recognition.
Testing Speech Emotion Recognition Machine Learning Models
Machine learning models for speech emotion recognition (SER) can be trained for different tasks and are usually evaluated on the basis of a few available datasets per task.
STAA-Net: A Sparse and Transferable Adversarial Attack for Speech Emotion Recognition
Speech contains rich information on the emotions of humans, and Speech Emotion Recognition (SER) has been an important topic in the area of human-computer interaction.
On Prompt Sensitivity of ChatGPT in Affective Computing
Recent studies have demonstrated the emerging capabilities of foundation models like ChatGPT in several fields, including affective computing.
Enhancing Suicide Risk Assessment: A Speech-Based Automated Approach in Emergency Medicine
The metadata integration yields a balanced accuracy of $94. 4\,\%$, marking an absolute improvement of $28. 2\,\%$, demonstrating the efficacy of our proposed approaches for automatic suicide risk assessment in emergency medicine.
