Search Results for author:
Found 31 papers, 14 papers with code
Egocentric Speaker Classification in Child-Adult Dyadic Interactions: From Sensing to Computational Modeling
In this study, we design an experiment to perform speech sampling in BOSCC interviews from an egocentric perspective using wearable sensors and explore pre-training Ego4D speech samples to enhance child-adult speaker classification in dyadic interactions.
Data Efficient Child-Adult Speaker Diarization with Simulated Conversations
We then train a Whisper Encoder-based model, achieving strong zero-shot performance on child-adult speaker diarization using real datasets.
ModalityMirror: Improving Audio Classification in Modality Heterogeneity Federated Learning with Multimodal Distillation
Multimodal Federated Learning frequently encounters challenges of client modality heterogeneity, leading to undesired performances for secondary modality in multimodal learning.
Creating a Lens of Chinese Culture: A Multimodal Dataset for Chinese Pun Rebus Art Understanding
Large vision-language models (VLMs) have demonstrated remarkable abilities in understanding everyday content.
Can Synthetic Audio From Generative Foundation Models Assist Audio Recognition and Speech Modeling?
In contrast, we aim to evaluate the quality of audio generation by examining the effectiveness of using them as training data.
Toward Fully-End-to-End Listened Speech Decoding from EEG Signals
Speech decoding from EEG signals is a challenging task, where brain activity is modeled to estimate salient characteristics of acoustic stimuli.
Exploring Speech Foundation Models for Speaker Diarization in Child-Adult Dyadic Interactions
Speech foundation models, trained on vast datasets, have opened unique opportunities in addressing challenging low-resource speech understanding, such as child speech.
POWQMIX: Weighted Value Factorization with Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning
Value function factorization methods are commonly used in cooperative multi-agent reinforcement learning, with QMIX receiving significant attention.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
TI-ASU: Toward Robust Automatic Speech Understanding through Text-to-speech Imputation Against Missing Speech Modality
Automatic Speech Understanding (ASU) aims at human-like speech interpretation, providing nuanced intent, emotion, sentiment, and content understanding from speech and language (text) content conveyed in speech.
Partial Federated Learning
Federated Learning (FL) is a popular algorithm to train machine learning models on user data constrained to edge devices (for example, mobile phones) due to privacy concerns.
Can Text-to-image Model Assist Multi-modal Learning for Visual Recognition with Visual Modality Missing?
Specifically, we propose a simple but effective multi-modal learning framework GTI-MM to enhance the data efficiency and model robustness against missing visual modality by imputing the missing data with generative transformers.
Audio-visual child-adult speaker classification in dyadic interactions
Building on the foundation of an audio-only child-adult speaker classification pipeline, we propose incorporating visual cues through active speaker detection and visual processing models.
FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Things
However, most existing FL works do not use datasets collected from authentic IoT devices and thus do not capture unique modalities and inherent challenges of IoT data.
Scaling Representation Learning from Ubiquitous ECG with State-Space Models
We train this model in a self-supervised manner with 275, 000 10s ECG recordings collected in the wild and evaluate it on a range of downstream tasks.
MM-AU:Towards Multimodal Understanding of Advertisement Videos
Advertisement videos (ads) play an integral part in the domain of Internet e-commerce as they amplify the reach of particular products to a broad audience or can serve as a medium to raise awareness about specific issues through concise narrative structures.
Robust Self Supervised Speech Embeddings for Child-Adult Classification in Interactions involving Children with Autism
We address the problem of detecting who spoke when in child-inclusive spoken interactions i. e., automatic child-adult speaker classification.
Learning Behavioral Representations of Routines From Large-scale Unlabeled Wearable Time-series Data Streams using Hawkes Point Process
In this work, we propose a novel wearable time-series mining framework, Hawkes point process On Time series clusters for ROutine Discovery (HOT-ROD), for uncovering behavioral routines from completely unlabeled wearable recordings.
FedMultimodal: A Benchmark For Multimodal Federated Learning
In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities.
GPT-FL: Generative Pre-trained Model-Assisted Federated Learning
Through comprehensive ablation analysis across various data modalities, we discover that the downstream model generated by synthetic data plays a crucial role in controlling the direction of gradient diversity during FL training, which enhances convergence speed and contributes to the notable accuracy boost observed with GPT-FL.
Understanding Spoken Language Development of Children with ASD Using Pre-trained Speech Embeddings
This paper proposes applications of speech processing technologies in support of automated assessment of children's spoken language development by classification between child and adult speech and between speech and nonverbal vocalization in NLS, with respective F1 macro scores of 82. 6% and 67. 8%, underscoring the potential for accurate and scalable tools for ASD research and clinical use.
Learning Sequence Descriptor based on Spatio-Temporal Attention for Visual Place Recognition
Visual Place Recognition (VPR) aims to retrieve frames from a geotagged database that are located at the same place as the query frame.
A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness
Speech-centric machine learning systems have revolutionized many leading domains ranging from transportation and healthcare to education and defense, profoundly changing how people live, work, and interact with each other.
Multimodal Estimation of Change Points of Physiological Arousal in Drivers
Detecting unsafe driving states, such as stress, drowsiness, and fatigue, is an important component of ensuring driving safety and an essential prerequisite for automatic intervention systems in vehicles.
Semi-FedSER: Semi-supervised Learning for Speech Emotion Recognition On Federated Learning using Multiview Pseudo-Labeling
In this work, we propose a semi-supervised federated learning framework, Semi-FedSER, that utilizes both labeled and unlabeled data samples to address the challenge of limited labeled data samples in FL.
Patch-NetVLAD+: Learned patch descriptor and weighted matching strategy for place recognition
Visual Place Recognition (VPR) in areas with similar scenes such as urban or indoor scenarios is a major challenge.
Attribute Inference Attack of Speech Emotion Recognition in Federated Learning Settings
To assess the information leakage of SER systems trained using FL, we propose an attribute inference attack framework that infers sensitive attribute information of the clients from shared gradients or model parameters, corresponding to the FedSGD and the FedAvg training algorithms, respectively.
Unsupervised Joint Learning of Depth, Optical Flow, Ego-motion from Video
Compared with independent training, joint training can make full use of the geometric relationship between geometric elements and provide dynamic and static information of the scene.
Screenplay Quality Assessment: Can We Predict Who Gets Nominated?
Toward that goal, in this work, we present a method to evaluate the quality of a screenplay based on linguistic cues.
TILES-2018, a longitudinal physiologic and behavioral data set of hospital workers
We designed the study to investigate the use of off-the-shelf wearable and environmental sensors to understand individual-specific constructs such as job performance, interpersonal interaction, and well-being of hospital workers over time in their natural day-to-day job settings.
Occlusion Aware Unsupervised Learning of Optical Flow From Video
In this paper, we proposed an unsupervised learning method for estimating the optical flow between video frames, especially to solve the occlusion problem.
Vision-based Semantic Mapping and Localization for Autonomous Indoor Parking
In this paper, we proposed a novel and practical solution for the real-time indoor localization of autonomous driving in parking lots.
