Search Results for author:
Found 104 papers, 24 papers with code
The Twins Corpus of Museum Visitor Questions
In addition to speech recordings, the corpus contains the outputs of speech recognition performed at the time of utterance as well as the system interpretation of the utterances.
Inferring object rankings based on noisy pairwise comparisons from multiple annotators
In this work, we propose Expectation-Maximization (EM) based algorithms that rely on the judgments from multiple annotators and the object attributes for inferring the latent ground truth.
Linguistic analysis of differences in portrayal of movie characters
We examine differences in portrayal of characters in movies using psycholinguistic and graph theoretic measures computed directly from screenplays.
Tweester at SemEval-2017 Task 4: Fusion of Semantic-Affective and pairwise classification models for sentiment analysis in Twitter
In this paper, we describe our submission to SemEval2017 Task 4: Sentiment Analysis in Twitter.
NTUA-SLP at SemEval-2018 Task 1: Predicting Affective Content in Tweets with Deep Attentive RNNs and Transfer Learning
In this paper we present deep-learning models that submitted to the SemEval-2018 Task~1 competition: "Affect in Tweets".
Towards an Unsupervised Entrainment Distance in Conversational Speech using Deep Neural Networks
Entrainment is a known adaptation mechanism that causes interaction participants to adapt or synchronize their acoustic characteristics.
Semi-supervised and Transfer learning approaches for low resource sentiment classification
Sentiment classification involves quantifying the affective reaction of a human to a document, media item or an event.
Measuring Conversational Productivity in Child Forensic Interviews
Child Forensic Interviewing (FI) presents a challenge for effective information retrieval and decision making.
A Multi-task Approach to Learning Multilingual Representations
We present a novel multi-task modeling approach to learning multilingual distributed representations of text.
Cross-Lingual Document Classification
Document Classification
+5
Tensor Embedding: A Supervised Framework for Human Behavioral Data Mining and Prediction
Today's densely instrumented world offers tremendous opportunities for continuous acquisition and analysis of multimodal sensor data providing temporal characterization of an individual's behaviors.
Multi-label Multi-task Deep Learning for Behavioral Coding
We propose a methodology for estimating human behaviors in psychotherapy sessions using mutli-label and multi-task learning paradigms.
On evaluating CNN representations for low resource medical image classification
Convolutional Neural Networks (CNNs) have revolutionized performances in several machine learning tasks such as image classification, object tracking, and keyword spotting.
Generating Labels for Regression of Subjective Constructs using Triplet Embeddings
In this work, we propose a novel annotation approach using triplet embeddings.
Multimodal Representation Learning using Deep Multiset Canonical Correlation
We propose Deep Multiset Canonical Correlation Analysis (dMCCA) as an extension to representation learning using CCA when the underlying signal is observed across multiple (more than two) modalities.
Modeling Interpersonal Linguistic Coordination in Conversations using Word Mover's Distance
We find that our proposed measure is correlated with the therapist's empathy towards their patient in Motivational Interviewing and with affective behaviors in Couples Therapy.
A system for the 2019 Sentiment, Emotion and Cognitive State Task of DARPAs LORELEI project
In this work, we describe our submission for the 2019 Sentiment, Emotion and Cognitive state (SEC) pilot task of the LORELEI project.
Towards Adapting NMF Dictionaries Using Total Variability Modeling for Noise-Robust Acoustic Features
We propose an algorithm to extract noise-robust acoustic features from noisy speech.
Behavior Gated Language Models
In this work we derive motivation from psycholinguistics and propose the addition of behavioral information into the context of language modeling.
The Ambiguous World of Emotion Representation
A key reason for this is the lack of a common mathematical framework to describe all the relevant elements of emotion representations.
Multimodal Embeddings from Language Models
Word embeddings such as ELMo have recently been shown to model word semantics with greater efficacy through contextualized learning on large-scale language corpora, resulting in significant improvement in state of the art across many natural language tasks.
RNN based Incremental Online Spoken Language Understanding
We introduce and analyze different recurrent neural network architectures for incremental and online processing of the ASR transcripts and compare it to the existing offline systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+8
Learning Domain Invariant Representations for Child-Adult Classification from Speech
Diagnostic procedures for ASD (autism spectrum disorder) involve semi-naturalistic interactions between the child and a clinician.
Robust speaker recognition using unsupervised adversarial invariance
In this paper, we address the problem of speaker recognition in challenging acoustic conditions using a novel method to extract robust speaker-discriminative speech representations.
Speaker-invariant Affective Representation Learning via Adversarial Training
In this paper, we propose a machine learning framework to obtain speech emotion representations by limiting the effect of speaker variability in the speech signals.
Characterizing dynamically varying acoustic scenes from egocentric audio recordings in workplace setting
In this paper, we address the task of characterizing acoustic scenes in a workplace setting from audio recordings collected with wearable microphones.
Learning Behavioral Representations from Wearable Sensors
Continuous collection of physiological data from wearable sensors enables temporal characterization of individual behaviors.
An analysis of observation length requirements for machine understanding of human behaviors from spoken language
In this paper, we investigate this link and present an analysis framework that determines appropriate window lengths for the task of behavior estimation.
Auto-Tuning Spectral Clustering for Speaker Diarization Using Normalized Maximum Eigengap
In this study, we propose a new spectral clustering framework that can auto-tune the parameters of the clustering algorithm in the context of speaker diarization.
Ranked #1 on
Speaker Diarization
on CALLHOME
(DER(ig olp) metric)
Cross modal video representations for weakly supervised active speaker localization
Avoiding the need for manual annotations for active speakers in visual frames, acquiring of which is very expensive, we present a weakly supervised system for the task of localizing active speakers in movie content.
A Label Proportions Estimation Technique for Adversarial Domain Adaptation in Text Classification
The DAN-LPE simultaneously trains a domain adversarial net and processes label proportions estimation by the confusion of the source domain and the predictions of the target domain.
TILES-2018, a longitudinal physiologic and behavioral data set of hospital workers
We designed the study to investigate the use of off-the-shelf wearable and environmental sensors to understand individual-specific constructs such as job performance, interpersonal interaction, and well-being of hospital workers over time in their natural day-to-day job settings.
Speaker Diarization with Lexical Information
This work presents a novel approach for speaker diarization to leverage lexical information provided by automatic speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Joint Multi-Dimensional Model for Global and Time-Series Annotations
In this work we address this by proposing a generative model for multi-dimensional annotation fusion, which models the dimensions jointly leading to more accurate ground truth estimates.
Generalized Multi-view Shared Subspace Learning using View Bootstrapping
A key objective in multi-view learning is to model the information common to multiple parallel views of a class of objects/events to improve downstream learning tasks.
Screenplay Quality Assessment: Can We Predict Who Gets Nominated?
Toward that goal, in this work, we present a method to evaluate the quality of a screenplay based on linguistic cues.
Feature Fusion Strategies for End-to-End Evaluation of Cognitive Behavior Therapy Sessions
We propose a novel method to augment the word-based features with the utterance level tags for subsequent CBT code estimation.
Sentence level estimation of psycholinguistic norms using joint multidimensional annotations
We then use this parameter at sentence level to estimate the norms.
Towards end-2-end learning for predicting behavior codes from spoken utterances in psychotherapy conversations
Spoken language understanding tasks usually rely on pipelines involving complex processing blocks such as voice activity detection, speaker diarization and Automatic speech recognition (ASR).
Evidence of Task-Independent Person-Specific Signatures in EEG using Subspace Techniques
These high dimensional statistics are then projected to a lower dimensional space where the biometric information is preserved.
Designing Neural Speaker Embeddings with Meta Learning
Our experiments illustrate the applicability of meta-learning as a generalized learning paradigm for training deep neural speaker embeddings.
Audio and Speech Processing Sound
Adversarial Attack and Defense Strategies for Deep Speaker Recognition Systems
Robust speaker recognition, including in the presence of malicious attacks, is becoming increasingly important and essential, especially due to the proliferation of several smart speakers and personal agents that interact with an individual's voice commands to perform diverse, and even sensitive tasks.
Victim or Perpetrator? Analysis of Violent Characters Portrayals from Movie Scripts
To date, we are the first to show that language used in movie scripts is a strong indicator of violent content, and that there are systematic portrayals of certain demographics as victims and perpetrators in a large dataset.
Robust Character Labeling in Movie Videos: Data Resources and Self-supervised Feature Adaptation
Our work in this paper focuses on two key aspects of this problem: the lack of domain-specific training or benchmark datasets, and adapting face embeddings learned on web images to long-form content, specifically movies.
A Review of Speaker Diarization: Recent Advances with Deep Learning
Speaker diarization is a task to label audio or video recordings with classes that correspond to speaker identity, or in short, a task to identify "who spoke when".
Confusion2vec 2.0: Enriching Ambiguous Spoken Language Representations with Subwords
Confusion2vec, motivated from human speech production and perception, is a word vector representation which encodes ambiguities present in human spoken language in addition to semantics and syntactic information.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
End-to-End Neural Systems for Automatic Children Speech Recognition: An Empirical Study
A key desiderata for inclusive and accessible speech recognition technology is ensuring its robust performance to children's speech.
Automated Evaluation Of Psychotherapy Skills Using Speech And Language Technologies
With the growing prevalence of psychological interventions, it is vital to have measures which rate the effectiveness of psychological care to assist in training, supervision, and quality assurance of services.
Automated Quality Assessment of Cognitive Behavioral Therapy Sessions Through Highly Contextualized Language Representations
In this work, we propose a BERT-based model for automatic behavioral scoring of a specific type of psychotherapy, called Cognitive Behavioral Therapy (CBT), where prior work is limited to frequency-based language features and/or short text excerpts which do not capture the unique elements involved in a spontaneous long conversational interaction.
Front-end Diarization for Percussion Separation in Taniavartanam of Carnatic Music Concerts
In this work, we address the problem of separating the percussive voices in the taniavartanam segments of Carnatic music.
Unsupervised Speech Representation Learning for Behavior Modeling using Triplet Enhanced Contextualized Networks
In this paper, we exploit the stationary properties of human behavior within an interaction and present a representation learning method to capture behavioral information from speech in an unsupervised way.
Acted vs. Improvised: Domain Adaptation for Elicitation Approaches in Audio-Visual Emotion Recognition
Key challenges in developing generalized automatic emotion recognition systems include scarcity of labeled data and lack of gold-standard references.
An Automated Quality Evaluation Framework of Psychotherapy Conversations with Local Quality Estimates
In particular, we model the global quality as a linear function of the local quality scores, which allows us to update the segment-level quality estimates based on the session-level quality prediction.
Perceptual-based deep-learning denoiser as a defense against adversarial attacks on ASR systems
In this paper we investigate speech denoising as a defense against adversarial attacks on automatic speech recognition (ASR) systems.
Phone Duration Modeling for Speaker Age Estimation in Children
In this paper, we propose features specific to children and focus on speaker's phone duration as an important biomarker of children's age.
Representation of professions in entertainment media: Insights into frequency and sentiment trends through computational text analysis
We analyze the frequency and sentiment trends of different occupations, study the effect of media attributes like genre, country of production, and title type on these trends, and investigate if the incidence of professions in media subtitles correlate with their real-world employment statistics.
Cross Domain Emotion Recognition using Few Shot Knowledge Transfer
Emotion recognition from text is a challenging task due to diverse emotion taxonomies, lack of reliable labeled data in different domains, and highly subjective annotation standards.
Understanding of Emotion Perception from Art
Computational modeling of the emotions evoked by art in humans is a challenging problem because of the subjective and nuanced nature of art and affective signals.
Semi-FedSER: Semi-supervised Learning for Speech Emotion Recognition On Federated Learning using Multiview Pseudo-Labeling
In this work, we propose a semi-supervised federated learning framework, Semi-FedSER, that utilizes both labeled and unlabeled data samples to address the challenge of limited labeled data samples in FL.
To train or not to train adversarially: A study of bias mitigation strategies for speaker recognition
In this paper, we systematically evaluate the biases present in speaker recognition systems with respect to gender across a range of system operating points.
Audio visual character profiles for detecting background characters in entertainment media
We curate a background character dataset which provides annotations for background character for a set of TV shows, and use it to evaluate the performance of the background character detection framework.
Mel Frequency Spectral Domain Defenses against Adversarial Attacks on Speech Recognition Systems
A variety of recent works have looked into defenses for deep neural networks against adversarial attacks particularly within the image processing domain.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Using Active Speaker Faces for Diarization in TV shows
Speaker diarization is one of the critical components of computational media intelligence as it enables a character-level analysis of story portrayals and media content understanding.
Multimodal Clustering with Role Induced Constraints for Speaker Diarization
Speaker clustering is an essential step in conventional speaker diarization systems and is typically addressed as an audio-only speech processing task.
Local dynamic mode of Cognitive Behavioral Therapy
In order to increase mental health equity among the most vulnerable and marginalized communities, it is important to increase access to high-quality therapists.
Automating Detection of Papilledema in Pediatric Fundus Images with Explainable Machine Learning
Papilledema is an ophthalmic neurologic disorder in which increased intracranial pressure leads to swelling of the optic nerves.
VAuLT: Augmenting the Vision-and-Language Transformer for Sentiment Classification on Social Media
VAuLT is an extension of the popular Vision-and-Language Transformer (ViLT), and improves performance on vision-and-language (VL) tasks that involve more complex text inputs than image captions while having minimal impact on training and inference efficiency.
The 2022 Far-field Speaker Verification Challenge: Exploring domain mismatch and semi-supervised learning under the far-field scenario
In addition, a supplementary set for the FFSVC2020 dataset is released this year.
Unsupervised active speaker detection in media content using cross-modal information
We leverage speaker identity information from speech and faces, and formulate active speaker detection as a speech-face assignment task such that the active speaker's face and the underlying speech identify the same person (character).
MovieCLIP: Visual Scene Recognition in Movies
Longform media such as movies have complex narrative structures, with events spanning a rich variety of ambient visual scenes.
Leveraging Open Data and Task Augmentation to Automated Behavioral Coding of Psychotherapy Conversations in Low-Resource Scenarios
In psychotherapy interactions, the quality of a session is assessed by codifying the communicative behaviors of participants during the conversation through manual observation and annotation.
Multimodal Estimation of Change Points of Physiological Arousal in Drivers
Detecting unsafe driving states, such as stress, drowsiness, and fatigue, is an important component of ensuring driving safety and an essential prerequisite for automatic intervention systems in vehicles.
Leveraging Label Correlations in a Multi-label Setting: A Case Study in Emotion
First, we develop two modeling approaches to the problem in order to capture word associations of the emotion words themselves, by either including the emotions in the input, or by leveraging Masked Language Modeling (MLM).
Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats
In this work, we study how we can build a single model that can transition between these different configurations by leveraging multilingual models and Demux, a transformer-based model whose input includes the emotions of interest, enabling us to dynamically change the emotions predicted by the model.
A Context-Aware Computational Approach for Measuring Vocal Entrainment in Dyadic Conversations
Vocal entrainment is a social adaptation mechanism in human interaction, knowledge of which can offer useful insights to an individual's cognitive-behavioral characteristics.
Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection
Active speaker detection in videos addresses associating a source face, visible in the video frames, with the underlying speech in the audio modality.
Ranked #1 on
Audio-Visual Active Speaker Detection
on VPCD
A dataset for Audio-Visual Sound Event Detection in Movies
In this work, we present a dataset of audio events called Subtitle-Aligned Movie Sounds (SAM-S).
Contextually-rich human affect perception using multimodal scene information
The process of human affect understanding involves the ability to infer person specific emotional states from various sources including images, speech, and language.
Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP
There is an imminent need for guidelines and standard test sets to allow direct and fair comparisons of speech emotion recognition (SER).
Signal Processing Grand Challenge 2023 -- e-Prevention: Sleep Behavior as an Indicator of Relapses in Psychotic Patients
This paper presents the approach and results of USC SAIL's submission to the Signal Processing Grand Challenge 2023 - e-Prevention (Task 2), on detecting relapses in psychotic patients.
Understanding Spoken Language Development of Children with ASD Using Pre-trained Speech Embeddings
This paper proposes applications of speech processing technologies in support of automated assessment of children's spoken language development by classification between child and adult speech and between speech and nonverbal vocalization in NLS, with respective F1 macro scores of 82. 6% and 67. 8%, underscoring the potential for accurate and scalable tools for ASD research and clinical use.
FedMultimodal: A Benchmark For Multimodal Federated Learning
In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities.
Learning Behavioral Representations of Routines From Large-scale Unlabeled Wearable Time-series Data Streams using Hawkes Point Process
In this work, we propose a novel wearable time-series mining framework, Hawkes point process On Time series clusters for ROutine Discovery (HOT-ROD), for uncovering behavioral routines from completely unlabeled wearable recordings.
Robust Self Supervised Speech Embeddings for Child-Adult Classification in Interactions involving Children with Autism
We address the problem of detecting who spoke when in child-inclusive spoken interactions i. e., automatic child-adult speaker classification.
MM-AU:Towards Multimodal Understanding of Advertisement Videos
Advertisement videos (ads) play an integral part in the domain of Internet e-commerce as they amplify the reach of particular products to a broad audience or can serve as a medium to raise awareness about specific issues through concise narrative structures.
Does Video Summarization Require Videos? Quantifying the Effectiveness of Language in Video Summarization
Video summarization remains a huge challenge in computer vision due to the size of the input videos to be summarized.
Scaling Representation Learning from Ubiquitous ECG with State-Space Models
We train this model in a self-supervised manner with 275, 000 10s ECG recordings collected in the wild and evaluate it on a range of downstream tasks.
Audio-visual child-adult speaker classification in dyadic interactions
Building on the foundation of an audio-only child-adult speaker classification pipeline, we propose incorporating visual cues through active speaker detection and visual processing models.
Context Unlocks Emotions: Text-based Emotion Classification Dataset Auditing with Large Language Models
In this work, we propose a formal definition of textual context to motivate a prompting strategy to enhance such contextual information.
Explainable Severity ranking via pairwise n-hidden comparison: a case study of glaucoma
Primary open-angle glaucoma (POAG) is a chronic and progressive optic nerve condition that results in an acquired loss of optic nerve fibers and potential blindness.
A Multi-Perspective Machine Learning Approach to Evaluate Police-Driver Interaction in Los Angeles
Interactions between the government officials and civilians affect public wellbeing and the state legitimacy that is necessary for the functioning of democratic society.
Can Text-to-image Model Assist Multi-modal Learning for Visual Recognition with Visual Modality Missing?
Specifically, we propose a simple but effective multi-modal learning framework GTI-MM to enhance the data efficiency and model robustness against missing visual modality by imputing the missing data with generative transformers.
Knowledge-guided EEG Representation Learning
We also propose a novel knowledge-guided pre-training objective that accounts for the idiosyncrasies of the EEG signal.
Evaluating Atypical Gaze Patterns through Vision Models: The Case of Cortical Visual Impairment
A wide range of neurological and cognitive disorders exhibit distinct behavioral markers aside from their clinical manifestations.
The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition
The promise of ICL is that the LLM can adapt to perform the present task at a competitive or state-of-the-art level at a fraction of the cost.
Joint Estimation and Analysis of Risk Behavior Ratings in Movie Scripts
Exposure to violent, sexual, or substance-abuse content in media increases the willingness of children and adolescents to imitate similar behaviors.
