Search Results for author:
Found 70 papers, 41 papers with code
Tutorial on Multimodal Machine Learning
Multimodal machine learning involves integrating and modeling information from multiple heterogeneous sources of data.
Diverse and Admissible Trajectory Prediction through Multimodal Context Understanding
Multi-agent trajectory forecasting in autonomous driving requires an agent to accurately anticipate the behaviors of the surrounding vehicles and pedestrians, for safe and reliable decision-making.
CMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French
It covers a diverse set topics and speakers, and carries supervision of 20 labels including sentiment (and subjectivity), emotions, and attributes.
Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions
Building socially-intelligent AI agents (Social-AI) is a multidisciplinary, multimodal research goal that involves creating agents that can sense, perceive, reason about, learn from, and respond to affect, behavior, and cognition of other agents (human or artificial).
Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Empirical results and human evaluations in a zero-shot setup demonstrate that our distillation method results in more precise VL models of reasoning compared to a baseline of passing a generated referring expression to an LLM.
MMOE: Mixture of Multimodal Interaction Experts
Multimodal machine learning, which studies the information and interactions across various input modalities, has made significant advancements in understanding the relationship between images and descriptive text.
Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities
Human interactions are deeply rooted in the interplay of thoughts, beliefs, and desires made possible by Theory of Mind (ToM): our cognitive ability to understand the mental states of ourselves and others.
MultiIoT: Towards Large-scale Multisensory Learning for the Internet of Things
The Internet of Things (IoT), the network integrating billions of smart physical devices embedded with sensors, software, and communication technologies for the purpose of connecting and exchanging data with other devices and systems, is a critical and rapidly expanding component of our modern world.
Comparative Knowledge Distillation
We observe that prevalent KD techniques and state of the art data augmentation strategies fall short in this constrained setting.
Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP
Furthermore, we release our BLIP causal tracing tool as open source to enable further experimentation in vision-language mechanistic interpretability by the community.
MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning
Learning multimodal representations involves integrating information from multiple heterogeneous sources of data.
Factorized Contrastive Learning: Going Beyond Multi-view Redundancy
How can we learn self-supervised multimodal representations to capture both shared and unique information relevant to downstream tasks?
Multimodal Fusion Interactions: A Study of Human and Automatic Quantification
In order to perform multimodal fusion of heterogeneous signals, we need to understand their interactions: how each modality individually provides information useful for a task and how this information changes in the presence of other modalities.
Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications
We propose two lower bounds based on the amount of shared information between modalities and the disagreement between separately trained unimodal classifiers, and derive an upper bound through connections to approximate algorithms for min-entropy couplings.
Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions
Since large-scale retraining of these models from scratch is both time and compute-expensive, a variety of approaches have been previously proposed that de-bias a pre-trained model.
Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition
Fusing multiple modalities has proven effective for multimodal information processing.
Difference-Masking: Choosing What to Mask in Continued Pretraining
The self-supervised objective of masking-and-predicting has led to promising performance gains on a variety of downstream tasks.
HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer
Accurately modeling affect dynamics, which refers to the changes and fluctuations in emotions and affective displays during human conversations, is crucial for understanding human interactions.
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities.
Lecture Presentations Multimodal Dataset: Towards Understanding Multimodality in Educational Videos
We introduce three research tasks, (1) figure-to-text retrieval, (2) text-to-figure retrieval, and (3) generation of slide explanations, which are grounded in multimedia learning and psychology principles to test a vision-language model's understanding of multimodal content.
Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment
To tackle this problem, we show that relation alignment can be enforced by encouraging the directed language attention from 'mug' to 'grass' (capturing the semantic relation 'in') to match the directed visual attention from the mug to the grass.
Does Structural Attention Improve Compositional Representations in Vision-Language Models?
Although scaling self-supervised approaches has gained widespread success in Vision-Language pre-training, a number of works providing structural knowledge of visually-grounded semantics have recently shown incremental performance gains.
Ranked #27 on
Visual Reasoning
on Winoground
Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
Existing techniques for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions.
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis
In particular, there are various considerations behind the pipeline: (1) the choice and (2) the size of PLM, (3) the choice of uncertainty quantifier, (4) the choice of fine-tuning loss, and many more.
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
With the recent interest in video understanding, embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, multimodal machine learning has brought unique computational and theoretical challenges to the machine learning community given the heterogeneity of data sources and the interconnections often found between modalities.
Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides
As a step toward developing AI to aid in student learning as intelligent teacher assistants, we introduce the Multimodal Lecture Presentations dataset as a large-scale benchmark testing the capabilities of machine learning models in multimodal understanding of educational content.
Face-to-Face Contrastive Learning for Social Intelligence Question-Answering
Creating artificial social intelligence - algorithms that can understand the nuances of multi-person interactions - is an exciting and emerging challenge in processing facial expressions and gestures from multimodal videos.
MultiViz: Towards Visualizing and Understanding Multimodal Models
How can we visualize the internal modeling of multimodal interactions in these models?
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness
Through discussing how Brainish is crucial for communication and coordination in order to achieve consciousness in the CTM, and by implementing a simple version of Brainish and evaluating its capability of demonstrating intelligence on multimodal prediction and retrieval tasks on several real-world image, text, and audio datasets, we argue that such an inner language will be important for advances in machine models of intelligence and consciousness.
PACS: A Dataset for Physical Audiovisual CommonSense Reasoning
Our paper takes a step towards real-world physical commonsense reasoning by contributing PACS: the first audiovisual benchmark annotated for physical commonsense attributes.
DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations
The ability for a human to understand an Artificial Intelligence (AI) model's decision-making process is critical in enabling stakeholders to visualize model behavior, perform model debugging, promote trust in AI models, and assist in collaborative human-AI decision-making.
High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on robots.
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas.
Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data
Using computational models, we find that language and multimodal representations of mobile typed text (spanning typed characters, words, keystroke timings, and app usage) are predictive of daily mood.
Towards Understanding and Mitigating Social Biases in Language Models
As machine learning methods are deployed in real-world settings such as healthcare, legal systems, and social science, it is crucial to recognize how they shape social biases and stereotypes in these sensitive decision-making processes.
Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
Federated learning is an emerging research paradigm enabling collaborative training of machine learning models among different organizations while keeping data private at each institution.
Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning
In this paper, we propose a Conditional Contrastive Learning (CCL) approach to improve the fairness of contrastive SSL methods.
Ask & Explore: Grounded Question Answering for Curiosity-Driven Exploration
In many real-world scenarios where extrinsic rewards to the agent are extremely sparse, curiosity has emerged as a useful concept providing intrinsic rewards that enable the agent to explore its environment and acquire information to achieve its goals.
StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer
Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes (e. g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence.
Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
As users increasingly rely on cloud-based computing services, it is important to ensure that uploaded speech data remains private.
Multimodal Privacy-preserving Mood Prediction from Mobile Data: A Preliminary Study
Mental health conditions remain under-diagnosed even in countries with common access to advanced medical care.
Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
In this work, we propose algorithms for cross-modal generalization: a learning paradigm to train a model that can (1) quickly perform new tasks in a target modality (i. e. meta-learning) and (2) doing so while being trained on a different source modality.
An Investigation of how Label Smoothing Affects Generalization
In this paper, as a step towards understanding why label smoothing is effective, we propose a theoretical framework to show how label smoothing provides in controlling the generalization loss.
Towards Debiasing Sentence Representations
As natural language processing methods are increasingly deployed in real-world scenarios such as healthcare, legal systems, and social science, it becomes necessary to recognize the role they potentially play in shaping social biases and stereotypes.
Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
In this paper, we design a simple and efficient embedding algorithm that learns a small set of anchor embeddings and a sparse transformation matrix.
Diverse and Admissible Trajectory Forecasting through Multimodal Context Understanding
Multi-agent trajectory forecasting in autonomous driving requires an agent to accurately anticipate the behaviors of the surrounding vehicles and pedestrians, for safe and reliable decision-making.
On Emergent Communication in Competitive Multi-Agent Teams
Several recent works have found the emergence of grounded compositional language in the communication protocols developed by mostly cooperative multi-agent systems when learned end-to-end to maximize performance on a downstream task.
Learning Not to Learn in the Presence of Noisy Labels
Learning in the presence of label noise is a challenging yet important task: it is crucial to design models that are robust in the presence of mislabeled datasets.
Think Locally, Act Globally: Federated Learning with Local and Global Representations
To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices.
Factorized Multimodal Transformer for Multimodal Sequential Learning
As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental research area.
Anchor & Transform: Learning Sparse Representations of Discrete Objects
Learning continuous representations of discrete objects such as text, users, and items lies at the heart of many applications including text and user modeling.
A Simple Approach to the Noisy Label Problem Through the Gambler's Loss
Learning in the presence of label noise is a challenging yet important task.
Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Our method is based on the observation that high-dimensional multimodal time series data often exhibit correlations across time and modalities which leads to low-rank tensor representations.
Deep Gamblers: Learning to Abstain with Portfolio Theory
We deal with the \textit{selective classification} problem (supervised-learning problem with a rejection option), where we want to achieve the best performance at a certain level of coverage of the data.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors.
Ranked #5 on
Multimodal Sentiment Analysis
on MOSI
Strong and Simple Baselines for Multimodal Utterance Embeddings
Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations.
Variational Auto-Decoder: A Method for Neural Generative Modeling from Incomplete Data
We study a specific implementation of the Auto-Encoding Variational Bayes (AEVB) algorithm, named in this paper as a Variational Auto-Decoder (VAD).
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Our method is based on the key insight that translation from a source to a target modality provides a method of learning joint representations using only the source modality as input.
An Empirical Evaluation of Sketched SVD and its Application to Leverage Score Ordering
We provide a comprehensive empirical evaluation of these algorithms and provide guidelines on how to ensure accurate deployment to real-world data.
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Humans convey their intentions through the usage of both verbal and nonverbal behaviors during face-to-face communication.
Multimodal Language Analysis with Recurrent Multistage Fusion
In this paper, we propose the Recurrent Multistage Fusion Network (RMFN) which decomposes the fusion problem into multiple stages, each of them focused on a subset of multimodal signals for specialized, effective fusion.
Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
Multimodal machine learning is a core research area spanning the language, visual and acoustic modalities.
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Analyzing human multimodal language is an emerging area of research in NLP.
Ranked #11 on
Multimodal Sentiment Analysis
on CMU-MOSEI
(using extra training data)
Learning Factorized Multimodal Representations
Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction.
Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
Previous research in this field has exploited the expressiveness of tensors for multimodal representation.
Multi-attention Recurrent Network for Human Communication Comprehension
AI must understand each modality and the interactions between them that shape human communication.
Ranked #9 on
Multimodal Sentiment Analysis
on MOSI
Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning
In this paper, we propose the Gated Multimodal Embedding LSTM with Temporal Attention (GME-LSTM(A)) model that is composed of 2 modules.
Memory Fusion Network for Multi-view Sequential Learning
In this paper, we present a new neural architecture for multi-view sequential learning called the Memory Fusion Network (MFN) that explicitly accounts for both interactions in a neural architecture and continuously models them through time.
