Search Results for author:
Found 70 papers, 25 papers with code
NICE: Neural Image Commenting with Empathy
Based on these annotations, we define two different tasks for the NICE dataset.
A Demonstration of Adaptive Collaboration of Large Language Models for Medical Decision-Making
Medical Decision-Making (MDM) is a multi-faceted process that requires clinicians to assess complex multi-modal patient data patient, often collaboratively.
SympCam: Remote Optical Measurement of Sympathetic Arousal
Furthermore, we show that the sympathetic arousal predicted by our method allows detecting physical stress with a balanced accuracy of 90% - an improvement of 61% compared to the rPPG method commonly used in related work, demonstrating the limitations of using rPPG alone.
Scaling Wearable Foundation Models
Wearable sensors have become ubiquitous thanks to a variety of health tracking features.
Evidence of Cognitive Deficits andDevelopmental Advances in Generative AI: A Clock Drawing Test Analysis
Generative AI's rapid advancement sparks interest in its cognitive abilities, especially given its capacity for tasks like language understanding and code generation.
The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks
There is increasing interest in tracking the capabilities of general intelligence foundation models.
Global atmospheric data assimilation with multi-modal masked autoencoders
We show that EarthNet performs a form of data assimilation producing a global 0. 16 degree reanalysis dataset of 3D atmospheric temperature and humidity at a fraction of the time compared to operational systems.
The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
Foundation model development attracts a rapidly expanding body of contributors, scientists, and applications.
The Potential and Perils of Generative Artificial Intelligence for Quality Improvement and Patient Safety
Generative artificial intelligence (GenAI) has the potential to improve healthcare through automation that enhances the quality and safety of patient care.
What Are the Odds? Language Models Are Capable of Probabilistic Reasoning
Language models (LM) are capable of remarkably complex linguistic tasks; however, numerical reasoning is an area in which they frequently struggle.
PARSE-Ego4D: Personal Action Recommendation Suggestions for Egocentric Videos
Based on our synthetic dataset and complete human annotations, we propose several new tasks for action suggestions based on ego-centric videos.
Transforming Wearable Data into Health Insights using Large Language Model Agents
Despite the proliferation of wearable health trackers and the importance of sleep and exercise to health, deriving actionable personalized insights from wearable data remains a challenge because doing so requires non-trivial open-ended analysis of these data.
Towards a Personal Health Large Language Model
Here we present Personal Health Large Language Model (PH-LLM), fine-tuned from Gemini for understanding and reasoning over numerical time-series personal health data.
Capabilities of Gemini Models in Medicine
We evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art (SoTA) performance on 10 of them, and surpass the GPT-4 model family on every benchmark where a direct comparison is viable, often by a wide margin.
Ranked #1 on
Question Answering
on MedQA
(using extra training data)
MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making
MDAgents achieved the best performance in seven out of ten benchmarks on tasks requiring an understanding of medical knowledge and multi-modal reasoning, showing a significant improvement of up to 4. 2% (p < 0. 05) compared to previous methods' best performances.
How Suboptimal is Training rPPG Models with Videos and Targets from Different Body Sites?
Using a recently released unique dataset with synchronized contact PPG and video measurements from both the hand and face, we can provide precise and quantitative answers to this question.
On the Standardization of Behavioral Use Clauses and Their Adoption for Responsible Licensing of AI
As of the end of 2023, on the order of 40, 000 software and model repositories have adopted responsible AI licenses licenses.
Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data
Notably, we observe that our context enhancement can yield up to 23. 8% improvement in performance.
Towards Accurate Differential Diagnosis with Large Language Models
Comparing the two assisted study arms, the DDx quality score was higher for clinicians assisted by our LLM (top-10 accuracy 51. 7%) compared to clinicians without its assistance (36. 1%) (McNemar's Test: 45. 7, p < 0. 01) and clinicians with search (44. 4%) (4. 75, p = 0. 03).
From Classification to Clinical Insights: Towards Analyzing and Reasoning About Mobile and Behavioral Health Data With Large Language Models
Passively collected behavioral health data from ubiquitous sensors holds significant promise to provide mental health professionals insights from patient's daily lives; however, developing analysis tools to use this data in clinical practice requires addressing challenges of generalization across devices and weak or ambiguous correlations between the measured signals and an individual's mental health.
Video-based sympathetic arousal assessment via peripheral blood flow estimation
We obtain median correlations of 0. 57 to 0. 63 between our inferred signals and the ground truth EDA using only videos of the participants' palms or foreheads or PPG signals from the foreheads or fingers.
The Capability of Large Language Models to Measure Psychiatric Functioning
The current work investigates the capability of Large language models (LLMs) that are explicitly trained on large corpuses of medical knowledge (Med-PaLM 2) to predict psychiatric functioning from patient interviews and clinical descriptions without being trained to do so.
Large Language Models are Few-Shot Health Learners
Large language models (LLMs) can capture rich representations of concepts that are useful for real-world tasks.
"Can't Take the Pressure?": Examining the Challenges of Blood Pressure Estimation via Pulse Wave Analysis
The use of observed wearable sensor data (e. g., photoplethysmograms [PPG]) to infer health measures (e. g., glucose level or blood pressure) is a very active area of research.
A Review of Deep Learning for Video Captioning
Video captioning (VC) is a fast-moving, cross-disciplinary area of research that bridges work in the fields of computer vision, natural language processing (NLP), linguistics, and human-computer interaction.
Synthetic Data in Healthcare
Synthetic data are becoming a critical tool for building artificially intelligent systems.
BigSmall: Efficient Multi-Task Learning for Disparate Spatial and Temporal Physiological Measurements
As an example, perception occurs at different scales both spatially and temporally, suggesting that the extraction of salient visual information may be made more effective by paying attention to specific features at varying scales.
Motion Matters: Neural Motion Transfer for Better Camera Physiological Measurement
Our findings illustrate the usefulness of motion transfer as a data augmentation technique for improving the generalization of models for camera-based physiological sensing.
MMPD: Multi-Domain Mobile Video Physiology Dataset
Second, most datasets are relatively small and therefore are limited in diversity, both in appearance (e. g., skin tone), behaviors (e. g., motion) and environment (e. g., lighting conditions).
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
SimPer: Simple Self-Supervised Learning of Periodic Targets
From human physiology to environmental evolution, important processes in nature often exhibit meaningful and strong periodic or quasi-periodic changes.
rPPG-Toolbox: Deep Remote PPG Toolbox
Camera-based physiological measurement is a fast growing field of computer vision.
SCAMPS: Synthetics for Camera Measurement of Physiological Signals
The use of cameras and computational algorithms for noninvasive, low-cost and scalable measurement of physiological (e. g., cardiac and pulmonary) vital signs is very attractive.
Federated Remote Physiological Measurement with Imperfect Data
The growing need for technology that supports remote healthcare is being acutely highlighted by an aging population and the COVID-19 pandemic.
MobilePhys: Personalized Mobile Camera-Based Contactless Physiological Sensing
Camera-based contactless photoplethysmography refers to a set of popular techniques for contactless physiological measurement.
Contrastive Learning of Global and Local Video Representations
In this work, we propose to learn video representations that generalize to both the tasks which require global semantic information (e. g., classification) and the tasks that require local fine-grained spatio-temporal information (e. g., localization).
Camera Measurement of Physiological Vital Signs
The need for remote tools for healthcare monitoring has never been more apparent.
RGB Camera-based Physiological Sensing: Challenges and Future Directions
Numerous real-world applications have been driven by the recent algorithmic advancement of artificial intelligence (AI).
Synthetic Data for Multi-Parameter Camera-Based Physiological Sensing
We present systematic experiments showing how physiologically-grounded synthetic data can be used in training camera-based multi-parameter cardiopulmonary sensing.
EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals Measurement
Camera-based physiological measurement is a growing field with neural models providing state-the-art-performance.
Face Detection
Photoplethysmography (PPG) heart rate estimation
Learning Higher-Order Dynamics in Video-Based Cardiac Measurement
Recent developments in camera-based vital sign measurement have shown that cardiac measurements can be recovered with impressive accuracy from videos; however, most of the research has focused on extracting summary statistics such as heart rate.
EfficientPhys: Enabling Simple, Fast, and Accurate Camera-Based Vitals Measurement
Camera-based physiological measurement is a growing field with neural models providing state-the-art-performance.
CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning
The ability to perform causal and counterfactual reasoning are central properties of human intelligence.
Contrastive Learning of Global-Local Video Representations
In this work, we propose to learn video representations that generalize to both the tasks which require global semantic information (e. g., classification) and the tasks that require local fine-grained spatio-temporal information (e. g., localization).
DeepFN: Towards Generalizable Facial Action Unit Recognition with Deep Face Normalization
We show that person-independent models yield significantly lower performance (55% average F1 and accuracy across 40 subjects) than person-dependent models (60. 3%), leading to a generalization gap of 5. 3%.
DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents
Creating presentation materials requires complex multimodal reasoning skills to summarize key concepts and arrange them in a logical and visually pleasing manner.
Contrastive Self-Supervised Learning of Global-Local Audio-Visual Representations
Contrastive self-supervised learning has delivered impressive results in many audio-visual recognition tasks.
The Benefit of Distraction: Denoising Camera-Based Physiological Measurements Using Inverse Attention
The core idea is that the signal of interest is stronger in some pixels ("foreground"), and by selectively focusing computation on these pixels, networks can extract subtle information buried in noise and other sources of corruption.
Advancing Non-Contact Vital Sign Measurement using Synthetic Avatars
Non-contact physiological measurement has the potential to provide low-cost, non-invasive health monitoring.
The Benefit of Distraction: Denoising Remote Vitals Measurements using Inverse Attention
A convolutional attention network is used to learn which regions of a video contain the physiological signal and generate a preliminary estimate.
Spectral Synthesis for Satellite-to-Satellite Translation
These satellites have different vantage points above the earth and different spectral imaging bands resulting in inconsistent imagery from one to another.
MetaPhys: Few-Shot Adaptation for Non-Contact Physiological Measurement
There are large individual differences in physiological processes, making designing personalized health sensing algorithms challenging.
Active Contrastive Learning of Audio-Visual Video Representations
Contrastive learning has been shown to produce generalizable representations of audio and visual data by maximizing the lower bound on the mutual information (MI) between different views of an instance.
Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement
Telehealth and remote health monitoring have become increasingly important during the SARS-CoV-2 pandemic and it is widely expected that this will have a lasting impact on healthcare practices.
Do Facial Expressions Predict Ad Sharing? A Large-Scale Observational Study
But while some actions associated with negative emotion (e. g., lip depressor, associated with sadness) were linked to decreased sharing, others (i. e., nose wrinkles, associated with disgust) were linked to increased sharing.
Modeling Affect-based Intrinsic Rewards for Exploration and Learning
Positive affect has been linked to increased interest, curiosity and satisfaction in human learning.
A Scalable Approach for Facial Action Unit Classifier Training UsingNoisy Data for Pre-Training
We show that pre-training on a large diverse set of noisy data can result in even a simple CNN model improving over the current state-of-the-art DNN architectures. The average F1-score achieved with our proposed method on the DISFA dataset is 0. 60, compared to a previous state-of-the-art of 0. 57.
Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistency
We use this method to transfer emotion from a dataset containing four emotions to a dataset with only a single emotion.
Designing Style Matching Conversational Agents
Advances in machine intelligence have enabled conversational interfaces that have the potential to radically change the way humans interact with machines.
A High-Fidelity Open Embodied Avatar with Lip Syncing and Expression Capabilities
Embodied avatars as virtual agents have many applications and provide benefits over disembodied agents, allowing non-verbal social and interactional cues to be leveraged, in a similar manner to how humans interact with each other.
Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck
We propose a multimodal information bottleneck approach that learns the correspondence between modalities from unpaired data (image and speech) by leveraging the shared modality (text).
M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention
Generative adversarial networks have led to significant advances in cross-modal/domain translation.
Characterizing Bias in Classifiers using Generative Models
Models that are learned from real-world data are often biased because the data used to train them is biased.
Visceral Machines: Reinforcement Learning with Intrinsic Physiological Rewards
The human autonomic nervous system has evolved over millions of years and is essential for survival and responding to threats.
Neural TTS Stylization with Adversarial and Collaborative Games
The synthesized audio waveform is expected to contain the verbal content of x_txt and the auditory style of x_aud.
iPhys: An Open Non-Contact Imaging-Based Physiological Measurement Toolbox
We present an open source imaging-based physiological measurement toolbox with implementations of many of the most frequently employed computational methods.
Identifying Bias in AI using Simulation
Machine learned models exhibit bias, often because the datasets used to train them are biased.
DeepMag: Source Specific Motion Magnification Using Gradient Ascent
Through systematic quantitative and qualitative evaluation of the approach on videos with different levels of head motion, we compare the magnification of pulse and respiration to existing state-of-the-art methods.
Visceral Machines: Risk-Aversion in Reinforcement Learning with Intrinsic Physiological Rewards
As people learn to navigate the world, autonomic nervous system (e. g., "fight or flight") responses provide intrinsic feedback about the potential consequence of action choices (e. g., becoming nervous when close to a cliff edge or driving fast around a bend.)
DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks
Non-contact video-based physiological measurement has many applications in health care and human-computer interaction.
