Search Results for author:
Found 89 papers, 37 papers with code
Recent Advances, Applications and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2024 Symposium
The fourth Machine Learning for Health (ML4H) symposium was held in person on December 15th and 16th, 2024, in the traditional, ancestral, and unceded territories of the Musqueam, Squamish, and Tsleil-Waututh Nations in Vancouver, British Columbia, Canada.
Pulse-PPG: An Open-Source Field-Trained PPG Foundation Model for Wearable Applications Across Lab and Field Settings
Photoplethysmography (PPG)-based foundation models are gaining traction due to the widespread use of PPG in biosignal monitoring and their potential to generalize across diverse health applications.
SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images
Generative methods handle uncertain regions better by modeling distributions, but are computationally expensive and the generation is often misaligned with visible surfaces.
Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
We address the problem of gaze target estimation, which aims to predict where a person is looking in a scene.
PyPulse: A Python Library for Biosignal Imputation
We introduce PyPulse, a Python package for imputation of biosignals in both clinical and wearable sensor settings.
Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation
We also introduce a relation regularization method to further disentangle image transformation features from irrelevant contents in exemplar images.
Optimization-Free Image Immunization Against Diffusion-Based Editing
Extensive qualitative and quantitative results demonstrate that our model is scalable, optimization-free, adaptable to various diffusion-based editing tools, robust against counter-attacks, and, for the first time, effectively protects video content from editing.
RelCon: Relative Contrastive Learning for a Motion Foundation Model for Wearable Data
We present RelCon, a novel self-supervised *Rel*ative *Con*trastive learning approach that uses a learnable distance measure in combination with a softened contrastive loss for training an motion foundation model from wearable sensors.
Symmetry Strikes Back: From Single-Image Symmetry Detection to 3D Generation
Symmetry is a ubiquitous and fundamental property in the visual world, serving as a critical cue for perception and structure interpretation.
Medical Video Generation for Disease Progression Simulation
To address this challenge, we propose the first Medical Video Generation (MVG) framework that enables controlled manipulation of disease-related image and video features, allowing precise, realistic, and personalized simulations of disease progression.
Human Action Anticipation: A Survey
Predicting future human behavior is an increasingly popular topic in computer vision, driven by the interest in applications such as autonomous vehicles, digital assistants and human-robot interactions.
Leveraging Object Priors for Point Tracking
Point tracking is a fundamental problem in computer vision with numerous applications in AR and robotics.
3x2: 3D Object Part Segmentation by 2D Semantic Correspondences
While substantial progress has been made in 2D object part segmentation, the 3D counterpart has received less attention, in part due to the scarcity of annotated 3D datasets, which are expensive to collect.
Temporally Multi-Scale Sparse Self-Attention for Physical Activity Data Imputation
Wearable sensors enable health researchers to continuously collect data pertaining to the physiological state of individuals in real-world settings.
MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs
Our findings illuminate the persistence of the reliance on spurious correlations from these models and underscore the urge for new methodologies to mitigate spurious biases.
What is the Visual Cognition Gap between Humans and Multimodal LLMs?
Recently, Multimodal Large Language Models (MLLMs) have shown great promise in language-guided perceptual tasks such as recognition, segmentation, and object detection.
PointInfinity: Resolution-Invariant Point Diffusion Models
We present PointInfinity, an efficient family of point cloud diffusion models.
Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations
Furthermore, we propose a novel multimodal baseline that leverages densely aligned language-visual representations by synchronizing visual features with their corresponding utterances.
MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding
Specifically we annotate and leverage large-scale broad-coverage traffic and map data extracted from huge HD map annotations and use CLIP and LLaMA-2 / Vicuna to finetune a baseline model with instruction-following data.
ZeroShape: Regression-based Zero-shot Shape Reconstruction
In contrast, the traditional approach to this problem is regression-based, where deterministic models are trained to directly regress the object shape.
The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective
We propose a unified multi-modal framework -- Audio-Visual Conversational Attention (AV-CONV), for the joint prediction of conversation behaviors -- speaking and listening -- for both the camera wearer as well as all other social partners present in the egocentric video.
LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs
Autonomous driving (AD) has made significant strides in recent years.
RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models
Recent advancements in diffusion-based models have demonstrated significant success in generating images from text.
LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning
Additionally, existing diffusion-based image manipulation models are sub-optimal in controlling the state transition of an action in egocentric image pixel space because of the domain gap.
Low-shot Object Learning with Mutual Exclusivity Bias
This paper introduces Low-shot Object Learning with Mutual Exclusivity Bias (LSME), the first computational framing of mutual exclusivity bias, a phenomenon commonly observed in infants during word learning.
Which way is `right'?: Uncovering limitations of Vision-and-Language Navigation model
The challenging task of Vision-and-Language Navigation (VLN) requires embodied agents to follow natural language instructions to reach a goal location or object (e. g. `walk down the hallway and turn left at the piano').
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning
The success of self-supervised contrastive learning hinges on identifying positive data pairs, such that when they are pushed together in embedding space, the space encodes useful information for subsequent downstream tasks.
Explaining a machine learning decision to physicians via counterfactuals
Machine learning models perform well on several healthcare tasks and can help reduce the burden on the healthcare system.
Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation
Motivated by this observation, we introduce the first model that leverages both the video and audio modalities for egocentric gaze anticipation.
ShapeClipper: Scalable 3D Shape Learning from Single-View Images via Geometric and CLIP-based Consistency
We present ShapeClipper, a novel method that reconstructs 3D object shapes from real-world single-view RGB images.
Egocentric Auditory Attention Localization in Conversations
In a noisy conversation environment such as a dinner party, people often exhibit selective auditory attention, or the ability to focus on a particular speaker while tuning out others.
Werewolf Among Us: A Multimodal Dataset for Modeling Persuasion Behaviors in Social Deduction Games
We also explore the generalization ability of language models for persuasion modeling and the role of persuasion strategies in predicting social deduction game outcomes.
PulseImpute: A Novel Benchmark Task for Pulsative Physiological Signal Imputation
The promise of Mobile Health (mHealth) is the ability to use wearable sensors to monitor participant physiology at high frequencies during daily life to enable temporally-precise health interventions.
Learning Dense Object Descriptors from Multiple Views for Low-shot Category Generalization
A hallmark of the deep learning era for computer vision is the successful use of large-scale labeled datasets to train feature representations for tasks ranging from object recognition and semantic segmentation to optical flow estimation and novel view synthesis of 3D scenes.
Transformer-based Localization from Embodied Dialog with Large-scale Pre-training
We address the challenging task of Localization via Embodied Dialog (LED).
In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation
To this end, we design the transformer encoder to embed the global context as one additional visual token and further propose a novel Global-Local Correlation (GLC) module to explicitly model the correlation of the global token and each local token.
Planes vs. Chairs: Category-guided 3D shape learning without any 3D cues
We present a novel 3D shape reconstruction method which learns to predict an implicit 3D shape representation from a single RGB image.
Generative Adversarial Network for Future Hand Segmentation from Egocentric Video
We introduce the novel problem of anticipating a time series of future hand masks from egocentric video.
Kernel Deformed Exponential Families for Sparse Continuous Attention
Theoretically, we show new existence results for both kernel exponential and deformed exponential families, and that the deformed case has similar approximation capabilities to kernel exponential families.
Transformers for prompt-level EMA non-response prediction
The transformer model achieves a non-response prediction AUC of 0. 77 and is significantly better than classical ML and LSTM-based deep learning models.
Efficient Learning and Decoding of the Continuous-Time Hidden Markov Model for Disease Progression Modeling
We solve the first challenge by reformulating the estimation problem as an equivalent discrete time-inhomogeneous hidden Markov model.
No RL, No Simulation: Learning to Navigate without Navigating
Most prior methods for learning navigation policies require access to simulation environments, as they need online policy interaction and rely on ground-truth maps for rewards.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Egocentric Activity Recognition and Localization on a 3D Map
Given a video captured from a first person perspective and the environment context of where the video is recorded, can we recognize what the person is doing and identify where the action occurs in the 3D space?
Discriminative Appearance Modeling with Multi-track Pooling for Real-time Multi-object Tracking
Many approaches model each target in isolation and lack the ability to use all the targets in the scene to jointly update the memory.
The Surprising Positive Knowledge Transfer in Continual 3D Object Shape Reconstruction
Continual learning has been extensively studied for classification tasks with methods developed to primarily avoid catastrophic forgetting, a phenomenon where earlier learned concepts are forgotten at the expense of more recent samples.
Using Shape to Categorize: Low-Shot Learning with an Explicit Shape Bias
It is widely accepted that reasoning about object shape is important for object recognition.
4D Human Body Capture from Egocentric Video via 3D Scene Grounding
We introduce a novel task of reconstructing a time series of second-person 3D human body meshes from monocular egocentric videos.
Where Are You? Localization from Embodied Dialog
In this paper, we focus on the LED task -- providing a strong baseline model with detailed ablations characterizing both dataset biases and the importance of various modeling choices.
3D Reconstruction of Novel Object Shapes from Single Images
This is challenging as it requires a model to learn a representation that can infer both the visible and occluded portions of any object using a limited training set.
In the Eye of the Beholder: Gaze and Actions in First Person Video
Moving beyond the dataset, we propose a novel deep model for joint gaze estimation and action recognition in FPV.
Approximate Inverse Reinforcement Learning from Vision-based Imitation Learning
In this work, we present a method for obtaining an implicit objective function for vision-based navigation.
Orthogonal Over-Parameterized Training
The inductive bias of a neural network is largely determined by the architecture and the training algorithm.
Detecting Attended Visual Targets in Video
We address the problem of detecting attention targets in video.
A Robust Functional EM Algorithm for Incomplete Panel Count Data
Panel count data describes aggregated counts of recurrent events observed at discrete time points.
Neural Similarity Learning
By generalizing inner product with a bilinear matrix, we propose the neural similarity which serves as a learnable parametric similarity measure for CNNs.
Regularizing Neural Networks via Minimizing Hyperspherical Energy
Inspired by the Thomson problem in physics where the distribution of multiple propelling electrons on a unit sphere can be modeled via minimizing some potential energy, hyperspherical energy minimization has demonstrated its potential in regularizing neural networks and improving their generalization power.
Locally Weighted Regression Pseudo-Rehearsal for Online Learning of Vehicle Dynamics
We consider the problem of online adaptation of a neural network designed to represent vehicle dynamics.
Tripping through time: Efficient Localization of Activities in Videos
Localizing moments in untrimmed videos via language queries is a new and interesting task that requires the ability to accurately ground language into video.
Learning to Generate Synthetic Data via Compositing
The synthesizer and target networks are trained in an adversarial manner wherein each network is updated with a goal to outdo the other.
Unsupervised 3D Pose Estimation with Geometric Self-Supervision
Additionally, to learn from 2D poses "in the wild", we train an unsupervised 2D domain adapter network to allow for an expansion of 2D data.
Ranked #78 on
3D Human Pose Estimation
on MPI-INF-3DHP
(AUC metric)
Attention Distillation for Learning Video Representations
To this end, we make use of attention modules that learn to highlight regions in the video and aggregate features for recognition.
Ranked #41 on
Action Recognition
on UCF101
Action2Vec: A Crossmodal Embedding Approach to Action Learning
We describe a novel cross-modal embedding space for actions, named Action2Vec, which combines linguistic cues from class labels with spatio-temporal features derived from video clips.
Taking a Deeper Look at the Inverse Compositional Algorithm
In this paper, we provide a modern synthesis of the classic inverse compositional algorithm for dense image alignment.
Learning to Localize and Align Fine-Grained Actions to Sparse Instructions
Automatic generation of textual video descriptions that are time-aligned with video content is a long-standing goal in computer vision.
Multi-object Tracking with Neural Gating Using Bilinear LSTM
We also propose novel data augmentation approaches to efficiently train recurrent models that score object tracks on both appearance and motion.
In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video
We address the task of jointly determining what a person is doing and where they are looking based on the analysis of video captured by a headworn camera.
3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare
Our method produces a compact 3D representation of the scene, which can be readily used for applications like autonomous driving.
Ranked #3 on
Vehicle Pose Estimation
on KITTI Cars Hard
(using extra training data)
Decoupled Networks
Inner product-based convolution has been a central component of convolutional neural networks (CNNs) and the key to learning visual representations.
Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
Estimation of 3D motion in a dynamic scene from a temporal pair of images is a core task in many scene understanding problems.
Towards Black-box Iterative Machine Teaching
We propose an active teacher model that can actively query the learner (i. e., make the learner take exams) for estimating the learner's status and provably guide the learner to achieve faster convergence.
Fine-Grained Head Pose Estimation Without Keypoints
Estimating the head pose of a person is a crucial problem that has a large amount of applications such as aiding in gaze estimation, modeling attention, fitting 3D models to video and performing face alignment.
Ranked #5 on
Head Pose Estimation
on AFLW
Dockerface: an Easy to Install and Use Faster R-CNN Face Detector in a Docker Container
Face detection is a very important task and a necessary pre-processing step for many applications such as facial landmark detection, pose estimation, sentiment analysis and face recognition.
iSurvive: An Interpretable, Event-time Prediction Model for mHealth
We present a parameter learning method for GLM emissions and survival model fitting, and present promising results on both synthetic data and an mHealth drug use dataset.
Information Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving
We present an information theoretic approach to stochastic optimal control problems that can be used to derive general sampling based optimization schemes.
Robotics
Iterative Machine Teaching
Different from traditional machine teaching which views the learners as batch algorithms, we study a new paradigm where the learner uses an iterative algorithm and a teacher can feed examples sequentially and intelligently based on the current performance of the learner.
Automatic Variational ABC
Approximate Bayesian Computation (ABC) is a framework for performing likelihood-free posterior inference for simulation models.
The Middle Child Problem: Revisiting Parametric Min-Cut and Seeds for Object Proposals
We propose a new energy minimization framework incorporating geodesic distances between segments which solves this problem.
Efficient Learning of Continuous-Time Hidden Markov Models for Disease Progression
The Continuous-Time Hidden Markov Model (CT-HMM) is an attractive approach to modeling disease progression due to its ability to describe noisy observations arriving irregularly in time.
Minimizing Human Effort in Interactive Tracking by Incremental Learning of Model Parameters
We address the problem of minimizing human effort in interactive tracking by learning sequence-specific model parameters.
Multiple Hypothesis Tracking Revisited
This paper revisits the classical multiple hypotheses tracking (MHT) algorithm in a tracking-by-detection framework.
Unsupervised Learning of Edges
In this work we present a simple yet effective approach for training edge detectors without human supervision.
Gaze-Enabled Egocentric Video Summarization via Constrained Submodular Maximization
Motivated by these applications, this paper focuses on the problem of egocentric video summarization.
Delving Into Egocentric Actions
We propose to utilize these mid-level egocentric cues for egocentric action recognition.
Robust Video Segment Proposals With Painless Occlusion Handling
We propose a robust algorithm to generate video segment proposals.
The Secrets of Salient Object Segmentation
The dataset design bias does not only create the discomforting disconnection between fixations and salient object segmentation, but also misleads the algorithm designing.
RIGOR: Reusing Inference in Graph Cuts for Generating Object Regions
By precomputing a graph which can be used for parametric min-cuts over different seeds, we speed up the generation of the segment pool.
Modeling Actions through State Changes
The key to differentiating these actions is the ability to identify how they change the state of objects and materials in the environment.
