Search Results for author:
Found 92 papers, 30 papers with code
Harmony4D: A Video Dataset for In-The-Wild Close Human Interactions
Understanding how humans interact with each other is key to building realistic multi-human virtual reality systems.
Human Action Anticipation: A Survey
Predicting future human behavior is an increasingly popular topic in computer vision, driven by the interest in applications such as autonomous vehicles, digital assistants and human-robot interactions.
Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos
Goal-oriented planning, or anticipating a series of actions that transition an agent from its current state to a predefined objective, is crucial for developing intelligent assistants aiding users in daily procedural tasks.
Multi-Modal Diffusion for Hand-Object Grasp Generation
Compared to previous works of generating hand poses with a given object, we aim to allow the generalization of both hand and object shapes by a single model.
Unlocking Exocentric Video-Language Data for Egocentric Video Representation Learning
We present EMBED (Egocentric Models Built with Exocentric Data), a method designed to transform exocentric video-language data for egocentric video representation learning.
ExpertAF: Expert Actionable Feedback from Video
Our method takes a video demonstration and its accompanying 3D body pose and generates (1) free-form expert commentary describing what the person is doing well and what they could improve, and (2) a visual expert demonstration that incorporates the required corrections.
DegustaBot: Zero-Shot Visual Preference Estimation for Personalized Multi-Object Rearrangement
De gustibus non est disputandum ("there is no accounting for others' tastes") is a common Latin maxim describing how many solutions in life are determined by people's personal preferences.
SMPLOlympics: Sports Environments for Physically Simulated Humanoids
We present SMPLOlympics, a collection of physically simulated environments that allow humanoids to compete in a variety of Olympic sports.
OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy.
Generalizable Neural Human Renderer
While recent advancements in animatable human rendering have achieved remarkable results, they require test-time optimization for each subject which can be a significant limitation for real-world applications.
G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesis
We propose G-HOP, a denoising diffusion based generative prior for hand-object interactions that allows modeling both the 3D object and a human hand, conditioned on the object category.
Bootstrapping Linear Models for Fast Online Adaptation in Human-Agent Collaboration
Initializing policies to maximize performance with unknown partners can be achieved by bootstrapping nonlinear models using imitation learning over large, offline datasets.
Zero-Shot Multi-Object Scene Completion
We present a 3D scene completion method that recovers the complete geometry of multiple unseen objects in complex scenes from a single RGB-D image.
Real-Time Simulated Avatar from Head-Mounted Sensors
We present SimXR, a method for controlling a simulated avatar from information (headset pose and cameras) obtained from AR / VR headsets.
Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation
We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time whole-body teleoperation of a full-sized humanoid robot with only an RGB camera.
Multi-Object Tracking by Hierarchical Visual Representations
We propose a new visual hierarchical representation paradigm for multi-object tracking.
Mixed Gaussian Flow for Diverse Trajectory Prediction
However, mapping from a standard Gaussian by a flow-based model hurts the capacity to capture complicated patterns of trajectories, ignoring the under-represented motion intentions in the training data.
Multi-Person 3D Pose Estimation from Multi-View Uncalibrated Depth Cameras
In order to evaluate our proposed pipeline, we collect three video sets of RGBD videos recorded from multiple sparse-view depth cameras and ground truth 3D poses are manually annotated.
Flexible Depth Completion for Sparse and Varying Point Densities
While recent depth completion methods have achieved remarkable results filling in relatively dense depth maps (e. g. projected 64-line LiDAR on KITTI or 500 sampled points on NYUv2) with RGB guidance their performance on very sparse input (e. g. 4-line LiDAR or 32 depth point measurements) is unverified.
Multi-View Person Matching and 3D Pose Estimation with Arbitrary Uncalibrated Camera Networks
The 2D human poses used in clustering are obtained through a pre-trained 2D pose detector, so our method does not require expensive 3D training data for each new scene.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
Universal Humanoid Motion Representations for Physics-Based Control
We close this gap by significantly increasing the coverage of our motion representation space.
TEMPO: Efficient Multi-View Pose Estimation, Tracking, and Forecasting
In doing so, our model is able to use spatiotemporal context to predict more accurate human poses without sacrificing efficiency.
EgoHumans: An Egocentric 3D Multi-Human Benchmark
We present EgoHumans, a new multi-view multi-human video benchmark to advance the state-of-the-art of egocentric human 3D pose estimation and tracking.
Type-to-Track: Retrieve Any Object via Prompt-based Tracking
This paper introduces a novel paradigm for Multiple Object Tracking called Type-to-Track, which allows users to track objects in videos by typing natural language descriptions.
Grounded Multiple Object Tracking
Multiple Object Tracking
+1
Joint Metrics Matter: A Better Standard for Trajectory Forecasting
In response to the limitations of marginal metrics, we present the first comprehensive evaluation of state-of-the-art (SOTA) trajectory forecasting methods with respect to multi-agent metrics (joint metrics): JADE, JFDE, and collision rate.
Perpetual Humanoid Control for Real-time Simulated Avatars
We present a physics-based humanoid controller that achieves high-fidelity motion imitation and fault-tolerant behavior in the presence of noisy input (e. g. pose estimates from video or generated from language) and unexpected falls.
Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion
We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet user-defined goals.
3D-CLFusion: Fast Text-to-3D Rendering with Contrastive Latent Diffusion
We note that the novelty of our model lies in that we introduce contrastive learning during training the diffusion prior which enables the generation of the valid view-invariant latent code.
Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification
Motion-based association for Multi-Object Tracking (MOT) has recently re-achieved prominence with the rise of powerful object detectors.
Ranked #5 on
Multi-Object Tracking
on MOT20
(using extra training data)
Ego-Humans: An Ego-Centric 3D Multi-Human Benchmark
We present EgoHumans, a new multi-view multi-human video benchmark to advance the state-of-the-art of egocentric human 3D pose estimation and tracking.
Azimuth Super-Resolution for FMCW Radar in Autonomous Driving
We also propose a hybrid super-resolution model (Hybrid-SR) combining our ADC-SR with a standard RAD super-resolution model, and show that performance can be improved by a large margin.
Learnable Spatio-Temporal Map Embeddings for Deep Inertial Localization
To address the robustness problem in map utilization, we propose a data-driven prior on possible user locations in a map by combining learned spatial map embeddings and temporal odometry embeddings.
3D-Aware Encoding for Style-based Neural Radiance Fields
In the first stage, we introduce a base encoder that converts the input image to a latent code.
Track Targets by Dense Spatio-Temporal Position Encoding
Experiments on video multi-object tracking (MOT) and multi-object tracking and segmentation (MOTS) datasets demonstrate the effectiveness of the proposed DST position encoding.
Multi-Object Tracking
Multi-Object Tracking and Segmentation
+2
Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection
While recent camera-only 3D detection methods leverage multiple timesteps, the limited history they use significantly hampers the extent to which temporal fusion can improve object perception.
Ranked #1 on
Robust Camera Only 3D Object Detection
on nuScenes-C
Embodied Scene-aware Human Pose Estimation
Since 2D third-person observations are coupled with the camera pose, we propose to disentangle the camera pose and use a multi-step projection gradient defined in the global coordinate frame as the movement cue for our embodied agent.
Ranked #324 on
3D Human Pose Estimation
on Human3.6M
Online No-regret Model-Based Meta RL for Personalized Navigation
Our theoretical analysis shows that our method is a no-regret algorithm and we provide the convergence rate in the agnostic setting.
Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking
Instead of relying only on the linear state estimate (i. e., estimation-centric approach), we use object observations (i. e., the measurements by object detector) to compute a virtual trajectory over the occlusion period to fix the error accumulation of filter parameters during the occlusion period.
Ranked #2 on
Multiple Object Tracking
on CroHD
Occluded Human Mesh Recovery
Along with the input image, we condition the top-down model on spatial context from the image in the form of body-center heatmaps.
Ranked #72 on
3D Human Pose Estimation
on 3DPW
(using extra training data)
Whose Track Is It Anyway? Improving Robustness to Tracking Errors With Affinity-Based Trajectory Prediction
This is typically caused by the propagation of errors from tracking to prediction, such as noisy tracks, fragments, and identity switches.
Modality-Agnostic Learning for Radar-Lidar Fusion in Vehicle Detection
To mitigate this problem, we propose the Self-Training Multimodal Vehicle Detection Network (ST-MVDNet) which leverages a Teacher-Student mutual learning framework and a simulated sensor noise model used in strong data augmentation for Lidar and Radar.
GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras
Since the joint reconstruction of human motions and camera poses is underconstrained, we propose a global trajectory predictor that generates global human trajectories based on local body movements.
Ranked #1 on
Global 3D Human Pose Estimation
on EMDB
DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion
A typical pipeline for multi-object tracking (MOT) is to use a detector for object localization, and following re-identification (re-ID) for object association.
Cross-Domain Adaptive Teacher for Object Detection
To mitigate this problem, we propose a teacher-student framework named Adaptive Teacher (AT) which leverages domain adversarial learning and weak-strong data augmentation to address the domain gap.
AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation
In this paper, we make an attempt to simulate that ability of a human by proposing Actor Environment Interaction (AEI) network to improve the video representation for temporal action proposals generation.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design
Specifically, we learn a conditional policy that, in an episode, first applies a sequence of transform actions to modify an agent's skeletal structure and joint attributes, and then applies control actions under the new design.
Multi-Echo LiDAR for 3D Object Detection
LiDAR sensors can be used to obtain a wide range of measurement signals other than a simple 3D point cloud, and those signals can be leveraged to improve perception tasks like 3D object detection.
Multi-Modality Task Cascade for 3D Object Detection
To provide a more integrated approach, we propose a novel Multi-Modality Task Cascade network (MTC-RCNN) that leverages 3D box proposals to improve 2D segmentation predictions, which are then used to further refine the 3D boxes.
Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation
By comparing the pose instructed by the kinematic model against the pose generated by the dynamics model, we can use their misalignment to further improve the kinematic model.
Egocentric Pose Estimation
Human-Object Interaction Detection
+2
Wide-Baseline Multi-Camera Calibration using Person Re-Identification
We address the problem of estimating the 3D pose of a network of cameras for large-environment wide-baseline scenarios, e. g., cameras for construction sites, sports stadiums, and public spaces.
SimPoE: Simulated Character Control for 3D Human Pose Estimation
Based on this refined kinematic pose, the policy learns to compute dynamics-based control (e. g., joint torques) of the character to advance the current-frame pose estimate to the pose estimate of the next frame.
Ranked #246 on
3D Human Pose Estimation
on Human3.6M
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting
Instead, we would prefer a method that allows an agent's state at one time to directly affect another agent's state at a future time.
Ranked #10 on
Trajectory Prediction
on ETH/UCY
Efficient Model Performance Estimation via Feature Histories
Using a sampling-based search algorithm and parallel computing, our method can find an architecture which is better than DARTS and with an 80% reduction in wall-clock search time.
Inverse Reinforcement Learning with Explicit Policy Estimates
Various methods for solving the inverse reinforcement learning (IRL) problem have been developed independently in machine learning and economics.
DeepBLE: Generalizing RSSI-based Localization Across Different Devices
Accurate smartphone localization (< 1-meter error) for indoor navigation using only RSSI received from a set of BLE beacons remains a challenging problem, due to the inherent noise of RSSI measurements.
IDOL: Inertial Deep Orientation-Estimation and Localization
Many smartphone applications use inertial measurement units (IMUs) to sense movement, but the use of these sensors for pedestrian localization can be challenging due to their noise characteristics.
AutoSelect: Automatic and Dynamic Detection Selection for 3D Multi-Object Tracking
Also, this threshold is sensitive to many factors such as target object category so we need to re-search the threshold if these factors change.
Rethinking Transformer-based Set Prediction for Object Detection
DETR is a recently proposed Transformer-based method which views object detection as a set prediction problem and achieves state-of-the-art performance but demands extra-long training time to converge.
Audio-Visual Self-Supervised Terrain Type Discovery for Mobile Platforms
The ability to both recognize and discover terrain characteristics is an important function required for many autonomous ground robots such as social robots, assistive robots, autonomous vehicles, and ground exploration robots.
End-to-End 3D Multi-Object Tracking and Trajectory Forecasting
To evaluate this hypothesis, we propose a unified solution for 3D MOT and trajectory forecasting which also incorporates two additional novel computational units.
Few-Shot Learning with Intra-Class Knowledge Transfer
Recent works have proposed to solve this task by augmenting the training data of the few-shot classes using generative models with the few-shot training samples as the seeds.
Graph Neural Networks for 3D Multi-Object Tracking
3D Multi-object tracking (MOT) is crucial to autonomous systems.
AB3DMOT: A Baseline for 3D Multi-Object Tracking and New Evaluation Metrics
Additionally, 3D MOT datasets such as KITTI evaluate MOT methods in 2D space and standardized 3D MOT evaluation tools are missing for a fair comparison of 3D MOT methods.
Efficient Non-Line-of-Sight Imaging from Transient Sinograms
Non-line-of-sight (NLOS) imaging techniques use light that diffusely reflects off of visible surfaces (e. g., walls) to see around corners.
Improving Lesion Segmentation for Diabetic Retinopathy using Adversarial Learning
Diabetic Retinopathy (DR) is a leading cause of blindness in working age adults.
Joint Object Detection and Multi-Object Tracking with Graph Neural Networks
Despite the fact that the two components are dependent on each other, prior works often design detection and data association modules separately which are trained with separate objectives.
Ranked #1 on
Multi-Object Tracking
on 2D MOT 2015
When We First Met: Visual-Inertial Person Localization for Co-Robot Rendezvous
We propose a method to learn a visual-inertial feature space in which the motion of a person in video can be easily matched to the motion measured by a wearable inertial measurement unit (IMU).
GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning
As a result, the feature of one object is informed of the features of other objects so that the object feature can lean towards the object with similar feature (i. e., object probably with a same ID) and deviate from objects with dissimilar features (i. e., object probably with different IDs), leading to a more discriminative feature for each object; (2) instead of obtaining the feature from either 2D or 3D space in prior work, we propose a novel joint feature extractor to learn appearance and motion features from 2D and 3D space simultaneously.
Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis
Our approach is the first humanoid control method that successfully learns from a large-scale human motion dataset (Human3. 6M) and generates diverse long-term motions.
No-Reference Image Quality Assessment via Feature Fusion and Multi-Task Learning
In our experiments, we demonstrate that by utilizing multi-task learning and our proposed feature fusion method, our model yields better performance for the NR-IQA task.
Optical Non-Line-of-Sight Physics-based 3D Human Pose Estimation
We bring together a diverse set of technologies from NLOS imaging, human pose estimation and deep reinforcement learning to construct an end-to-end data processing pipeline that converts a raw stream of photon measurements into a full 3D human pose sequence estimate.
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction
To obtain samples from a pretrained generative model, most existing generative human motion prediction methods draw a set of independent Gaussian latent codes and convert them to motion samples.
Ranked #1 on
Human Pose Forecasting
on AMASS
(APD metric)
Inverting the Pose Forecasting Pipeline with SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting
Through experiments on a robotic manipulation dataset and two driving datasets, we show that SPFNet is effective for the SPF task, our forecast-then-detect pipeline outperforms the detect-then-forecast approaches to which we compared, and that pose forecasting performance improves with the addition of unlabeled data.
PTP: Parallelized Tracking and Prediction with Graph Neural Networks and Diversity Sampling
We evaluate on KITTI and nuScenes datasets showing that our method with socially-aware feature learning and diversity sampling achieves new state-of-the-art performance on 3D MOT and trajectory prediction.
Estimating 3D Camera Pose from 2D Pedestrian Trajectories
We propose an alternative strategy for extracting 3D information to solve for camera pose by using pedestrian trajectories.
Incremental Class Discovery for Semantic Segmentation with RGBD Sensing
This work addresses the task of open world semantic segmentation using RGBD sensing to discover new semantic classes over time.
Diverse Trajectory Forecasting with Determinantal Point Processes
To learn the parameters of the DSF, the diversity of the trajectory samples is evaluated by a diversity loss based on a determinantal point process (DPP).
Ranked #5 on
Human Pose Forecasting
on HumanEva-I
3D Multi-Object Tracking: A Baseline and New Evaluation Metrics
Additionally, 3D MOT datasets such as KITTI evaluate MOT methods in the 2D space and standardized 3D MOT evaluation tools are missing for a fair comparison of 3D MOT methods.
Ranked #16 on
Multiple Object Tracking
on KITTI Test (Online Methods)
Ego-Pose Estimation and Forecasting as Real-Time PD Control
We propose the use of a proportional-derivative (PD) control based policy learned via reinforcement learning (RL) to estimate and forecast 3D human pose from egocentric videos.
Doctor of Crosswise: Reducing Over-parametrization in Neural Networks
Dr. of Crosswise proposes a new architecture to reduce over-parametrization in Neural Networks.
Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading
We evaluate different combinations of front-end and back-end modules with the grayscale video and optical flow inputs on the LRW dataset.
PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings
For autonomous vehicles (AVs) to behave appropriately on roads populated by human-driven vehicles, they must be able to reason about the uncertain intentions and decisions of other drivers from rich perceptual information.
Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud
Following the pipeline of two-stage 3D detection algorithms, we detect 2D object proposals in the input image and extract a point cloud frustum from the pseudo-LiDAR for each proposal.
MGpi: A Computational Model of Multiagent Group Perception and Interaction
Toward enabling next-generation robots capable of socially intelligent interaction with humans, we present a $\mathbf{computational\; model}$ of interactions in a social environment of multiple agents and multiple groups.
GroundNet: Monocular Ground Plane Normal Estimation with Geometric Consistency
We focus on estimating the 3D orientation of the ground plane from a single image.
3D Ego-Pose Estimation via Imitation Learning
Motivated by this, we propose a novel control-based approach to model human motion with physics simulation and use imitation learning to learn a video-conditioned control policy for ego-pose estimation.
Understanding hand-object manipulation by modeling the contextual relationship between actions, grasp types and object attributes
In the proposed model, we explore various semantic relationships between actions, grasp types and object attributes, and show how the context can be used to boost the recognition of each component.
Personalized Dynamics Models for Adaptive Assistive Navigation Systems
Consider an assistive system that guides visually impaired users through speech and haptic feedback to their destination.
Model-based Reinforcement Learning
Reinforcement Learning
+1
Rotational Rectification Network: Enabling Pedestrian Detection for Mobile Vision
To address this issue, we propose a Rotational Rectification Network (R2N) that can be inserted into any CNN-based pedestrian (or object) detector to adapt it to significant changes in camera rotation.
Visual Compiler: Synthesizing a Scene-Specific Pedestrian Detector and Pose Estimator
We introduce the concept of a Visual Compiler that generates a scene specific pedestrian detector and pose estimator without any pedestrian observations.
