Search Results for author:
Found 80 papers, 16 papers with code
Applications of fractional calculus in learned optimization
Fractional gradient descent has been studied extensively, with a focus on its ability to extend traditional gradient descent methods by incorporating fractional-order derivatives.
Learned Neural Physics Simulation for Articulated 3D Human Pose Reconstruction
We propose a novel neural network approach, LARP (Learned Articulated Rigid body Physics), to model the dynamics of articulated human motion with contact.
Instant 3D Human Avatar Generation using Image Diffusion Models
We present AvatarPopUp, a method for fast, high quality 3D human avatar generation from different input modalities, such as images and text prompts and with control over the generated pose and shape.
VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis
We propose VLOGGER, a method for audio-driven human video generation from a single input image of a person, which builds on the success of recent generative diffusion models.
Score Distillation Sampling with Learned Manifold Corrective
Score Distillation Sampling (SDS) is a recent but already widely popular method that relies on an image diffusion model to control optimization problems using text prompts.
DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans
Our experiments show that DiffHuman can produce diverse and detailed reconstructions for the parts of the person that are unseen or uncertain in the input image while remaining competitive with the state-of-the-art when reconstructing visible surfaces.
Personalized Pose Forecasting
Human pose forecasting is the task of predicting articulated human motion given past human motion.
SPHEAR: Spherical Head Registration for Complete Statistical 3D Modeling
We present \emph{SPHEAR}, an accurate, differentiable parametric statistical 3D human head model, enabled by a novel 3D registration method based on spherical embeddings.
Blendshapes GHUM: Real-time Monocular Facial Blendshape Prediction
We present Blendshapes GHUM, an on-device ML pipeline that predicts 52 facial blendshape coefficients at 30+ FPS on modern mobile phones, from a single monocular RGB image and enables facial motion capture applications like virtual avatars.
Reconstructing Three-Dimensional Models of Interacting Humans
Understanding 3d human interactions is fundamental for fine-grained scene analysis and behavioural modeling.
HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving
Autonomous driving is an exciting new industry, posing important research questions.
PhoMoH: Implicit Photorealistic 3D Models of Human Heads
We present PhoMoH, a neural network methodology to construct generative models of photo-realistic 3D geometry and appearance of human heads including hair, beards, an oral cavity, and clothing.
Structured 3D Features for Reconstructing Controllable Avatars
We introduce Structured 3D Features, a model based on a novel implicit 3D representation that pools pixel-aligned image features onto dense 3D points sampled from a parametric, statistical human mesh surface.
Transformer-Based Learned Optimization
We propose a new approach to learned optimization where we represent the computation of an optimizer's update step using a neural network.
BlazePose GHUM Holistic: Real-time 3D Human Landmarks and Pose Estimation
We present BlazePose GHUM Holistic, a lightweight neural network pipeline for 3D human body landmarks and pose estimation, specifically tailored to real-time on-device inference.
Differentiable Dynamics for Articulated 3d Human Motion Reconstruction
We introduce DiffPhy, a differentiable physics-based model for articulated 3d human motion reconstruction from video.
Ranked #58 on
3D Human Pose Estimation
on Human3.6M
Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Video
We focus on the task of estimating a physically plausible articulated human motion from monocular video.
Ranked #320 on
3D Human Pose Estimation
on Human3.6M
Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing
We present PHORHUM, a novel, end-to-end trainable, deep neural network methodology for photorealistic 3D human reconstruction given just a monocular RGB image.
BEHAVE: Dataset and Method for Tracking Human Object Interactions
We present BEHAVE dataset, the first full body human- object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them.
Learning Online Multi-Sensor Depth Fusion
Multi-sensor depth fusion is able to substantially improve the robustness and accuracy of 3D reconstruction methods, but existing techniques are not robust enough to handle sensors which operate with diverse value ranges as well as noise and outlier statistics.
Discriminating Against Unrealistic Interpolations in Generative Adversarial Networks
By reusing the discriminator network to modify the metric on the latent space, we propose a lightweight solution for improved interpolations in pre-trained GANs.
Embodied Learning for Lifelong Visual Perception
As we study this task in a lifelong learning context, the agents should use knowledge gained in earlier visited environments in order to guide their exploration and active learning strategy in successively visited buildings.
HSPACE: Synthetic Parametric Humans Animated in Complex Environments
We combine a hundred diverse individuals of varying ages, gender, proportions, and ethnicity, with hundreds of motions and scenes, as well as parametric variations in body shape (for a total of 1, 600 different humans), in order to generate an initial dataset of over 1 million frames.
Ranked #1 on
3D Human Pose Estimation
on HSPACE
REMIPS: Physically Consistent 3D Reconstruction of Multiple Interacting People under Weak Supervision
We introduce a novel unified model for self- and interpenetration-collisions based on a mesh approximation computed by applying decimation operators.
H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion
This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training.
imGHUM: Implicit Generative Models of 3D Human Shape and Articulated Pose
We present imGHUM, the first holistic generative model of 3D human shape and articulated pose, represented as a signed distance function.
A Real-Time Online Learning Framework for Joint 3D Reconstruction and Semantic Segmentation of Indoor Scenes
This paper presents a real-time online vision framework to jointly recover an indoor scene's 3D structure and semantic label.
RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection
These larger detection ranges require more efficient and accurate detection models.
AIFit: Automatic 3D Human-Interpretable Feedback Models for Fitness Training
AIFit is able to reconstruct 3d human pose and motion, reliably segment exercise repetitions, and identify in real-time the deviations between standards learnt from trainers, and the execution of a trainee.
THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers
We present THUNDR, a transformer-based deep neural network methodology to reconstruct the 3d pose and shape of people, given monocular RGB images.
Ranked #50 on
3D Human Pose Estimation
on 3DPW
(MPJPE metric)
Local Metrics for Multi-Object Tracking
This paper introduces temporally local metrics for Multi-Object Tracking.
TropEx: An Algorithm for Extracting Linear Terms in Deep Neural Networks
Deep neural networks with rectified linear (ReLU) activations are piecewise linear functions, where hyperplanes partition the input space into an astronomically high number of linear regions.
Learning Complex 3D Human Self-Contact
Monocular estimation of three dimensional human self-contact is fundamental for detailed scene analysis including body language understanding and behaviour modeling.
Embodied Visual Active Learning for Semantic Segmentation
We study the task of embodied visual active learning, where an agent is set to explore a 3d environment with the goal to acquire visual scene understanding by actively selecting views for which to request annotation.
LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration
Formulating this closed loop is not straightforward because it is not trivial to force the output of the NN to be on the surface of the human model - outside this surface the human model is not even defined.
Neural Descent for Visual 3D Human Pose and Shape
We present deep neural network methodology to reconstruct the 3d pose and shape of people, given an input RGB image.
Ranked #71 on
3D Human Pose Estimation
on 3DPW
(MPJPE metric)
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction
In this work, we present methodology that combines detail-rich implicit functions and parametric representations in order to reconstruct 3D models of people that remain controllable and accurate even in the presence of clothing.
Calibration of Neural Networks using Splines
From this, by approximating the empirical cumulative distribution using a differentiable function via splines, we obtain a recalibration function, which maps the network outputs to actual (calibrated) class assignment probabilities.
Post-hoc Calibration of Neural Networks by g-Layers
Calibration of neural networks is a critical aspect to consider when incorporating machine learning models in real-world decision-making systems where the confidence of decisions are equally important as the decisions themselves.
Consistency Guided Scene Flow Estimation
To handle inherent modeling error in the consistency loss (e. g. Lambertian assumptions) and for better generalization, we further introduce a learned, output refinement network, which takes the initial predictions, the loss, and the gradient as input, and efficiently predicts a correlated output update.
Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detection
This paper presents a novel 3D object detection framework that processes LiDAR data directly on its native representation: range images.
Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flows
Monocular 3D human pose and shape estimation is challenging due to the many degrees of freedom of the human body and thedifficulty to acquire training data for large-scale supervised learning in complex visual scenes.
Ranked #63 on
3D Human Pose Estimation
on 3DPW
(PA-MPJPE metric)
Deep Reinforcement Learning for Active Human Pose Estimation
Most 3d human pose estimation methods assume that input -- be it images of a scene collected from one or several viewpoints, or from a video -- is given.
Relative Flatness and Generalization
Flatness of the loss curve is conjectured to be connected to the generalization ability of machine learning models, in particular neural networks.
Domes to Drones: Self-Supervised Active Triangulation for 3D Human Pose Reconstruction
In order to address the view selection problem in a principled way, we here introduce ACTOR, an active triangulation agent for 3d human pose reconstruction.
A Reparameterization-Invariant Flatness Measure for Deep Neural Networks
The performance of deep neural networks is often attributed to their automated, task-related feature construction.
Feature-Robustness, Flatness and Generalization Error for Deep Neural Networks
With this, the generalization error of a model trained on representative data can be bounded by its feature robustness which depends on our novel flatness measure.
Human Synthesis and Scene Compositing
Generating good quality and geometrically plausible synthetic images of humans with the ability to control appearance, pose and shape parameters, has become increasingly important for a variety of tasks ranging from photo editing, fashion virtual try-on, to special effects and image compression.
Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera
We present GLNet, a self-supervised framework for learning depth, optical flow, camera pose and intrinsic parameters from monocular video - addressing the difficulty of acquiring realistic ground-truth for such tasks.
Non-attracting Regions of Local Minima in Deep and Wide Neural Networks
For extremely wide neural networks of decreasing width after the wide layer, we prove that every suboptimal local minimum belongs to such a connected set.
Deep Network for the Integrated 3D Sensing of Multiple People in Natural Images
The final stage of 3d pose and shape prediction is based on a learned attention process where information from different human body parts is optimally integrated.
3D Human Sensing, Action and Emotion Recognition in Robot Assisted Therapy of Children With Autism
We introduce new, fine-grained action and emotion recognition tasks defined on non-staged videos, recorded during robot-assisted therapy sessions of children with autism.
Human Appearance Transfer
We propose an automatic person-to-person appearance transfer model based on explicit parametric 3d human representations and learned, constrained deep translation network architectures for photographic image synthesis.
Monocular 3D Pose and Shape Estimation of Multiple People in Natural Scenes - The Importance of Multiple Scene Constraints
Human sensing has greatly benefited from recent advances in deep learning, parametric human modeling, and large scale 2d and 3d datasets.
Deep Reinforcement Learning of Region Proposal Networks for Object Detection
We propose drl-RPN, a deep reinforcement learning-based visual recognition model consisting of a sequential region proposal network (RPN) and an object detector.
Deep Learning of Graph Matching
The problem of graph matching under node and pair-wise constraints is fundamental in areas as diverse as combinatorial optimization, machine learning or computer vision, where representing both the relations between nodes and their neighborhood structure is essential.
Ranked #19 on
Graph Matching
on Willow Object Class
Deep Multitask Architecture for Integrated 2D and 3D Human Sensing
We propose a deep multitask architecture for \emph{fully automatic 2d and 3d human sensing} (DMHS), including \emph{recognition and reconstruction}, in \emph{monocular images}.
Ranked #23 on
3D Human Pose Estimation
on HumanEva-I
Semantic Video Segmentation by Gated Recurrent Flow Propagation
In this paper we present a deep, end-to-end trainable methodology to video segmentation that is capable of leveraging information present in unlabeled data in order to improve semantic estimates.
Ranked #6 on
Video Semantic Segmentation
on CamVid
Spatio-Temporal Attention Models for Grounded Video Captioning
Automatic video captioning is challenging due to the complex interactions in dynamic real scenes.
Reinforcement Learning for Visual Object Detection
One of the most widely used strategies for visual object detection is based on exhaustive spatial hypothesis search.
Large Displacement 3D Scene Flow With Occlusion Reasoning
In this paper we propose a novel coarse to fine correspondence-based scene flow approach to account for the effects of large displacements and to model occlusion, based on explicit geometric reasoning.
Matrix Backpropagation for Deep Networks With Structured Layers
Deep neural network architectures have recently produced excellent results in a variety of areas in artificial intelligence and visual recognition, well surpassing traditional shallow architectures trained using hand-designed features.
Training Deep Networks with Structured Layers by Matrix Backpropagation
Deep neural network architectures have recently produced excellent results in a variety of areas in artificial intelligence and visual recognition, well surpassing traditional shallow architectures trained using hand-designed features.
A Parallel Framework for Parametric Maximum Flow Problems in Image Segmentation
This paper presents a framework that supports the implementation of parallel solutions for the widespread parametric maximum flow computational routines used in image segmentation algorithms.
Second-Order Constrained Parametric Proposals and Sequential Search-Based Structured Prediction for Semantic Segmentation in RGB-D Images
We focus on the problem of semantic segmentation based on RGB-D data, with emphasis on analyzing cluttered indoor scenes containing many visual categories and instances.
A Framework for Symmetric Part Detection in Cluttered Scenes
The role of symmetry in computer vision has waxed and waned in importance during the evolution of the field from its earliest days.
Parametric Image Segmentation of Humans with Structural Shape Priors
The figure-ground segmentation of humans in images captured in natural environments is an outstanding open problem due to the presence of complex backgrounds, articulation, varying body proportions, partial views and viewpoint changes.
Multiple Instance Reinforcement Learning for Efficient Weakly-Supervised Detection in Images
State-of-the-art visual recognition and detection systems increasingly rely on large amounts of training data and complex classifiers.
Iterated Second-Order Label Sensitive Pooling for 3D Human Pose Estimation
Recently, the emergence of Kinect systems has demonstrated the benefits of predicting an intermediate body part labeling for 3D human pose estimation, in conjunction with RGB-D imagery.
Constrained Parametric Proposals and Pooling Methods for Semantic Segmentation in RGB-D Images
Our contributions can be summarized as proposing the following: (1) a generalization of parametric max flow figure-ground proposal methodology to take advantage of intensity and depth information, in order to systematically and efficiently generate the breakpoints of an underlying spatial model in polynomial time, (2) new region description methods based on second-order pooling over multiple features constructed using both intensity and depth channels, (3) an inference procedure that can resolve conflicts in overlapping spatial partitions, and handles scenes with a large number of objects category instances, of very different scales, (4) extensive evaluation of the impact of depth, as well as the effectiveness of a large number of descriptors, both pre-designed and automatically obtained using deep learning, in a difficult RGB-D semantic segmentation problem with 92 classes.
Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition
Systems based on bag-of-words models from image features collected at maxima of sparse interest point operators have been used successfully for both computer visual object and action recognition tasks.
Human3.6m: Large scale datasets and predictive methods for 3D human sensing in natural environments
We introduce a new dataset, Human3. 6M, of 3. 6 Million accurate 3D Human poses, acquired by recording the performance of 5 female and 6 male subjects, under 4 different viewpoints, for training realistic human sensing systems and for evaluating the next generation of human pose estimation models and algorithms.
Action from Still Image Dataset and Inverse Optimal Control to Learn Task Specific Visual Scanpaths
Finally, we leverage our large scale dataset in conjunction with powerful machine learning techniques and computer vision features, to introduce novel dynamic eye movement prediction methods which learn task-sensitive reward functions from eye movement data and efficiently integrate these rewards to plan future saccades based on inverse optimal control.
Composite Statistical Inference for Semantic Segmentation
In this paper we present an inference procedure for the semantic segmentation of images.
A Linear Approximation to the chi^2 Kernel with Geometric Convergence
We propose a new analytical approximation to the $\chi^2$ kernel that converges geometrically.
Probabilistic Joint Image Segmentation and Labeling
We present a joint image segmentation and labeling model (JSL) which, given a bag of figure-ground segment hypotheses extracted at multiple image locations and scales, constructs a joint probability distribution over both the compatible image interpretations (tilings or image segmentations) composed from those segments, and over their labeling into categories.
Convex Multiple-Instance Learning by Estimating Likelihood Ratio
In this work, we propose an approach that recasts it as a convex likelihood ratio estimation problem.
Twin gaussian processes for structured prediction
We describe twin Gaussian processes (TGP), a generic structured prediction method that uses Gaussian process (GP) priors on both covariates and responses, both multivariate, and estimates outputs by minimizing the Kullback-Leibler divergence between two GP modeled as normal distributions over finite index sets of training and testing examples, emphasizing the goal that similar inputs should produce similar percepts and this should hold, on average, between their marginal distributions.
Ranked #25 on
3D Human Pose Estimation
on HumanEva-I
Efficient Match Kernel between Sets of Features for Visual Recognition
To address this problem, we propose an efficient match kernel (EMK), which maps local features to a low dimensional feature space, average the resulting feature vectors to form a set-level feature, then apply a linear classifier.
People Tracking with the Laplacian Eigenmaps Latent Variable Model
Reliably recovering 3D human pose from monocular video requires constraints that bias the estimates towards typical human poses and motions.
