Search Results for author:
Found 88 papers, 45 papers with code
Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loop
In this way, hand-eye coordination emerges as the eye looks towards regions which allow the hand to complete the task.
Visual Imitation Enables Contextual Humanoid Control
How can we teach humanoids to climb staircases and sit on chairs using the surrounding environment context?
St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World
Dynamic 3D reconstruction and point tracking in videos are typically treated as separate tasks, despite their deep connection.
Segment Any Motion in Videos
Moving object segmentation is a crucial task for achieving a high-level understanding of visual scenes and has numerous downstream applications.
Fillerbuster: Multi-View Scene Completion for Casual Captures
Our solution is to train a generative model that can consume a large context of input frames while generating unknown target views and recovering image poses when desired.
Continuous 3D Perception Model with Persistent State
We present a unified framework capable of solving a broad range of 3D tasks.
Decentralized Diffusion Models
We propose Decentralized Diffusion Models, a scalable framework for distributing diffusion model training across independent clusters or datacenters by eliminating the dependence on a centralized, high-bandwidth networking fabric.
MegaSaM: Accurate, Fast and Robust Structure and Motion from Casual Dynamic Videos
We present a system that allows for accurate, fast, and robust estimation of camera parameters and depth maps from casual monocular videos of dynamic scenes.
Reconstructing People, Places, and Cameras
We present "Humans and Structure from Motion" (HSfM), a method for jointly reconstructing multiple human meshes, scene point clouds, and camera parameters in a metric world coordinate system from a sparse set of uncalibrated multi-view images featuring people.
MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos
We present a system that allows for accurate, fast, and robust estimation of camera parameters and depth maps from casual monocular videos of dynamic scenes.
SOAR: Self-Occluded Avatar Recovery from a Single Video In the Wild
SOAR leverages structural normal prior and generative diffusion prior to address such an ill-posed reconstruction problem.
Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos
We present Agent-to-Sim (ATS), a framework for learning interactive behavior models of 3D agents from casual longitudinal video collections.
Estimating Body and Hand Motion in an Ego-sensed World
We present EgoAllo, a system for human motion estimation from a head-mounted device.
Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction
This analysis-by-synthesis approach uses part-centric feature fields in an iterative optimization which enables the use of geometric regularizers to recover 3D motions from only a single video.
gsplat: An Open-Source Library for Gaussian Splatting
gsplat is an open-source library designed for training and developing Gaussian Splatting methods.
Synergy and Synchrony in Couple Dances
Our contributions are a demonstration of the advantages of socially conditioned future motion prediction and an in-the-wild, couple dance video dataset to enable future research in this direction.
Shape of Motion: 4D Reconstruction from a Single Video
Monocular dynamic reconstruction is a challenging and long-standing vision problem due to the highly ill-posed nature of the task.
Splatfacto-W: A Nerfstudio Implementation of Gaussian Splatting for Unconstrained Photo Collections
Novel view synthesis from unconstrained in-the-wild image collections remains a significant yet challenging task due to photometric variations and transient occluders that complicate accurate scene reconstruction.
Rethinking Score Distillation as a Bridge Between Image Distributions
We compare our method to existing approaches for score distillation sampling and show that it can produce high-frequency details with realistic colors.
Toon3D: Seeing Cartoons from New Perspectives
We recover the underlying 3D structure from images of cartoons and anime depicting the same scene.
NurtureNet: A Multi-task Video-based Approach for Newborn Anthropometry
Our goal is to equip health workers and public health systems with a solution for contactless newborn anthropometry in the community.
NeRF-XL: Scaling NeRFs with Multiple GPUs
We present NeRF-XL, a principled method for distributing Neural Radiance Fields (NeRFs) across multiple GPUs, thus enabling the training and rendering of NeRFs with an arbitrarily large capacity.
Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mind
In this paper, we aim to mimic this spatial cognition ability.
The More You See in 2D, the More You Perceive in 3D
Inspired by this behavior, we introduce SAP3D, a system for 3D reconstruction and novel view synthesis from an arbitrary number of unposed images.
GARField: Group Anything with Radiance Fields
We optimize this field from a set of 2D masks provided by Segment Anything (SAM) in a way that respects coarse-to-fine hierarchy, using scale to consistently fuse conflicting masks from different viewpoints.
From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
We present a framework for generating full-bodied photorealistic avatars that gesture according to the conversational dynamics of a dyadic interaction.
The More You See in 2D the More You Perceive in 3D
Inspired by this behavior we introduce SAP3D a system for 3D reconstruction and novel view synthesis from an arbitrary number of unposed images.
Reconstructing Hands in 3D with Transformers
The key to HaMeR's success lies in scaling up both the data used for training and the capacity of the deep network for hand reconstruction.
NeRFiller: Completing Scenes via Generative 3D Inpainting
In contrast to related works, we focus on completing scenes rather than deleting foreground objects, and our approach does not require tight 2D object masks or text.
Mathematical Supplement for the $\texttt{gsplat}$ Library
This report provides the mathematical details of the gsplat library, a modular toolbox for efficient differentiable Gaussian splatting, as proposed by Kerbl et al.
State of the Art on Diffusion Models for Visual Computing
The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes.
Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping
Instead, we propose LERF-TOGO, Language Embedded Radiance Fields for Task-Oriented Grasping of Objects, which uses vision-language models zero-shot to output a grasp distribution over an object given a natural language query.
Can Language Models Learn to Listen?
We present a framework for generating appropriate facial responses from a listener in dyadic social interactions based on the speaker's words.
Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives
We compare our approach to the state of the art on diverse scenes from DTU, and demonstrate its robustness on real-life captures from BlendedMVS and Nerfstudio.
Generative Proxemics: A Prior for 3D Social Interaction from Images
To address this, we present a novel approach that learns a prior over the 3D proxemics two people in close social interaction and demonstrate its use for single-view 3D reconstruction.
Humans in 4D: Reconstructing and Tracking Humans with Transformers
To analyze video, we use 3D reconstructions from HMR 2. 0 as input to a tracking system that operates in 3D.
Ranked #3 on
Pose Tracking
on PoseTrack2018
NerfAcc: Efficient Sampling Accelerates NeRFs
Optimizing and rendering Neural Radiance Fields is computationally expensive due to the vast number of samples required by volume rendering.
Nerfbusters: Removing Ghostly Artifacts from Casually Captured NeRFs
Casually captured Neural Radiance Fields (NeRFs) suffer from artifacts such as floaters or flawed geometry when rendered outside the camera trajectory.
Generating Continual Human Motion in Diverse 3D Scenes
We decompose the continual motion synthesis problem into walking along paths and transitioning in and out of the actions specified by the keypoints, which enables long generation of motions that satisfy scene constraints without explicitly incorporating scene information.
On the Benefits of 3D Pose and Tracking for Human Action Recognition
Subsequently, we propose a Lagrangian Action Recognition model by fusing 3D pose and contextualized appearance over tracklets.
Ranked #1 on
Action Recognition
on AVA v2.2
(using extra training data)
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions
We propose a method for editing NeRF scenes with text-instructions.
LERF: Language Embedded Radiance Fields
Humans describe the physical world using natural language to refer to specific 3D locations based on a vast range of properties: visual appearance, semantics, abstract associations, or actionable affordances.
Decoupling Human and Camera Motion from Videos in the Wild
Our method robustly recovers the global 3D trajectories of people in challenging in-the-wild videos, such as PoseTrack.
Nerfstudio: A Modular Framework for Neural Radiance Field Development
Neural Radiance Fields (NeRF) are a rapidly growing area of research with wide-ranging applications in computer vision, graphics, robotics, and more.
Ranked #7 on
Novel View Synthesis
on RefRef
K-Planes: Explicit Radiance Fields in Space, Time, and Appearance
We introduce k-planes, a white-box model for radiance fields in arbitrary dimensions.
Ranked #2 on
Novel View Synthesis
on LLFF
Monocular Dynamic View Synthesis: A Reality Check
We study the recent progress on dynamic view synthesis (DVS) from monocular video.
NerfAcc: A General NeRF Acceleration Toolbox
We propose NerfAcc, a toolbox for efficient volumetric rendering of radiance fields.
Studying Bias in GANs through the Lens of Race
In this work, we study how the performance and evaluation of generative image models are impacted by the racial composition of their training datasets.
The One Where They Reconstructed 3D Humans and Environments in TV Shows
TV shows depict a wide variety of human behaviors and have been studied extensively for their potential to be a rich source of data for many applications.
InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images
We present a method for learning to generate unbounded flythrough videos of natural scenes starting from a single view, where this capability is learned from a collection of single photographs, without requiring camera poses or even multiple views of each scene.
Domain Adaptive 3D Pose Augmentation for In-the-wild Human Mesh Recovery
The ability to perceive 3D human bodies from a single image has a multitude of applications ranging from entertainment and robotics to neuroscience and healthcare.
TAVA: Template-free Animatable Volumetric Actors
Since TAVA does not require a body template, it is applicable to humans as well as other creatures such as animals.
Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion
We present a framework for modeling interactional communication in dyadic conversations: given multimodal inputs of a speaker, we autoregressively output multiple possibilities of corresponding listener motion.
Deformable Sprites for Unsupervised Video Decomposition
We describe a method to extract persistent elements of a dynamic scene from an input video.
Tracking People by Predicting 3D Appearance, Location and Pose
For a future frame, we compute the similarity between the predicted state of a tracklet and the single frame observations in a probabilistic manner.
Plenoxels: Radiance Fields without Neural Networks
We introduce Plenoxels (plenoptic voxels), a system for photorealistic view synthesis.
Tracking People by Predicting 3D Appearance, Location & Pose
For a future frame, we compute the similarity between the predicted state of a tracklet and the single frame observations in a probabilistic manner.
Tracking People with 3D Representations
We find that 3D representations are more effective than 2D representations for tracking in these settings, and we obtain state-of-the-art performance.
Differentiable Gradient Sampling for Learning Implicit 3D Scene Reconstructions from a Single Image
Existing approaches for single object reconstruction impose supervision signals based on the loss of the signed distance value from all locations in a scene, posing difficulties when extending to real-world scenarios.
KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control
We cast this as the problem of aligning a source 3D object to a target 3D object from the same object category.
De-rendering the World's Revolutionary Artefacts
Recent works have shown exciting results in unsupervised image de-rendering -- learning to decompose 3D shape, appearance, and lighting from single-image collections without explicit supervision.
AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
Our system produces high-quality motions that are comparable to those achieved by state-of-the-art tracking-based techniques, while also being able to easily accommodate large datasets of unstructured motion clips.
PlenOctrees for Real-time Rendering of Neural Radiance Fields
We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects.
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
We present AIST++, a new multi-modal dataset of 3D dance motion and music, along with FACT, a Full-Attention Cross-modal Transformer network for generating 3D dance motion conditioned on music.
Ranked #2 on
Motion Synthesis
on BRACE
Human Mesh Recovery from Multiple Shots
The tools we develop open the door to processing and analyzing in 3D content from a large library of edited media, which could be helpful for many downstream applications.
Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image
We introduce the problem of perpetual view generation - long-range generation of novel views corresponding to an arbitrarily long camera trajectory given a single image.
Reconstructing Hand-Object Interactions in the Wild
In this work we explore reconstructing hand-object interactions in the wild.
pixelNeRF: Neural Radiance Fields from One or Few Images
This allows the network to be trained across multiple scenes to learn a scene prior, enabling it to perform novel view synthesis in a feed-forward manner from a sparse set of views (as few as one).
Ranked #2 on
Generalizable Novel View Synthesis
on NERDS 360
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild
We present a method that infers spatial arrangements and shapes of humans and objects in a globally consistent 3D scene, all from a single image in-the-wild captured in an uncontrolled environment.
Ranked #3 on
3D Object Reconstruction
on BEHAVE
Shape and Viewpoint without Keypoints
We present a learning framework that learns to recover the 3D shape, pose and texture from a single image, trained on an image collection without any ground truth 3D shape, multi-view, camera viewpoints or keypoint supervision.
An Analysis of SVD for Deep Rotation Estimation
Symmetric orthogonalization via SVD, and closely related procedures, are well-known techniques for projecting matrices onto $O(n)$ or $SO(n)$.
Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images "In the Wild"
In contrast to research on human pose, shape and texture estimation, training data for endangered species is limited, the animals are in complex natural scenes with occlusion, they are naturally camouflaged, travel in herds, and look similar to each other.
Predicting 3D Human Dynamics from Video
In this work, we present perhaps the first approach for predicting a future 3D mesh model sequence of a person from past video input.
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
We introduce Pixel-aligned Implicit Function (PIFu), a highly effective implicit representation that locally aligns pixels of 2D images with the global context of their corresponding 3D object.
Ranked #1 on
3D Object Reconstruction
on RenderPeople
Learning 3D Human Dynamics from Video
We present a framework that can similarly learn a representation of 3D dynamics of humans from video via a simple but effective temporal encoding of image features.
Ranked #17 on
3D Human Pose Estimation
on 3DPW
(Acceleration Error metric)
SFV: Reinforcement Learning of Physical Skills from Videos
In this paper, we propose a method that enables physically simulated characters to learn skills from videos (SFV).
Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
By enforcing a constraint on the mutual information between the observations and the discriminator's internal representation, we can effectively modulate the discriminator's accuracy and maintain useful and informative gradients.
SfSNet: Learning Shape, Reflectance and Illuminance of Faces `in the Wild'
SfSNet learns from a mixture of labeled synthetic and unlabeled real world images.
Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape From Images
Animals are widespread in nature and the analysis of their shape and motion is important in many fields and industries.
Learning Category-Specific Mesh Reconstruction from Image Collections
The shape is represented as a deformable 3D mesh model of an object category where a shape is parameterized by a learned mean shape and per-instance predicted deformation.
End-to-end Recovery of Human Shape and Pose
The main objective is to minimize the reprojection loss of keypoints, which allow our model to be trained using images in-the-wild that only have ground truth 2D annotations.
Ranked #1 on
Weakly-supervised 3D Human Pose Estimation
on Human3.6M
(3D Annotations metric)
SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild
SfSNet learns from a mixture of labeled synthetic and unlabeled real world images.
Towards Accurate Markerless Human Shape and Pose Estimation over Time
Existing marker-less motion capture methods often assume known backgrounds, static cameras, and sequence specific motion priors, which narrows its application scenarios.
3D Menagerie: Modeling the 3D shape and pose of animals
The best human body models are learned from thousands of 3D scans of people in specific poses, which is infeasible with live animals.
Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image
We then fit (top-down) a recently published statistical body shape model, called SMPL, to the 2D joints.
Ranked #31 on
3D Human Pose Estimation
on HumanEva-I
WarpNet: Weakly Supervised Matching for Single-view Reconstruction
This is in contrast to prior works that require part annotations, since matching objects across class and pose variations is challenging with appearance features alone.
Learning 3D Deformation of Animals from 2D Images
In this paper, we show that such information can be learned from user-clicked 2D images and a template 3D model of the target animal.
Locally Scale-Invariant Convolutional Neural Networks
We show on a modified MNIST dataset that when faced with scale variation, building in scale-invariance allows ConvNets to learn more discriminative features with reduced chances of over-fitting.
