Search Results for author:
Found 225 papers, 72 papers with code
UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting
We address the challenge of relighting a single image or video, a task that demands precise scene intrinsic understanding and high-quality light transport synthesis.
Align Your Flow: Scaling Continuous-Time Flow Map Distillation
Diffusion- and flow-based models have emerged as state-of-the-art generative modeling approaches, but they require many sampling steps.
Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models
To address this challenge, we introduce the Cosmos-Drive-Dreams - a synthetic data generation (SDG) pipeline that aims to generate challenging scenarios to facilitate downstream tasks such as perception and driving policy training.
Socratic-MCTS: Test-Time Visual Reasoning by Asking the Right Questions
Recent research in vision-language models (VLMs) has centered around the possibility of equipping them with implicit long-form chain-of-thought reasoning -- akin to the success observed in language models -- via distillation and reinforcement learning.
Controllable Weather Synthesis and Removal with Video Diffusion Models
Generating realistic and controllable weather effects in videos is valuable for many applications.
VideoPanda: Video Panoramic Diffusion with Multi-view Attention
High resolution panoramic video content is paramount for immersive experiences in Virtual Reality, but is non-trivial to collect as it requires specialized equipment and intricate camera setups.
PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyond
We propose PartField, a feedforward approach for learning part-based 3D features, which captures the general concept of parts and their hierarchy without relying on predefined templates or text-based names, and can be applied to open-world 3D shapes across various modalities.
Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control
We introduce Cosmos-Transfer, a conditional world generation model that can generate world simulations based on multiple spatial control inputs of various modalities such as segmentation, depth, and edge.
GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control
Our results demonstrate more precise camera control than prior work, as well as state-of-the-art results in sparse-view novel view synthesis, even in challenging settings such as driving scenes and monocular dynamic video.
Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models
At the core of our approach is Difix, a single-step image diffusion model trained to enhance and remove artifacts in rendered novel views caused by underconstrained regions of the 3D representation.
DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models
Classic physically-based rendering (PBR) accurately simulates the light transport, but relies on precise scene representations--explicit 3D geometry, high-quality material properties, and lighting conditions--that are often impractical to obtain in real-world scenarios.
Cosmos World Foundation Model Platform for Physical AI
In this paper, we present the Cosmos World Foundation Model Platform to help developers build customized world models for their Physical AI setups.
Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?
Improving semantic grounding in Vision-Language Models (VLMs) often involves collecting domain-specific training data, refining the network architectures, or modifying the training recipes.
Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models
Classic physically-based rendering (PBR) accurately simulates the light transport, but relies on precise scene representations--explicit 3D geometry, high-quality material properties, and lighting conditions--that are often impractical to obtain in real-world scenarios.
InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models
We present InfiniCube, a scalable method for generating unbounded dynamic 3D driving scenes with high fidelity and controllability.
Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos
Recent advancements in static feed-forward scene reconstruction have demonstrated significant progress in high-quality novel view synthesis.
LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models
This work explores expanding the capabilities of large language models (LLMs) pretrained on text to generate 3D meshes within a unified model.
ReMatching Dynamic Reconstruction Flow
Reconstructing a dynamic scene from image inputs is a fundamental computer vision task with many downstream applications.
SCube: Instant Large-Scale Scene Reconstruction using VoxSplats
We present SCube, a novel method for reconstructing large-scale 3D scenes (geometry, appearance, and semantics) from a sparse set of posed images.
SpaceMesh: A Continuous Representation for Learning Manifold Surface Meshes
In particular, our vertex embeddings generate cyclic neighbor relationships in a halfedge mesh representation, which gives a guarantee of edge-manifoldness and the ability to represent general polygonal meshes.
Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models
Inspired by this observation, we develop a zero-shot prompting technique, SpatialPrompt, that encourages VLMs to answer quantitative spatial questions using reference objects as visual cues.
OmniRe: Omni Urban Scene Reconstruction
We introduce OmniRe, a comprehensive system for efficiently creating high-fidelity digital twins of dynamic real-world scenes from on-device logs.
Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering
The correct insertion of virtual objects in images of real-world scenes requires a deep understanding of the scene's lighting, geometry and materials, as well as the image formation process.
Wolf: Captioning Everything with a World Summarization Framework
Finally, we establish a benchmark for video captioning and introduce a leaderboard, aiming to accelerate advancements in video understanding, captioning, and data alignment.
SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillation
Inspired by these successes, in this work we introduce SuperPADL, a scalable framework for physics-based text-to-motion that leverages both RL and supervised learning to train controllers on thousands of diverse motion clips.
3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes
The benefits of ray tracing are well-known in computer graphics: processing incoherent rays for secondary lighting effects such as shadows and reflections, rendering from highly-distorted cameras common in robotics, stochastically sampling rays, and more.
fVDB: A Deep-Learning Framework for Sparse, Large-Scale, and High-Performance Spatial Intelligence
We present fVDB, a novel GPU-optimized framework for deep learning on large-scale 3D data.
DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features
Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs with limited view overlap, and is trained self-supervised with differentiable rendering to reconstruct RGB, depth, or feature images.
L4GM: Large 4D Gaussian Reconstruction Model
We present L4GM, the first 4D Large Reconstruction Model that produces animated objects from a single-view video input -- in a single feed-forward pass that takes only a second.
Outdoor Scene Extrapolation with Hierarchical Generative Cellular Automata
We aim to generate fine-grained 3D geometry from large-scale sparse LiDAR scans, abundantly captured by autonomous vehicles (AV).
NeRF-XL: Scaling NeRFs with Multiple GPUs
We present NeRF-XL, a principled method for distributing Neural Radiance Fields (NeRFs) across multiple GPUs, thus enabling the training and rendering of NeRFs with an arbitrarily large capacity.
Align Your Steps: Optimizing Sampling Schedules in Diffusion Models
Diffusion models (DMs) have established themselves as the state-of-the-art generative modeling approach in the visual domain and beyond.
RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting
In this work, we propose an approach for 3D scene inpainting -- the task of coherently replacing parts of the reconstructed scene with desired content.
Can Feedback Enhance Semantic Grounding in Large Vision-Language Models?
We find that if prompted appropriately, VLMs can utilize feedback both in a single step and iteratively, showcasing the potential of feedback as an alternative technique to improve grounding in internet-scale VLMs.
Augmented Reality based Simulated Data (ARSim) with multi-view consistency for AV perception networks
The resulting augmented multi-view consistent dataset is used to train a multi-camera perception network for autonomous vehicles.
LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis
Recent text-to-3D generation approaches produce impressive 3D results but require time-consuming optimization that can take up to an hour per prompt.
EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models
Diffusion models have recently received increasing research attention for their remarkable transfer abilities in semantic segmentation tasks.
Compact Neural Graphics Primitives with Learned Hash Probing
Neural graphics primitives are faster and achieve higher quality when their neural networks are augmented by spatial data structures that hold trainable features arranged in a grid.
Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models
We also propose a motion amplification mechanism as well as a new autoregressive synthesis scheme to generate and combine multiple 4D sequences for longer generation.
Trajeglish: Traffic Modeling as Next-Token Prediction
A longstanding challenge for self-driving development is simulating dynamic driving scenarios seeded from recorded driving logs.
XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies
We present XCube (abbreviated as $\mathcal{X}^3$), a novel generative model for high-resolution sparse 3D voxel grids with arbitrary attributes.
WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space
Then, we train a diffusion model in the 3D-aware latent space, thereby enabling synthesis of high-quality 3D-consistent image samples, outperforming recent state-of-the-art GAN-based methods.
Adaptive Shells for Efficient Neural Radiance Field Rendering
We then extract an explicit mesh of a narrow band around the surface, with width determined by the kernel size, and fine-tune the radiance field within this band.
3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features
However, these features are initially trained on paired text and image data, which are not optimized for 3D tasks, and often exhibit a domain gap when applied to the target data.
EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision
We present EmerNeRF, a simple yet powerful approach for learning spatial-temporal representations of dynamic driving scenes.
ViR: Towards Efficient Vision Retention Backbones
Inspired by this trend, we propose a new class of computer vision models, dubbed Vision Retention Networks (ViR), with dual parallel and recurrent formulations, which strike an optimal balance between fast inference and parallel training with competitive performance.
TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models
We present TexFusion (Texture Diffusion), a new method to synthesize textures for given 3D geometries, using large-scale text-guided image diffusion models.
Towards Viewpoint Robustness in Bird's Eye View Segmentation
We introduce a technique for novel view synthesis and use it to transform collected data to the viewpoint of target rigs, allowing us to train BEV segmentation models for diverse target rigs without any additional data collection or labeling cost.
Flexible Isosurface Extraction for Gradient-Based Mesh Optimization
This work considers gradient-based mesh optimization, where we iteratively optimize for a 3D surface mesh by representing it as the isosurface of a scalar field, an increasingly common paradigm in applications including photogrammetry, generative modeling, and inverse physics.
DreamTeacher: Pretraining Image Backbones with Deep Generative Models
In this work, we introduce a self-supervised feature representation learning framework DreamTeacher that utilizes generative networks for pre-training downstream image backbones.
ATT3D: Amortized Text-to-3D Object Synthesis
Text-to-3D modelling has seen exciting progress by combining generative text-to-image models with image-to-3D methods like Neural Radiance Fields.
Neural Kernel Surface Reconstruction
We present a novel method for reconstructing a 3D implicit surface from a large-scale, sparse, and noisy point cloud.
Neural LiDAR Fields for Novel View Synthesis
We present Neural Fields for LiDAR (NFL), a method to optimise a neural field scene representation from LiDAR measurements, with the goal of synthesizing realistic LiDAR scans from novel viewpoints.
NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models
We first train a scene auto-encoder to express a set of image and pose pairs as a neural field, represented as density and feature voxel grids that can be projected to produce novel views of the scene.
Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
We first pre-train an LDM on images only; then, we turn the image generator into a video generator by introducing a temporal dimension to the latent space diffusion model and fine-tuning on encoded image sequences, i. e., videos.
Ranked #5 on
Text-to-Video Generation
on MSR-VTT
(CLIP-FID metric)
Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes
Reconstruction and intrinsic decomposition of scenes from captured imagery would enable many applications such as relighting and virtual object insertion.
Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion
We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet user-defined goals.
VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion
To enable such capability in AI systems, we propose VoxFormer, a Transformer-based semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images.
3D geometry
3D Semantic Scene Completion from a single RGB image
+1
Bridging the Sim2Real gap with CARE: Supervised Detection Adaptation with Conditional Alignment and Reweighting
Sim2Real domain adaptation (DA) research focuses on the constrained setting of adapting from a labeled synthetic source domain to an unlabeled or sparsely labeled real target domain.
Synthesizing Physical Character-Scene Interactions
These scene interactions are learned using an adversarial discriminator that evaluates the realism of a motion within the context of a scene.
PADL: Language-Directed Physics-Based Character Control
In this work, we present PADL, which leverages recent innovations in NLP in order to take steps towards developing language-directed controllers for physics-based character animation.
Neural Fields Meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes
Reconstruction and intrinsic decomposition of scenes from captured imagery would enable many applications such as relighting and virtual object insertion.
Learning Human Dynamics in Autonomous Driving Scenarios
In this work, we propose a holistic framework for learning physically plausible human dynamics from real driving scenarios, narrowing the gap between real and simulated human behavior in safety-critical applications.
End-to-end 3D Tracking with Decoupled Queries
In this work, we present an end-to-end framework for camera-based 3D multi-object tracking, called DQTrack.
Magic3D: High-Resolution Text-to-3D Content Creation
DreamFusion has recently demonstrated the utility of a pre-trained text-to-image diffusion model to optimize Neural Radiance Fields (NeRF), achieving remarkable text-to-3D synthesis results.
Ranked #2 on
Text to 3D
on T$^3$Bench
LION: Latent Point Diffusion Models for 3D Shape Generation
To advance 3D DDMs and make them useful for digital artists, we require (i) high generation quality, (ii) flexibility for manipulation and applications such as conditional synthesis and shape interpolation, and (iii) the ability to output smooth surfaces or meshes.
Ranked #1 on
Point Cloud Generation
on ShapeNet Airplane
XDGAN: Multi-Modal 3D Shape Generation in 2D Space
Specifically, we propose a novel method to convert 3D shapes into compact 1-channel geometry images and leverage StyleGAN3 and image-to-image translation networks to generate 3D objects in 2D space.
Optimizing Data Collection for Machine Learning
Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what kind of data to collect.
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations
VISOR annotates videos from EPIC-KITCHENS, which comes with a new set of challenges not encountered in current video segmentation datasets.
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
As several industries are moving towards modeling massive 3D virtual worlds, the need for content creation tools that can scale in terms of the quantity, quality, and diversity of 3D content is becoming evident.
Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion
In this work, we propose a neural approach that estimates the 5D HDR light field from a single image, and a differentiable object insertion formulation that enables end-to-end training with image-based losses that encourage realism.
MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation
As a result, the learned 2D representations are view-invariant and geometrically consistent, leading to better generalization when trained on a limited number of labeled shapes compared to alternatives that utilize self-supervision in 2D or 3D alone.
Improving Semantic Segmentation in Transformers using Hierarchical Inter-Level Attention
HILA extends hierarchical vision transformer architectures by adding local connections between features of higher and lower levels to the backbone encoder.
How Much More Data Do I Need? Estimating Requirements for Downstream Tasks
Given a small training data set and a learning algorithm, how much more data is necessary to reach a target validation or test performance?
Scalable Neural Data Server: A Data Recommender for Transfer Learning
NDS uses a mixture of experts trained on data sources to estimate similarity between each source and the downstream task.
Variable Bitrate Neural Fields
Neural approximations of scalar and vector fields, such as signed distance functions and radiance fields, have emerged as accurate, high-quality representations.
Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps
Modern image generative models show remarkable sample quality when trained on a single domain or class of objects.
Generative Adversarial Network
Image-to-Image Translation
+1
ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters
By leveraging a massively parallel GPU-based simulator, we are able to train skill embeddings using over a decade of simulated experiences, enabling our model to learn a rich and versatile repertoire of skills.
M$^2$BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation
In this paper, we propose M$^2$BEV, a unified framework that jointly performs 3D object detection and map segmentation in the Birds Eye View~(BEV) space with multi-camera image inputs.
AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis
In this paper, we address the problem of texture representation for 3D shapes for the challenging and underexplored tasks of texture transfer and synthesis.
Learning Smooth Neural Functions via Lipschitz Regularization
The latent descriptor of a neural field acts as a deformation handle for the 3D shape it represents.
Domain Adversarial Training: A Game Perspective
Defining optimal solutions in domain-adversarial training as a local Nash equilibrium, we show that gradient descent in domain-adversarial training can violate the asymptotic convergence guarantees of the optimizer, oftentimes hindering the transfer performance.
Causal Scene BERT: Improving object detection by searching for challenging groups of data
Our main contribution is a pseudo-automatic method to discover such groups in foresight by performing causal interventions on simulated scenes.
Federated Learning with Heterogeneous Architectures using Graph HyperNetworks
Standard Federated Learning (FL) techniques are limited to clients with identical network architectures.
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations
By training an effective feature segmentation architecture on top of BigGAN, we turn BigGAN into a labeled dataset generator.
Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior
Scenario generation is formulated as an optimization in the latent space of this traffic model, perturbing an initial real-world scene to produce trajectories that collide with a given planner.
Frame Averaging for Equivariant Shape Space Learning
A natural way to incorporate symmetries in shape space learning is to ask that the mapping to the shape space (encoder) and mapping from the shape space (decoder) are equivariant to the relevant symmetries.
Hierarchical Neural Implicit Pose Network for Animation and Motion Retargeting
We present HIPNet, a neural implicit pose network trained on multiple subjects across many poses.
Don’t Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence
Generative models trained with privacy constraints on private data can sidestep this challenge, providing indirect access to private data instead.
Neural Fields as Learnable Kernels for 3D Reconstruction
We present Neural Kernel Fields: a novel method for reconstructing implicit 3D shapes based on a learned kernel ridge regression.
Extracting Triangular 3D Models, Materials, and Lighting From Images
We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations.
Ranked #2 on
Depth Prediction
on Stanford-ORB
Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation
Alternative solutions seek to exploit driving simulators that can generate large amounts of labeled data with a plethora of content variations.
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
The core of DMTet includes a deformable tetrahedral grid that encodes a discretized signed distance function and a differentiable marching tetrahedra layer that converts the implicit signed distance representation to the explicit surface mesh representation.
EditGAN: High-Precision Semantic Image Editing
EditGAN builds on a GAN framework that jointly models images and their semantic segmentations, requiring only a handful of labeled examples, making it a scalable tool for editing.
Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence
Generative models trained with privacy constraints on private data can sidestep this challenge, providing indirect access to private data instead.
DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable renderers.
ATISS: Autoregressive Transformers for Indoor Scene Synthesis
The ability to synthesize realistic and diverse indoor furniture layouts automatically or based on partial input, unlocks many applications, from better interactive 3D tools to data synthesis for training and simulation.
Ranked #3 on
Indoor Scene Synthesis
on PRO-teXt
2D Semantic Segmentation task 1 (8 classes)
3D Semantic Scene Completion
+1
Low-Budget Active Learning via Wasserstein Distance: An Integer Programming Approach
Active learning is the process of training a model with limited labeled data by selecting a core subset of an unlabeled data pool to label.
Causal Scene BERT: Improving object detection by searching for challenging groups
We verify that the prioritized groups found via intervention are challenging for the object detector and show that retraining with data collected from these groups helps inordinately compared to adding more IID data.
Physics-based Human Motion Estimation and Synthesis from Videos
By enabling learning of motion synthesis from video, our method paves the way for large-scale, realistic and diverse motion synthesis.
Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting
In this paper, we propose a unified, learning-based inverse rendering framework that formulates 3D spatially-varying lighting.
3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations
Given a small set of high-quality textured objects, our method can create many novel stylized shapes, resulting in effortless 3D content creation and style-ware data augmentation.
NP-DRAW: A Non-Parametric Structured Latent Variable Model for Image Generation
1) We propose a non-parametric prior distribution over the appearance of image parts so that the latent variable ``what-to-draw'' per step becomes a categorical random variable.
f-Domain-Adversarial Learning: Theory and Algorithms
Unsupervised domain adaptation is used in many machine learning applications where, during training, a model has access to unlabeled data in the target domain, and a related labeled dataset.
Low Budget Active Learning via Wasserstein Distance: An Integer Programming Approach
Active learning is the process of training a model with limited labeled data by selecting a core subset of an unlabeled data pool to label.
DriveGAN: Towards a Controllable High-Quality Neural Simulation
Realistic simulators are critical for training and verifying robotics systems.
Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets
This is expensive, and guaranteeing the quality of the labels is a major challenge.
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort
To showcase the power of our approach, we generated datasets for 7 image segmentation tasks which include pixel-level labels for 34 human face parts, and 32 car parts.
Image-Level or Object-Level? A Tale of Two Resampling Strategies for Long-Tailed Detection
As a result, image resampling alone is not enough to yield a sufficiently balanced distribution at the object level.
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
Training deep networks with limited labeled data while achieving a strong generalization ability is key in the quest to reduce human annotation efforts.
gradSim: Differentiable simulation for system identification and visuomotor control
We consider the problem of estimating an object's physical properties such as mass, friction, and elasticity directly from video sequences.
Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks
The INN allows us to compute the inverse mapping of the homeomorphism, which in turn, enables the efficient computation of both the implicit surface function of a primitive and its mesh, without any additional post-processing.
Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes
We introduce an efficient neural representation that, for the first time, enables real-time rendering of high-fidelity neural SDFs, while achieving state-of-the-art geometry reconstruction quality.
f-Domain-Adversarial Learning: Theory and Algorithms for Unsupervised Domain Adaptation with Neural Networks
We provide empirical results for several f-divergences and show that some, not considered previously in domain-adversarial learning, achieve state-of-the-art results in practice.
Differentially Private Generative Models Through Optimal Transport
Generative models trained with privacy constraints on private data can sidestep this challenge and provide indirect access to the private data instead.
Personalized Federated Learning with First Order Model Optimization
While federated learning traditionally aims to train a single global model across decentralized local datasets, one model may not always be ideal for all participating clients.
Variational Amodal Object Completion
In images of complex scenes, objects are often occluding each other which makes perception tasks such as object detection and tracking, or robotic control tasks such as planning, challenging.
UniCon: Universal Neural Controller For Physics-based Character Motion
The field of physics-based animation is gaining importance due to the increasing demand for realism in video games and films, and has recently seen wide adoption of data-driven techniques, such as deep reinforcement learning (RL), which learn control from (human) demonstrations.
Emergent Road Rules In Multi-Agent Driving Environments
For autonomous vehicles to safely share the road with human drivers, autonomous vehicles must abide by specific "road rules" that human drivers have agreed to follow.
Learning Deformable Tetrahedral Meshes for 3D Reconstruction
We introduce Deformable Tetrahedral Meshes (DefTet) as a particular parameterization that utilizes volumetric tetrahedral meshes for the reconstruction problem.
The efficacy of Neural Planning Metrics: A meta-analysis of PKL on nuScenes
A high-performing object detection system plays a crucial role in autonomous driving (AD).
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents.
Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
Key to our approach is to exploit GANs as a multi-view data generator to train an inverse graphics network using an off-the-shelf differentiable renderer, and the trained inverse graphics network as a teacher to disentangle the GAN's latent code into interpretable 3D properties.
Fed-Sim: Federated Simulation for Medical Imaging
Since the model of geometry and material is disentangled from the imaging sensor, it can effectively be trained across multiple medical centers.
Expressive Telepresence via Modular Codec Avatars
It is important to note that traditional person-specific CAs are learned from few training samples, and typically lack robustness as well as limited expressiveness when transferring facial expressions.
Interactive Annotation of 3D Object Geometry using 2D Scribbles
We implement our framework as a web service and conduct a user study, where we show that user annotated data using our method effectively facilitates real-world learning tasks.
ScribbleBox: Interactive Annotation Framework for Video Object Segmentation
Our approach tolerates a modest amount of noise in the box placements, thus typically only a few clicks are needed to annotate tracked boxes to a sufficient accuracy.
Beyond Fixed Grid: Learning Geometric Image Representation with a Deformable Grid
We also utilize DefGrid at the output layers for the task of object mask annotation, and show that reasoning about object boundaries on our predicted polygonal grid leads to more accurate results over existing pixel-wise and curve-based approaches.
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation
In Meta-Sim2, we aim to learn the scene structure in addition to parameters, which is a challenging problem due to its discrete nature.
Learning to Generate Diverse Dance Motions with Transformer
We also introduce new evaluation metrics for the quality of synthesized dance motions, and demonstrate that our system can outperform state-of-the-art methods.
Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D
By training on the entire camera rig, we provide evidence that our model is able to learn not only how to represent images but how to fuse predictions from all cameras into a single cohesive representation of the scene while being robust to calibration error.
Ranked #6 on
Bird's-Eye View Semantic Segmentation
on OBAT PENGGUGUR KANDUNGAN DI BANJARMASIN (087776558899)
(IoU ped - 224x480 - Vis filter. - 100x100 at 0.5 metric)
Autonomous Vehicles
Bird's-Eye View Semantic Segmentation
+1
Learning to Simulate Dynamic Environments with GameGAN
Simulation is a crucial component of any robotic system.
Efficient and Information-Preserving Future Frame Prediction and Beyond
Applying resolution-preserving blocks is a common practice to maximize information preservation in video prediction, yet their high memory consumption greatly limits their application scenarios.
Ranked #16 on
Video Prediction
on Moving MNIST
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines
Our dataset features 55 hours of video consisting of 11. 5M frames, which we densely labelled for a total of 39. 6K action segments and 454. 2K object bounding boxes.
Learning to Evaluate Perception Models Using Planner-Centric Metrics
The downside of these metrics is that, at worst, they penalize all incorrect detections equally without conditioning on the task or scene, and at best, heuristics need to be chosen to ensure that different mistakes count differently.
Neural Data Server: A Large-Scale Search Engine for Transfer Learning Data
NDS consists of a dataserver which indexes several large popular image datasets, and aims to recommend data to a client, an end-user with a target application with its own small labeled dataset.
The Shmoop Corpus: A Dataset of Stories with Loosely Aligned Summaries
Understanding stories is a challenging reading comprehension problem for machines as it requires reading a large volume of text and following long-range dependencies.
Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
We present Kaolin, a PyTorch library aiming to accelerate 3D deep learning research.
CrevNet: Conditionally Reversible Video Prediction
Applying resolution-preserving blocks is a common practice to maximize information preservation in video prediction, yet their high memory consumption greatly limits their application scenarios.
Neural Turtle Graphics for Modeling City Road Layouts
We propose Neural Turtle Graphics (NTG), a novel generative model for spatial graphs, and demonstrate its applications in modeling city road layouts.
DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation
In practice, it performs similarly to the Hungarian algorithm during inference.
A Theoretical Analysis of the Number of Shots in Few-Shot Learning
We introduce a theoretical analysis of the impact of the shot number on Prototypical Networks, a state-of-the-art few-shot classification method.
Video Face Clustering with Unknown Number of Clusters
Understanding videos such as TV series and movies requires analyzing who the characters are and what they are doing.
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting with light, in a process called rendering.
Gated-SCNN: Gated Shape CNNs for Semantic Segmentation
Here, we propose a new two-stream CNN architecture for semantic segmentation that explicitly wires shape information as a separate processing branch, i. e. shape stream, that processes information in parallel to the classical stream.
Ranked #24 on
Semantic Segmentation
on Cityscapes test
Neural Graph Evolution: Towards Efficient Automatic Robot Design
To address the two challenges, we formulate automatic robot design as a graph search problem and perform evolution search in graph space.
DARNet: Deep Active Ray Network for Building Segmentation
In this paper, we propose a Deep Active Ray Network (DARNet) for automatic building segmentation.
EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
Reducing the test time resource requirements of a neural network while preserving test accuracy is crucial for running inference on resource-constrained devices.
Neural Graph Evolution: Automatic Robot Design
To address the two challenges, we formulate automatic robot design as a graph search problem and perform evolution search in graph space.
ACTRCE: Augmenting Experience via Teacher’s Advice
Sparse reward is one of the most challenging problems in reinforcement learning (RL).
Meta-Sim: Learning to Generate Synthetic Datasets
Training models to high-end performance requires availability of large labeled datasets, which are expensive to get.
Devil is in the Edges: Learning Semantic Boundaries from Noisy Annotations
We further reason about true object boundaries during training using a level set formulation, which allows the network to learn from misaligned labels in an end-to-end fashion.
Action Recognition from Single Timestamp Supervision in Untrimmed Videos
We propose a method that is supervised by single timestamps located around each action instance, in untrimmed videos.
Mimicking the In-Camera Color Pipeline for Camera-Aware Object Compositing
We present a method for compositing virtual objects into a photograph such that the object colors appear to have been processed by the photo's camera imaging pipeline.
Fast Interactive Object Annotation with Curve-GCN
Our model runs at 29. 3ms in automatic, and 2. 6ms in interactive mode, making it 10x and 100x faster than Polygon-RNN++.
ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning
We first analyze the differences among goal representation, and show that ACTRCE can efficiently solve difficult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn.
SurfConv: Bridging 3D and 2D Convolution for RGBD Images
On the other hand, 3D convolution wastes a large amount of memory on mostly unoccupied 3D space, which consists of only the surface visible to the sensor.
A Face-to-Face Neural Conversation Model
The decoder consists of two layers, where the lower layer aims at generating the verbal response and coarse facial expressions, while the second layer fills in the subtle gestures, making the generated output more smooth and natural.
Learning to Caption Images through a Lifetime by Asking Questions
In order to bring artificial agents into our lives, we will need to go beyond supervised learning on closed datasets to having the ability to continuously expand knowledge.
A Neural Compositional Paradigm for Image Captioning
Mainstream captioning models often follow a sequential structure to generate captions, leading to issues such as introduction of irrelevant semantics, lack of diversity in the generated captions, and inadequate generalization performance.
Pose Estimation for Objects with Rotational Symmetry
In particular, our aim is to infer poses for objects not seen at training time, but for which their 3D CAD models are available at test time.
VirtualHome: Simulating Household Activities via Programs
We then implement the most common atomic (inter)actions in the Unity3D game engine, and use our programs to "drive" an artificial agent to execute tasks in a simulated household environment.
Color Sails: Discrete-Continuous Palettes for Deep Color Exploration
Then, the user can adjust color sail parameters to change the base colors, their blending behavior and the number of colors, exploring a wide range of options for the original design.
Visual Reasoning by Progressive Module Networks
Thus, a module for a new task learns to query existing modules and composes their outputs in order to produce its own output.
Now You Shake Me: Towards Automatic 4D Cinema
We are interested in enabling automatic 4D cinema by parsing physical and special effects from untrimmed movies.
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
First-person vision is gaining interest as it offers a unique viewpoint on people's interaction with objects, their attention, and even intention.
Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++
Manually labeling datasets with object masks is extremely time consuming.
NerveNet: Learning Structured Policy with Graph Neural Networks
We address the problem of learning structured policies for continuous control.
Learning to Act Properly: Predicting and Explaining Affordances from Images
We address the problem of affordance reasoning in diverse scenes that appear in the real world.
MovieGraphs: Towards Understanding Human-Centric Situations from Videos
Towards this goal, we introduce a novel dataset called MovieGraphs which provides detailed, graph-based annotations of social situations depicted in movie clips.
Teaching Machines to Describe Images with Natural Language Feedback
Robots will eventually be part of every household.
Be Your Own Prada: Fashion Synthesis with Structural Coherence
In the second stage, a generative model with a newly proposed compositional mapping layer is used to render the final image with precise regions and textures conditioned on this map.
SGN: Sequential Grouping Networks for Instance Segmentation
By exploiting two-directional information, the second network groups horizontal and vertical lines into connected components.
3D Graph Neural Networks for RGBD Semantic Segmentation
Each node in the graph corresponds to a set of points and is associated with a hidden representation vector initialized with an appearance feature extracted by a unary CNN from 2D images.
Ranked #38 on
Semantic Segmentation
on SUN-RGBD
(using extra training data)
Situation Recognition with Graph Neural Networks
We address the problem of recognizing situations in images.
Ranked #10 on
Situation Recognition
on imSitu
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
We present a new technique for learning visual-semantic embeddings for cross-modal retrieval.
Ranked #24 on
Cross-Modal Retrieval
on Flickr30k
Sports Field Localization via Deep Structured Models
In this work, we propose a novel way of efficiently localizing a sports field from a single broadcast image of the game.
Scene Parsing Through ADE20K Dataset
A novel network design called Cascade Segmentation Module is proposed to parse a scene into stuff, objects, and object parts in a cascade and improve over the baselines.
Teaching Machines to Describe Images via Natural Language Feedback
Robots will eventually be part of every household.
Annotating Object Instances with a Polygon-RNN
We show that our approach speeds up the annotation process by a factor of 4. 7 across all classes in Cityscapes, while achieving 78. 4% agreement in IoU with original ground-truth, matching the typical agreement between human annotators.
Open Vocabulary Scene Parsing
Recognizing arbitrary objects in the wild has been a challenging problem due to the limitations of existing classification models and datasets.
Towards Diverse and Natural Image Descriptions via a Conditional GAN
Despite the substantial progress in recent years, the image captioning techniques are still far from being perfect. Sentences produced by existing methods, e. g. those based on RNNs, are often overly rigid and lacking in variability.
Proximal Deep Structured Models
Many problems in real-world applications involve predicting continuous-valued random variables that are statistically related.
TorontoCity: Seeing the World with a Million Eyes
In this paper we introduce the TorontoCity benchmark, which covers the full greater Toronto area (GTA) with 712. 5 $km^2$ of land, 8439 $km$ of road and around 400, 000 buildings.
Efficient Summarization with Read-Again and Copy Mechanism
Towards this goal, we first introduce a simple mechanism that first reads the input sequence before committing to a representation of each word.
Song From PI: A Musically Plausible Network for Pop Music Generation
We present a novel framework for generating pop music.
3D Object Proposals using Stereo Imagery for Accurate Object Class Detection
We then exploit a CNN on top of these proposals to perform object detection.
Semantic Understanding of Scenes through the ADE20K Dataset
Scene parsing, or recognizing and segmenting objects and stuff in an image, is one of the key problems in computer vision.
Find your Way by Observing the Sun and Other Semantic Cues
In this paper we present a robust, efficient and affordable approach to self-localization which does not require neither GPS nor knowledge about the appearance of the world.
Monocular 3D Object Detection for Autonomous Driving
The focus of this paper is on proposal generation.
Ranked #8 on
Vehicle Pose Estimation
on KITTI Cars Hard
HD Maps: Fine-Grained Road Segmentation by Parsing Ground and Aerial Images
In this paper we present an approach to enhance existing maps with fine grained segmentation categories such as parking spots and sidewalk, as well as the number and location of road lanes.
Soccer Field Localization from a Single Image
In this work, we propose a novel way of efficiently localizing a soccer field from a single broadcast image of the game.
Instance-Level Segmentation for Autonomous Driving with Deep Densely Connected MRFs
Our aim is to provide a pixel-wise instance-level labeling of a monocular image in the context of autonomous driving.
MovieQA: Understanding Stories in Movies through Question-Answering
We introduce the MovieQA dataset which aims to evaluate automatic story comprehension from both video and text.
Enhancing Road Maps by Parsing Aerial Images Around the World
In recent years, contextual models that exploit maps have been shown to be very effective for many recognition and localization tasks.
3D Object Proposals for Accurate Object Class Detection
The goal of this paper is to generate high-quality 3D object proposals in the context of autonomous driving.
Ranked #10 on
Vehicle Pose Estimation
on KITTI Cars Hard
Lost Shopping! Monocular Localization in Large Indoor Spaces
In this paper we propose a novel approach to localization in very large indoor spaces (i. e., 200+ store shopping malls) that takes a single image and a floor plan of the environment as input.
Learning to Combine Mid-Level Cues for Object Proposal Generation
In this paper, we introduce Parametric Min-Loss (PML), a novel structured learning framework for parametric energy functions.
Order-Embeddings of Images and Language
Hypernymy, textual entailment, and image captioning can be seen as special cases of a single visual-semantic hierarchy over words, sentences, and images.
Ranked #88 on
Natural Language Inference
on SNLI
