Search Results for author:
Found 206 papers, 100 papers with code
UFO²: A Unified Framework towards Omni-supervised Object Detection
Existing work on object detection often relies on a single form of annotation: the model is trained using either accurate yet costly bounding boxes or cheaper but less expressive image-level tags.
Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation
We tackle open-vocabulary 3D scene understanding by introducing a novel data generation pipeline and training framework.
Parallel Sequence Modeling via Generalized Spatial Propagation Network
We present the Generalized Spatial Propagation Network (GSPN), a new attention mechanism optimized for vision tasks that inherently captures 2D spatial structures.
Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models
Recently, promising progress has been made by open-source vision-language models (VLMs) in bringing their capabilities closer to those of proprietary frontier models.
FoundationStereo: Zero-Shot Stereo Matching
However, achieving strong zero-shot generalization - a hallmark of foundation models in other computer vision tasks - remains challenging for stereo matching.
SimAvatar: Simulation-Ready Avatars with Layered Hair and Clothing
Specifically, we first employ three text-conditioned 3D generative models to generate garment mesh, body shape and hair strands from the given text prompt.
StreamChat: Chatting with Streaming Video
This paper presents StreamChat, a novel approach that enhances the interaction capabilities of Large Multimodal Models (LMMs) with streaming video content.
RADIO Amplified: Improved Baselines for Agglomerative Vision Foundation Models
Agglomerative models have recently emerged as a powerful approach to training vision foundation models, leveraging multi-teacher distillation from existing models such as CLIP, DINO, and SAM.
Gated Delta Networks: Improving Mamba2 with Delta Rule
Linear Transformers have gained attention as efficient alternatives to standard Transformers, but their performance in retrieval and long-context tasks has been limited.
NVILA: Efficient Frontier Visual Language Models
This paper introduces NVILA, a family of open VLMs designed to optimize both efficiency and accuracy.
NaVILA: Legged Robot Vision-Language-Action Model for Navigation
This paper proposes to solve the problem of Vision-and-Language Navigation with legged robots, which not only provides a flexible way for humans to command but also allows the robot to navigate through more challenging and cluttered scenes.
Hymba: A Hybrid-head Architecture for Small Language Models
We propose Hymba, a family of small language models featuring a hybrid-head parallel architecture that integrates transformer attention mechanisms with state space models (SSMs) for enhanced efficiency.
EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation
In this work, we re-formulate the model compression problem into the customized compensation problem: Given a compressed model, we aim to introduce residual low-rank paths to compensate for compression errors under customized requirements from users (e. g., tasks, compression ratios), resulting in greater flexibility in adjusting overall capacity without being constrained by specific compression formats.
nvTorchCam: An Open-source Library for Camera-Agnostic Differentiable Geometric Vision
We introduce nvTorchCam, an open-source library under the Apache 2. 0 license, designed to make deep learning algorithms camera model-independent.
Exploring the design space of deep-learning-based weather forecasting systems
Interestingly, we see a strong positive effect of using a larger dataset when training a smaller model as compared to training on a smaller dataset for longer.
MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models
This approach facilitates end-to-end training on large-scale datasets and offers two notable advantages: 1) High-quality Masks - our method effectively scales to large datasets and learns accurate masks; 2) Transferability - the probabilistic modeling of mask distribution enables the transfer learning of sparsity across domains or tasks.
COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimation
Experiments on three challenging benchmarks demonstrate the effectiveness of COIN, which outperforms the state-of-the-art methods in terms of global human motion estimation and camera motion estimation.
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies.
LLM Pruning and Distillation in Practice: The Minitron Approach
We present a comprehensive report on compressing the Llama 3. 1 8B and Mistral NeMo 12B models to 4B and 8B parameters, respectively, using pruning and distillation.
LongVILA: Scaling Long-Context Visual Language Models for Long Videos
We introduce the long-context Multi-Modal Sequence Parallelism (MM-SP) system that efficiently parallelizes long video training and inference, enabling 2M context length training on 256 GPUs without any gradient checkpointing.
VILA$^2$: VILA Augmented VILA
In the self-augment step, the instruction-finetuned VLM recaptions its pretraining caption datasets and then retrains from scratch leveraging refined data.
Ranked #63 on
Visual Question Answering
on MM-Vet
A deeper look at depth pruning of LLMs
Large Language Models (LLMs) are not only resource-intensive to train but even more costly to deploy in production.
Compact Language Models via Pruning and Knowledge Distillation
Large language models (LLMs) targeting different deployment scales and sizes are currently produced by training each variant from scratch; this is extremely compute-intensive.
MambaVision: A Hybrid Mamba-Transformer Vision Backbone
We propose a novel hybrid Mamba-Transformer backbone, denoted as MambaVision, which is specifically tailored for vision applications.
Ranked #246 on
Image Classification
on ImageNet
An Empirical Study of Mamba-based Language Models
On an additional 23 long-context tasks, the hybrid model continues to closely match or exceed the Transformer on average.
Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation
We propose Hydra-MDP, a novel paradigm employing multiple teachers in a teacher-student model.
Flextron: Many-in-One Flexible Large Language Model
Training modern LLMs is extremely resource intensive, and customizing them for various deployment scenarios characterized by limited compute and memory resources through repeated training is impractical.
CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
Recently video diffusion models have emerged as expressive generative tools for high-quality video content creation readily available to general users.
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models
Vision Language Models (VLMs) have demonstrated remarkable performance in 2D vision and language tasks.
X-VILA: Cross-Modality Alignment for Large Language Model
We introduce X-VILA, an omni-modality model designed to extend the capabilities of large language models (LLMs) by incorporating image, video, and audio modalities.
OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning
We further propose OmniDrive-nuScenes, a new visual question-answering dataset challenging the true 3D situational awareness of a model with comprehensive visual question-answering (VQA) tasks, including scene description, traffic regulation, 3D grounding, counterfactual reasoning, decision making and planning.
LITA: Language Instructed Temporal-Localization Assistant
In addition to leveraging existing video datasets with timestamps, we propose a new task, Reasoning Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for learning and evaluating this task.
FoVA-Depth: Field-of-View Agnostic Depth Estimation for Cross-Dataset Generalization
Unfortunately, most of the GT data is for pinhole cameras, making it impossible to properly train depth estimation models for large-FoV cameras.
AM-RADIO: Agglomerative Vision Foundation Model Reduce All Domains Into One
A handful of visual foundation models (VFMs) have recently emerged as the backbones for numerous downstream tasks.
GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning
Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations.
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
We present FoundationPose, a unified foundation model for 6D object pose estimation and tracking, supporting both model-based and model-free setups.
COLMAP-Free 3D Gaussian Splatting
While neural rendering has led to impressive advances in scene reconstruction and novel view synthesis, it relies heavily on accurately pre-computed camera poses.
VILA: On Pre-training for Visual Language Models
Visual language models (VLMs) rapidly progressed with the recent success of large language models.
Ranked #4 on
Zero-Shot Video Question Answer
on MSVD-QA
AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One
A handful of visual foundation models (VFMs) have recently emerged as the backbones for numerous downstream tasks.
Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?
We initially observed that the nuScenes dataset, characterized by relatively simple driving scenarios, leads to an under-utilization of perception information in end-to-end models incorporating ego status, such as the ego vehicle's velocity.
DiffiT: Diffusion Vision Transformers for Image Generation
In this paper, we study the effectiveness of ViTs in diffusion-based generative learning and propose a new model denoted as Diffusion Vision Transformers (DiffiT).
Ranked #17 on
Image Generation
on ImageNet 256x256
ViR: Towards Efficient Vision Retention Backbones
Inspired by this trend, we propose a new class of computer vision models, dubbed Vision Retention Networks (ViR), with dual parallel and recurrent formulations, which strike an optimal balance between fast inference and parallel training with competitive performance.
PACE: Human and Camera Motion Estimation from in-the-wild Videos
This design combines the strengths of SLAM and motion priors, which leads to significant improvements in human and camera motion estimation.
SMRD: SURE-based Robust MRI Reconstruction with Diffusion Models
However, diffusion models require careful tuning of inference hyperparameters on a validation set and are still sensitive to distribution shifts during testing.
3D Reconstruction with Generalizable Neural Fields using Scene Priors
NFP not only demonstrates SOTA scene reconstruction performance and efficiency, but it also supports single-image novel-view synthesis, which is underexplored in neural fields.
Residual Corrective Diffusion Modeling for Km-scale Atmospheric Downscaling
The model is trained to predict 2km data from a regional weather model over Taiwan, conditioned on a 25km global reanalysis.
Online Overexposed Pixels Hallucination in Videos with Adaptive Reference Frame Selection
We present a learning-based system to reduce these artifacts without resorting to complex acquisition mechanisms like alternating exposures or costly processing that are typical of high dynamic range (HDR) imaging.
FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation
This technical report summarizes the winning solution for the 3D Occupancy Prediction Challenge, which is held in conjunction with the CVPR 2023 Workshop on End-to-End Autonomous Driving and CVPR 23 Workshop on Vision-Centric Autonomous Driving Workshop.
Ranked #2 on
Prediction Of Occupancy Grid Maps
on Occ3D-nuScenes
Heterogeneous Continual Learning
We propose a novel framework and a solution to tackle the continual learning (CL) problem with changing network architectures.
Generalizable One-shot Neural Head Avatar
We present a method that reconstructs and animates a 3D head avatar from a single-view portrait image.
FasterViT: Fast Vision Transformers with Hierarchical Attention
At a high level, global self-attentions enable the efficient cross-window communication at lower costs.
Ranked #197 on
Image Classification
on ImageNet
Zero-shot Pose Transfer for Unrigged Stylized 3D Characters
We present a zero-shot approach that requires only the widely available deformed non-stylized avatars in training, and deforms stylized characters of significantly different shapes at inference.
The Best Defense is a Good Offense: Adversarial Augmentation against Adversarial Attacks
Many defenses against adversarial attacks (\eg robust classifiers, randomization, or image purification) use countermeasures put to work only after the attack has been crafted.
A Variational Perspective on Solving Inverse Problems with Diffusion Models
To cope with this challenge, we propose a variational approach that by design seeks to approximate the true posterior distribution.
Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization
To tackle this problem, we propose a novel method for constructing implicit 3D morphable face models that are both generalizable and intuitive for editing.
Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Models
Our Landmark DEQ (LDEQ) achieves state-of-the-art performance on the challenging WFLW facial landmark dataset, reaching $3. 92$ NME with fewer parameters and a training memory cost of $\mathcal{O}(1)$ in the number of recurrent modules.
Ranked #2 on
Face Alignment
on WFLW
BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects
We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object.
Score-based Diffusion Models in Function Space
They consist of a forward process that perturbs input data with Gaussian white noise and a reverse process that learns a score function to generate samples by denoising.
RANA: Relightable Articulated Neural Avatars
We propose RANA, a relightable and articulated neural avatar for the photorealistic synthesis of humans under arbitrary viewpoints, body poses, and lighting.
PhysDiff: Physics-Guided Human Motion Diffusion Model
Specifically, we propose a physics-based motion projection module that uses motion imitation in a physics simulator to project the denoised motion of a diffusion step to a physically-plausible motion.
Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation
An effective approach is to supervise the training of deep neural networks with a high-fidelity dataset of desired input-output pairs, captured with a light stage.
Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion
In this work, we propose a neural approach that estimates the 5D HDR light field from a single image, and a differentiable object insertion formulation that enables end-to-end training with image-based losses that encourage realism.
Global Context Vision Transformers
Pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation using MS COCO and ADE20K datasets outperform prior work consistently.
Ranked #135 on
Semantic Segmentation
on ADE20K
RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis
We present a large-scale synthetic dataset for novel view synthesis consisting of ~300k images rendered from nearly 2000 complex scenes using high-quality ray tracing at high resolution (1600 x 1600 pixels).
Ranked #1 on
Novel View Synthesis
on RTMV
CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs
We represent the correspondence maps of different images as warped coordinate frames transformed from a canonical coordinate frame, i. e., the correspondence map, which describes the structure (e. g., the shape of a face), is controlled via a transformation.
DRaCoN -- Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars
In this work, we present, DRaCoN, a framework for learning full-body volumetric avatars which exploits the advantages of both the 2D and 3D neural rendering techniques.
GradViT: Gradient Inversion of Vision Transformers
In this work we demonstrate the vulnerability of vision transformers (ViTs) to gradient-based inversion attacks.
FreeSOLO: Learning to Segment Objects without Annotations
FreeSOLO further demonstrates superiority as a strong pre-training method, outperforming state-of-the-art self-supervised pre-training methods by +9. 8% AP when fine-tuning instance segmentation with only 5% COCO masks.
Physics Informed RNN-DCT Networks for Time-Dependent Partial Differential Equations
This efficiently and flexibly produces a compressed representation which is used for additional conditioning of physics-informed models.
GroupViT: Semantic Segmentation Emerges from Text Supervision
With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i. e., without any further fine-tuning.
Do Gradient Inversion Attacks Make Federated Learning Unsafe?
Federated learning (FL) allows the collaborative training of AI models without needing to share raw data.
Federated Learning with Heterogeneous Architectures using Graph HyperNetworks
Standard Federated Learning (FL) techniques are limited to clients with identical network architectures.
AdaViT: Adaptive Tokens for Efficient Vision Transformer
A-ViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds.
Ranked #34 on
Efficient ViTs
on ImageNet-1K (with DeiT-S)
GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras
Since the joint reconstruction of human motions and camera poses is underconstrained, we propose a global trajectory predictor that generates global human trajectories based on local body movements.
Ranked #1 on
Global 3D Human Pose Estimation
on EMDB
Coupled Segmentation and Edge Learning via Dynamic Graph Propagation
It is therefore interesting to study how these two tasks can be coupled to benefit each other.
Learning Continuous Environment Fields via Implicit Functions
We propose a novel scene representation that encodes reaching distance -- the distance between any position in the scene to a goal along a feasible trajectory.
Self-Supervised Object Detection via Generative Image Synthesis
We present SSOD, the first end-to-end analysis-by synthesis framework with controllable GANs for the task of self-supervised object detection.
Global Vision Transformer Pruning with Hessian-Aware Saliency
This work aims on challenging the common design philosophy of the Vision Transformer (ViT) model with uniform dimension across all the stacked blocks in a model stage, where we redistribute the parameters both across transformer blocks and between different structures within the block via the first systematic attempt on global structural pruning.
Hardware-Aware Network Transformation
In the second phase, it solves the combinatorial selection of efficient operations using a novel constrained integer linear optimization approach.
Learning Contrastive Representation for Semantic Correspondence
Dense correspondence across semantically related images has been extensively studied, but still faces two challenges: 1) large variations in appearance, scale and pose exist even for objects from the same category, and 2) labeling pixel-level dense correspondences is labor intensive and infeasible to scale.
Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting
In this paper, we propose a unified, learning-based inverse rendering framework that formulates 3D spatially-varying lighting.
Privacy Vulnerability of Split Computing to Data-Free Model Inversion Attacks
Prior works usually assume that SC offers privacy benefits as only intermediate features, instead of private data, are shared from devices to the cloud.
LANA: Latency Aware Network Acceleration
We analyze three popular network architectures: EfficientNetV1, EfficientNetV2 and ResNeST, and achieve accuracy improvement for all models (up to $3. 0\%$) when compressing larger models to the latency level of smaller models.
Self-Supervised Learning on 3D Point Clouds by Learning Discrete Generative Models
In this work, we introduce a general method for 3D self-supervised representation learning that 1) remains agnostic to the underlying neural network architecture, and 2) specifically leverages the geometric nature of 3D point cloud data.
Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation
Hand pose estimation is difficult due to different environmental conditions, object- and self-occlusion as well as diversity in hand shape and appearance.
Score-based Generative Modeling in Latent Space
Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space, resulting in fewer network evaluations and faster sampling.
Ranked #4 on
Image Generation
on CelebA 256x256
Weakly-Supervised Physically Unconstrained Gaze Estimation
A major challenge for physically unconstrained gaze estimation is acquiring training data with 3D gaze annotations for in-the-wild and outdoor scenarios.
Ranked #3 on
Gaze Estimation
on Gaze360
KAMA: 3D Keypoint Aware Body Mesh Articulation
We present KAMA, a 3D Keypoint Aware Mesh Articulation approach that allows us to estimate a human body mesh from the positions of 3D body keypoints.
Ranked #57 on
3D Human Pose Estimation
on 3DPW
See through Gradients: Image Batch Recovery via GradInversion
In this work, we introduce GradInversion, using which input images from a larger batch (8 - 48 images) can also be recovered for large networks such as ResNets (50 layers), on complex datasets such as ImageNet (1000 classes, 224x224 px).
DexYCB: A Benchmark for Capturing Hand Grasping of Objects
We introduce DexYCB, a new dataset for capturing hand grasping of objects.
Learning to Track Instances without Video Annotations
Tracking segmentation masks of multiple instances has been intensively studied, but still faces two fundamental challenges: 1) the requirement of large-scale, frame-wise annotation, and 2) the complexity of two-stage approaches.
Neural 3D Clothes Retargeting from a Single Image
In this paper, we present a method of clothes retargeting; generating the potential poses and deformations of a given 3D clothing template model to fit onto a person in a single RGB image.
Binary TTC: A Temporal Geofence for Autonomous Navigation
Time-to-contact (TTC), the time for an object to collide with the observer's plane, is a powerful tool for path planning: it is potentially more informative than the depth, velocity, and acceleration of objects in the scene -- even for humans.
Parameter Efficient Multimodal Transformers for Video Representation Learning
The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model.
Online Adaptation for Consistent Mesh Reconstruction in the Wild
This paper presents an algorithm to reconstruct temporally consistent 3D meshes of deformable object instances from videos in the wild.
Displacement-Invariant Cost Computation for Efficient Stereo Matching
A common way to speed up the computation is to downsample the feature volume, but this loses high-frequency details.
UFO$^2$: A Unified Framework towards Omni-supervised Object Detection
Existing work on object detection often relies on a single form of annotation: the model is trained using either accurate yet costly bounding boxes or cheaper but less expressive image-level tags.
A Contrastive Learning Approach for Training Variational Autoencoder Priors
To tackle this issue, we propose an energy-based prior defined by the product of a base prior distribution and a reweighting factor, designed to bring the base closer to the aggregate posterior.
Ranked #6 on
Image Generation
on CelebA 256x256
(FID metric)
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
VAEBM captures the overall mode structure of the data distribution using a state-of-the-art VAE and it relies on its EBM component to explicitly exclude non-data-like regions from the model and refine the image samples.
Ranked #1 on
Image Generation
on Stacked MNIST
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors
To tackle this issue, we propose an energy-based prior defined by the product of a base prior distribution and a reweighting factor, designed to bring the base closer to the aggregate posterior.
Improving Deep Stereo Network Generalization with Geometric Priors
End-to-end deep learning methods have advanced stereo vision in recent years and obtained excellent results when the training and test data are similar.
DeepGMR: Learning Latent Gaussian Mixture Models for Registration
Point cloud registration is a fundamental problem in 3D computer vision, graphics and robotics.
Joint Disentangling and Adaptation for Cross-Domain Person Re-Identification
To this end, we propose a joint learning framework that disentangles id-related/unrelated features and enforces adaptation to work on the id-related feature space exclusively.
Ranked #7 on
Unsupervised Domain Adaptation
on Market to MSMT
Hierarchical Contrastive Motion Learning for Video Action Recognition
Thus, the motion features at higher levels are trained to gradually capture semantic dynamics and evolve more discriminative for action recognition.
NVAE: A Deep Hierarchical Variational Autoencoder
For example, on CIFAR-10, NVAE pushes the state-of-the-art from 2. 98 to 2. 91 bits per dimension, and it produces high-quality images on CelebA HQ.
Ranked #3 on
Image Generation
on FFHQ 256 x 256
(bits/dimension metric)
Contrastive Learning for Weakly Supervised Phrase Grounding
Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions.
Bi3D: Stereo Depth Estimation via Binary Classifications
Given a strict time budget, Bi3D can detect objects closer than a given distance in as little as a few milliseconds, or estimate depth with arbitrarily coarse quantization, with complexity linear with the number of quantization levels.
How to Close Sim-Real Gap? Transfer with Segmentation!
Combining the control policy learned from simulation with the perception model, we achieve an impressive $\bf{88\%}$ success rate in grasping a tiny sphere with a real robot.
Robotics
Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detection
Weakly supervised learning has emerged as a compelling tool for object detection by reducing the need for strong supervision during training.
Ranked #1 on
Weakly Supervised Object Detection
on COCO test-dev
Self-Supervised Viewpoint Learning From Image Collections
Training deep neural networks to estimate the viewpoint of objects requires large labeled training datasets.
Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera
Our insight is that although its scale and quality are inconsistent with other views, the depth estimation from a single view can be used to reason about the globally coherent geometry of dynamic contents.
Two-shot Spatially-varying BRDF and Shape Estimation
Extensive experiments on both synthetic and real-world datasets show that our network trained on a synthetic dataset can generalize well to real-world images.
Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints
Estimating 3D hand pose from 2D images is a difficult, inverse problem due to the inherent scale and depth ambiguities.
Ranked #10 on
3D Hand Pose Estimation
on DexYCB
Gaze-Sensing LEDs for Head Mounted Displays
We introduce a new gaze tracker for Head Mounted Displays (HMDs).
Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild
One major challenge for monocular 3D human pose estimation in-the-wild is the acquisition of training data that contains unconstrained images annotated with accurate 3D poses.
Monocular 3D Human Pose Estimation
Weakly-superavised 3D Human Pose Estimation
+1
Self-supervised Single-view 3D Reconstruction via Semantic Consistency
To the best of our knowledge, we are the first to try and solve the single-view reconstruction problem without a category-specific template mesh or semantic keypoints.
Convolutional Tensor-Train LSTM for Spatio-temporal Learning
Learning from spatio-temporal data has numerous applications such as human-behavior analysis, object tracking, video compression, and physics simulation. However, existing methods still perform poorly on challenging video tasks such as long-term forecasting.
Ranked #1 on
Video Prediction
on KTH
(Cond metric)
Learning to Generate Multiple Style Transfer Outputs for an Input Sentence
Specifically, we decompose the latent representation of the input sentence to a style code that captures the language style variation and a content code that encodes the language style-independent content.
Exploiting Semantics for Face Image Deblurring
Specifically, we first use a coarse deblurring network to reduce the motion blur on the input face image.
Discovering Nonlinear Relations with Minimum Predictive Information Regularization
Identifying the underlying directional relations from observational time series with nonlinear interactions and complex relational structures is key to a wide range of applications, yet remains a hard problem.
Meshlet Priors for 3D Mesh Reconstruction
Meshlets act as a dictionary of local features and thus allow to use learned priors to reconstruct object meshes in any pose and from unseen classes, even when the noise is large and the samples sparse.
Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
We introduce DeepInversion, a new method for synthesizing images from the image distribution used to train a deep neural network.
UNAS: Differentiable Architecture Search Meets Reinforcement Learning
Our framework brings the best of both worlds, and it enables us to search for architectures with both differentiable and non-differentiable criteria in one unified framework while maintaining a low search cost.
Angular Visual Hardness
We also find that AVH has a statistically significant correlation with human visual hardness.
Dancing to Music
In the analysis phase, we decompose a dance into a series of basic dance units, through which the model learns how to move.
Ranked #3 on
Motion Synthesis
on BRACE
Few-shot Video-to-Video Synthesis
To address the limitations, we propose a few-shot vid2vid framework, which learns to synthesize videos of previously unseen subjects or scenes by leveraging few example images of the target at test time.
Ranked #1 on
Video-to-Video Synthesis
on YouTube Dancing
SENSE: a Shared Encoder Network for Scene-flow Estimation
We introduce a compact network for holistic scene flow estimation, called SENSE, which shares common encoder features among four closely-related tasks: optical flow estimation, disparity estimation from stereo, occlusion estimation, and semantic segmentation.
Joint-task Self-supervised Learning for Temporal Correspondence
Our learning process integrates two highly related tasks: tracking large image regions \emph{and} establishing fine-grained pixel-level associations between consecutive video frames.
Long History Short-Term Memory for Long-Term Video Prediction
While video prediction approaches have advanced considerably in recent years, learning to predict long-term future is challenging — ambiguous future or error propagation over time yield blurry predictions.
Convolutional Tensor-Train LSTM for Long-Term Video Prediction
Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across a long range of image frames. Standard recurrent models are ineffective since they are prone to error propagation and cannot effectively capture higher-order correlations.
Learning Propagation for Arbitrarily-structured Data
We experiment with semantic segmentation networks, where we use our propagation module to jointly train on different data -- images, superpixels and point clouds.
Video Stitching for Linear Camera Arrays
Despite the long history of image and video stitching research, existing academic and commercial solutions still produce strong artifacts.
Importance Estimation for Neural Network Pruning
On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0. 02% in the top-1 accuracy on ImageNet.
Unsupervised Video Interpolation Using Cycle Consistency
We further introduce a pseudo supervised loss term that enforces the interpolated frames to be consistent with predictions of a pre-trained interpolation model.
Ranked #1 on
Video Frame Interpolation
on UCF101
(PSNR (sRGB) metric)
Few-Shot Viewpoint Estimation
Through extensive experimentation on the ObjectNet3D and Pascal3D+ benchmark datasets, we demonstrate that our framework, which we call MetaView, significantly outperforms fine-tuning the state-of-the-art models with few examples, and that the specific architectural innovations of our method are crucial to achieving good performance.
Few-Shot Adaptive Gaze Estimation
Inter-personal anatomical differences limit the accuracy of person-independent gaze estimation networks.
Ranked #1 on
Gaze Estimation
on MPII Gaze
(using extra training data)
Few-Shot Unsupervised Image-to-Image Translation
Unsupervised image-to-image translation methods learn to map images in a given class to an analogous image in a different class, drawing on unstructured (non-registered) datasets of images.
SCOPS: Self-Supervised Co-Part Segmentation
Parts provide a good intermediate representation of objects that is robust with respect to the camera, pose and appearance variations.
Ranked #4 on
Unsupervised Keypoint Estimation
on CUB
Neural Causal Discovery with Learnable Input Noise
Learning causal relations from observational time series with nonlinear interactions and complex causal structures is a key component of human intelligence, and has a wide range of applications.
STEP: Spatio-Temporal Progressive Learning for Video Action Detection
In this paper, we propose Spatio-TEmporal Progressive (STEP) action detector---a progressive learning framework for spatio-temporal action detection in videos.
Ranked #7 on
Action Detection
on UCF101-24
Joint Discriminative and Generative Learning for Person Re-identification
To this end, we propose a joint learning framework that couples re-id learning and data generation end-to-end.