Search Results for author:
Found 72 papers, 31 papers with code
Proposal-based Video Completion
Video inpainting is an important technique for a wide variety of applications from video content editing to video restoration.
Studying Classifier(-Free) Guidance From a Classifier-Centric Perspective
Classifier-free guidance has become a staple for conditional generation with denoising diffusion models.
Variational Rectified Flow Matching
We study Variational Rectified Flow Matching, a framework that enhances classic rectified flow matching by modeling multi-modal velocity vector-fields.
LIFe-GoM: Generalizable Human Rendering with Learned Iterative Feedback Over Multi-Resolution Gaussians-on-Mesh
Generalizable rendering of an animatable human avatar from sparse inputs relies on data priors and inductive biases extracted from training on large data to avoid scene-specific optimization and to enable fast reconstruction.
Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers
As suggested by our analysis, in our experiments, we hence find that simply adding TD3 gradients to the finetuning process of ODT effectively improves the online finetuning performance of ODT, especially if ODT is pretrained with low-reward offline data.
GoMAvatar: Efficient Animatable Human Modeling from Monocular Video Using Gaussians-on-Mesh
We introduce GoMAvatar, a novel approach for real-time, memory-efficient, high-quality animatable human modeling.
OW-VISCapTor: Abstractors for Open-World Video Instance Segmentation and Captioning
Concretely, we connect a multi-scale visual feature extractor and a large language model (LLM) by developing an object abstractor and an object-to-text abstractor.
StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D
Our analysis identifies the core of these challenges as the interaction among noise levels in the 2D diffusion process, the architecture of the diffusion network, and the 3D model representation.
Offline Imitation from Observation via Primal Wasserstein State Occupancy Matching
It minimizes the primal Wasserstein distance between the learner and expert state occupancies and leverages a contrastively learned distance metric.
Pseudo-Generalized Dynamic View Synthesis from a Video
In contrast, for dynamic scenes, scene-specific optimization techniques exist, but, to our best knowledge, there is currently no generalized method for dynamic novel view synthesis from a given monocular video.
Robust Model-Based Optimization for Challenging Fitness Landscapes
Protein design, a grand challenge of the day, involves optimization on a fitness landscape, and leading methods adopt a model-based approach where a model is trained on a training set (protein sequences and fitness) and proposes candidates to explore next.
Context-Aware Relative Object Queries To Unify Video Instance and Panoptic Segmentation
We evaluate the proposed approach across three challenging tasks: video instance segmentation, multi-object tracking and segmentation, and video panoptic segmentation.
CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations
To better benefit from available demonstrations, we develop a method to Combine Explicit and Implicit Priors (CEIP).
Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks
We propose learnable polyphase sampling (LPS), a pair of learnable down/upsampling layers that enable truly shift-invariant and equivariant convolutional networks.
Controllable Radiance Fields for Dynamic Face Synthesis
Recent work on 3D-aware image synthesis has achieved compelling results using advances in neural rendering.
Learning to Decompose Visual Features with Latent Textual Prompts
Recent advances in pre-training vision-language models like CLIP have shown great potential in learning transferable visual representations.
Occupancy Planes for Single-view RGB-D Human Reconstruction
Specifically, a set of 3D locations within the view-frustum of the camera are first projected independently onto the image and a corresponding feature is subsequently extracted for each 3D location.
Initialization and Alignment for Adversarial Texture Optimization
While recovery of geometry from image and video data has received a lot of attention in computer vision, methods to capture the texture for a given geometry are less mature.
Generative Multiplane Images: Making a 2D GAN 3D-Aware
What is really needed to make an existing 2D GAN 3D-aware?
XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model
We present XMem, a video object segmentation architecture for long videos with unified feature memory stores inspired by the Atkinson-Shiffrin memory model.
Ranked #1 on
Video Object Segmentation
on YouTube-VOS 2019
(using extra training data)
Asking for Knowledge: Training RL Agents to Query External Knowledge Using Language
In order to study how agents can be taught to query external knowledge via language, we first introduce two new environments: the grid-world-based Q-BabyAI and the text-based Q-TextWorld.
Equivariance Discovery by Learned Parameter-Sharing
Designing equivariance as an inductive bias into deep-nets has been a prominent approach to build effective models, e. g., a convolutional neural network incorporates translation equivariance.
Total Variation Optimization Layers for Computer Vision
To study question (a), in this work, we propose total variation (TV) minimization as a layer for computer vision.
Mask2Former for Video Instance Segmentation
We find Mask2Former also achieves state-of-the-art performance on video instance segmentation without modifying the architecture, the loss or even the training pipeline.
Class-agnostic Reconstruction of Dynamic Objects from Videos
We introduce REDO, a class-agnostic framework to REconstruct the Dynamic Objects from RGBD or calibrated videos.
Masked-attention Mask Transformer for Universal Image Segmentation
While only the semantics of each task differ, current research focuses on designing specialized architectures for each task.
Ranked #3 on
Semantic Segmentation
on Mapillary val
Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning
We evaluate `semantic tracklets' on the visual multi-agent particle environment (VMPE) and on the challenging visual multi-agent GFootball environment.
Multi-agent Reinforcement Learning
reinforcement-learning
+2
Cooperative Exploration for Multi-Agent Deep Reinforcement Learning
To address this shortcoming, in this paper, we propose cooperative multi-agent exploration (CMAE): agents share a common goal while exploring.
Per-Pixel Classification is Not All You Need for Semantic Segmentation
Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results.
Ranked #4 on
Semantic Segmentation
on Mapillary val
Robustifying $\ell_\infty$ Adversarial Training to the Union of Perturbation Models
Classical adversarial training (AT) frameworks are designed to achieve high adversarial accuracy against a single attack type, typically $\ell_\infty$ norm-bounded perturbations.
SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Data
Moreover, existing image-based datasets for mesh reconstruction don't permit to study models which integrate temporal information.
3D Spatial Recognition without Spatially Labeled 3D
We introduce WyPR, a Weakly-supervised framework for Point cloud Recognition, requiring only scene-level class tags as supervision.
DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization
The recent growth of web video sharing platforms has increased the demand for systems that can efficiently browse, retrieve and summarize video content.
Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation
Controllable semantic image editing enables a user to change entire image attributes with a few clicks, e. g., gradually making a summer scene look like it was taken in winter.
Assignment-Space-Based Multi-Object Tracking and Segmentation
In contrast, we formulate a global method for MOTS over the space of assignments rather than detections: First, we find all top-k assignments of objects detected and segmented between any two consecutive frames and develop a structured prediction formulation to score assignment sequences across any number of consecutive frames.
Multi-Object Tracking
Multi-Object Tracking and Segmentation
+4
High-Throughput Synchronous Deep RL
In contrast, asynchronous methods achieve high throughput but suffer from stability issues and lower sample efficiency due to `stale policies.'
UFO$^2$: A Unified Framework towards Omni-supervised Object Detection
Existing work on object detection often relies on a single form of annotation: the model is trained using either accurate yet costly bounding boxes or cheaper but less expressive image-level tags.
Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning
Existing semi-supervised learning (SSL) algorithms use a single weight to balance the loss of labeled and unlabeled examples, i. e., all unlabeled examples are equally weighted.
Can We Learn Heuristics For Graphical Model Inference Using Reinforcement Learning?
In this paper, we show that we can learn program heuristics, i. e., policies, for solving inference in higher order CRFs for the task of semantic segmentation, using reinforcement learning.
Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detection
Weakly supervised learning has emerged as a compelling tool for object detection by reducing the need for strong supervision during training.
Ranked #1 on
Weakly Supervised Object Detection
on COCO test-dev
Chirality Nets for Human Pose Regression
We propose Chirality Nets, a family of deep nets that is equivariant to the "chirality transform," i. e., the transformation to create a chiral pair.
PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning
Sample efficiency and scalability to a large number of agents are two important goals for multi-agent reinforcement learning systems.
Deep Reinforcement Learning
Multi-agent Reinforcement Learning
+2
Co-Generation with GANs using AIS based HMC
Inferring the most likely configuration for a subset of variables of a joint distribution given the remaining ones - which we refer to as co-generation - is an important challenge that is computationally demanding for all but the simplest settings.
TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
Despite impressive recent progress that has been reported on tasks that necessitate reasoning, such as visual question answering and visual dialog, models often exploit biases in datasets.
FMRI data augmentation via synthesis
Thus, our results suggest that data augmentation via synthesis is a promising approach to address the limited availability of fMRI data, and to improve the quality of predictive fMRI models.
Knowledge Flow: Improve Upon Your Teachers
A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model.
Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
Given a question-image pair, deep network techniques have been employed to successively reduce the large set of facts until one of the two entities of the final remaining fact is predicted as the answer.
Structural Consistency and Controllability for Diverse Colorization
Colorizing a given gray-level image is an important task in the media and advertising industry.
Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering
Question answering is an important task for autonomous agents and virtual assistants alike and was shown to support the disabled in efficiently navigating an overwhelming environment.
VideoMatch: Matching based Video Object Segmentation
Due to the formulation as a prediction task, most of these methods require fine-tuning during test time, such that the deep nets memorize the appearance of the objects of interest in the given video.
Unsupervised Video Object Segmentation using Motion Saliency-Guided Spatio-Temporal Propagation
We even demonstrate competitive results comparable to deep learning based methods in the semi-supervised setting on the DAVIS dataset.
Ranked #4 on
Video Salient Object Detection
on DAVSOD-Difficult20
(using extra training data)
Diverse and Coherent Paragraph Generation from Images
Paragraph generation from images, which has gained popularity recently, is an important task for video summarization, editing, and support of the disabled.
Unsupervised Textual Grounding: Linking Words to Image Concepts
Textual grounding, i. e., linking words to objects in images, is a challenging but important task for robotics and human-computer interaction.
Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts
Textual grounding is an important but challenging task for human-computer interaction, robotics and knowledge mining.
MaskRNN: Instance Level Video Object Segmentation
Instance level video object segmentation is an important technique for video editing and compression.
Fast and Accurate Text Classification: Skimming, Rereading and Early Stopping
Recent advances in recurrent neural nets (RNNs) have shown much promise in many applications in natural language processing.
Hallucinating brains with artificial brains
Our classification results provide a quantitative evaluation of the quality of the generated images, and also serve as an additional contribution of this manuscript.
Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space
This paper explores image caption generation using conditional variational auto-encoders (CVAEs).
High-Order Attention Models for Visual Question Answering
The quest for algorithms that enable cognitive abilities is an important part of machine learning.
Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening
We propose a novel training algorithm for reinforcement learning which combines the strength of deep Q-learning with a constrained optimization approach to tighten optimality and encourage faster reward propagation.
Semantic Image Inpainting with Deep Generative Models
In this paper, we propose a novel method for semantic image inpainting, which generates the missing content by conditioning on the available data.
Efficient Deep Learning for Stereo Matching
In the past year, convolutional neural networks have been shown to perform extremely well for stereo estimation.
Training Deep Neural Networks via Direct Loss Minimization
Supervised training of deep neural nets typically relies on minimizing cross-entropy.
Rent3D: Floor-Plan Priors for Monocular Layout Estimation
What sets us apart from past work in layout estimation is the use of floor plans as a source of prior knowledge, as well as localization of each image within a bigger space (apartment).
Learning to Segment Under Various Forms of Weak Supervision
Despite the promising performance of conventional fully supervised algorithms, semantic segmentation has remained an important, yet challenging task.
Monocular Object Instance Segmentation and Depth Ordering with CNNs
In this paper we tackle the problem of instance-level segmentation and depth ordering from a single monocular image.
Fully Connected Deep Structured Networks
Convolutional neural networks with many layers have recently been shown to achieve excellent results on many high-level tasks such as image classification, object detection and more recently also semantic segmentation.
Learning Deep Structured Models
Towards this goal, we propose a training algorithm that is able to learn structured models jointly with deep features that form the MRF potentials.
Efficient Structured Parsing of Facades Using Dynamic Programming
We propose a sequential optimization technique for segmenting a rectified image of a facade into semantic categories.
Tell Me What You See and I will Show You Where It Is
We tackle the problem of weakly labeled semantic segmentation, where the only source of annotation are image tags encoding which classes are present in the scene.
