Search Results for author:
Found 123 papers, 64 papers with code
Urban Scene Diffusion through Semantic Occupancy Map
Generating unbounded 3D scenes is crucial for large-scale scene understanding and simulation.
A Holistic Framework Towards Vision-based Traffic Signal Control with Microscopic Simulation
Traffic signal control (TSC) is crucial for reducing traffic congestion that leads to smoother traffic flow, reduced idling time, and mitigated CO2 emissions.
Unsupervised Discovery of Steerable Factors When Graph Deep Generative Models Are Entangled
Deep generative models (DGMs) have been widely developed for graph data.
SceneWiz3D: Towards Text-guided 3D Scene Composition
We are witnessing significant breakthroughs in the technology for generating 3D objects from text.
FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition
Recent approaches such as ControlNet offer users fine-grained spatial control over text-to-image (T2I) diffusion models.
BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation
Generating large-scale 3D scenes cannot simply apply existing 3D object synthesis technique since 3D scenes usually hold complex spatial configurations and consist of a number of objects at varying scales.
CAT: Closed-loop Adversarial Training for Safe End-to-End Driving
Driving safety is a top priority for autonomous vehicles.
In-Domain GAN Inversion for Faithful Reconstruction and Editability
This work fills in this gap by proposing in-domain GAN inversion, which consists of a domain-guided encoder and a domain-regularized optimizer, to regularize the inverted code in the native latent space of the pre-trained GAN model.
Efficient 3D Articulated Human Generation with Layered Surface Volumes
These layers are rendered using alpha compositing with fast differentiable rasterization, and they can be interpreted as a volumetric representation that allocates its capacity to a manifold of finite thickness around the template.
Next Steps for Human-Centered Generative AI: A Technical Perspective
Through iterative, cross-disciplinary discussions, we define and propose next-steps for Human-centered Generative AI (HGAI).
V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception
To facilitate the development of cooperative perception, we present V2V4Real, the first large-scale real-world multi-modal dataset for V2V perception.
Guarded Policy Optimization with Imperfect Online Demonstrations
Assuming optimal, the teacher policy has the perfect timing and capability to intervene in the learning process of the student agent, providing safety guarantee and exploration guidance.
Spatial Steerability of GANs via Self-Supervision from Discriminator
In this work, we propose a self-supervised approach to improve the spatial steerability of GANs without searching for steerable directions in the latent space or requiring extra annotations.
GH-Feat: Learning Versatile Generative Hierarchical Features from GANs
In this work we investigate that such a generative feature learned from image synthesis exhibits great potentials in solving a wide range of computer vision tasks, including both generative ones and more importantly discriminative ones.
Street-View Image Generation from a Bird's-Eye View Layout
Instead of using perception data from real-life scenarios, an ideal model for simulation would generate realistic street-view images that align with a given HD map and traffic layout, a task that is critical for visualizing complex traffic scenarios and developing robust perception models for autonomous driving.
DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis
Existing 3D-aware image synthesis approaches mainly focus on generating a single canonical object and show limited capacity in composing a complex scene containing a variety of objects.
Towards Smooth Video Composition
Video generation requires synthesizing consistent and persistent frames with dynamic content over time.
Ranked #1 on
Video Generation
on YouTube Driving
V2XP-ASG: Generating Adversarial Scenes for Vehicle-to-Everything Perception
Recent advancements in Vehicle-to-Everything communication technology have enabled autonomous vehicles to share sensory information to obtain better perception performance.
Improving GANs with A Dynamic Discriminator
Two capacity adjusting schemes are developed for training GANs under different data regimes: i) given a sufficient amount of training data, the discriminator benefits from a progressively increased learning capacity, and ii) when the training data is limited, gradually decreasing the layer width mitigates the over-fitting issue of the discriminator.
Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping
We validate our insight on a range of RL tasks and show its improvement over baselines: (1) In offline RL, the conservative exploitation leads to improved performance based on off-the-shelf algorithms; (2) In online continuous control, multiple value functions with different shifting constants can be used to tackle the exploration-exploitation dilemma for better sample efficiency; (3) In discrete control tasks, a negative reward shifting yields an improvement over the curiosity-based exploration method.
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers
The extensive experiments on the V2V perception dataset, OPV2V, demonstrate that CoBEVT achieves state-of-the-art performance for cooperative BEV semantic segmentation.
Human-AI Shared Control via Policy Dissection
Inspired by the neuroscience approach to investigate the motor cortex in primates, we develop a simple yet effective frequency-based approach called \textit{Policy Dissection} to align the intermediate representation of the learned neural controller with the kinematic attributes of the agent behavior.
Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining
Specifically, we train an inverse dynamic model with a small amount of labeled data and use it to predict action labels for all the YouTube video frames.
Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation
To enhance the quality of synthesized gestures, we develop a contrastive learning strategy based on audio-text alignment for better audio representations.
Ranked #3 on
Gesture Generation
on TED Gesture Dataset
LocATe: End-to-end Localization of Actions in 3D with Transformers
An important component of this problem is 3D Temporal Action Localization (3D-TAL), which involves recognizing what actions a person is performing, and when.
Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization
HACO can train agents to drive in unseen traffic scenarios with a handful of human intervention budget and achieve high safety and generalizability, outperforming both reinforcement learning and imitation learning baselines with a large margin.
Visual Sound Localization in the Wild by Cross-Modal Interference Erasing
Specifically, we observe that the previous practice of learning only a single audio representation is insufficient due to the additive nature of audio signals.
Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation
Moreover, to enable portrait rendering in one unified neural radiance field, a Torso Deformation module is designed to stabilize the large-scale non-rigid torso motions.
AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detection
This map enables our model to automate the alignment of non-homogenous features in a dynamic and data-driven manner.
3D-aware Image Synthesis via Learning Structural and Textural Representations
The feature field is further accumulated into a 2D feature map as the textural representation, followed by a neural renderer for appearance synthesis.
Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognition
Typically in recent work, the pseudo-labels are obtained by training a model on the labeled data, and then using confident predictions from the model to teach itself.
SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations
To bridge this gap, we aim to learn a spatial-aware visual representation that can describe the three-dimensional space and is more suitable and effective for these tasks.
Improving GAN Equilibrium by Raising Spatial Awareness
We further propose to align the spatial awareness of G with the attention map induced from D. Through this way we effectively lessen the information gap between D and G. Extensive results show that our method pushes the two-player game in GANs closer to the equilibrium, leading to a better synthesis performance.
One-Shot Generative Domain Adaptation
We then equip the well-learned discriminator backbone with an attribute classifier to ensure that the generator captures the appropriate characters from the reference.
Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization
Self-Driven Particles (SDP) describe a category of multi-agent systems common in everyday life, such as flocking birds and traffic flows.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
The Nuts and Bolts of Adopting Transformer in GANs
Transformer becomes prevalent in computer vision, especially for high-level vision tasks.
Safe Driving via Expert Guided Policy Optimization
Offline RL technique is further used to learn from the partial demonstration generated by the expert.
Improving Out-of-Distribution Robustness of Classifiers Through Interpolated Generative Models
In this paper, we employ interpolated generative models to generate OoD samples at training time via data augmentation.
SPLID: Self-Imitation Policy Learning through Iterative Distillation
To this end, we propose a novel meta-algorithm Self-Imitation Policy Learning through Iterative Distillation (SPLID) which relies on the concept of $\delta$-distilled policy to iteratively level up the quality of the target data and agent mimics from the relabeled target data.
Reward Shifting for Optimistic Exploration and Conservative Exploitation
We verify our insight on a range of tasks: (1) In offline RL, the conservative exploitation leads to improved learning performance based on off-the-shelf algorithms; (2) In online continuous control, multiple value functions with different shifting constants can be used to trade-off between exploration and exploitation thus improving learning efficiency; (3) In online RL with discrete action space, a negative reward shifting brings an improvement over the previous curiosity-based exploration method.
Interpreting Molecule Generative Models for Interactive Molecule Discovery
In this work, we develop a simple yet effective method to interpret the latent space of the learned generative models with various molecular properties for more interactive molecule generation and discovery.
MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
Based on MetaDrive, we construct a variety of RL tasks and baselines in both single-agent and multi-agent settings, including benchmarking generalizability across unseen scenes, safe exploration, and learning multi-agent traffic.
Interpreting Generative Adversarial Networks for Interactive Image Generation
Significant progress has been made by the advances in Generative Adversarial Networks (GANs) for image generation.
Safe Exploration by Solving Early Terminated MDP
Safe exploration is crucial for the real-world application of reinforcement learning (RL).
Data-Efficient Instance Generation from Instance Discrimination
Meanwhile, the learned instance discrimination capability from the discriminator is in turn exploited to encourage the generator for diverse generation.
Ranked #6 on
Image Generation
on FFHQ 256 x 256
(FD metric)
TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization
TS-CAM finally couples the patch tokens with the semantic-agnostic attention map to achieve semantic-aware localization.
Multimodal Motion Prediction with Stacked Transformers
Predicting multiple plausible future trajectories of the nearby vehicles is crucial for the safety of autonomous driving.
Unsupervised Image Transformation Learning via Generative Adversarial Networks
In this work, we study the image transformation problem, which targets at learning the underlying transformations (e. g., the transition of seasons) from a collection of unlabeled images.
Instance Localization for Self-supervised Detection Pretraining
The pretext task is to predict the instance category given the composited images as well as the foreground bounding boxes.
Deep Learning for Scene Classification: A Survey
Scene classification, aiming at classifying a scene image to one of the predefined scene categories by comprehending the entire image, is a longstanding, fundamental and challenging problem in computer vision.
GAN Inversion: A Survey
GAN inversion aims to invert a given image back into the latent space of a pretrained GAN model, for the image to be faithfully reconstructed from the inverted code by the generator.
Self-Supervised Continuous Control without Policy Gradient
Despite the remarkable progress made by the policy gradient algorithms in reinforcement learning (RL), sub-optimal policies usually result from the local exploration property of the policy gradient update.
Improving the Generalization of End-to-End Driving through Procedural Generation
We validate that training with the increasing number of procedurally generated scenes significantly improves the generalization of the agent across scenarios of different traffic densities and road networks.
Positional Encoding as Spatial Inductive Bias in GANs
In this work, taking SinGAN and StyleGAN2 as examples, we show that such capability, to a large extent, is brought by the implicit positional encoding when using zero padding in the generators.
Improving the Fairness of Deep Generative Models without Retraining
Generative Adversarial Networks (GANs) advance face synthesis through learning the underlying distribution of observed data.
Texture Memory-Augmented Deep Patch-Based Image Inpainting
By bringing together the best of both paradigms, we propose a new deep inpainting framework where texture generation is guided by a texture memory of patch samples extracted from unmasked regions.
Understanding the Role of Individual Units in a Deep Neural Network
Second, we use a similar analytic method to analyze a generative adversarial network (GAN) model trained to generate scenes.
A Unified Framework for Shot Type Classification Based on Subject Centric Lens
Shots are key narrative elements of various videos, e. g. movies, TV series, and user-generated videos that are thriving over the Internet.
Generative Hierarchical Features from Synthesizing Images
Generative Adversarial Networks (GANs) have recently advanced image synthesis by learning the underlying distribution of the observed data.
Closed-Form Factorization of Latent Semantics in GANs
A rich set of interpretable dimensions has been shown to emerge in the latent space of the Generative Adversarial Networks (GANs) trained for synthesizing images.
Unsupervised Landmark Learning from Unpaired Data
Recent attempts for unsupervised landmark learning leverage synthesized image pairs that are similar in appearance but different in poses.
Video Representation Learning with Visual Tempo Consistency
Visual tempo, which describes how fast an action goes, has shown its potential in supervised action recognition.
Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
Conventional Reinforcement Learning (RL) algorithms usually have one single agent learning to solve the task independently.
Zeroth-Order Supervised Policy Improvement
However, PG algorithms rely on exploiting the value function being learned with the first-order update locally, which results in limited sample efficiency.
Novel Policy Seeking with Constrained Optimization
In problem-solving, we humans can come up with multiple novel solutions to the same problem.
InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs
In this work, we propose a framework called InterFaceGAN to interpret the disentangled face representation learned by the state-of-the-art GAN models and study the properties of the facial semantics encoded in the latent space.
Semantic Photo Manipulation with a Generative Image Prior
First, it is hard for GANs to precisely reproduce an input image.
Evolutionary Stochastic Policy Distillation
Solving the Goal-Conditioned Reward Sparse (GCRS) task is a challenging reinforcement learning problem due to the sparsity of reward signals.
TPNet: Trajectory Proposal Network for Motion Prediction
However, it remains difficult for these methods to provide multimodal predictions as well as integrate physical constraints such as traffic rules and movable areas.
Temporal Pyramid Network for Action Recognition
Previous works often capture the visual tempo through sampling raw videos at multiple rates and constructing an input-level frame pyramid, which usually requires a costly multi-branch network to handle.
Ranked #105 on
Action Recognition
on Something-Something V2
A Local-to-Global Approach to Multi-modal Movie Scene Segmentation
Scene, as the crucial unit of storytelling in movies, contains complex activities of actors and their interactions in a physical environment.
In-Domain GAN Inversion for Real Image Editing
A common practice of feeding a real image to a trained GAN generator is to invert it back to a latent code.
TransMoMo: Invariance-Driven Unsupervised Video Motion Retargeting
We present a lightweight video motion retargeting approach TransMoMo that is capable of transferring motion of a person in a source video realistically to another video of a target person.
Image Processing Using Multi-Code GAN Prior
Such an over-parameterization of the latent space significantly improves the image reconstruction quality, outperforming existing competitors.
Ranked #7 on
Blind Face Restoration
on CelebA-Test
Learning a Decision Module by Imitating Driver's Control Behaviors
In this work, we propose a hybrid framework to learn neural decisions in the classical modular pipeline through end-to-end imitation learning.
Every Frame Counts: Joint Learning of Video Segmentation and Optical Flow
Moreover, our framework is able to utilize both labeled and unlabeled frames in the video through joint training, while no additional calculation is required in inference.
Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what generative models have learned inside the deep generative representations and how photo-realistic images are able to be composed of the layer-wise stochasticity introduced in recent GANs.
Policy Continuation with Hindsight Inverse Dynamics
This approach learns from Hindsight Inverse Dynamics based on Hindsight Experience Replay, enabling the learning process in a self-imitated manner and thus can be trained with supervised learning.
Seeing What a GAN Cannot Generate
Differences in statistics reveal object classes that are omitted by a GAN.
A Graph-Based Framework to Bridge Movies and Synopses
On top of this dataset, we develop a framework to perform matching between movie segments and synopsis paragraphs.
Semantic Hierarchy Emerges in the Deep Generative Representations for Scene Synthesis
Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what networks have learned inside the deep generative representations and how photo-realistic images are able to be composed from random noises.
Learning with Social Influence through Interior Policy Differentiation
In this work we model the social influence into the scheme of reinforcement learning, enabling the agents to learn both from the environment and from their peers.
LIA: Latently Invertible Autoencoder with Adversarial Learning
A two-stage stochasticity-free training scheme is designed to train LIA via adversarial learning, in the sense that the decoder of LIA is first trained as a standard GAN with the invertible network and then the partial encoder is learned from an autoencoder by detaching the invertible network from LIA.
Reasoning About Human-Object Interactions Through Dual Attention Networks
The model not only finds when an action is happening and which object is being manipulated, but also identifies which part of the object is being interacted with.
Interpreting the Latent Space of GANs for Semantic Face Editing
In this work, we propose a novel framework, called InterFaceGAN, for semantic face editing by interpreting the latent semantics learned by GANs.
Disentangled Inference for GANs with Latently Invertible Autoencoder
In this paper, we show that the entanglement of the latent space for the VAE/GAN framework poses the main challenge for encoder learning.
Cross-view Semantic Segmentation for Sensing Surroundings
Our further experiment on a LoCoBot robot shows that our model enables the surrounding sensing capability from 2D image input.
Deep Flow-Guided Video Inpainting
Then the synthesized flow field is used to guide the propagation of pixels to fill up the missing regions in the video.
Ranked #8 on
Video Inpainting
on DAVIS
One-shot visual object segmentation
Optical Flow Estimation
+2
Visualizing and Understanding GANs
We present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level.
On the Units of GANs (Extended Abstract)
We quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.
Proceedings of AAAI 2019 Workshop on Network Interpretability for Deep Learning
This is the Proceedings of AAAI 2019 Workshop on Network Interpretability for Deep Learning
FaceFeat-GAN: a Two-Stage Approach for Identity-Preserving Face Synthesis
In the second stage, they compete in the image domain to render photo-realistic images that contain high diversity but preserve identity.
GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.
Single Image Intrinsic Decomposition without a Single Intrinsic Image
At inference time, our model can be easily reduced to a single stream module that performs intrinsic decomposition on a single input image.
Interpretable Basis Decomposition for Visual Explanation
Explanations of the decisions made by a deep neural network are important for human end-users to be able to understand and diagnose the trustworthiness of the system.
Real-Time Object Pose Estimation with Pose Interpreter Networks
In this work, we introduce pose interpreter networks for 6-DoF object pose estimation.
Unified Perceptual Parsing for Scene Understanding
In this paper, we study a new task called Unified Perceptual Parsing, which requires the machine vision systems to recognize as many visual concepts as possible from a given image.
Ranked #88 on
Semantic Segmentation
on ADE20K val
Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation
Generating scene graph to describe all the relations inside an image gains increasing interests these years.
Ranked #1 on
Scene Graph Generation
on VRD
Revisiting the Importance of Individual Units in CNNs via Ablation
We confirm that unit attributes such as class selectivity are a poor predictor for impact on overall accuracy as found previously in recent work \cite{morcos2018importance}.
DeepMiner: Discovering Interpretable Representations for Mammogram Classification and Explanation
We propose DeepMiner, a framework to discover interpretable representations in deep neural networks and to build explanations for medical predictions.
Expert identification of visual primitives used by CNNs during mammogram classification
This work interprets the internal representations of deep neural networks trained for classification of diseased tissue in 2D mammograms.
Recurrent Residual Module for Fast Inference in Videos
Deep convolutional neural networks (CNNs) have made impressive progress in many video recognition tasks such as video pose estimation and video object detection.
Moments in Time Dataset: one million videos for event understanding
We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds.
Temporal Relational Reasoning in Videos
Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is a fundamental property of intelligent species.
Ranked #2 on
Hand Gesture Recognition
on Jester test
Interpreting Deep Visual Representations via Network Dissection
In this work, we describe Network Dissection, a method that interprets networks by providing labels for the units of their deep visual representations.
Visual Question Generation as Dual Task of Visual Question Answering
Recently visual question answering (VQA) and visual question generation (VQG) are two trending topics in the computer vision, which have been explored separately.
Scene Graph Generation from Objects, Phrases and Region Captions
Object detection, scene graph generation and region captioning, which are three scene understanding tasks at different semantic levels, are tied together: scene graphs are generated on top of objects detected in an image with their pairwise relationship predicted, while region captioning gives a language description of the objects, their attributes, relations, and other context information.
Ranked #2 on
Object Detection
on Visual Genome
Scene Parsing Through ADE20K Dataset
A novel network design called Cascade Segmentation Module is proposed to parse a scene into stuff, objects, and object parts in a cascade and improve over the baselines.
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Given any CNN model, the proposed method draws on a broad data set of visual concepts to score the semantics of hidden units at each intermediate convolutional layer.
Open Vocabulary Scene Parsing
Recognizing arbitrary objects in the wild has been a challenging problem due to the limitations of existing classification models and datasets.
SegICP: Integrated Deep Semantic Segmentation and Pose Estimation
Recent robotic manipulation competitions have highlighted that sophisticated robots still struggle to achieve fast and reliable perception of task-relevant objects in complex, realistic scenarios.
Person Search with Natural Language Description
Searching persons in large-scale image databases with the query of natural language description has important applications in video surveillance.
Places: An Image Database for Deep Scene Understanding
The rise of multi-million-item dataset initiatives has enabled data-hungry machine learning algorithms to reach near-human semantic classification at tasks such as object and scene recognition.
Semantic Understanding of Scenes through the ADE20K Dataset
Scene parsing, or recognizing and segmenting objects and stuff in an image, is one of the key problems in computer vision.
Learning Deep Features for Discriminative Localization
In this work, we revisit the global average pooling layer proposed in [13], and shed light on how it explicitly enables the convolutional neural network to have remarkable localization ability despite being trained on image-level labels.
Simple Baseline for Visual Question Answering
We describe a very simple bag-of-words baseline for visual question answering.
Optimization as Estimation with Gaussian Processes in Bandit Settings
Recently, there has been rising interest in Bayesian optimization -- the optimization of an unknown function with assumptions usually expressed by a Gaussian Process (GP) prior.
Understanding Intra-Class Knowledge Inside CNN
Convolutional Neural Network (CNN) has been successful in image recognition tasks, and recent works shed lights on how CNN separates different classes with the learned inter-class knowledge through visualization.
Object Detectors Emerge in Deep Scene CNNs
With the success of new computational architectures for visual processing, such as convolutional neural networks (CNN) and access to image databases with millions of labeled examples (e. g., ImageNet, Places), the state of the art in computer vision is advancing rapidly.
Learning Deep Features for Scene Recognition using Places Database
Whereas the tremendous recent progress in object recognition tasks is due to the availability of large datasets like ImageNet and the rise of Convolutional Neural Networks (CNNs) for learning high-level features, performance at scene recognition has not attained the same level of success.
ConceptLearner: Discovering Visual Concepts from Weakly Labeled Image Collections
Discovering visual knowledge from weakly labeled data is crucial to scale up computer vision recognition system, since it is expensive to obtain fully labeled data for a large number of concept categories.
Measuring Crowd Collectiveness
Collective motions are common in crowd systems and have attracted a great deal of attention in a variety of multidisciplinary fields.
