Search Results for author:
Found 228 papers, 93 papers with code
Towards Synthesized and Editable Motion In-Betweening Through Part-Wise Phase Representation
Styled motion in-betweening is crucial for computer animation and gaming.
Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games
Multi-agent reinforcement learning (MARL) lies at the heart of a plethora of applications involving the interaction of a group of agents in a shared unknown environment.
Multi-agent Reinforcement Learning
Uncertainty Quantification
Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates
The proofs are based on novel findings about action sampling rates and the relationship between cumulative progress and noise, and extend the current understanding of how simple stochastic gradient methods behave in bandit settings.
GAS: Generative Avatar Synthesis from a Single Image
Additionally, we propose a unified framework that enables the generalization learned from novel pose synthesis on in-the-wild videos to naturally transfer to novel view synthesis.
EdgeTAM: On-Device Track Anything Model
Given that video segmentation is a dense prediction task, we find preserving the spatial structure of the memories is essential so that the queries are split into global-level and patch-level groups.
Learning 3D Garment Animation from Trajectories of A Piece of Cloth
Without the priors from analytical physics models and differentiable simulation engines, EUNet is able to directly capture the constitutive behaviors from the observed piece of cloth and uniformly describes the change of energy caused by deformations, such as stretching and bending.
GausSim: Foreseeing Reality by Gaussian Simulator for Elastic Objects
We introduce GausSim, a novel neural network-based simulator designed to capture the dynamic behaviors of real-world elastic objects represented through Gaussian kernels.
ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model
In this paper, we introduce a scalable motion generation framework that includes the motion tokenizer Motion FSQ-VAE and a text-prefix autoregressive transformer.
Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models
Recent studies have indicated that effectively utilizing inference-time compute is crucial for attaining better performance from large language models (LLMs).
ChatDyn: Language-Driven Multi-Actor Dynamics Generation in Street Scenes
Generating realistic and interactive dynamics of traffic participants according to specific instruction is critical for street scene simulation.
Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians
Buildings are primary components of cities, often featuring repeated elements such as windows and doors.
Horizon-GS: Unified 3D Gaussian Splatting for Large-Scale Aerial-to-Ground Scenes
Seamless integration of both aerial and street view images remains a significant challenge in neural scene reconstruction and rendering.
On Domain-Specific Post-Training for Multimodal Large Language Models
This paper systematically investigates domain adaptation of MLLMs through post-training, focusing on data synthesis, training pipelines, and task evaluation.
DRiVE: Diffusion-based Rigging Empowers Generation of Versatile and Expressive Characters
Recent advances in generative models have enabled high-quality 3D character reconstruction from multi-modal.
Boosting 3D Object Generation through PBR Materials
By analyzing the components of PBR materials, we choose to consider albedo, roughness, metalness, and bump maps.
GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation
While 3D content generation has advanced significantly, existing methods still face challenges with input formats, latent space design, and output representations.
Matryoshka: Learning to Drive Black-Box LLMs with LLMs
To address this challenge, we introduce Matryoshika, a lightweight white-box LLM controller that guides a large-scale black-box LLM generator by decomposing complex tasks into a series of intermediate outputs.
Faster WIND: Accelerating Iterative Best-of-$N$ Distillation for LLM Alignment
Recent advances in aligning large language models with human preferences have corroborated the growing importance of best-of-N distillation (BOND).
Primal-Dual Spectral Representation for Off-policy Evaluation
We highlight that our algorithm, SpectralDICE, is the first to leverage the linear representation of primal-dual variables that is both computation and sample efficient, the performance of which is supported by a rigorous theoretical sample complexity guarantee and a thorough empirical evaluation on various benchmarks.
Scalable spectral representations for multi-agent reinforcement learning in network MDPs
Network Markov Decision Processes (MDPs), a popular model for multi-agent control, pose a significant challenge to efficient learning due to the exponential growth of the global state-action space with the number of agents.
VideoAgent: Self-Improving Video Generation
Instead of directly executing the generated video plan, VideoAgent first refines the generated video plans using a novel procedure which we call self-conditioning consistency, utilizing feedback from a pretrained vision-language model (VLM).
Scaling Laws For Diffusion Transformers
Diffusion transformers (DiT) have already achieved appealing synthesis and scaling properties in content recreation, e. g., image and video generation.
EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing
Diffusion transformers have been widely adopted for text-to-image synthesis.
Fisheye-GS: Lightweight and Extensible Gaussian Splatting Module for Fisheye Cameras
However, adapting 3DGS to different camera models, particularly fisheye lenses, poses challenges due to the unique 3D to 2D projection calculation.
CinePreGen: Camera Controllable Video Previsualization via Engine-powered Diffusion
With advancements in video generative AI models (e. g., SORA), creators are increasingly using these techniques to enhance video previsualization.
FlashGS: Efficient 3D Gaussian Splatting for Large-scale and High-resolution Rendering
This work introduces FlashGS, an open-source CUDA Python library, designed to facilitate the efficient differentiable rasterization of 3D Gaussian Splatting through algorithmic and kernel-level optimizations.
HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation
Notably, we introduce a rule-based camera trajectory generation method, enabling the synthetic pipeline to incorporate diverse and precise camera motion annotation, which can rarely be found in real-world data.
Spectral Representation for Causal Estimation with Hidden Confounders
We address the problem of causal effect estimation where hidden confounders are present, with a focus on two settings: instrumental variable regression with additional observed confounders, and proxy causal learning.
Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text
To achieve this, (1) we first utilize a Trajectory Diffusion Transformer, acting as the Cinematographer, to model the distribution of camera trajectories based on textual descriptions.
UQE: A Query Engine for Unstructured Databases
This engine accepts queries in a Universal Query Language (UQL), a dialect of SQL that provides full natural language flexibility in specifying conditions and operators.
Diffusion Spectral Representation for Reinforcement Learning
Diffusion-based models have achieved notable empirical successes in reinforcement learning (RL) due to their expressiveness in modeling complex distributions.
HYDRA: Model Factorization Framework for Black-Box LLM Personalization
To address these challenges, we propose HYDRA, a model factorization framework that captures both user-specific behavior patterns from historical data and shared general knowledge among all users to deliver personalized generation.
RoomTex: Texturing Compositional Indoor Scenes via Iterative Inpainting
In the coarse stage, RoomTex first unwraps the scene mesh to a panoramic depth map and leverages ControlNet to generate a room panorama, which is regarded as the coarse reference to ensure the global texture consistency.
Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation
We prove that the combination of a target network and over-parameterized linear function approximation establishes a weaker convergence condition for bootstrapped value estimation in certain cases, even with off-policy data.
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
A key bottleneck is understanding how to incorporate uncertainty estimation in the reward function learned from the preference data for RLHF, regardless of how the preference data is collected.
Dual3D: Efficient and Consistent Text-to-3D Generation with Dual-mode Multi-view Latent Diffusion
We present Dual3D, a novel text-to-3D generation framework that generates high-quality 3D assets from texts in only $1$ minute. The key component is a dual-mode multi-view latent diffusion model.
PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarios
We address the challenge of content diversity and controllability in pedestrian simulation for driving scenarios.
MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model
By adopting one-step (or few-step) inference, we further improve the runtime efficiency of the motion latent diffusion model for motion generation.
Ranked #27 on
Motion Synthesis
on HumanML3D
PhyRecon: Physically Plausible Neural Scene Reconstruction
In this paper, we introduce PHYRECON, the first approach to leverage both differentiable rendering and differentiable physics simulation to learn implicit surface representations.
TELA: Text to Layer-wise 3D Clothed Human Generation
Previous works usually encode the human body and clothes as a holistic model and generate the whole model in a single-stage optimization, which makes them struggle for clothing editing and meanwhile lose fine-grained control over the whole generation process.
Efficient Duple Perturbation Robustness in Low-rank MDPs
The pursuit of robustness has recently been a popular topic in reinforcement learning (RL) research, yet the existing methods generally suffer from efficiency issues that obstruct their real-world implementation.
Skill Transfer and Discovery for Sim-to-Real Learning: A Representation-Based Viewpoint
Moreover, to handle the sim-to-real gap in the dynamics, we propose a skill discovery algorithm that learns new skills caused by the sim-to-real gap from real-world data.
SemGrasp: Semantic Grasp Generation via Language Aligned Discretization
We introduce a discrete representation that aligns the grasp space with semantic space, enabling the generation of grasp postures in accordance with language instructions.
CameraCtrl: Enabling Camera Control for Text-to-Video Generation
Controllability plays a crucial role in video generation since it allows users to create desired content.
Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians
The recent 3D Gaussian splatting (3D-GS) has shown remarkable rendering fidelity and efficiency compared to NeRF-based neural scene representations.
Make-It-Vivid: Dressing Your Animatable Biped Cartoon Characters from Text
Creating and animating 3D biped cartoon characters is crucial and valuable in various applications.
GSDF: 3DGS Meets SDF for Improved Rendering and Reconstruction
We show on diverse scenes that our design unlocks the potential for more accurate and detailed surface reconstructions, and at the meantime benefits 3DGS rendering with structures that are more aligned with the underlying geometry.
GetMesh: A Controllable Model for High-quality Mesh Generation and Manipulation
Mesh is a fundamental representation of 3D assets in various industrial applications, and is widely supported by professional softwares.
LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation
The latent is decoded by a transformer-based decoder into a high-capacity 3D neural field.
GenAD: Generalized Predictive Model for Autonomous Driving
In this paper, we introduce the first large-scale video prediction model in the autonomous driving discipline.
Stochastic Gradient Succeeds for Bandits
We show that the \emph{stochastic gradient} bandit algorithm converges to a \emph{globally optimal} policy at an $O(1/t)$ rate, even with a \emph{constant} step size.
BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models
BBox-Adapter distinguishes target and source domain data by treating target data as positive and source data as negative.
Beyond Expectations: Learning with Stochastic Dominance Made Practical
Stochastic dominance models risk-averse preferences for decision making with uncertain outcomes, which naturally captures the intrinsic structure of the underlying uncertainty, in contrast to simply resorting to the expectations.
DiffMorpher: Unleashing the Capability of Diffusion Models for Image Morphing
Our key idea is to capture the semantics of the two images by fitting two LoRAs to them respectively, and interpolate between both the LoRA parameters and the latent noises to ensure a smooth semantic transition, where correspondence automatically emerges without the need for annotation.
EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion
Generating multiview images from a single view facilitates the rapid generation of a 3D mesh conditioned on a single image.
EdgeSAM: Prompt-In-the-Loop Distillation for On-Device Deployment of SAM
It is also the first SAM variant that can run at over 30 FPS on an iPhone 14.
BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation
Generating large-scale 3D scenes cannot simply apply existing 3D object synthesis technique since 3D scenes usually hold complex spatial configurations and consist of a number of objects at varying scales.
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering
Neural rendering methods have significantly advanced photo-realistic 3D scene rendering in various academic and industrial applications.
Cinematic Behavior Transfer via NeRF-based Differentiable Filming
In the evolving landscape of digital media and video production, the precise manipulation and reproduction of visual elements like camera movements and character actions are highly desired.
SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models
The development of text-to-video (T2V), i. e., generating videos with a given text prompt, has been significantly advanced in recent years.
InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint
In our approach, we strive to explore this problem by synthesizing human motion with interactions for a group of characters of any size in a zero-shot manner.
Point Cloud Pre-training with Diffusion Models
This generator aggregates the features extracted by the backbone and employs them as the condition to guide the point-to-point recovery from the noisy point cloud, thereby assisting the backbone in capturing both local and global geometric priors as well as the global point density distribution of the object.
Provable Representation with Efficient Planning for Partial Observable Reinforcement Learning
In most real-world reinforcement learning applications, state information is only partially observable, which breaks the Markov decision process assumption and leads to inferior performance for algorithms that conflate observations with state.
Partially Observable Reinforcement Learning
reinforcement-learning
+1
On Task-personalized Multimodal Few-shot Learning for Visually-rich Document Entity Retrieval
However, prior works for Few-shot VDER mainly address the problem at the document level with a predefined global entity space, which doesn't account for the entity-level few-shot scenario: target entity types are locally personalized by each task and entity occurrences vary significantly among documents.
MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond
While most of recent neural rendering works focus on objects and small-scale scenes, developing neural rendering methods for city-scale scenes is of great potential in many real-world applications.
OrthoPlanes: A Novel Representation for Better 3D-Awareness of GANs
We present a new method for generating realistic and view-consistent images with fine geometry from 2D image collections.
LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models
To this end, we propose LaVie, an integrated video generation framework that operates on cascaded video latent diffusion models, comprising a base T2V model, a temporal interpolation model, and a video super-resolution model.
Ranked #4 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
Interpret Vision Transformers as ConvNets with Dynamic Convolutions
There has been a debate about the superiority between vision Transformers and ConvNets, serving as the backbone of computer vision models.
Unified Human-Scene Interaction via Prompted Chain-of-Contacts
Based on the definition, UniHSI constitutes a Large Language Model (LLM) Planner to translate language prompts into task plans in the form of CoC, and a Unified Controller that turns CoC into uniform task execution.
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior
We present DiffBIR, a general restoration pipeline that could handle different blind image restoration tasks in a unified framework.
Ranked #1 on
Blind Face Restoration
on LFW
X-VoE: Measuring eXplanatory Violation of Expectation in Physical Events
Intuitive physics is pivotal for human understanding of the physical world, enabling prediction and interpretation of events even in infancy.
MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR
In this work, we propose Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR (\textbf{MATLABER}) that leverages a novel latent BRDF auto-encoder for material generation.
DF2: Distribution-Free Decision-Focused Learning
However, existing end-to-end DFL methods are hindered by three significant bottlenecks: model mismatch error, sample average approximation error, and gradient approximation error.
DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Rendering
Realistic human-centric rendering plays a key role in both computer vision and computer graphics.
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
Once trained, the motion module can be inserted into a personalized T2I model to form a personalized animation generator.
HireVAE: An Online and Adaptive Factor Model Based on Hierarchical and Regime-Switch VAE
Factor model is a fundamental investment tool in quantitative investment, which can be empowered by deep learning to become more flexible and efficient in practical complicated investing situations.
Probabilistic Adaptation of Text-to-Video Models
Large text-to-video models trained on internet-scale data have demonstrated exceptional capabilities in generating high-fidelity videos from arbitrary textual descriptions.
Interactive Character Control with Auto-Regressive Motion Diffusion Models
The success of diffusion models for image synthesis has led to the use of these models for motion synthesis.
AdaPlanner: Adaptive Planning from Feedback with Language Models
We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback.
E2EAI: End-to-End Deep Learning Framework for Active Investing
Active investing aims to construct a portfolio of assets that are believed to be relatively profitable in the markets, with one popular method being to construct a portfolio via factor-based strategies.
RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars
It is a large-scale digital library for head avatars with three key attributes: 1) High Fidelity: all subjects are captured by 60 synchronized, high-resolution 2K cameras in 360 degrees.
Towards Multi-Layered 3D Garments Animation
In this paper, we propose a novel data-driven method, called LayersNet, to model garment-level animations as particle-wise interactions in a micro physics system.
LEO: Generative Latent Image Animator for Human Video Synthesis
Our key idea is to represent motion as a sequence of flow maps in the generation process, which inherently isolate motion from appearance.
HyperStyle3D: Text-Guided 3D Portrait Stylization via Hypernetworks
While the use of 3D-aware GANs bypasses the requirement of 3D data, we further alleviate the necessity of style images with the CLIP model being the stylization guidance.
Stochastic Nonlinear Control via Finite-dimensional Spectral Dynamic Embedding
This paper presents an approach, Spectral Dynamics Embedding Control (SDEC), to optimal control for nonlinear stochastic systems.
Generative Diffusion Prior for Unified Image Restoration and Enhancement
Besides, we devise hierarchical guidance and patch-based methods, enabling the GDP to generate images of arbitrary resolutions.
SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and Modeling
Synthetic data has emerged as a promising source for 3D human research as it offers low-cost access to large-scale human datasets.
AssetField: Assets Mining and Reconfiguration in Ground Feature Plane Representation
Traditional modeling pipelines keep an asset library storing unique object templates, which is both versatile and memory efficient in practice.
Grid-guided Neural Radiance Fields for Large Urban Scenes
An alternative solution is to use a feature grid representation, which is computationally efficient and can naturally scale to a large scene with increased grid resolutions.
Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations
Generating task-oriented human-object interaction motions in simulation is challenging.
3D Data Augmentation for Driving Scenes on Camera
The proposed data augmentation approach contributes to a gain of 1. 7% and 1. 4% in terms of detection accuracy, on Waymo and nuScences respectively.
Controllable Mesh Generation Through Sparse Latent Point Diffusion Models
In this work, we design a novel sparse latent point diffusion model for mesh generation.
Prototype-based Embedding Network for Scene Graph Generation
Current Scene Graph Generation (SGG) methods explore contextual information to predict relationships among entity pairs.
Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production
Amateurs working on mini-films and short-form videos usually spend lots of time and effort on the multi-round complicated process of setting and adjusting scenes, plots, and cameras to deliver satisfying video shots.
The Role of Baselines in Policy Gradient Optimization
Instead, the analysis reveals that the primary effect of the value baseline is to \textbf{reduce the aggressiveness of the updates} rather than their variance.
LinkGAN: Linking GAN Latents to Pixels for Controllable Image Synthesis
This work presents an easy-to-use regularizer for GAN training, which helps explicitly link some axes of the latent space to a set of pixels in the synthesized image.
Correspondence Distillation from NeRF-based GAN
The neural radiance field (NeRF) has shown promising results in preserving the fine details of objects and scenes.
Latent Variable Representation for Reinforcement Learning
Theoretically, we establish the sample complexity of the proposed approach in the online and offline settings.
Model-based Reinforcement Learning
reinforcement-learning
+2
Self-Supervised Geometry-Aware Encoder for Style-Based 3D GAN Inversion
In this paper, we study the challenging problem of 3D GAN inversion where a latent code is predicted given a single face image to faithfully recover its 3D shapes and detailed textures.
3DHumanGAN: 3D-Aware Human Image Generation with 3D Pose Mapping
We present 3DHumanGAN, a 3D-aware generative adversarial network that synthesizes photorealistic images of full-body humans with consistent appearances under different view-angles and body-poses.
Score-based Continuous-time Discrete Diffusion Models
Score-based modeling through stochastic differential equations (SDEs) has provided a new perspective on diffusion models, and demonstrated superior performance on continuous data.
Learning to Optimize with Stochastic Dominance Constraints
In real-world decision-making, uncertainty is important yet difficult to handle.
Oracle Inequalities for Model Selection in Offline Reinforcement Learning
We propose the first model selection algorithm for offline RL that achieves minimax rate-optimal oracle inequalities up to logarithmic factors.
Factor Investing with a Deep Multi-Factor Model
Modeling and characterizing multiple factors is perhaps the most important step in achieving excess returns over market benchmarks.
Rethinking Trajectory Prediction via "Team Game"
To accurately predict trajectories in multi-agent settings, e. g. team games, it is important to effectively model the interactions among agents.
Temporal and Contextual Transformer for Multi-Camera Editing of TV Shows
The ability to choose an appropriate camera view among multiple cameras plays a vital role in TV shows delivery.
Improving GANs with A Dynamic Discriminator
Two capacity adjusting schemes are developed for training GANs under different data regimes: i) given a sufficient amount of training data, the discriminator benefits from a progressively increased learning capacity, and ii) when the training data is limited, gradually decreasing the layer width mitigates the over-fitting issue of the discriminator.
Spectral Decomposition Representation for Reinforcement Learning
Representation learning often plays a critical role in reinforcement learning by managing the curse of dimensionality.
Transformer with Implicit Edges for Particle-based Physics Simulation
Consequently, in this paper we propose a novel Transformer-based method, dubbed as Transformer with Implicit Edges (TIE), to capture the rich semantics of particle interactions in an edge-free manner.
BRACE: The Breakdancing Competition Dataset for Dance Motion Synthesis
Estimating human keypoints from these videos is difficult due to the complexity of the dance, as well as the multiple moving cameras recording setup.
Monocular 3D Object Reconstruction with GAN Inversion
Reconstruction is achieved by searching for a latent space in the 3D GAN that best resembles the target mesh in accordance with the single view observation.
Making Linear MDPs Practical via Contrastive Representation Learning
It is common to address the curse of dimensionality in Markov decision processes (MDPs) by exploiting low-rank representations.
Discrete Langevin Sampler via Wasserstein Gradient Flow
It is known that gradient-based MCMC samplers for continuous spaces, such as Langevin Monte Carlo (LMC), can be derived as particle versions of a gradient flow that minimizes KL divergence on a Wasserstein manifold.
Guided Diffusion Model for Adversarial Purification
In this paper, we propose a novel purification approach, referred to as guided diffusion model for purification (GDMP), to help protect classifiers from adversarial attacks.
Accelerating Diffusion Models via Early Stop of the Diffusion Process
By modeling the reverse process of gradually diffusing the data distribution into a Gaussian distribution, generating a sample in DDPMs can be regarded as iteratively denoising a randomly sampled Gaussian noise.
Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis
The ability to synthesize long-term human motion sequences in real-world scenes can facilitate numerous applications.
TransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editing
In this study, we highlight the importance of interaction in a dual-space GAN for more controllable editing.
Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation
To enhance the quality of synthesized gestures, we develop a contrastive learning strategy based on audio-text alignment for better audio representations.
Ranked #3 on
Gesture Generation
on TED Gesture Dataset
DeciWatch: A Simple Baseline for 10x Efficient 2D and 3D Pose Estimation
This paper proposes a simple baseline framework for video-based 2D/3D human pose estimation that can achieve 10 times efficiency improvement over existing works without any performance degradation, named DeciWatch.
Ranked #1 on
2D Human Pose Estimation
on JHMDB (2D poses only)
SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition
However, we identify these techniques are not well equipped for safe policy learning because they ignore negative experiences(e. g., unsafe or unsuccessful), focusing only on positive experiences, which harms their ability to generalize to new tasks safely.
Model Selection in Batch Policy Optimization
We formalize the problem in the contextual bandit setting with linear model classes by identifying three sources of error that any model selection algorithm should optimally trade-off in order to be competitive: (1) approximation error, (2) statistical complexity, and (3) coverage.
Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognition
Typically in recent work, the pseudo-labels are obtained by training a model on the labeled data, and then using confident predictions from the model to teach itself.
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering
The wide span of viewing positions within these scenes yields multi-scale renderings with very different levels of detail, which poses great challenges to neural radiance field and biases it towards compromised results.
Extract Free Dense Labels from CLIP
Contrastive Language-Image Pre-training (CLIP) has made a remarkable breakthrough in open-vocabulary zero-shot image recognition.
Neural Stochastic Dual Dynamic Programming
Stochastic dual dynamic programming (SDDP) is a state-of-the-art method for solving multi-stage stochastic optimization, widely used for modeling real-world process optimization tasks.
Towards understanding retrosynthesis by energy-based models
In this paper, we propose a framework that unifies sequence- and graph-based methods as energy-based models (EBMs) with different energy functions.
A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning
Representation learning lies at the heart of the empirical success of deep learning for dealing with the curse of dimensionality.
Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data
Generative adversarial networks (GANs) typically require ample data for training in order to synthesize high-fidelity images.
Generative Occupancy Fields for 3D Surface-Aware Image Synthesis
In this paper, we propose Generative Occupancy Fields (GOF), a novel model based on generative radiance fields that can learn compact object surfaces without impeding its training convergence.
A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis
Motivated by the observation that a 3D object should look realistic from multiple viewpoints, these methods introduce a multi-view constraint as regularization to learn valid 3D radiance fields from 2D images.
Understanding the Effect of Stochasticity in Policy Optimization
We study the effect of stochasticity in on-policy policy optimization, and make the following four contributions.
SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs
There are two important reasoning tasks on KGs: (1) single-hop knowledge graph completion, which involves predicting individual links in the KG; and (2), multi-hop reasoning, where the goal is to predict which KG entities satisfy a given logical query.
SAFER: Data-Efficient and Safe Reinforcement Learning Through Skill Acquisition
Though many reinforcement learning (RL) problems involve learning policies in settings that are difficult to specify safety constraints and sparse rewards, current methods struggle to rapidly and safely acquire successful policies.
Understanding and Leveraging Overparameterization in Recursive Value Estimation
To better understand the utility of deep models in RL we present an analysis of recursive value estimation using overparameterized linear representations that provides useful, transferable findings.
MeshInversion: 3D textured mesh reconstruction with generative prior
Reconstruction is achieved by searching for a latent space in the 3D GAN that best resembles the target mesh in accordance with the single view observation.
SiT: Simulation Transformer for Particle-based Physics Simulation
However, they force particles to interact with all neighbors without selection, and they fall short in capturing material semantics for different particles, leading to unsatisfactory performance, especially in generalization.
Combiner: Full Attention Transformer with Sparse Computation Cost
However, the key limitation of transformers is their quadratic memory and time complexity $\mathcal{O}(L^2)$ with respect to the sequence length in attention layers, which restricts application in extremely long sequences.
Ranked #2 on
Language Modelling
on Wiki-40B
Safe Exploration by Solving Early Terminated MDP
Safe exploration is crucial for the real-world application of reinforcement learning (RL).
The Curse of Passive Data Collection in Batch Reinforcement Learning
In high stake applications, active experimentation may be considered too risky and thus data are often collected passively.
Optimization Variance: Exploring Generalization Properties of DNNs
Unlike the conventional wisdom in statistical learning theory, the test error of a deep neural network (DNN) often demonstrates double descent: as the model complexity increases, it first follows a classical U-shaped curve and then shows a second descent.
Scene-aware Generative Network for Human Motion Synthesis
We revisit human motion synthesis, a task useful in various real world applications, in this paper.
Leveraging Non-uniformity in First-order Non-convex Optimization
Classical global convergence results for first-order methods rely on uniform smoothness and the \L{}ojasiewicz inequality.
Revisiting Skeleton-based Action Recognition
In this work, we propose PoseC3D, a new approach to skeleton-based action recognition, which relies on a 3D heatmap stack instead of a graph sequence as the base representation of human skeletons.
Ranked #1 on
Action Recognition
on Volleyball
Unsupervised 3D Shape Completion through GAN Inversion
In contrast to previous fully supervised approaches, in this paper we present ShapeInversion, which introduces Generative Adversarial Network (GAN) inversion to shape completion for the first time.
Visually Informed Binaural Audio Generation without Binaural Audios
Moreover, combined with binaural recordings, our method is able to further boost the performance of binaural audio generation under supervised settings.
On the Optimality of Batch Policy Optimization Algorithms
First, we introduce a class of confidence-adjusted index algorithms that unifies optimistic and pessimistic principles in a common framework, which enables a general analysis.
Nearly Horizon-Free Offline Reinforcement Learning
To the best of our knowledge, these are the \emph{first} set of nearly horizon-free bounds for episodic time-homogeneous offline tabular MDP and linear MDP with anchor points.
DeepStyle: User Style Embedding for Authorship Attribution of Short Texts
Authorship attribution (AA), which is the task of finding the owner of a given text, is an important and widely studied research topic with many applications.
Off-Policy Imitation Learning from Observations
To further accelerate the learning procedure, we regulate the policy update with an inverse action model, which assists distribution matching from the perspective of mode-covering.
Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
An ideal environment for evaluating dialog systems, also known as the Turing test, needs to involve human interaction, which is usually not affordable for large-scale experiments.
Self-Supervised Continuous Control without Policy Gradient
Despite the remarkable progress made by the policy gradient algorithms in reinforcement learning (RL), sub-optimal policies usually result from the local exploration property of the policy gradient update.
BlockPlanner: City Block Generation With Vectorized Graph Representation
City modeling is the foundation for computational urban planning, navigation, and entertainment.
Slow Control System for PandaX-III experiment
The PandaX-III experiment uses high pressure gaseous time projection chamber to search for the neutrinoless double beta decay of $^{136}$Xe.
Anomaly Detection
High Energy Physics - Experiment
Instrumentation and Detectors
Focal Frequency Loss for Image Reconstruction and Synthesis
In this study, we show that narrowing gaps in the frequency domain can ameliorate image reconstruction and synthesis quality further.
Offline Policy Selection under Uncertainty
More importantly, we show how the belief distribution estimated by BayesDICE may be used to rank policies with respect to any arbitrary downstream policy selection metric, and we empirically demonstrate that this selection procedure significantly outperforms existing approaches, such as ranking policies according to mean or high-confidence lower bound value estimates.
Differentiable Top-k with Optimal Transport
Finding the k largest or smallest elements from a collection of scores, i. e., top-k operation, is an important model component widely used in information retrieval, machine learning, and data mining.
Escaping the Gravitational Pull of Softmax
Both findings are based on an analysis of convergence rates using the Non-uniform \L{}ojasiewicz (N\L{}) inequalities.
Provably Efficient Neural Estimation of Structural Equation Models: An Adversarial Approach
We study estimation in a class of generalized SEMs where the object of interest is defined as the solution to a linear operator equation.
Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
In this paper we propose ALOE, a new algorithm for learning conditional and unconditional EBMs for discrete structured data, where parameter gradients are estimated using a learned sampler that mimics local search.
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs
Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner.
Named Entity Recognition for Social Media Texts with Semantic Augmentation
In particular, we obtain the augmented semantic information from a large-scale corpus, and propose an attentive semantic augmentation module and a gate module to encode and aggregate such information, respectively.
Ranked #4 on
Named Entity Recognition (NER)
on WNUT 2016
Chinese Named Entity Recognition
named-entity-recognition
+3
CoinDICE: Off-Policy Confidence Interval Estimation
We study high-confidence behavior-agnostic off-policy evaluation in reinforcement learning, where the goal is to estimate a confidence interval on a target policy's value, given only access to a static experience dataset collected by unknown behavior policies.
Differentiable Top-$k$ with Optimal Transport
The top-$k$ operation, i. e., finding the $k$ largest or smallest elements from a collection of scores, is an important model component, which is widely used in information retrieval, machine learning, and data mining.
Small Towers Make Big Differences
A delicate balance between multi-task generalization and multi-objective optimization is therefore needed for finding a better trade-off between efficiency and generalization.
Energy-based View of Retrosynthesis
Retrosynthesis -- the process of identifying a set of reactants to synthesize a target molecule -- is of vital importance to material design and drug discovery.
Ranked #5 on
Single-step retrosynthesis
on USPTO-50k
Off-Policy Evaluation via the Regularized Lagrangian
The recently proposed distribution correction estimation (DICE) family of estimators has advanced the state of the art in off-policy evaluation from behavior-agnostic data.
Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach
We study estimation in a class of generalized SEMs where the object of interest is defined as the solution to a linear operator equation.
Unsupervised Landmark Learning from Unpaired Data
Recent attempts for unsupervised landmark learning leverage synthesized image pairs that are similar in appearance but different in poses.
Video Representation Learning with Visual Tempo Consistency
Visual tempo, which describes how fast an action goes, has shown its potential in supervised action recognition.
Scalable Deep Generative Modeling for Sparse Graphs
Based on this, we develop a novel autoregressive model, named BiGG, that utilizes this sparsity to avoid generating the full adjacency matrix, and importantly reduces the graph generation time complexity to $O((n + m)\log n)$.
Zeroth-Order Supervised Policy Improvement
However, PG algorithms rely on exploiting the value function being learned with the first-order update locally, which results in limited sample efficiency.
Intra- and Inter-Action Understanding via Temporal Action Parsing
Current methods for action recognition primarily rely on deep convolutional networks to derive feature embeddings of visual and motion features.
Evolutionary Stochastic Policy Distillation
Solving the Goal-Conditioned Reward Sparse (GCRS) task is a challenging reinforcement learning problem due to the sparsity of reward signals.
FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding
To take action recognition to a new level, we develop FineGym, a new dataset built on top of gymnastic videos.
Temporal Pyramid Network for Action Recognition
Previous works often capture the visual tempo through sampling raw videos at multiple rates and constructing an input-level frame pyramid, which usually requires a costly multi-branch network to handle.
Ranked #105 on
Action Recognition
on Something-Something V2
Self-Supervised Scene De-occlusion
This is achieved via Partial Completion Network (PCNet)-mask (M) and -content (C), that learn to recover fractions of object masks and contents, respectively, in a self-supervised manner.
Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
SAIL bridges the advantages of IL and RL to reduce the sample complexity substantially, by effectively exploiting sup-optimal demonstrations and efficiently exploring the environment to surpass the demonstrated performance.
Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation
Learning a good image prior is a long-term goal for image restoration and manipulation.
Energy-Based Processes for Exchangeable Data
Recently there has been growing interest in modeling sets with exchangeability such as point clouds.
Batch Stationary Distribution Estimation
We consider the problem of approximating the stationary distribution of an ergodic Markov chain given a set of sampled transitions.
GenDICE: Generalized Offline Estimation of Stationary Values
An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain.
Differentiable Top-k Operator with Optimal Transport
The top-k operation, i. e., finding the k largest or smallest elements from a collection of scores, is an important model component, which is widely used in information retrieval, machine learning, and data mining.
Real or Not Real, that is the Question
While generative adversarial networks (GAN) have been widely adopted in various topics, in this paper we generalize the standard GAN to a new perspective by treating realness as a random variable that can be estimated from multiple angles.
Reinforcement Learning via Fenchel-Rockafellar Duality
We review basic concepts of convex duality, focusing on the very general and supremely useful Fenchel-Rockafellar duality.
Retrosynthesis Prediction with Conditional Graph Logic Network
Retrosynthesis is one of the fundamental problems in organic chemistry.
Ranked #25 on
Single-step retrosynthesis
on USPTO-50k
AlgaeDICE: Policy Gradient from Arbitrary Experience
In many real-world applications of reinforcement learning (RL), interactions with the environment are limited due to cost or feasibility.
Overcoming Catastrophic Forgetting by Generative Regularization
In this paper, we propose a new method to overcome catastrophic forgetting by adding generative regularization to Bayesian inference framework.
Energy-Inspired Models: Learning with Sampler-Induced Distributions
Motivated by this, we consider the sampler-induced distribution as the model of interest and maximize the likelihood of this model.
Learning with Social Influence through Interior Policy Differentiation
In this work we model the social influence into the scheme of reinforcement learning, enabling the agents to learn both from the environment and from their peers.
Recursive Visual Sound Separation Using Minus-Plus Net
Sounds provide rich semantics, complementary to visual data, for many tasks.
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
In contrast to previous approaches, our algorithm is agnostic to knowledge of the behavior policy (or policies) used to generate the dataset.
Cooperative neural networks (CoNN): Exploiting prior independence structure for improved classification
We propose a new approach, called cooperative neural networks (CoNN), which uses a set of cooperatively trained neural networks to capture latent representations that exploit prior given independence structure.
Exponential Family Estimation via Adversarial Dynamics Embedding
We present an efficient algorithm for maximum likelihood estimation (MLE) of exponential family models, with a general parametrization of the energy function that includes neural networks.
Feature Intertwiner for Object Detection
We argue that the reliable set could guide the feature learning of the less reliable set during training - in spirit of student mimicking teacher behavior and thus pushing towards a more compact class centroid in the feature space.
Ranked #146 on
Object Detection
on COCO test-dev
Learning to Defense by Learning to Attack
From the perspective of generative learning, our proposed method can be viewed as learning a deep generative model for generating adversarial samples, which is adaptive to the robust classification.
Revisiting Auxiliary Latent Variables in Generative Models
The success of enriching the variational family with auxiliary latent variables motivates applying the same techniques to the generative model.
Learning to Plan in High Dimensions via Neural Exploration-Exploitation Trees
We propose a meta path planning algorithm named \emph{Neural Exploration-Exploitation Trees~(NEXT)} for learning from prior experience for solving new path planning problems in high dimensional continuous state and action spaces.
Meta Architecture Search
Neural Architecture Search (NAS) has been quite successful in constructing state-of-the-art models on a variety of tasks.
Predictive Approximate Bayesian Computation via Saddle Points
Approximate Bayesian computation (ABC) is an important methodology for Bayesian inference when the likelihood function is intractable.
