Search Results for author:
Found 58 papers, 13 papers with code
Natural Language Guided Ligand-Binding Protein Design
Given a textual description of the desired function and a ligand formula in SMILES, InstructPro generates protein sequences that are functionally consistent with the specified instructions.
Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity
Experiments confirm that Meerkat and Meerkat-vp significantly improve the efficiency and effectiveness of ZO federated LLM fine-tuning.
Latent Adaptive Planner for Dynamic Manipulation
This paper presents Latent Adaptive Planner (LAP), a novel approach for dynamic nonprehensile manipulation tasks that formulates planning as latent space inference, effectively learned from human demonstration videos.
SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning
Despite recent advances on multi-modal models, 3D spatial reasoning remains a challenging task for state-of-the-art open-source and proprietary models.
Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark
Recent advancements in language multimodal models (LMMs) for video have demonstrated their potential for understanding video content, yet the task of comprehending multi-discipline lectures remains largely unexplored.
A Materials Foundation Model via Hybrid Invariant-Equivariant Architectures
Machine learning interatomic potentials (MLIPs) can predict energy, force, and stress of materials and enable a wide range of downstream discovery tasks.
Scalable Language Models with Posterior Inference of Latent Thought Vectors
We propose a novel family of language models, Latent-Thought Language Models (LTMs), which incorporate explicit latent thought vectors that follow an explicit prior model in latent space.
Latent Space Energy-based Neural ODEs
This family of models generates each data point in the time series by a neural emission model, which is a non-linear transformation of a latent state vector.
Latent Energy-Based Odyssey: Black-Box Optimization via Expanded Exploration in the Energy-Based Latent Space
To this end, we formulate an learnable energy-based latent space, and propose Noise-intensified Telescoping density-Ratio Estimation (NTRE) scheme for variational learning of an accurate latent space model without costly Markov Chain Monte Carlo.
Molecule Design by Latent Prompt Transformer
We propose the Latent Prompt Transformer (LPT), a novel generative model comprising three components: (1) a latent vector with a learnable prior distribution modeled by a neural transformation of Gaussian white noise; (2) a molecule generation model based on a causal Transformer, which uses the latent vector as a prompt; and (3) a property prediction model that predicts a molecule's target properties and/or constraint values using the latent prompt.
Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference
We introduce the Latent Plan Transformer (LPT), a novel model that leverages a latent variable to connect a Transformer-based trajectory generator and the final return.
STANLEY: Stochastic Gradient Anisotropic Langevin Dynamics for Learning Energy-Based Models
We propose in this paper, STANLEY, a STochastic gradient ANisotropic LangEvin dYnamics, for sampling high dimensional data.
Molecule Design by Latent Prompt Transformer
This paper proposes a latent prompt Transformer model for solving challenging optimization problems such as molecule design, where the goal is to find molecules with optimal values of a target chemical or biological property that can be computed by an existing software.
Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood
Training energy-based models (EBMs) on high-dimensional data can be both challenging and time-consuming, and there exists a noticeable gap in sample quality between EBMs and other generative frameworks like GANs and diffusion models.
Progressive Energy-Based Cooperative Learning for Multi-Domain Image-to-Image Translation
The components of translator, style encoder, and style generator constitute a diversified image generator.
Likelihood-Based Generative Radiance Field with Latent Space Energy-Based Model for 3D-Aware Disentangled Image Representation
We propose the NeRF-LEBM, a likelihood-based top-down 3D-aware 2D image generative model that incorporates 3D representation via Neural Radiance Fields (NeRF) and 2D imaging process via differentiable volume rendering.
CoopInit: Initializing Generative Adversarial Networks via Cooperative Learning
The proposed algorithm consists of two learning stages: (i) Cooperative initialization stage: The discriminator of GAN is treated as an energy-based model (EBM) and is optimized via maximum likelihood estimation (MLE), with the help of the GAN's generator to provide synthetic data to approximate the learning gradients.
A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference
We study a normalizing flow in the latent space of a top-down generator model, in which the normalizing flow model plays the role of the informative prior model of the generator.
CoopHash: Cooperative Learning of Multipurpose Descriptor and Contrastive Pair Generator via Variational MCMC Teaching for Supervised Image Hashing
Leveraging supervised information can lead to superior retrieval performance in the image hashing domain but the performance degrades significantly without enough labeled data.
A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model
Under the short-run non-mixing MCMC scenario, the estimation of the energy-based model is shown to follow the perturbation of maximum likelihood, and the short-run Langevin flow and the normalizing flow form a two-flow generator that we call CoopFlow.
An Energy-Based Prior for Generative Saliency
We propose a novel generative saliency prediction framework that adopts an informative energy-based model as a prior distribution.
Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction
In this paper, we take a step further by proposing a novel generative vision transformer with latent variables following an informative energy-based prior for salient object detection.
Unsupervised Contrastive Learning for Signal-Dependent Noise Synthesis
We present a simple yet robust noise synthesis framework based on unsupervised contrastive learning.
Blind Image Super-resolution with Elaborate Degradation Modeling on Noise and Kernel
To address the above issues, this paper proposes a model-based blind SISR method under the probabilistic framework, which elaborately models image degradation from the perspectives of noise and blur kernel.
Energy-Based Generative Cooperative Saliency Prediction
In this paper, to model the uncertainty of visual saliency, we study the saliency prediction problem from the perspective of generative models by learning a conditional probability distribution over the saliency map given an input image, and treating the saliency prediction as a sampling process from the learned distribution.
Learning Deep Latent Variable Models by Short-Run MCMC Inference With Optimal Transport Correction
Learning latent variable models with deep top-down architectures typically requires inferring the latent variables for each training example based on the posterior distribution of these latent variables.
Patchwise Generative ConvNet: Training Energy-Based Models From a Single Natural Image for Internal Learning
Exploiting internal statistics of a single natural image has long been recognized as a significant research paradigm where the goal is to learn the distribution of patches within the image without relying on external training data.
Semi-Supervised Video Deraining with Dynamical Rain Generator
Firstly, most of them do not sufficiently model the characteristics of rain layers of rainy videos.
Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translation
This paper studies the unsupervised cross-domain translation problem by proposing a generative framework, in which the probability distribution of each domain is represented by a generative cooperative network that consists of an energy-based model and a latent variable model.
Learning Energy-Based Generative Models via Coarse-to-Fine Expanding and Sampling
Energy-based models (EBMs) for generative modeling parametrize a single net and can be directly trained by maximum likelihood estimation.
Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler
In this paper, we propose to learn a variational auto-encoder (VAE) to initialize the finite-step MCMC, such as Langevin dynamics that is derived from the energy function, for efficient amortized sampling of the EBM.
Generative VoxelNet: Learning Energy-Based Models for 3D Shape Synthesis and Analysis
3D data that contains rich geometry information of objects and scenes is valuable for understanding 3D physical world.
A Representational Model of Grid Cells' Path Integration Based on Matrix Lie Algebras
The grid cells in the mammalian medial entorhinal cortex exhibit striking hexagon firing patterns when the agent navigates in the open field.
Learning Noise-Aware Encoder-Decoder from Noisy Labels by Alternating Back-Propagation for Saliency Detection
The proposed model consists of two sub-models parameterized by neural networks: (1) a saliency predictor that maps input images to clean saliency maps, and (2) a noise generator, which is a latent variable model that produces noises from Gaussian latent vectors.
On Path Integration of Grid Cells: Group Representation and Isotropic Scaling
In this paper, we conduct theoretical analysis of a general representation model of path integration by grid cells, where the 2D self-position is encoded as a higher dimensional vector, and the 2D self-motion is represented by a general transformation of the vector.
Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification
We propose a generative model of unordered point sets, such as point clouds, in the form of an energy-based model, where the energy function is parameterized by an input-permutation-invariant bottom-up neural network.
Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns
To model the motions explicitly, it is natural for the model to be based on the motions or the displacement fields of the pixels.
Representation Learning: A Statistical Perspective
Learning representations of data is an important problem in statistics and machine learning.
Learning Energy-based Spatial-Temporal Generative ConvNets for Dynamic Patterns
We show that an energy-based spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns.
Learning Pose Grammar for Monocular 3D Pose Estimation
In this paper, we propose a pose grammar to tackle the problem of 3D human pose estimation from a monocular RGB image.
Ranked #15 on
3D Human Pose Estimation
on HumanEva-I
Learning Feature-to-Feature Translator by Alternating Back-Propagation for Generative Zero-Shot Learning
We investigate learning feature-to-feature translator networks by alternating back-propagation as a general-purpose solution to zero-shot learning (ZSL) problems.
Energy-Based Continuous Inverse Optimal Control
The problem of continuous inverse optimal control (over finite time horizon) is to learn the unknown cost function over the sequence of continuous control variables from expert demonstrations.
Semantic-Guided Multi-Attention Localization for Zero-Shot Learning
Zero-shot learning extends the conventional object classification to the unseen class recognition by introducing semantic representations of classes.
Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning
This paper studies the problem of learning the conditional distribution of a high-dimensional output given an input, where the output and input may belong to two different domains, e. g., the output is a photo image and the input is a sketch image.
Learning V1 Simple Cells with Vector Representation of Local Content and Matrix Representation of Local Motion
This paper proposes a representational model for image pairs such as consecutive video frames that are related by local pixel displacements, in the hope that the model may shed light on motion perception in primary visual cortex (V1).
Learning Dynamic Generator Model by Alternating Back-Propagation Through Time
The non-linear transformation of this transition model can be parametrized by a feedforward neural network.
Segregated Temporal Assembly Recurrent Networks for Weakly Supervised Multiple Action Detection
Finally, we transform the output of recurrent neural network into the corresponding action distribution.
Learning Grid Cells as Vector Representation of Self-Position Coupled with Matrix Representation of Self-Motion
In this model, the 2D self-position of the agent is represented by a high-dimensional vector, and the 2D self-motion or displacement of the agent is represented by a matrix that transforms the vector.
Weakly and Semi Supervised Human Body Part Parsing via Pose-Guided Knowledge Transfer
In this paper, we present a novel method to generate synthetic human part segmentation data using easily-obtained human keypoint annotations.
Ranked #4 on
Human Part Segmentation
on PASCAL-Part
(using extra training data)
Learning Descriptor Networks for 3D Shape Synthesis and Analysis
This paper proposes a 3D shape descriptor network, which is a deep convolutional energy-based model, for modeling volumetric shape patterns.
A Causal And-Or Graph Model for Visibility Fluent Reasoning in Tracking Interacting Objects
We introduce a Causal And-Or Graph (C-AOG) to represent the causal-effect relations between an object's visibility fluent and its activities, and develop a probabilistic graph model to jointly reason the visibility fluent change (e. g., from visible to invisible) and track humans in videos.
Generative Hierarchical Learning of Sparse FRAME Models
This paper proposes a method for generative learning of hierarchical random field models.
Super-Trajectory for Video Segmentation
We introduce a novel semi-supervised video segmentation approach based on an efficient video representation, called as "super-trajectory".
Cooperative Training of Descriptor and Generator Networks
Specifically, within each iteration of the cooperative learning algorithm, the generator model generates initial synthesized examples to initialize a finite-step MCMC that samples and trains the energy-based descriptor model.
Decoding the Encoding of Functional Brain Networks: an fMRI Classification Comparison of Non-negative Matrix Factorization (NMF), Independent Component Analysis (ICA), and Sparse Coding Algorithms
Spatial sparse coding algorithms ($L1$ Regularized Learning and K-SVD) would impose local specialization and a discouragement of multitasking, where the total observed activity in a single voxel originates from a restricted number of possible brain networks.
Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet
We show that a spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns.
A Theory of Generative ConvNet
If we further assume that the non-linearity in the ConvNet is Rectified Linear Unit (ReLU) and the reference distribution is Gaussian white noise, then we obtain a generative ConvNet model that is unique among energy-based models: The model is piecewise Gaussian, and the means of the Gaussian pieces are defined by an auto-encoder, where the filters in the bottom-up encoding become the basis functions in the top-down decoding, and the binary activation variables detected by the filters in the bottom-up convolution process become the coefficients of the basis functions in the top-down deconvolution process.
Learning Inhomogeneous FRAME Models for Object Patterns
We investigate an inhomogeneous version of the FRAME (Filters, Random field, And Maximum Entropy) model and apply it to modeling object patterns.
