Search Results for author:
Found 108 papers, 35 papers with code
Learning Mixtures of Bernoulli Templates by Two-Round EM with Performance Guarantee
Each template is a binary vector, and a template generates examples by randomly switching its binary components independently with a certain probability.
Unsupervised Learning of Dictionaries of Hierarchical Compositional Models
Given a set of unannotated training images, a dictionary of such hierarchical templates are learned so that each training image can be represented by a small number of templates that are spatially translated, rotated and scaled versions of the templates in the learned dictionary.
Learning Inhomogeneous FRAME Models for Object Patterns
We investigate an inhomogeneous version of the FRAME (Filters, Random field, And Maximum Entropy) model and apply it to modeling object patterns.
Learning FRAME Models Using CNN Filters
We explain that each learned model corresponds to a new CNN unit at a layer above the layer of filters employed by the model.
Mining And-Or Graphs for Graph Matching and Object Discovery
This paper reformulates the theory of graph mining on the technical basis of graph matching, and extends its scope of applications to computer vision.
A Theory of Generative ConvNet
If we further assume that the non-linearity in the ConvNet is Rectified Linear Unit (ReLU) and the reference distribution is Gaussian white noise, then we obtain a generative ConvNet model that is unique among energy-based models: The model is piecewise Gaussian, and the means of the Gaussian pieces are defined by an auto-encoder, where the filters in the bottom-up encoding become the basis functions in the top-down decoding, and the binary activation variables detected by the filters in the bottom-up convolution process become the coefficients of the basis functions in the top-down deconvolution process.
Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet
We show that a spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns.
Alternating Back-Propagation for Generator Network
This paper proposes an alternating back-propagation algorithm for learning the generator network model.
Decoding the Encoding of Functional Brain Networks: an fMRI Classification Comparison of Non-negative Matrix Factorization (NMF), Independent Component Analysis (ICA), and Sparse Coding Algorithms
Spatial sparse coding algorithms ($L1$ Regularized Learning and K-SVD) would impose local specialization and a discouragement of multitasking, where the total observed activity in a single voxel originates from a restricted number of possible brain networks.
Cooperative Training of Descriptor and Generator Networks
Specifically, within each iteration of the cooperative learning algorithm, the generator model generates initial synthesized examples to initialize a finite-step MCMC that samples and trains the energy-based descriptor model.
Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning
This paper proposes a learning strategy that extracts object-part concepts from a pre-trained convolutional neural network (CNN), in an attempt to 1) explore explicit semantics hidden in CNN units and 2) gradually grow a semantically interpretable graphical model on the pre-trained CNN for hierarchical object understanding.
Mining Object Parts from CNNs via Active Question-Answering
We use an active human-computer communication to incrementally grow such an AOG on the pre-trained CNN as follows.
Generative Hierarchical Learning of Sparse FRAME Models
This paper proposes a method for generative learning of hierarchical random field models.
Interactively Transferring CNN Patterns for Part Localization
In the scenario of one/multi-shot learning, conventional end-to-end learning strategies without sufficient supervision are usually not powerful enough to learn correct patterns from noisy signals.
Interpreting CNN Knowledge via an Explanatory Graph
Considering that each filter in a conv-layer of a pre-trained CNN usually represents a mixture of object parts, we propose a simple yet efficient method to automatically disentangles different part patterns from each filter, and construct an explanatory graph.
Mining Deep And-Or Object Structures via Cost-Sensitive Question-Answer-Based Active Annotations
The loss is defined for nodes in all layers of the AOG, including the generative loss (measuring the likelihood of the images) and the discriminative loss (measuring the fitness to human answers).
Learning Energy-Based Models as Generative ConvNets via Multi-grid Modeling and Sampling
Within each iteration of our learning algorithm, for each observed training image, we generate synthesized images at multiple grids by initializing the finite-step MCMC sampling from a minimal 1 x 1 version of the training image.
Interpretable Convolutional Neural Networks
Instead, the interpretable CNN automatically assigns each filter in a high conv-layer with an object part during the learning process.
Ranked #1 on
single catogory classification
on ILSVRC Part
Interpreting CNNs via Decision Trees
We propose to learn a decision tree, which clarifies the specific reason for each prediction made by the CNN at the semantic level.
Learning Descriptor Networks for 3D Shape Synthesis and Analysis
This paper proposes a 3D shape descriptor network, which is a deep convolutional energy-based model, for modeling volumetric shape patterns.
Network Transplanting
This paper focuses on a new task, i. e., transplanting a category-and-task-specific neural network to a generic, modular network without strong supervision.
Replicating Active Appearance Model by Generator Network
A recent Cell paper [Chang and Tsao, 2017] reports an interesting discovery.
Unsupervised Learning of Neural Networks to Explain Neural Networks
Given feature maps of a certain conv-layer of the CNN, the explainer performs like an auto-encoder, which first disentangles the feature maps into object-part features and then inverts object-part features back to features of higher conv-layers of the CNN.
Deformable Generator Networks: Unsupervised Disentanglement of Appearance and Geometry
We present a deformable generator model to disentangle the appearance and geometric information for both image and video data in a purely unsupervised manner.
Interactive Agent Modeling by Learning to Probe
In particular, the probing agent (i. e. a learner) learns to interact with the environment and with a target agent (i. e., a demonstrator) to maximize the change in the observed behaviors of that agent.
A Tale of Three Probabilistic Families: Discriminative, Descriptive and Generative Models
In this paper, we review three families of probability models, namely, the discriminative models, the descriptive models, and the generative models.
Learning Grid Cells as Vector Representation of Self-Position Coupled with Matrix Representation of Self-Motion
In this model, the 2D self-position of the agent is represented by a high-dimensional vector, and the 2D self-motion or displacement of the agent is represented by a matrix that transforms the vector.
Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation
Holistic 3D indoor scene understanding refers to jointly recovering the i) object bounding boxes, ii) room layout, and iii) camera pose, all in 3D.
Ranked #5 on
Monocular 3D Object Detection
on SUN RGB-D
Mining Interpretable AOG Representations from Convolutional Networks via Active Question Answering
The AOG associates each object part with certain neural units in feature maps of conv-layers.
Explanatory Graphs for CNNs
This paper introduces a graphical model, namely an explanatory graph, which reveals the knowledge hierarchy hidden inside conv-layers of a pre-trained CNN.
Learning Dynamic Generator Model by Alternating Back-Propagation Through Time
The non-linear transformation of this transition model can be parametrized by a feedforward neural network.
Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Model
This paper proposes the divergence triangle as a framework for joint training of generator model, energy-based model and inference model.
Interpretable CNNs for Object Classification
This paper proposes a generic method to learn interpretable convolutional filters in a deep convolutional neural network (CNN) for object classification, where each interpretable filter encodes features of a specific object part.
Inducing Sparse Coding and And-Or Grammar from Generator Network
We introduce an explainable generative model by applying sparse operation on the feature maps of the generator network.
Network Transplanting (extended abstract)
This paper focuses on a new task, i. e., transplanting a category-and-task-specific neural network to a generic, modular network without strong supervision.
Unsupervised Learning of Neural Networks to Explain Neural Networks (extended abstract)
This paper presents an unsupervised method to learn a neural network, namely an explainer, to interpret a pre-trained convolutional neural network (CNN), i. e., the explainer uses interpretable visual concepts to explain features in middle conv-layers of a CNN.
Learning V1 Simple Cells with Vector Representation of Local Content and Matrix Representation of Local Motion
This paper proposes a representational model for image pairs such as consecutive video frames that are related by local pixel displacements, in the hope that the model may shed light on motion perception in primary visual cortex (V1).
Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning
This paper studies the problem of learning the conditional distribution of a high-dimensional output given an input, where the output and input may belong to two different domains, e. g., the output is a photo image and the input is a sketch image.
On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models
On the other hand, ConvNet potentials learned with non-convergent MCMC do not have a valid steady-state and cannot be considered approximate unnormalized densities of the training data because long-run MCMC samples differ greatly from observed images.
Multi-Agent Tensor Fusion for Contextual Trajectory Prediction
Specifically, the model encodes multiple agents' past trajectories and the scene context into a Multi-Agent Tensor, then applies convolutional fusion to capture multiagent interactions while retaining the spatial structure of agents and the scene context.
Energy-Based Continuous Inverse Optimal Control
The problem of continuous inverse optimal control (over finite time horizon) is to learn the unknown cost function over the sequence of continuous control variables from expert demonstrations.
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
We treat this non-convergent short-run MCMC as a learned generator model or a flow model.
Neural Architecture Search for Joint Optimization of Predictive Power and Biological Knowledge
We report a neural architecture search framework, BioNAS, that is tailored for biomedical researchers to easily build, evaluate, and uncover novel knowledge from interpretable deep learning models.
Inducing Hierarchical Compositional Model by Sparsifying Generator Network
To realize this AND-OR hierarchy in image synthesis, we learn a generator network that consists of the following two components: (i) Each layer of the hierarchy is represented by an over-complete set of convolutional basis functions.
Finding Deep Local Optima Using Network Pruning
However, in non-vision sparse datasets, especially with many irrelevant features where a standard neural network would overfit, this might not be the case and there might be many non-equivalent local optima.
Learning Energy-based Spatial-Temporal Generative ConvNets for Dynamic Patterns
We show that an energy-based spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns.
Deep Unsupervised Clustering with Clustered Generator Model
We propose the clustered generator model for clustering which contains both continuous and discrete latent variables.
Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns
To model the motions explicitly, it is natural for the model to be based on the motions or the displacement fields of the pixels.
Representation Learning: A Statistical Perspective
Learning representations of data is an important problem in statistics and machine learning.
Flow Contrastive Estimation of Energy-Based Models
(2) The update of the flow model approximately minimizes the Jensen-Shannon divergence between the flow model and the data distribution.
Learning Multi-layer Latent Variable Model via Variational Optimization of Short Run MCMC for Approximate Inference
Learning such a generative model requires inferring the latent variables for each training example based on the posterior distribution of these latent variables.
Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification
We propose a generative model of unordered point sets, such as point clouds, in the form of an energy-based model, where the energy function is parameterized by an input-permutation-invariant bottom-up neural network.
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility inference, and social learning.
Joint Training of Variational Auto-Encoder and Latent Energy-Based Model
This paper proposes a joint training method to learn both the variational auto-encoder (VAE) and the latent energy-based model (EBM).
Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning
In this paper, we address these issues and close the loop of neural-symbolic learning by (1) introducing the \textbf{grammar} model as a \textit{symbolic prior} to bridge neural perception and symbolic reasoning, and (2) proposing a novel \textbf{back-search} algorithm which mimics the top-down human-like learning procedure to propagate the error through the symbolic reasoning module efficiently.
MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMC
Learning energy-based model (EBM) requires MCMC sampling of the learned model as an inner loop of the learning algorithm.
Learning Latent Space Energy-Based Prior Model
Due to the low dimensionality of the latent space and the expressiveness of the top-down network, a simple EBM in latent space can capture regularities in the data effectively, and MCMC sampling in latent space is efficient and mixes well.
On Path Integration of Grid Cells: Group Representation and Isotropic Scaling
In this paper, we conduct theoretical analysis of a general representation model of path integration by grid cells, where the 2D self-position is encoded as a higher dimensional vector, and the 2D self-motion is represented by a general transformation of the vector.
A Representational Model of Grid Cells' Path Integration Based on Matrix Lie Algebras
The grid cells in the mammalian medial entorhinal cortex exhibit striking hexagon firing patterns when the agent navigates in the open field.
Learning Latent Space Energy-Based Prior Model for Molecule Generation
Deep generative models have recently been applied to molecule design.
Semi-supervised Learning by Latent Space Energy-Based Model of Symbol-Vector Coupling
This paper proposes a latent space energy-based prior model for semi-supervised learning.
From em-Projections to Variational Auto-Encoder
This paper reviews the em-projections in information geometry and the recent understanding of variational auto-encoder, and explains that they share a common formulation as joint minimization of the Kullback-Leibler divergence between two manifolds of probability distributions, and the joint minimization can be implemented by alternating projections or alternating gradient descent.
Learning Energy-Based Models by Diffusion Recovery Likelihood
Inspired by recent progress on diffusion probabilistic models, we present a diffusion recovery likelihood method to tractably learn and sample from a sequence of EBMs trained on increasingly noisy versions of a dataset.
Ranked #18 on
Image Generation
on CelebA 64x64
Generative VoxelNet: Learning Energy-Based Models for 3D Shape Synthesis and Analysis
3D data that contains rich geometry information of objects and scenes is valuable for understanding 3D physical world.
Learning Algebraic Representation for Abstract Spatial-Temporal Reasoning
We further show that the algebraic representation learned can be decoded by isomorphism and used to generate an answer.
HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
Leveraging these concepts, they could understand the internal structure of this task, without seeing all of the problem instances.
A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
We believe the HINT dataset and the experimental findings are of great interest to the learning community on systematic generalization.
Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translation
This paper studies the unsupervised cross-domain translation problem by proposing a generative framework, in which the probability distribution of each domain is represented by a generative cooperative network that consists of an energy-based model and a latent variable model.
Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance
Predicting agents' future trajectories plays a crucial role in modern AI systems, yet it is challenging due to intricate interactions exhibited in multi-agent systems, especially when it comes to collision avoidance.
Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis
Specifically, the camera pose and 3D scene are represented as vectors and the local camera movement is represented as a matrix operating on the vector of the camera pose.
Trajectory Prediction with Latent Belief Energy-Based Model
Sampling from or optimizing the learned LB-EBM yields a belief vector which is used to make a path plan, which then in turn helps to predict a long-range trajectory.
Generative Text Modeling through Short Run Inference
It is initialized from the prior distribution of the latent variable and then runs a small number (e. g., 20) of Langevin dynamics steps guided by its posterior distribution.
SCRIPT: Self-Critic PreTraining of Transformers
We introduce Self-CRItic Pretraining Transformers (SCRIPT) for representation learning of text.
SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues
Inferring social relations from dialogues is vital for building emotionally intelligent robots to interpret human language better and act accordingly.
Ranked #5 on
Dialog Relation Extraction
on DialogRE
SAS: Self-Augmentation Strategy for Language Model Pre-training
In this paper, we propose a self-augmentation strategy (SAS) where a single network is utilized for both regular pre-training and contextualized data augmentation for the training in later epochs.
Proceedings of ICML 2021 Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI
This workshop pays a special interest in theoretic foundations, limitations, and new application trends in the scope of XAI.
Latent Space Energy-Based Model of Symbol-Vector Coupling for Text Generation and Classification
The energy term of the prior model couples a continuous latent vector and a symbolic one-hot vector, so that discrete category can be inferred from the observed example based on the continuous latent vector.
MCMC Should Mix: Learning Energy-Based Model with Flow-Based Backbone
However, MCMC sampling of EBMs in high-dimensional data space is generally not mixing, because the energy function, which is usually parametrized by deep network, is highly multi-modal in the data space.
Unsupervised Meta-Learning via Latent Space Energy-based Model of Symbol Vector Coupling
We propose to learn an energy-based model (EBM) in the latent space of a top-down generative model such that the EBM in the low dimensional latent space is able to be learned efficiently and adapt to each task rapidly.
Iterative Teacher-Aware Learning
Recently, the benefits of integrating this cooperative pedagogy into machine concept learning in discrete spaces have been proved by multiple works.
Unsupervised Foreground Extraction via Deep Region Competition
Foreground extraction can be viewed as a special case of generic image segmentation that focuses on identifying and disentangling objects from the background.
Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning
Extensive experiments show that by incorporating an algebraic treatment, the ALANS learner outperforms various pure connectionist models in domains requiring systematic generalization.
Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering
Most previous works address the problem by first fusing the image and question in the multi-modal space, which is inflexible for further fusion with a vast amount of external knowledge.
Ranked #19 on
Visual Question Answering (VQA)
on OK-VQA
A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering
Outside-knowledge visual question answering (OK-VQA) requires the agent to comprehend the image, make use of relevant knowledge from the entire web, and digest all the information to answer the question.
Deep Generative model with Hierarchical Latent Factors for Time Series Anomaly Detection
Multivariate time series anomaly detection has become an active area of research in recent years, with Deep Learning models outperforming previous approaches on benchmark datasets.
Latent Diffusion Energy-Based Model for Interpretable Text Modeling
Latent space Energy-Based Models (EBMs), also known as energy-based priors, have drawn growing interests in generative modeling.
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning
However, it is unknown if the models can handle more complex problems that involve math reasoning over heterogeneous information, such as tabular data.
Neural-Symbolic Recursive Machine for Systematic Generalization
In experiments, NSR achieves state-of-the-art performance in three benchmarks from different domains: SCAN for semantic parsing, PCFG for string manipulation, and HINT for arithmetic reasoning.
Conformal Isometry of Lie Group Representation in Recurrent Network of Grid Cells
Recurrent neural networks have been proposed to explain the properties of the grid cells by updating the neural activity vector based on the velocity input of the animal.
SpectraNet: Multivariate Forecasting and Imputation under Distribution Shifts and Missing Data
In this work, we tackle two widespread challenges in real applications for time-series forecasting that have been largely understudied: distribution shifts and missing data.
On the Complexity of Bayesian Generalization
Specifically, at the $representational \ level$, we seek to answer how the complexity varies when a visual concept is mapped to the representation space.
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response.
Learning for Transductive Threshold Calibration in Open-World Recognition
In deep metric learning for visual recognition, the calibration of distance thresholds is crucial for achieving desired model performance in the true positive rates (TPR) or true negative rates (TNR).
Diverse and Faithful Knowledge-Grounded Dialogue Generation via Sequential Posterior Inference
The capability to generate responses with diversity and faithfulness using factual knowledge is paramount for creating a human-like, trustworthy dialogue system.
Molecule Design by Latent Space Energy-Based Modeling and Gradual Distribution Shifting
To search for molecules with desired properties, we propose a sampling with gradual distribution shifting (SGDS) algorithm, so that after learning the model initially on the training data of existing molecules and their properties, the proposed algorithm gradually shifts the model distribution towards the region supported by molecules with desired values of properties.
Learning Joint Latent Space EBM Prior Model for Multi-layer Generator
To tackle this issue and learn more expressive prior models, we propose an energy-based model (EBM) on the joint latent space over all layers of latent variables with the multi-layer generator as its backbone.
Threshold-Consistent Margin Loss for Open-World Deep Metric Learning
Existing losses used in deep metric learning (DML) for image retrieval often lead to highly non-uniform intra-class and inter-class representation structures across test classes and data distributions.
Molecule Design by Latent Prompt Transformer
This paper proposes a latent prompt Transformer model for solving challenging optimization problems such as molecule design, where the goal is to find molecules with optimal values of a target chemical or biological property that can be computed by an existing software.
Learning Concept-Based Causal Transition and Symbolic Reasoning for Visual Planning
In this paper, we propose an interpretable and generalizable visual planning framework consisting of i) a novel Substitution-based Concept Learner (SCL) that abstracts visual inputs into disentangled concept representations, ii) symbol abstraction and reasoning that performs task planning via the self-learned symbols, and iii) a Visual Causal Transition model (ViCT) that grounds visual causal transitions to semantically similar real-world actions.
Learning Hierarchical Features with Joint Latent Space Energy-Based Prior
In this paper, we propose a joint latent space EBM prior model with multi-layer latent variables for effective hierarchical representation learning.
Emergence of Grid-like Representations by Training Recurrent Networks with Conformal Normalization
As the agent moves, the vector is transformed by an RNN that takes the velocity of the agent as input.
Differentiable VQ-VAE's for Robust White Matter Streamline Encodings
Given the complex geometry of white matter streamlines, Autoencoders have been proposed as a dimension-reduction tool to simplify the analysis streamlines in a low-dimensional latent spaces.
Image Translation as Diffusion Visual Programmers
We introduce the novel Diffusion Visual Programmer (DVP), a neuro-symbolic image translation framework.
Latent Plan Transformer: Planning as Latent Variable Inference
We introduce the Latent Plan Transformer (LPT), a novel model that leverages a latent space to connect a Transformer-based trajectory generator and the final return.
Efficient and Scalable Fine-Tune of Language Models for Genome Understanding
Impressively, applying these adapters on natural language foundation models matched or even exceeded the performance of DNA foundation models.
Dual-Space Optimization: Improved Molecule Sequence Design by Latent Prompt Transformer
Designing molecules with desirable properties, such as drug-likeliness and high binding affinities towards protein targets, is a challenging problem.
LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoning
Through a series of simulations in a box-packing domain, we quantitatively demonstrate the effectiveness of LLM^3 in solving TAMP problems and the efficiency in selecting action parameters.
Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Models
Text-to-image diffusion models have shown great success in generating high-quality text-guided images.
