Search Results for author:
Found 201 papers, 63 papers with code
Mind the Context: The Impact of Contextualization in Neural Module Networks for Grounding Visual Referring Expressions
Our model outperforms the state-of-the-art NMN model on CLEVR-Ref+ dataset with +8. 1% improvement in accuracy on the single-referent test set and +4. 3% on the full test set.
CrossVQA: Scalably Generating Benchmarks for Systematically Testing VQA Generalization
One challenge in evaluating visual question answering (VQA) models in the cross-dataset adaptation setting is that the distribution shifts are multi-modal, making it difficult to identify if it is the shifts in visual or language features that play a key role.
Towards Socially Intelligent Agents with Mental State Transition and Human Value
One of which is to track the agent’s mental state transition and teach the agent to make decisions guided by its value like a human.
An Embodied Generalist Agent in 3D World
Leveraging massive knowledge and learning schemes from large language models (LLMs), recent machine learning models show notable successes in building generalist agents that exhibit the capability of general-purpose task solving in diverse domains, including natural language processing, computer vision, and robotics.
AI Alignment: A Comprehensive Survey
AI alignment aims to make AI systems behave in line with human intentions and values.
Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World
We even designed a neuro-symbolic reasoning approach that reconciles LLMs & VLMs with logical reasoning to emulate the human problem-solving process for Bongard Problems.
Ranked #1 on
Visual Reasoning
on Bongard-OpenWorld
CORE: Common Random Reconstruction for Distributed Optimization with Provable Low Communication Complexity
With distributed machine learning being a prominent technique for large-scale machine learning tasks, communication complexity has become a major bottleneck for speeding up training and scaling up machine numbers.
MindAgent: Emergent Gaming Interaction
Large Language Models (LLMs) have the capacity of performing complex scheduling in a multi-agent system and can coordinate these agents into completing sophisticated tasks that require extensive collaboration.
ProAgent: Building Proactive Cooperative AI with Large Language Models
Experimental evaluations conducted within the framework of \textit{Overcook-AI} unveil the remarkable performance superiority of ProAgent, outperforming five methods based on self-play and population-based training in cooperation with AI agents.
X-VoE: Measuring eXplanatory Violation of Expectation in Physical Events
Intuitive physics is pivotal for human understanding of the physical world, enabling prediction and interpretation of events even in infancy.
Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Models
In this perspective paper, we first comprehensively review existing evaluations of Large Language Models (LLMs) using both standardized tests and ability-oriented benchmarks.
MindDial: Belief Dynamics Tracking with Theory-of-Mind Modeling for Situated Neural Dialogue Generation
We design an explicit mind module that can track three-level beliefs -- the speaker's belief, the speaker's prediction of the listener's belief, and the common belief based on the gap between the first two.
Heterogeneous Value Evaluation for Large Language Models
In this paper, we propose A2EHV, an Automated Alignment Evaluation with a Heterogeneous Value system that (1) is automated to minimize individual human biases, and (2) allows assessments against various target values to foster heterogeneous agents.
Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
On the whole, our analysis provides a novel perspective on the role of semantics in developing and evaluating language models' reasoning abilities.
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response.
ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes
To tackle these challenges, we present ARNOLD, a benchmark that evaluates language-grounded task learning with continuous states in realistic 3D scenes.
Rearrange Indoor Scenes for Human-Robot Co-Activity
We present an optimization-based framework for rearranging indoor furniture to accommodate human-robot co-activities better.
Diffusion-based Generation, Optimization, and Planning in 3D Scenes
SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization, and planning.
A Reconfigurable Data Glove for Reconstructing Physical and Virtual Grasps
In this work, we present a reconfigurable data glove design to capture different modes of human hand-object interactions, which are critical in training embodied artificial intelligence (AI) agents for fine manipulation tasks.
On the Complexity of Bayesian Generalization
Specifically, at the $representational \ level$, we seek to answer how the complexity varies when a visual concept is mapped to the representation space.
Learning Probabilistic Models from Generator Latent Spaces with Hat EBM
This work proposes a method for using any generator network as the foundation of an Energy-Based Model (EBM).
RulE: Neural-Symbolic Knowledge Graph Reasoning with Rule Embedding
In this paper, we propose a novel and principled framework called \textbf{RulE} (stands for {Rul}e {E}mbedding) to effectively leverage logical rules to enhance KG reasoning.
SQA3D: Situated Question Answering in 3D Scenes
We propose a new task to benchmark scene understanding of embodied agents: Situated Question Answering in 3D Scenes (SQA3D).
Ranked #1 on
Referring Expression
on SQA3D
EgoTaskQA: Understanding Human Tasks in Egocentric Videos
The challenges of such capability lie in the difficulty of generating a detailed understanding of situated actions, their effects on object states (i. e., state changes), and their causal dependencies.
Neural-Symbolic Recursive Machine for Systematic Generalization
In experiments, NSR achieves state-of-the-art performance in three benchmarks from different domains: SCAN for semantic parsing, PCFG for string manipulation, and HINT for arithmetic reasoning.
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning
However, it is unknown if the models can handle more complex problems that involve math reasoning over heterogeneous information, such as tabular data.
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions.
Ranked #4 on
Science Question Answering
on ScienceQA
Sequential Manipulation Planning on Scene Graph
We devise a 3D scene graph representation, contact graph+ (cg+), for efficient sequential task planning.
Understanding Physical Effects for Effective Tool-use
We present a robot learning and planning framework that produces an effective tool-use strategy with the least joint efforts, capable of handling objects different from training.
VRKitchen2.0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse
With the recent progress of simulations by 3D modeling software and game engines, many researchers have focused on Embodied AI tasks in the virtual environment.
Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning
In this study, we propose the Bimanual Dexterous Hands Benchmark (Bi-DexHands), a simulator that involves two dexterous hands with tens of bimanual manipulation tasks and thousands of target objects.
Latent Diffusion Energy-Based Model for Interpretable Text Modeling
Latent space Energy-Based Models (EBMs), also known as energy-based priors, have drawn growing interests in generative modeling.
Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions
A significant gap remains between today's visual pattern recognition models and human-level visual cognition especially when it comes to few-shot learning and compositional reasoning of novel concepts.
Ranked #1 on
Few-Shot Image Classification
on Bongard-HOI
EBM Life Cycle: MCMC Strategies for Synthesis, Defense, and Density Modeling
This work presents strategies to learn an Energy-Based Model (EBM) according to the desired length of its MCMC sampling trajectories.
RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning
This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of entities, and 3) generalizing to novel object-relation combinations, i. e., systematic generalization.
Ranked #1 on
Zero-Shot Human-Object Interaction Detection
on HICO
Triangular Character Animation Sampling with Motion, Emotion, and Relation
We propose a Spatial-Temporal And-Or graph (ST-AOG), a stochastic grammar model, to encode the contextual relationship between motion, emotion, and relation, forming a triangle in a conditional random field.
PartAfford: Part-level Affordance Discovery from 3D Objects
We propose a novel learning framework for PartAfford, which discovers part-level representations by leveraging only the affordance set supervision and geometric primitive regularization, without dense supervision.
Attention cannot be an Explanation
Motivated by this, we ask a follow-up question: "Assuming that we only consider the tasks where attention weights correlate well with feature importance, how effective are these attention based explanations in increasing human trust and reliance in the underlying models?".
Discourse Analysis for Evaluating Coherence in Video Paragraph Captions
We also introduce DisNet, a novel dataset containing the proposed visual discourse annotations of 3000 videos and their paragraphs.
Learning from the Tangram to Solve Mini Visual Tasks
Current pre-training methods in computer vision focus on natural images in the daily-life context.
ValueNet: A New Dataset for Human Value Driven Dialogue System
To the best of our knowledge, ValueNet is the first large-scale text dataset for human value modeling, and we are the first one trying to incorporate a value model into emotionally intelligent dialogue systems.
Robust Visual Reasoning via Language Guided Neural Module Networks
Neural module networks (NMN) are a popular approach for solving multi-modal tasks such as visual question answering (VQA) and visual referring expression recognition (REF).
Emergent Graphical Conventions in a Visual Communication Game
Humans communicate with graphical sketches apart from symbolic languages.
Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning
Extensive experiments show that by incorporating an algebraic treatment, the ALANS learner outperforms various pure connectionist models in domains requiring systematic generalization.
Unsupervised Foreground Extraction via Deep Region Competition
Foreground extraction can be viewed as a special case of generic image segmentation that focuses on identifying and disentangling objects from the background.
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
Also, we develop a strong IconQA baseline Patch-TRM that applies a pyramid cross-modal Transformer with input diagram embeddings pre-trained on the icon dataset.
Ranked #1 on
Visual Question Answering (VQA)
on IconQA
Iterative Teacher-Aware Learning
Recently, the benefits of integrating this cooperative pedagogy into machine concept learning in discrete spaces have been proved by multiple works.
Emergence of Theory of Mind Collaboration in Multiagent Systems
Currently, in the study of multiagent systems, the intentions of agents are usually ignored.
MCMC Should Mix: Learning Energy-Based Model with Flow-Based Backbone
However, MCMC sampling of EBMs in high-dimensional data space is generally not mixing, because the energy function, which is usually parametrized by deep network, is highly multi-modal in the data space.
YouRefIt: Embodied Reference Understanding with Language and Gesture
To the best of our knowledge, this is the first embodied reference dataset that allows us to study referring expressions in daily physical scenes to understand referential behavior, human communication, and human-robot interaction.
CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Models
More concretely, our CX-ToM framework generates sequence of explanations in a dialog by mediating the differences between the minds of machine and human user.
Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds
To date, various 3D scene understanding tasks still lack practical and generalizable pre-trained models, primarily due to the intricate nature of 3D scene understanding tasks and their immense variations introduced by camera views, lighting, occlusions, etc.
Ranked #4 on
3D Object Detection
on SUN-RGBD
STAR: Sparse Transformer-based Action Recognition
The cognitive system for human action and behavior has evolved into a deep learning regime, and especially the advent of Graph Convolution Networks has transformed the field in recent years.
Transformer-based Machine Learning for Fast SAT Solvers and Logic Synthesis
Our model has a scale-free structure which could process varying size of instances.
SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues
Inferring social relations from dialogues is vital for building emotionally intelligent robots to interpret human language better and act accordingly.
Ranked #5 on
Dialog Relation Extraction
on DialogRE
Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
We further propose a novel geometry solving approach with formal language and symbolic reasoning, called Interpretable Geometry Problem Solver (Inter-GPS).
Ranked #1 on
Mathematical Question Answering
on GeoS
VersaGNN: a Versatile accelerator for Graph neural networks
As GNNs operate on non-Euclidean data, their irregular data access patterns cause considerable computational costs and overhead on conventional architectures, such as GPU and CPU.
Learning Triadic Belief Dynamics in Nonverbal Communication from Videos
By aggregating different beliefs and true world states, our model essentially forms "five minds" during the interactions between two agents.
Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis
Specifically, the camera pose and 3D scene are represented as vectors and the local camera movement is represented as a matrix operating on the vector of the camera pose.
Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments
In this paper, we rethink the problem of scene reconstruction from an embodied agent's perspective: While the classic view focuses on the reconstruction accuracy, our new perspective emphasizes the underlying functions and constraints such that the reconstructed scenes provide \em{actionable} information for simulating \em{interactions} with agents.
ACRE: Abstract Causal REasoning Beyond Covariation
Causal induction, i. e., identifying unobservable mechanisms that lead to the observable relations among variables, has played a pivotal role in modern scientific discovery, especially in scenarios with only sparse and limited data.
Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance
Predicting agents' future trajectories plays a crucial role in modern AI systems, yet it is challenging due to intricate interactions exhibited in multi-agent systems, especially when it comes to collision avoidance.
Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution
To fill in this gap, we propose a neuro-symbolic Probabilistic Abduction and Execution (PrAE) learner; central to the PrAE learner is the process of probabilistic abduction and execution on a probabilistic scene representation, akin to the mental manipulation of objects.
VLGrammar: Grounded Grammar Induction of Vision and Language
In this work, we study grounded grammar induction of vision and language in a joint learning framework.
Towards Socially Intelligent Agents with Mental State Transition and Human Utility
One of which is to track the agent's mental state transition and teach the agent to make decisions guided by its value like a human.
Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translation
This paper studies the unsupervised cross-domain translation problem by proposing a generative framework, in which the probability distribution of each domain is represented by a generative cooperative network that consists of an energy-based model and a latent variable model.
Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration
Our experiments with human participants demonstrate that a short calibration using REMP can effectively bridge the gap between what a non-expert user thinks a robot can reach and the ground truth.
A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
We believe the HINT dataset and the experimental findings are of great interest to the learning community on systematic generalization.
HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
Leveraging these concepts, they could understand the internal structure of this task, without seeing all of the problem instances.
Transformers satisfy
The Propositional Satisfiability Problem (SAT), and more generally, the Constraint Satisfaction Problem (CSP), are mathematical questions defined as finding an assignment to a set of objects that satisfies a series of constraints.
Learning Algebraic Representation for Abstract Spatial-Temporal Reasoning
We further show that the algebraic representation learned can be decoded by isomorphism and used to generate an answer.
SMART: A Situation Model for Algebra Story Problems via Attributed Grammar
Solving algebra story problems remains a challenging task in artificial intelligence, which requires a detailed understanding of real-world situations and a strong mathematical reasoning capability.
Generative VoxelNet: Learning Energy-Based Models for 3D Shape Synthesis and Analysis
3D data that contains rich geometry information of objects and scenes is valuable for understanding 3D physical world.
Learning by Fixing: Solving Math Word Problems with Weak Supervision
To generate more diverse solutions, \textit{tree regularization} is applied to guide the efficient shrinkage and exploration of the solution space, and a \textit{memory buffer} is designed to track and save the discovered various fixes for each problem.
Ranked #1 on
Math Word Problem Solving
on Math23K
(weakly-supervised metric)
Weighted Entropy Modification for Soft Actor-Critic
We generalize the existing principle of the maximum Shannon entropy in reinforcement learning (RL) to weighted entropy by characterizing the state-action pairs with some qualitative weights, which can be connected with prior knowledge, experience replay, and evolution process of the policy.
Generalized Inverse Planning: Learning Lifted non-Markovian Utility for Generalizable Task Representation
Seeing that the proposed generalization problem has not been widely studied yet, we carefully define an evaluation protocol, with which we illustrate the effectiveness of MEIP on two proof-of-concept domains and one challenging task: learning to fold from demonstrations.
A Representational Model of Grid Cells' Path Integration Based on Matrix Lie Algebras
The grid cells in the mammalian medial entorhinal cortex exhibit striking hexagon firing patterns when the agent navigates in the open field.
Structured Attention for Unsupervised Dialogue Structure Induction
Inducing a meaningful structural representation from one or a set of dialogues is a crucial but challenging task in computational linguistics.
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities
Understanding and interpreting human actions is a long-standing challenge and a critical indicator of perception in artificial intelligence.
Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks
Thus, in this paper, we propose a novel explainable AI (XAI) framework for achieving human-like communication in human-robot collaborations, where the robot builds a hierarchical mind model of the human user and generates explanations of its own mind as a form of communications based on its online Bayesian inference of the user's mental state.
Bayesian Inference
Explainable Artificial Intelligence (XAI)
+1
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
Humans can progressively learn visual concepts from easy to hard questions.
On Path Integration of Grid Cells: Group Representation and Isotropic Scaling
In this paper, we conduct theoretical analysis of a general representation model of path integration by grid cells, where the 2D self-position is encoded as a higher dimensional vector, and the 2D self-motion is represented by a general transformation of the vector.
Learning Latent Space Energy-Based Prior Model
Due to the low dimensionality of the latent space and the expressiveness of the top-down network, a simple EBM in latent space can capture regularities in the data effectively, and MCMC sampling in latent space is efficient and mixes well.
MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMC
Learning energy-based model (EBM) requires MCMC sampling of the learned model as an inner loop of the learning algorithm.
Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning
In this paper, we address these issues and close the loop of neural-symbolic learning by (1) introducing the \textbf{grammar} model as a \textit{symbolic prior} to bridge neural perception and symbolic reasoning, and (2) proposing a novel \textbf{back-search} algorithm which mimics the top-down human-like learning procedure to propagate the error through the symbolic reasoning module efficiently.
Joint Training of Variational Auto-Encoder and Latent Energy-Based Model
This paper proposes a joint training method to learn both the variational auto-encoder (VAE) and the latent energy-based model (EBM).
Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Models
Our contributions are 1) an improved method for training EBM's with realistic long-run MCMC samples, 2) an Expectation-Over-Transformation (EOT) defense that resolves theoretical ambiguities for stochastic defenses and from which the EOT attack naturally follows, and 3) state-of-the-art adversarial defense for naturally-trained classifiers and competitive defense compared to adversarially-trained classifiers on Cifar-10, SVHN, and Cifar-100.
Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressions
To measure the true progress of existing models, we split the test set into two sets, one which requires reasoning on linguistic structure and the other which doesn't.
Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning
To endow such a crucial cognitive ability to machine intelligence, we propose a dataset, Machine Number Sense (MNS), consisting of visual arithmetic problems automatically generated using a grammar model--And-Or Graph (AOG).
Congestion-aware Evacuation Routing using Augmented Reality Devices
We present a congestion-aware routing solution for indoor evacuation, which produces real-time individual-customized evacuation routes among multiple destinations while keeping tracks of all evacuees' locations.
Joint Inference of States, Robot Knowledge, and Human (False-)Beliefs
Aiming to understand how human (false-)belief--a core socio-cognitive ability--would affect human interactions with robots, this paper proposes to adopt a graphical model to unify the representation of object states, robot knowledge, and human (false-)beliefs.
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility inference, and social learning.
Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification
We propose a generative model of unordered point sets, such as point clouds, in the form of an energy-based model, where the energy function is parameterized by an input-permutation-invariant bottom-up neural network.
Emergence of Pragmatics from Referential Game between Theory of Mind Agents
Pragmatics studies how context can contribute to language meanings.
PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points
Detecting 3D objects from a single RGB image is intrinsically ambiguous, thus requiring appropriate prior knowledge and intermediate representations as constraints to reduce the uncertainties and improve the consistencies between the 2D image plane and the 3D world coordinate.
Ranked #2 on
Monocular 3D Object Detection
on SUN RGB-D
(AP@0.15 (10 / PNet-30) metric)
Learning Multi-layer Latent Variable Model via Variational Optimization of Short Run MCMC for Approximate Inference
Learning such a generative model requires inferring the latent variables for each training example based on the posterior distribution of these latent variables.
Learning Perceptual Inference by Contrasting
"Thinking in pictures," [1] i. e., spatial-temporal reasoning, effortless and instantaneous for humans, is believed to be a significant ability to perform logical induction and a crucial factor in the intellectual history of technology development.
Representation Learning: A Statistical Perspective
Learning representations of data is an important problem in statistics and machine learning.
Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns
To model the motions explicitly, it is natural for the model to be based on the motions or the displacement fields of the pixels.
Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning
Given these general theories, the goal is to train an agent by interactively exploring the problem space to (i) discover, form, and transfer useful abstract and structural knowledge, and (ii) induce useful knowledge from the instance-level attributes observed in the environment.
DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare
We present DenseRaC, a novel end-to-end framework for jointly estimating 3D human pose and body shape from a monocular RGB image.
Ranked #75 on
3D Human Pose Estimation
on MPI-INF-3DHP
(using extra training data)
Learning Energy-based Spatial-Temporal Generative ConvNets for Dynamic Patterns
We show that an energy-based spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns.
TWIN GRAPH CONVOLUTIONAL NETWORKS: GCN WITH DUAL GRAPH SUPPORT FOR SEMI-SUPERVISED LEARNING
Graph Neural Networks as a combination of Graph Signal Processing and Deep Convolutional Networks shows great power in pattern recognition in non-Euclidean domains.
X-ToM: Explaining with Theory-of-Mind for Gaining Justified Human Trust
We present a new explainable AI (XAI) framework aimed at increasing justified human trust and reliance in the AI machine through explanations.
Action Recognition
Explainable Artificial Intelligence (XAI)
+2
Inducing Hierarchical Compositional Model by Sparsifying Generator Network
To realize this AND-OR hierarchy in image synthesis, we learn a generator network that consists of the following two components: (i) Each layer of the hierarchy is represented by an over-complete set of convolutional basis functions.
Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
This paper addresses a new problem of understanding human gaze communication in social videos from both atomic-level and event-level, which is significant for studying human social interactions.
Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
We propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction---3D estimations of object bounding boxes, camera pose, and room layout, and (ii) 3D human pose estimation.
3D Human Pose Estimation
Human-Object Interaction Detection
+1
HUGE2: a Highly Untangled Generative-model Engine for Edge-computing
In this work, we propose a Highly Untangled Generative-model Engine for Edge-computing or HUGE2 for accelerating these two special convolutions on the edge-computing platform by decomposing the kernels and untangling these smaller convolutions by performing basic matrix multiplications.
Learning Pose Grammar for Monocular 3D Pose Estimation
In this paper, we propose a pose grammar to tackle the problem of 3D human pose estimation from a monocular RGB image.
Ranked #13 on
3D Human Pose Estimation
on HumanEva-I
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
We treat this non-convergent short-run MCMC as a learned generator model or a flow model.
Reasoning Visual Dialogs with Structural and Partial Observations
The answer to a given question is represented by a node with missing value.
Ranked #14 on
Visual Dialog
on VisDial v0.9 val
On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models
On the other hand, ConvNet potentials learned with non-convergent MCMC do not have a valid steady-state and cannot be considered approximate unnormalized densities of the training data because long-run MCMC samples differ greatly from observed images.
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
One of the main challenges of advancing task-oriented learning such as visual task planning and reinforcement learning is the lack of realistic and standardized environments for training and testing AI agents.
Natural Language Interaction with Explainable AI Models
This paper presents an explainable AI (XAI) system that provides explanations for its predictions.
RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
In this work, we propose a new dataset, built in the context of Raven's Progressive Matrices (RPM) and aimed at lifting machine intelligence by associating vision with structural, relational, and analogical reasoning in a hierarchical representation.
Discourse Parsing in Videos: A Multi-modal Appraoch
We propose the task of Visual Discourse Parsing, which requires understanding discourse relations among scenes in a video.
Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning
This paper studies the problem of learning the conditional distribution of a high-dimensional output given an input, where the output and input may belong to two different domains, e. g., the output is a photo image and the input is a sketch image.
Learning V1 Simple Cells with Vector Representation of Local Content and Matrix Representation of Local Motion
This paper proposes a representational model for image pairs such as consecutive video frames that are related by local pixel displacements, in the hope that the model may shed light on motion perception in primary visual cortex (V1).
Inducing Sparse Coding and And-Or Grammar from Generator Network
We introduce an explainable generative model by applying sparse operation on the feature maps of the generator network.
Explaining AlphaGo: Interpreting Contextual Effects in Neural Networks
In this paper, we propose to disentangle and interpret contextual effects that are encoded in a pre-trained deep neural network.
Interpretable CNNs for Object Classification
This paper proposes a generic method to learn interpretable convolutional filters in a deep convolutional neural network (CNN) for object classification, where each interpretable filter encodes features of a specific object part.
Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Model
This paper proposes the divergence triangle as a framework for joint training of generator model, energy-based model and inference model.
Learning Dynamic Generator Model by Alternating Back-Propagation Through Time
The non-linear transformation of this transition model can be parametrized by a feedforward neural network.
Mining Interpretable AOG Representations from Convolutional Networks via Active Question Answering
The AOG associates each object part with certain neural units in feature maps of conv-layers.
Explanatory Graphs for CNNs
This paper introduces a graphical model, namely an explanatory graph, which reveals the knowledge hierarchy hidden inside conv-layers of a pre-trained CNN.
MetaStyle: Three-Way Trade-Off Among Speed, Flexibility, and Quality in Neural Style Transfer
An unprecedented booming has been witnessed in the research area of artistic style transfer ever since Gatys et al. introduced the neural method.
Deeper Interpretability of Deep Networks
However, understanding the information represented and processed in CNNs remains in most cases challenging.
Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation
Holistic 3D indoor scene understanding refers to jointly recovering the i) object bounding boxes, ii) room layout, and iii) camera pose, all in 3D.
Ranked #5 on
Monocular 3D Object Detection
on SUN RGB-D
Learning Grid Cells as Vector Representation of Self-Position Coupled with Matrix Representation of Self-Motion
In this model, the 2D self-position of the agent is represented by a high-dimensional vector, and the 2D self-motion or displacement of the agent is represented by a matrix that transforms the vector.
A Tale of Three Probabilistic Families: Discriminative, Descriptive and Generative Models
In this paper, we review three families of probability models, namely, the discriminative models, the descriptive models, and the generative models.
Sparse Winograd Convolutional neural networks on small-scale systolic arrays
The reconfigurability, energy-efficiency, and massive parallelism on FPGAs make them one of the best choices for implementing efficient deep learning accelerators.
Interactive Agent Modeling by Learning to Probe
In particular, the probing agent (i. e. a learner) learns to interact with the environment and with a target agent (i. e., a demonstrator) to maximize the change in the observed behaviors of that agent.
Human-centric Indoor Scene Synthesis Using Stochastic Grammar
We present a human-centric method to sample and synthesize 3D room layouts and 2D images thereof, to obtain large-scale 2D/3D image data with perfect per-pixel ground truth.
Learning Human-Object Interactions by Graph Parsing Neural Networks
For a given scene, GPNN infers a parse graph that includes i) the HOI graph structure represented by an adjacency matrix, and ii) the node labels.
Ranked #31 on
Human-Object Interaction Detection
on V-COCO
Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image
We propose a computational framework to jointly parse a single RGB image and reconstruct a holistic 3D configuration composed by a set of CAD models using a stochastic grammar model.
Ranked #4 on
Monocular 3D Object Detection
on SUN RGB-D
(AP@0.15 (10 / PNet-30) metric)
Deformable Generator Networks: Unsupervised Disentanglement of Appearance and Geometry
We present a deformable generator model to disentangle the appearance and geometric information for both image and video data in a purely unsupervised manner.
Generalized Earley Parser: Bridging Symbolic Grammars and Sequence Data for Future Prediction
Future predictions on sequence data (e. g., videos or audios) require the algorithms to capture non-Markovian and compositional properties of high-level semantics.
Where and Why Are They Looking? Jointly Inferring Human Attention and Intentions in Complex Tasks
We built a new video dataset of tasks, intentions, and attention.
Inferring Shared Attention in Social Scene Videos
We collect a new dataset VideoCoAtt from public TV show videos, containing 380 complex video sequences with more than 492, 000 frames that include diverse social scenes for shared attention study.
Attentive Fashion Grammar Network for Fashion Landmark Detection and Clothing Category Classification
This paper proposes a knowledge-guided fashion network to solve the problem of visual fashion analysis, e. g., fashion landmark localization and clothing category classification.
Unsupervised Learning of Neural Networks to Explain Neural Networks
Given feature maps of a certain conv-layer of the CNN, the explainer performs like an auto-encoder, which first disentangles the feature maps into object-part features and then inverts object-part features back to features of higher conv-layers of the CNN.
Learning Descriptor Networks for 3D Shape Synthesis and Analysis
This paper proposes a 3D shape descriptor network, which is a deep convolutional energy-based model, for modeling volumetric shape patterns.
Intent-aware Multi-agent Reinforcement Learning
We experiment our algorithm in a real-world problem that is non-episodic, and the number of agents and goals can vary over time.
Multi-agent Reinforcement Learning
reinforcement-learning
+1
Building a Telescope to Look Into High-Dimensional Image Spaces
However, characterizing a learned probability density to uncover the Hopfield memories of the model, encoded by the structure of the local modes, remains an open challenge.
Visual Interpretability for Deep Learning: a Survey
This paper reviews recent studies in understanding neural-network representations and learning neural networks with interpretable/disentangled middle-layer representations.
Examining CNN Representations with respect to Dataset Bias
We aim to discover representation flaws caused by potential dataset bias.
Learning Pose Grammar to Encode Human Body Configuration for 3D Pose Estimation
In this paper, we propose a pose grammar to tackle the problem of 3D human pose estimation.
Ranked #1 on
3D Absolute Human Pose Estimation
on Human3.6M
(Average MPJPE (mm) metric)
Interpretable Convolutional Neural Networks
Instead, the interpretable CNN automatically assigns each filter in a high conv-layer with an object part during the learning process.
Ranked #1 on
single catogory classification
on ILSVRC Part
Monocular 3D Human Pose Estimation by Predicting Depth on Joints
This paper aims at estimating full-body 3D human poses from monocular images of which the biggest challenge is the inherent ambiguity introduced by lifting the 2D pose into 3D space.
Ranked #109 on
3D Human Pose Estimation
on Human3.6M
(PA-MPJPE metric)
Jointly Recognizing Object Fluents and Tasks in Egocentric Videos
Given an egocentric video, a beam search algorithm is applied to jointly recognizing the object fluents in each frame, and the task of the entire video.
Learning Energy-Based Models as Generative ConvNets via Multi-grid Modeling and Sampling
Within each iteration of our learning algorithm, for each observed training image, we generate synthesized images at multiple grids by initializing the finite-step MCMC sampling from a minimal 1 x 1 version of the training image.
A Causal And-Or Graph Model for Visibility Fluent Reasoning in Tracking Interacting Objects
We introduce a Causal And-Or Graph (C-AOG) to represent the causal-effect relations between an object's visibility fluent and its activities, and develop a probabilistic graph model to jointly reason the visibility fluent change (e. g., from visible to invisible) and track humans in videos.
Scene-centric Joint Parsing of Cross-view Videos
The proposed joint parsing framework represents such correlations and constraints explicitly and generates semantic scene-centric parse graphs.
Mining Deep And-Or Object Structures via Cost-Sensitive Question-Answer-Based Active Annotations
The loss is defined for nodes in all layers of the AOG, including the generative loss (measuring the likelihood of the images) and the discriminative loss (measuring the fitness to human answers).
Interpreting CNN Knowledge via an Explanatory Graph
Considering that each filter in a conv-layer of a pre-trained CNN usually represents a mixture of object parts, we propose a simple yet efficient method to automatically disentangles different part patterns from each filter, and construct an explanatory graph.
Interactively Transferring CNN Patterns for Part Localization
In the scenario of one/multi-shot learning, conventional end-to-end learning strategies without sufficient supervision are usually not powerful enough to learn correct patterns from noisy signals.
Predicting Human Activities Using Stochastic Grammar
This paper presents a novel method to predict future human activities from partially observed RGB-D videos.
Generative Hierarchical Learning of Sparse FRAME Models
This paper proposes a method for generative learning of hierarchical random field models.
Mining Object Parts from CNNs via Active Question-Answering
We use an active human-computer communication to incrementally grow such an AOG on the pre-trained CNN as follows.
CERN: Confidence-Energy Recurrent Network for Group Activity Recognition
This work is about recognizing human activities occurring in videos at distinct semantic levels, including individual actions, interactions, and group activities.
Ranked #11 on
Group Activity Recognition
on Volleyball
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammars
We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algorithms.
Learning Social Affordance Grammar from Videos: Transferring Human Interactions to Human-Robot Interactions
In this paper, we present a general framework for learning social affordance grammar as a spatiotemporal AND-OR graph (ST-AOG) from RGB-D videos of human interactions, and transfer the grammar to humanoids to enable a real-time motion inference for human-robot interaction (HRI).
Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning
This paper proposes a learning strategy that extracts object-part concepts from a pre-trained convolutional neural network (CNN), in an attempt to 1) explore explicit semantics hidden in CNN units and 2) gradually grow a semantically interpretable graphical model on the pre-trained CNN for hierarchical object understanding.
Cooperative Training of Descriptor and Generator Networks
Specifically, within each iteration of the cooperative learning algorithm, the generator model generates initial synthesized examples to initialize a finite-step MCMC that samples and trains the energy-based descriptor model.
Alternating Back-Propagation for Generator Network
This paper proposes an alternating back-propagation algorithm for learning the generator network model.
Modeling and Inferring Human Intents and Latent Functional Objects for Trajectory Prediction
This paper is about detecting functional objects and inferring human intentions in surveillance videos of public spaces.
Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet
We show that a spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns.
Inferring Forces and Learning Human Utilities From Videos
We propose a notion of affordance that takes into account physical quantities generated when the human body interacts with real-world objects, and introduce a learning framework that incorporates the concept of human utilities, which in our opinion provides a deeper and finer-grained account not only of object affordance but also of people's interaction with objects.