Search Results for author:
Found 218 papers, 70 papers with code
CrossVQA: Scalably Generating Benchmarks for Systematically Testing VQA Generalization
One challenge in evaluating visual question answering (VQA) models in the cross-dataset adaptation setting is that the distribution shifts are multi-modal, making it difficult to identify if it is the shifts in visual or language features that play a key role.
Mind the Context: The Impact of Contextualization in Neural Module Networks for Grounding Visual Referring Expressions
Our model outperforms the state-of-the-art NMN model on CLEVR-Ref+ dataset with +8. 1% improvement in accuracy on the single-referent test set and +4. 3% on the full test set.
Towards Socially Intelligent Agents with Mental State Transition and Human Value
One of which is to track the agent’s mental state transition and teach the agent to make decisions guided by its value like a human.
AdaSociety: An Adaptive Environment with Social Structures for Multi-Agent Decision-Making
As agents progress, the environment adaptively generates new tasks with social structures for agents to undertake.
Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games
LASE allocates a portion of its rewards to co-players as gifts, with this allocation adapting dynamically based on the social relationship -- a metric evaluating the friendliness of co-players estimated by counterfactual reasoning.
Mars: Situated Inductive Reasoning in an Open-World Environment
In this paper, we design Mars, an interactive environment devised for situated inductive reasoning.
M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenes
We propose M^3Bench, a new benchmark of whole-body motion generation for mobile manipulation tasks.
FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models
Vision language models (VLMs) have achieved impressive progress in diverse applications, becoming a prevalent research direction.
LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments
Recent advances in Large Language Models (LLMs) have shown inspiring achievements in constructing autonomous agents that rely on language descriptions as inputs.
Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning
To address these issues, we propose Hierarchical Opponent modeling and Planning (HOP), a novel multi-agent decision-making algorithm that enables few-shot adaptation to unseen policies in mixed-motive environments.
SocialGFs: Learning Social Gradient Fields for Multi-Agent Reinforcement Learning
Multi-agent systems (MAS) need to adaptively cope with dynamic environments, changing agent populations, and diverse tasks.
Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations
Autonomous robotic systems capable of learning novel manipulation tasks are poised to transform industries from manufacturing to service automation.
PhyRecon: Physically Plausible Neural Scene Reconstruction
In this paper, we introduce PHYRECON, the first approach to leverage both differentiable rendering and differentiable physics simulation to learn implicit surface representations.
LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoning
Through a series of simulations in a box-packing domain, we quantitatively demonstrate the effectiveness of LLM^3 in solving TAMP problems and the efficiency in selecting action parameters.
Fast Peer Adaptation with Context-aware Exploration
To do so, it is crucial for the agent to probe and identify the peer's strategy efficiently, as this is the prerequisite for carrying out the best response in adaptation.
On the Emergence of Symmetrical Reality
Artificial intelligence (AI) has revolutionized human cognitive abilities and facilitated the development of new AI entities capable of interacting with humans in both physical and virtual environments.
CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents
Within CivRealm, we provide interfaces for two typical agent types: tensor-based agents that focus on learning, and language-based agents that emphasize reasoning.
CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update
However, these methods often overlook the potential for continual learning, typically by freezing the utilized tools, thus limiting their adaptation to environments requiring new knowledge.
Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models
We introduce Aligner, a novel Parameter-Efficient Fine-Tuning (PEFT) method for aligning multi-billion-parameter-sized Large Language Models (LLMs).
An Embodied Generalist Agent in 3D World
However, several significant challenges remain: (i) most of these models rely on 2D images yet exhibit a limited capacity for 3D input; (ii) these models rarely explore the tasks inherently defined in 3D world, e. g., 3D grounding, embodied reasoning and acting.
AI Alignment: A Comprehensive Survey
The former aims to make AI systems aligned via alignment training, while the latter aims to gain evidence about the systems' alignment and govern them appropriately to avoid exacerbating misalignment risks.
Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World
We even conceived a neuro-symbolic reasoning approach that reconciles LLMs & VLMs with logical reasoning to emulate the human problem-solving process for Bongard Problems.
Ranked #1 on
Visual Reasoning
on Bongard-OpenWorld
Learning Energy-Based Prior Model with Diffusion-Amortized MCMC
To remedy this sampling issue, in this paper we introduce a simple but effective diffusion-based amortization method for long-run MCMC sampling and develop a novel learning algorithm for the latent space EBM based on it.
CORE: Common Random Reconstruction for Distributed Optimization with Provable Low Communication Complexity
With distributed machine learning being a prominent technique for large-scale machine learning tasks, communication complexity has become a major bottleneck for speeding up training and scaling up machine numbers.
MindAgent: Emergent Gaming Interaction
Large Language Models (LLMs) have the capacity of performing complex scheduling in a multi-agent system and can coordinate these agents into completing sophisticated tasks that require extensive collaboration.
ProAgent: Building Proactive Cooperative Agents with Large Language Models
Building agents with adaptive behavior in cooperative tasks stands as a paramount goal in the realm of multi-agent systems.
X-VoE: Measuring eXplanatory Violation of Expectation in Physical Events
Intuitive physics is pivotal for human understanding of the physical world, enabling prediction and interpretation of events even in infancy.
Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Models
In this perspective paper, we first comprehensively review existing evaluations of Large Language Models (LLMs) using both standardized tests and ability-oriented benchmarks.
MindDial: Belief Dynamics Tracking with Theory-of-Mind Modeling for Situated Neural Dialogue Generation
Experiments show that models with mind modeling can achieve higher task outcomes when aligning and negotiating common ground.
Heterogeneous Value Alignment Evaluation for Large Language Models
We conduct evaluations with new auto-metric \textit{value rationality} to represent the ability of LLMs to align with specific values.
Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
On the whole, our analysis provides a novel perspective on the role of semantics in developing and evaluating language models' reasoning abilities.
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response.
ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes
To tackle these challenges, we present ARNOLD, a benchmark that evaluates language-grounded task learning with continuous states in realistic 3D scenes.
Rearrange Indoor Scenes for Human-Robot Co-Activity
We present an optimization-based framework for rearranging indoor furniture to accommodate human-robot co-activities better.
Diffusion-based Generation, Optimization, and Planning in 3D Scenes
SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization, and planning.
A Reconfigurable Data Glove for Reconstructing Physical and Virtual Grasps
In this work, we present a reconfigurable data glove design to capture different modes of human hand-object interactions, which are critical in training embodied artificial intelligence (AI) agents for fine manipulation tasks.
On the Complexity of Bayesian Generalization
Specifically, at the $representational \ level$, we seek to answer how the complexity varies when a visual concept is mapped to the representation space.
Learning Probabilistic Models from Generator Latent Spaces with Hat EBM
This work proposes a method for using any generator network as the foundation of an Energy-Based Model (EBM).
RulE: Knowledge Graph Reasoning with Rule Embedding
In this paper, we propose a novel and principled framework called \textbf{RulE} (stands for {Rul}e {E}mbedding) to effectively leverage logical rules to enhance KG reasoning.
SQA3D: Situated Question Answering in 3D Scenes
We propose a new task to benchmark scene understanding of embodied agents: Situated Question Answering in 3D Scenes (SQA3D).
Ranked #1 on
Referring Expression
on SQA3D
EgoTaskQA: Understanding Human Tasks in Egocentric Videos
The challenges of such capability lie in the difficulty of generating a detailed understanding of situated actions, their effects on object states (i. e., state changes), and their causal dependencies.
Neural-Symbolic Recursive Machine for Systematic Generalization
We evaluate NSR's efficacy across four challenging benchmarks designed to probe systematic generalization capabilities: SCAN for semantic parsing, PCFG for string manipulation, HINT for arithmetic reasoning, and a compositional machine translation task.
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning
However, it is unknown if the models can handle more complex problems that involve math reasoning over heterogeneous information, such as tabular data.
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions.
Ranked #6 on
Science Question Answering
on ScienceQA
Sequential Manipulation Planning on Scene Graph
We devise a 3D scene graph representation, contact graph+ (cg+), for efficient sequential task planning.
Understanding Physical Effects for Effective Tool-use
We present a robot learning and planning framework that produces an effective tool-use strategy with the least joint efforts, capable of handling objects different from training.
VRKitchen2.0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse
With the recent progress of simulations by 3D modeling software and game engines, many researchers have focused on Embodied AI tasks in the virtual environment.
Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning
In this study, we propose the Bimanual Dexterous Hands Benchmark (Bi-DexHands), a simulator that involves two dexterous hands with tens of bimanual manipulation tasks and thousands of target objects.
Latent Diffusion Energy-Based Model for Interpretable Text Modeling
Latent space Energy-Based Models (EBMs), also known as energy-based priors, have drawn growing interests in generative modeling.
Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions
A significant gap remains between today's visual pattern recognition models and human-level visual cognition especially when it comes to few-shot learning and compositional reasoning of novel concepts.
Ranked #1 on
Few-Shot Image Classification
on Bongard-HOI
EBM Life Cycle: MCMC Strategies for Synthesis, Defense, and Density Modeling
This work presents strategies to learn an Energy-Based Model (EBM) according to the desired length of its MCMC sampling trajectories.
RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning
This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of entities, and 3) generalizing to novel object-relation combinations, i. e., systematic generalization.
Ranked #1 on
Zero-Shot Human-Object Interaction Detection
on HICO
Triangular Character Animation Sampling with Motion, Emotion, and Relation
We propose a Spatial-Temporal And-Or graph (ST-AOG), a stochastic grammar model, to encode the contextual relationship between motion, emotion, and relation, forming a triangle in a conditional random field.
PartAfford: Part-level Affordance Discovery from 3D Objects
We propose a novel learning framework for PartAfford, which discovers part-level representations by leveraging only the affordance set supervision and geometric primitive regularization, without dense supervision.
Attention cannot be an Explanation
Motivated by this, we ask a follow-up question: "Assuming that we only consider the tasks where attention weights correlate well with feature importance, how effective are these attention based explanations in increasing human trust and reliance in the underlying models?".
Discourse Analysis for Evaluating Coherence in Video Paragraph Captions
We also introduce DisNet, a novel dataset containing the proposed visual discourse annotations of 3000 videos and their paragraphs.
Learning from the Tangram to Solve Mini Visual Tasks
Current pre-training methods in computer vision focus on natural images in the daily-life context.
ValueNet: A New Dataset for Human Value Driven Dialogue System
To the best of our knowledge, ValueNet is the first large-scale text dataset for human value modeling, and we are the first one trying to incorporate a value model into emotionally intelligent dialogue systems.
Robust Visual Reasoning via Language Guided Neural Module Networks
Neural module networks (NMN) are a popular approach for solving multi-modal tasks such as visual question answering (VQA) and visual referring expression recognition (REF).
Emergent Graphical Conventions in a Visual Communication Game
Humans communicate with graphical sketches apart from symbolic languages.
Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning
Extensive experiments show that by incorporating an algebraic treatment, the ALANS learner outperforms various pure connectionist models in domains requiring systematic generalization.
Unsupervised Foreground Extraction via Deep Region Competition
Foreground extraction can be viewed as a special case of generic image segmentation that focuses on identifying and disentangling objects from the background.
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
Also, we develop a strong IconQA baseline Patch-TRM that applies a pyramid cross-modal Transformer with input diagram embeddings pre-trained on the icon dataset.
Ranked #1 on
Visual Question Answering (VQA)
on IconQA
Iterative Teacher-Aware Learning
Recently, the benefits of integrating this cooperative pedagogy into machine concept learning in discrete spaces have been proved by multiple works.
Emergence of Theory of Mind Collaboration in Multiagent Systems
Currently, in the study of multiagent systems, the intentions of agents are usually ignored.
MCMC Should Mix: Learning Energy-Based Model with Flow-Based Backbone
However, MCMC sampling of EBMs in high-dimensional data space is generally not mixing, because the energy function, which is usually parametrized by deep network, is highly multi-modal in the data space.
YouRefIt: Embodied Reference Understanding with Language and Gesture
To the best of our knowledge, this is the first embodied reference dataset that allows us to study referring expressions in daily physical scenes to understand referential behavior, human communication, and human-robot interaction.
CX-ToM: Counterfactual Explanations with Theory-of-Mind for Enhancing Human Trust in Image Recognition Models
More concretely, our CX-ToM framework generates sequence of explanations in a dialog by mediating the differences between the minds of machine and human user.
Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds
To date, various 3D scene understanding tasks still lack practical and generalizable pre-trained models, primarily due to the intricate nature of 3D scene understanding tasks and their immense variations introduced by camera views, lighting, occlusions, etc.
Ranked #4 on
3D Object Detection
on SUN-RGBD
STAR: Sparse Transformer-based Action Recognition
The cognitive system for human action and behavior has evolved into a deep learning regime, and especially the advent of Graph Convolution Networks has transformed the field in recent years.
Transformer-based Machine Learning for Fast SAT Solvers and Logic Synthesis
Our model has a scale-free structure which could process varying size of instances.
SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues
Inferring social relations from dialogues is vital for building emotionally intelligent robots to interpret human language better and act accordingly.
Ranked #5 on
Dialog Relation Extraction
on DialogRE
Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
We further propose a novel geometry solving approach with formal language and symbolic reasoning, called Interpretable Geometry Problem Solver (Inter-GPS).
Ranked #1 on
Mathematical Question Answering
on GeoS
VersaGNN: a Versatile accelerator for Graph neural networks
As GNNs operate on non-Euclidean data, their irregular data access patterns cause considerable computational costs and overhead on conventional architectures, such as GPU and CPU.
Learning Triadic Belief Dynamics in Nonverbal Communication from Videos
By aggregating different beliefs and true world states, our model essentially forms "five minds" during the interactions between two agents.
Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis
Specifically, the camera pose and 3D scene are represented as vectors and the local camera movement is represented as a matrix operating on the vector of the camera pose.
Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments
In this paper, we rethink the problem of scene reconstruction from an embodied agent's perspective: While the classic view focuses on the reconstruction accuracy, our new perspective emphasizes the underlying functions and constraints such that the reconstructed scenes provide \em{actionable} information for simulating \em{interactions} with agents.
Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance
Predicting agents' future trajectories plays a crucial role in modern AI systems, yet it is challenging due to intricate interactions exhibited in multi-agent systems, especially when it comes to collision avoidance.
ACRE: Abstract Causal REasoning Beyond Covariation
Causal induction, i. e., identifying unobservable mechanisms that lead to the observable relations among variables, has played a pivotal role in modern scientific discovery, especially in scenarios with only sparse and limited data.
Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution
To fill in this gap, we propose a neuro-symbolic Probabilistic Abduction and Execution (PrAE) learner; central to the PrAE learner is the process of probabilistic abduction and execution on a probabilistic scene representation, akin to the mental manipulation of objects.
VLGrammar: Grounded Grammar Induction of Vision and Language
In this work, we study grounded grammar induction of vision and language in a joint learning framework.
Towards Socially Intelligent Agents with Mental State Transition and Human Utility
One of which is to track the agent's mental state transition and teach the agent to make decisions guided by its value like a human.
Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translation
This paper studies the unsupervised cross-domain translation problem by proposing a generative framework, in which the probability distribution of each domain is represented by a generative cooperative network that consists of an energy-based model and a latent variable model.
Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration
Our experiments with human participants demonstrate that a short calibration using REMP can effectively bridge the gap between what a non-expert user thinks a robot can reach and the ground truth.
A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
We believe the HINT dataset and the experimental findings are of great interest to the learning community on systematic generalization.
HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
Leveraging these concepts, they could understand the internal structure of this task, without seeing all of the problem instances.
Learning Algebraic Representation for Abstract Spatial-Temporal Reasoning
We further show that the algebraic representation learned can be decoded by isomorphism and used to generate an answer.
Transformers satisfy
The Propositional Satisfiability Problem (SAT), and more generally, the Constraint Satisfaction Problem (CSP), are mathematical questions defined as finding an assignment to a set of objects that satisfies a series of constraints.
SMART: A Situation Model for Algebra Story Problems via Attributed Grammar
Solving algebra story problems remains a challenging task in artificial intelligence, which requires a detailed understanding of real-world situations and a strong mathematical reasoning capability.
Generative VoxelNet: Learning Energy-Based Models for 3D Shape Synthesis and Analysis
3D data that contains rich geometry information of objects and scenes is valuable for understanding 3D physical world.
Learning by Fixing: Solving Math Word Problems with Weak Supervision
To generate more diverse solutions, \textit{tree regularization} is applied to guide the efficient shrinkage and exploration of the solution space, and a \textit{memory buffer} is designed to track and save the discovered various fixes for each problem.
Ranked #1 on
Math Word Problem Solving
on Math23K
(weakly-supervised metric)
Weighted Entropy Modification for Soft Actor-Critic
We generalize the existing principle of the maximum Shannon entropy in reinforcement learning (RL) to weighted entropy by characterizing the state-action pairs with some qualitative weights, which can be connected with prior knowledge, experience replay, and evolution process of the policy.
Generalized Inverse Planning: Learning Lifted non-Markovian Utility for Generalizable Task Representation
Seeing that the proposed generalization problem has not been widely studied yet, we carefully define an evaluation protocol, with which we illustrate the effectiveness of MEIP on two proof-of-concept domains and one challenging task: learning to fold from demonstrations.
A Representational Model of Grid Cells' Path Integration Based on Matrix Lie Algebras
The grid cells in the mammalian medial entorhinal cortex exhibit striking hexagon firing patterns when the agent navigates in the open field.
Structured Attention for Unsupervised Dialogue Structure Induction
Inducing a meaningful structural representation from one or a set of dialogues is a crucial but challenging task in computational linguistics.
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities
Understanding and interpreting human actions is a long-standing challenge and a critical indicator of perception in artificial intelligence.
Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks
Thus, in this paper, we propose a novel explainable AI (XAI) framework for achieving human-like communication in human-robot collaborations, where the robot builds a hierarchical mind model of the human user and generates explanations of its own mind as a form of communications based on its online Bayesian inference of the user's mental state.
Bayesian Inference
Explainable Artificial Intelligence (XAI)
+1
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
Humans can progressively learn visual concepts from easy to hard questions.
On Path Integration of Grid Cells: Group Representation and Isotropic Scaling
In this paper, we conduct theoretical analysis of a general representation model of path integration by grid cells, where the 2D self-position is encoded as a higher dimensional vector, and the 2D self-motion is represented by a general transformation of the vector.
Learning Latent Space Energy-Based Prior Model
Due to the low dimensionality of the latent space and the expressiveness of the top-down network, a simple EBM in latent space can capture regularities in the data effectively, and MCMC sampling in latent space is efficient and mixes well.
MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMC
Learning energy-based model (EBM) requires MCMC sampling of the learned model as an inner loop of the learning algorithm.
Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning
In this paper, we address these issues and close the loop of neural-symbolic learning by (1) introducing the \textbf{grammar} model as a \textit{symbolic prior} to bridge neural perception and symbolic reasoning, and (2) proposing a novel \textbf{back-search} algorithm which mimics the top-down human-like learning procedure to propagate the error through the symbolic reasoning module efficiently.
Joint Training of Variational Auto-Encoder and Latent Energy-Based Model
This paper proposes a joint training method to learn both the variational auto-encoder (VAE) and the latent energy-based model (EBM).
Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Models
Our contributions are 1) an improved method for training EBM's with realistic long-run MCMC samples, 2) an Expectation-Over-Transformation (EOT) defense that resolves theoretical ambiguities for stochastic defenses and from which the EOT attack naturally follows, and 3) state-of-the-art adversarial defense for naturally-trained classifiers and competitive defense compared to adversarially-trained classifiers on Cifar-10, SVHN, and Cifar-100.
Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressions
To measure the true progress of existing models, we split the test set into two sets, one which requires reasoning on linguistic structure and the other which doesn't.
Congestion-aware Evacuation Routing using Augmented Reality Devices
We present a congestion-aware routing solution for indoor evacuation, which produces real-time individual-customized evacuation routes among multiple destinations while keeping tracks of all evacuees' locations.
Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning
To endow such a crucial cognitive ability to machine intelligence, we propose a dataset, Machine Number Sense (MNS), consisting of visual arithmetic problems automatically generated using a grammar model--And-Or Graph (AOG).
Joint Inference of States, Robot Knowledge, and Human (False-)Beliefs
Aiming to understand how human (false-)belief--a core socio-cognitive ability--would affect human interactions with robots, this paper proposes to adopt a graphical model to unify the representation of object states, robot knowledge, and human (false-)beliefs.
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility inference, and social learning.
Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification
We propose a generative model of unordered point sets, such as point clouds, in the form of an energy-based model, where the energy function is parameterized by an input-permutation-invariant bottom-up neural network.
Emergence of Pragmatics from Referential Game between Theory of Mind Agents
Pragmatics studies how context can contribute to language meanings.
PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points
Detecting 3D objects from a single RGB image is intrinsically ambiguous, thus requiring appropriate prior knowledge and intermediate representations as constraints to reduce the uncertainties and improve the consistencies between the 2D image plane and the 3D world coordinate.
Ranked #2 on
Monocular 3D Object Detection
on SUN RGB-D
(AP@0.15 (10 / PNet-30) metric)
Learning Multi-layer Latent Variable Model via Variational Optimization of Short Run MCMC for Approximate Inference
Learning such a generative model requires inferring the latent variables for each training example based on the posterior distribution of these latent variables.
Learning Perceptual Inference by Contrasting
"Thinking in pictures," [1] i. e., spatial-temporal reasoning, effortless and instantaneous for humans, is believed to be a significant ability to perform logical induction and a crucial factor in the intellectual history of technology development.
Representation Learning: A Statistical Perspective
Learning representations of data is an important problem in statistics and machine learning.
Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns
To model the motions explicitly, it is natural for the model to be based on the motions or the displacement fields of the pixels.
Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning
Given these general theories, the goal is to train an agent by interactively exploring the problem space to (i) discover, form, and transfer useful abstract and structural knowledge, and (ii) induce useful knowledge from the instance-level attributes observed in the environment.
DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare
We present DenseRaC, a novel end-to-end framework for jointly estimating 3D human pose and body shape from a monocular RGB image.
Ranked #82 on
3D Human Pose Estimation
on MPI-INF-3DHP
(using extra training data)
Learning Energy-based Spatial-Temporal Generative ConvNets for Dynamic Patterns
We show that an energy-based spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns.
TWIN GRAPH CONVOLUTIONAL NETWORKS: GCN WITH DUAL GRAPH SUPPORT FOR SEMI-SUPERVISED LEARNING
Graph Neural Networks as a combination of Graph Signal Processing and Deep Convolutional Networks shows great power in pattern recognition in non-Euclidean domains.
X-ToM: Explaining with Theory-of-Mind for Gaining Justified Human Trust
We present a new explainable AI (XAI) framework aimed at increasing justified human trust and reliance in the AI machine through explanations.
Action Recognition
Explainable Artificial Intelligence (XAI)
+2
Inducing Hierarchical Compositional Model by Sparsifying Generator Network
To realize this AND-OR hierarchy in image synthesis, we learn a generator network that consists of the following two components: (i) Each layer of the hierarchy is represented by an over-complete set of convolutional basis functions.
Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
This paper addresses a new problem of understanding human gaze communication in social videos from both atomic-level and event-level, which is significant for studying human social interactions.
Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
We propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction---3D estimations of object bounding boxes, camera pose, and room layout, and (ii) 3D human pose estimation.
3D Human Pose Estimation
Human-Object Interaction Detection
+1
HUGE2: a Highly Untangled Generative-model Engine for Edge-computing
In this work, we propose a Highly Untangled Generative-model Engine for Edge-computing or HUGE2 for accelerating these two special convolutions on the edge-computing platform by decomposing the kernels and untangling these smaller convolutions by performing basic matrix multiplications.
Learning Pose Grammar for Monocular 3D Pose Estimation
In this paper, we propose a pose grammar to tackle the problem of 3D human pose estimation from a monocular RGB image.
Ranked #15 on
3D Human Pose Estimation
on HumanEva-I
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
We treat this non-convergent short-run MCMC as a learned generator model or a flow model.
Reasoning Visual Dialogs with Structural and Partial Observations
The answer to a given question is represented by a node with missing value.
Ranked #14 on
Visual Dialog
on VisDial v0.9 val
On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models
On the other hand, ConvNet potentials learned with non-convergent MCMC do not have a valid steady-state and cannot be considered approximate unnormalized densities of the training data because long-run MCMC samples differ greatly from observed images.
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
One of the main challenges of advancing task-oriented learning such as visual task planning and reinforcement learning is the lack of realistic and standardized environments for training and testing AI agents.
Natural Language Interaction with Explainable AI Models
This paper presents an explainable AI (XAI) system that provides explanations for its predictions.
RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
In this work, we propose a new dataset, built in the context of Raven's Progressive Matrices (RPM) and aimed at lifting machine intelligence by associating vision with structural, relational, and analogical reasoning in a hierarchical representation.
Discourse Parsing in Videos: A Multi-modal Appraoch
We propose the task of Visual Discourse Parsing, which requires understanding discourse relations among scenes in a video.
Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning
This paper studies the problem of learning the conditional distribution of a high-dimensional output given an input, where the output and input may belong to two different domains, e. g., the output is a photo image and the input is a sketch image.
Learning V1 Simple Cells with Vector Representation of Local Content and Matrix Representation of Local Motion
This paper proposes a representational model for image pairs such as consecutive video frames that are related by local pixel displacements, in the hope that the model may shed light on motion perception in primary visual cortex (V1).
Inducing Sparse Coding and And-Or Grammar from Generator Network
We introduce an explainable generative model by applying sparse operation on the feature maps of the generator network.
Interpretable CNNs for Object Classification
This paper proposes a generic method to learn interpretable convolutional filters in a deep convolutional neural network (CNN) for object classification, where each interpretable filter encodes features of a specific object part.
Explaining AlphaGo: Interpreting Contextual Effects in Neural Networks
In this paper, we propose to disentangle and interpret contextual effects that are encoded in a pre-trained deep neural network.
Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Model
This paper proposes the divergence triangle as a framework for joint training of generator model, energy-based model and inference model.
Learning Dynamic Generator Model by Alternating Back-Propagation Through Time
The non-linear transformation of this transition model can be parametrized by a feedforward neural network.
Mining Interpretable AOG Representations from Convolutional Networks via Active Question Answering
The AOG associates each object part with certain neural units in feature maps of conv-layers.
Explanatory Graphs for CNNs
This paper introduces a graphical model, namely an explanatory graph, which reveals the knowledge hierarchy hidden inside conv-layers of a pre-trained CNN.
MetaStyle: Three-Way Trade-Off Among Speed, Flexibility, and Quality in Neural Style Transfer
An unprecedented booming has been witnessed in the research area of artistic style transfer ever since Gatys et al. introduced the neural method.
Deeper Interpretability of Deep Networks
However, understanding the information represented and processed in CNNs remains in most cases challenging.
Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation
Holistic 3D indoor scene understanding refers to jointly recovering the i) object bounding boxes, ii) room layout, and iii) camera pose, all in 3D.
Ranked #5 on
Monocular 3D Object Detection
on SUN RGB-D
Learning Grid Cells as Vector Representation of Self-Position Coupled with Matrix Representation of Self-Motion
In this model, the 2D self-position of the agent is represented by a high-dimensional vector, and the 2D self-motion or displacement of the agent is represented by a matrix that transforms the vector.
A Tale of Three Probabilistic Families: Discriminative, Descriptive and Generative Models
In this paper, we review three families of probability models, namely, the discriminative models, the descriptive models, and the generative models.
Sparse Winograd Convolutional neural networks on small-scale systolic arrays
The reconfigurability, energy-efficiency, and massive parallelism on FPGAs make them one of the best choices for implementing efficient deep learning accelerators.
Interactive Agent Modeling by Learning to Probe
In particular, the probing agent (i. e. a learner) learns to interact with the environment and with a target agent (i. e., a demonstrator) to maximize the change in the observed behaviors of that agent.
Human-centric Indoor Scene Synthesis Using Stochastic Grammar
We present a human-centric method to sample and synthesize 3D room layouts and 2D images thereof, to obtain large-scale 2D/3D image data with perfect per-pixel ground truth.
Learning Human-Object Interactions by Graph Parsing Neural Networks
For a given scene, GPNN infers a parse graph that includes i) the HOI graph structure represented by an adjacency matrix, and ii) the node labels.
Ranked #32 on
Human-Object Interaction Detection
on V-COCO
Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image
We propose a computational framework to jointly parse a single RGB image and reconstruct a holistic 3D configuration composed by a set of CAD models using a stochastic grammar model.
Ranked #4 on
Monocular 3D Object Detection
on SUN RGB-D
(AP@0.15 (10 / PNet-30) metric)
Deformable Generator Networks: Unsupervised Disentanglement of Appearance and Geometry
We present a deformable generator model to disentangle the appearance and geometric information for both image and video data in a purely unsupervised manner.
Generalized Earley Parser: Bridging Symbolic Grammars and Sequence Data for Future Prediction
Future predictions on sequence data (e. g., videos or audios) require the algorithms to capture non-Markovian and compositional properties of high-level semantics.
Where and Why Are They Looking? Jointly Inferring Human Attention and Intentions in Complex Tasks
We built a new video dataset of tasks, intentions, and attention.
Inferring Shared Attention in Social Scene Videos
We collect a new dataset VideoCoAtt from public TV show videos, containing 380 complex video sequences with more than 492, 000 frames that include diverse social scenes for shared attention study.
Attentive Fashion Grammar Network for Fashion Landmark Detection and Clothing Category Classification
This paper proposes a knowledge-guided fashion network to solve the problem of visual fashion analysis, e. g., fashion landmark localization and clothing category classification.
Unsupervised Learning of Neural Networks to Explain Neural Networks
Given feature maps of a certain conv-layer of the CNN, the explainer performs like an auto-encoder, which first disentangles the feature maps into object-part features and then inverts object-part features back to features of higher conv-layers of the CNN.
Learning Descriptor Networks for 3D Shape Synthesis and Analysis
This paper proposes a 3D shape descriptor network, which is a deep convolutional energy-based model, for modeling volumetric shape patterns.
Intent-aware Multi-agent Reinforcement Learning
We experiment our algorithm in a real-world problem that is non-episodic, and the number of agents and goals can vary over time.
Multi-agent Reinforcement Learning
reinforcement-learning
+2