Search Results for author:
Found 239 papers, 83 papers with code
Help or Hinder: Bayesian Models of Social Goal Inference
Everyday social interactions are heavily influenced by our snap judgments about others goals.
Perceptual Multistability as Markov Chain Monte Carlo Inference
While many perceptual and cognitive phenomena are well described in terms of Bayesian inference, the necessary computations are intractable at the scale of real-world tasks, and it remains unclear how the human mind approximates Bayesian inference algorithmically.
Modelling Relational Data using Bayesian Clustered Tensor Factorization
We consider the problem of learning probabilistic models for complex relational structures between various types of objects.
Explaining human multiple object tracking as resource-constrained approximate inference in a dynamic probabilistic model
Multiple object tracking is a task commonly used to investigate the architecture of human visual attention.
Nonparametric Bayesian Policy Priors for Reinforcement Learning
We consider reinforcement learning in partially observable domains where the agent can query an expert for demonstrations.
Dynamic Infinite Relational Model for Time-varying Relational Data Analysis
We propose a new probabilistic model for analyzing dynamic evolutions of relational data, such as additions, deletions and split & merge, of relation clusters like communities in social networks.
Learning to Learn with Compound HD Models
We introduce HD (or ``Hierarchical-Deep'') models, a new compositional learning architecture that integrates deep learning models with structured hierarchical Bayesian models.
Church: a language for generative models
We introduce Church, a universal language for describing stochastic generative processes.
Towards common-sense reasoning via conditional simulation: legacies of Turing in Artificial Intelligence
In the intervening years, the idea of cognition as computation has emerged as a fundamental tenet of Artificial Intelligence (AI) and cognitive science.
Structure Discovery in Nonparametric Regression through Compositional Kernel Search
Despite its importance, choosing the structural form of the kernel in nonparametric regression remains a black art.
Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs
The idea of computer vision as the Bayesian inverse problem to computer graphics has a long history and an appealing elegance, but it has proved difficult to directly implement.
Automatic Construction and Natural-Language Description of Nonparametric Regression Models
This paper presents the beginnings of an automatic statistician, focusing on regression problems.
Inverse Graphics with Probabilistic CAD Models
We show that it is possible to solve challenging, real-world 3D vision problems by approximate inference in generative models for images based on rendering the outputs of probabilistic CAD (PCAD) programs.
Deep Convolutional Inverse Graphics Network
This paper presents the Deep Convolution Inverse Graphics Network (DC-IGN), a model that learns an interpretable representation of images.
Risk and Regret of Hierarchical Bayesian Learners
Common statistical practice has shown that the full power of Bayesian methods is not realized until hierarchical priors are used, as these allow for greater "robustness" and the ability to "share statistical strength."
Picture: A Probabilistic Programming Language for Scene Perception
Recent progress on probabilistic modeling and statistical learning, coupled with the availability of large training datasets, has led to remarkable progress in computer vision.
Modeling Human Understanding of Complex Intentional Action with a Bayesian Nonparametric Subgoal Model
Here we model how humans infer subgoals from observations of complex action sequences using a nonparametric Bayesian model, which assumes that observed actions are generated by approximately rational planning over unknown subgoal sequences.
CrossCat: A Fully Bayesian Nonparametric Method for Analyzing Heterogeneous, High Dimensional Data
CrossCat infers multiple non-overlapping views of the data, each consisting of a subset of the variables, and uses a separate nonparametric mixture to model each view.
Modeling Human Ad Hoc Coordination
Whether in groups of humans or groups of computer agents, collaboration is most effective between individuals who have the ability to coordinate on a joint strategy for collective action.
Understanding Visual Concepts with Continuation Learning
We introduce a neural network architecture and a learning algorithm to produce factorized symbolic representations.
Building Machines That Learn and Think Like People
Recent progress in artificial intelligence (AI) has renewed interest in building systems that learn and think like people.
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms.
Single Image 3D Interpreter Network
In this work, we propose 3D INterpreter Network (3D-INN), an end-to-end framework which sequentially estimates 2D keypoint heatmaps and 3D object structure, trained on both real 2D-annotated images and synthetic 3D data.
A Comparative Evaluation of Approximate Probabilistic Simulation and Deep Neural Networks as Accounts of Human Physical Scene Understanding
Humans demonstrate remarkable abilities to predict physical events in complex scenes.
Human collective intelligence as distributed Bayesian inference
To address this gap, we introduce a new analytical framework: We propose that groups arrive at accurate shared beliefs via distributed Bayesian inference.
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
We study the problem of 3D object generation.
Ranked #3 on
3D Shape Classification
on Pix3D
3D Object Recognition
3D Point Cloud Linear Classification
+3
The Emergence of Organizing Structure in Conceptual Representation
While this approach can learn intuitive organizations, including a tree for animals and a ring for the color circle, it assumes a strong inductive bias that considers only these particular forms, and each form is explicitly provided as initial knowledge.
A Compositional Object-Based Approach to Learning Physical Dynamics
By comparing to less structured architectures, we show that the NPE's compositional representation of the structure in physical interactions improves its ability to predict movement, generalize across variable object count and different scene configurations, and infer latent properties of objects such as mass.
Beating the World's Best at Super Smash Bros. with Deep Reinforcement Learning
There has been a recent explosion in the capabilities of game-playing artificial intelligence.
Neural Scene De-Rendering
Our approach employs a deterministic rendering function as the decoder, mapping a naturally structured and disentangled scene description, which we named scene XML, to an image.
Synthesizing 3D Shapes via Modeling Multi-View Depth Maps and Silhouettes With Deep Generative Networks
We take an alternative approach: learning a generative model over multi-view depth maps or their corresponding silhouettes, and using a deterministic rendering function to produce 3D shapes from these images.
Learning to Infer Graphics Programs from Hand-Drawn Images
These drawing primitives are like a trace of the set of primitive commands issued by a graphics program.
Generative Modeling of Audible Shapes for Object Perception
Humans infer rich knowledge of objects from both auditory and visual cues.
A First Step in Combining Cognitive Event Features and Natural Language Representations to Predict Emotions
Cognitive science has proposed appraisal theory as a view on human emotion with previous research showing how human-rated abstract event features can predict fine-grained emotions and capture the similarity space of neural patterns in mentalizing brain regions.
MarrNet: 3D Shape Reconstruction via 2.5D Sketches
First, compared to full 3D shape, 2. 5D sketches are much easier to be recovered from a 2D image; models that recover 2. 5D sketches are also more likely to transfer from synthetic to real data.
Ranked #2 on
3D Shape Classification
on Pix3D
3D Object Reconstruction From A Single Image
3D Reconstruction
+3
Self-Supervised Intrinsic Image Decomposition
Intrinsic decomposition from a single image is a highly challenging task, due to its inherent ambiguity and the scarcity of training data.
The Variational Homoencoder: Learning to Infer High-Capacity Generative Models from Few Examples
Hierarchical Bayesian methods have the potential to unify many related tasks (e. g. k-shot classification, conditional, and unconditional generation) by framing each as inference within a single generative model.
Meta-Learning for Semi-Supervised Few-Shot Classification
To address this paradigm, we propose novel extensions of Prototypical Networks (Snell et al., 2017) that are augmented with the ability to use unlabeled examples when producing prototypes.
Blaming humans in autonomous vehicle accidents: Shared responsibility across levels of automation
However, when both drivers make errors in cases of shared control between a human and a machine, the blame and responsibility attributed to the machine is reduced.
The Three Pillars of Machine Programming
In this position paper, we describe our vision of the future of machine programming through a categorical examination of three pillars of research.
3D Interpreter Networks for Viewer-Centered Wireframe Modeling
3D-INN is trained on real images to estimate 2D keypoint heatmaps from an input image; it then predicts 3D object structure from heatmaps using knowledge learned from synthetic 3D shapes.
Discovery and usage of joint attention in images
Second, we use a human study to demonstrate the sensitivity of humans to joint attention, suggesting that the detection of such a configuration in an image can be useful for understanding the image, including the goals of the agents and their joint activity, and therefore can contribute to image captioning and related tasks.
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
We study 3D shape modeling from a single image and make contributions to it in three aspects.
Ranked #1 on
3D Shape Classification
on Pix3D
Word learning and the acquisition of syntactic--semantic overhypotheses
Children learning their first language face multiple problems of induction: how to learn the meanings of words, and how to build meaningful phrases from those words according to syntactic rules.
Relational inductive bias for physical construction in humans and machines
While current deep learning systems excel at tasks such as object classification, language processing, and gameplay, few can construct or modify a complex system such as a tower of blocks.
Flexible Neural Representation for Physics Prediction
Humans have a remarkable capacity to understand the physical dynamics of objects in their environment, flexibly capturing complex structures and interactions at multiple levels of detail.
Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networks
We propose a framework for the completely unsupervised learning of latent object properties from their interactions: the perception-prediction network (PPN).
The Variational Homoencoder: Learning to learn high capacity generative models from few examples
However, when this generative model is expressed as a powerful neural network such as a PixelCNN, we show that existing learning techniques typically fail to effectively use latent variables.
Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncing
An efficient, generalizable physical simulator with universal uncertainty estimates has wide applications in robot state estimation, planning, and control.
3D Shape Perception from Monocular Vision, Touch, and Shape Priors
Perceiving accurate 3D object shape is important for robots to interact with the physical world.
3D-Aware Scene Manipulation via Inverse Graphics
In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model.
Modeling human intuitions about liquid flow with particle-based simulation
Humans can easily describe, imagine, and, crucially, predict a wide variety of behaviors of liquids--splashing, squirting, gushing, sloshing, soaking, dripping, draining, trickling, pooling, and pouring--despite tremendous variability in their material and dynamical properties.
Seeing Tree Structure from Vibration
Humans recognize object structure from both their appearance and motion; often, motion helps to resolve ambiguities in object structure that arise when we observe object appearance only.
Learning Shape Priors for Single-View 3D Completion and Reconstruction
The problem of single-view 3D shape completion or reconstruction is challenging, because among the many possible shapes that explain an observation, most are implausible and do not correspond to natural objects.
Physical Primitive Decomposition
As annotated data for object parts and physics are rare, we propose a novel formulation that learns physical primitives by explaining both an object's appearance and its behaviors in physical events.
Propagation Networks for Model-Based Control Under Partial Observation
There has been an increasing interest in learning dynamics simulators for model-based control.
ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Robotics
The underlying physical laws of deformable objects are more complex, and the resulting systems have orders of magnitude more degrees of freedom and therefore they are significantly more computationally expensive to simulate.
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
In this paper, we propose to learn a particle-based simulator for complex control tasks.
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
Second, the model is more data- and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering.
Ranked #1 on
Visual Question Answering (VQA)
on CLEVR
GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.
Library Learning for Neurally-Guided Bayesian Program Induction
Successful approaches to program induction require a hand-engineered domain-specific language (DSL), constraining the space of allowed programs and imparting prior knowledge of the domain.
Visual Object Networks: Image Generation with Disentangled 3D Representation
Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes.
Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
Object-based factorizations provide a useful level of abstraction for interacting with the world.
Learning to Reconstruct Shapes from Unseen Classes
From a single image, humans are able to perceive the full 3D shape of an object by exploiting learned shape priors from everyday life.
Learning to Infer and Execute 3D Shape Programs
Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higher-level shape structure such as the repetition and reflective symmetry of object parts.
Theory of Minds: Understanding Behavior in Groups Through Inverse Planning
This representation is grounded in the formalism of stochastic games and multi-agent reinforcement learning.
On the Units of GANs (Extended Abstract)
We quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.
The Omniglot challenge: a 3-year progress report
Three years ago, we released the Omniglot dataset for one-shot learning, along with five challenge tasks and a computational model that addresses these tasks.
Infinite Mixture Prototypes for Few-Shot Learning
We propose infinite mixture prototypes to adaptively represent both simple and complex data distributions for few-shot learning.
Stochastic Prediction of Multi-Agent Interactions from Partial Observations
We present a method that learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents.
Unsupervised Discovery of Parts, Structure, and Dynamics
Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future.
Visualizing and Understanding GANs
We present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level.
Combining Physical Simulators and Object-Based Networks for Control
Physics engines play an important role in robot planning and control; however, many real-world control problems involve complex contact dynamics that cannot be characterized analytically.
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation.
Ranked #5 on
Visual Question Answering (VQA)
on CLEVR
Reasoning About Physical Interactions with Object-Centric Models
Object-based factorizations provide a useful level of abstraction for interacting with the world.
Learning to Describe Scenes with Programs
We are able to understand the higher-level, abstract regularities within the scene such as symmetry and repetition.
Modeling Parts, Structure, and System Dynamics via Predictive Learning
Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future.
Predicting the Present and Future States of Multi-agent Systems from Partially-observed Visual Data
We present a method which learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents.
Finding Friend and Foe in Multi-Agent Games
Here we develop the DeepRole algorithm, a multi-agent reinforcement learning agent that we test on The Resistance: Avalon, the most popular hidden role game.
Neurally-Guided Structure Inference
In this paper, we propose a hybrid inference algorithm, the Neurally-Guided Structure Inference (NG-SI), keeping the advantages of both search-based and data-driven methods.
Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
But human beings remain distinctive in their capacity for flexible, creative tool use -- using objects in new ways to act on the world, achieve a goal, or solve a problem.
Program-Guided Image Manipulators
Humans are capable of building holistic representations for images at various levels, from local objects, to pairwise relations, to global structures.
DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs
A major difficulty of solving continuous POMDPs is to infer the multi-modal distribution of the unobserved true states and to make the planning algorithm dependent on the perceived uncertainty.
CLEVRER: CoLlision Events for Video REpresentation and Reasoning
While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations.
Entity Abstraction in Visual Model-Based Reinforcement Learning
This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before.
Accurate Vision-based Manipulation through Contact Reasoning
Planning contact interactions is one of the core challenges of many robotic tasks.
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation
In this paper, we attempt to approach the problem of Audio-Visual Embodied Navigation, the task of planning the shortest path from a random starting location in a scene to the sound source in an indoor environment, given only raw egocentric visual and audio sensory data.
Visual Concept-Metaconcept Learning
Humans reason with concepts and metaconcepts: we recognize red and green from visual input; we also understand that they describe the same property of objects (i. e., the color).
Learning Compositional Rules via Neural Program Synthesis
Many aspects of human reasoning, including language, require learning rules from very little data.
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
The focus of recent meta-learning research has been on the development of learning algorithms that can quickly adapt to test time tasks with limited data and low computational cost.
Too many cooks: Bayesian inference for coordinating multi-agent collaboration
Underlying the human ability to collaborate is theory-of-mind, the ability to infer the hidden mental states that drive others to act.
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility inference, and social learning.
Music Gesture for Visual Sound Separation
Recent deep learning approaches have achieved impressive performance on visual sound separation tasks.
Visual Grounding of Learned Physical Models
The abilities to perform physical reasoning and to adapt to new environments, while intrinsic to humans, remain challenging to state-of-the-art computational models.
Deep Audio Priors Emerge From Harmonic Convolutional Networks
We show that networks using Harmonic Convolution can reliably model audio priors and achieve high performance in unsupervised audio restoration tasks.
Online Bayesian Goal Inference for Boundedly-Rational Planning Agents
These models are specified as probabilistic programs, allowing us to represent and perform efficient Bayesian inference over an agent's goals and internal planning processes.
DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning
It builds expertise by creating programming languages for expressing domain concepts, together with neural networks to guide the search for programs within these languages.
Learning with AMIGo: Adversarially Motivated Intrinsic Goals
A key challenge for reinforcement learning (RL) consists of learning in environments with sparse extrinsic rewards.
Learning Physical Graph Representations from Visual Scenes
To overcome these limitations, we introduce the idea of Physical Scene Graphs (PSGs), which represent scenes as hierarchical graphs, with nodes in the hierarchy corresponding intuitively to object parts at different scales, and edges to physical connections between parts.
Perspective Plane Program Induction from a Single Image
We study the inverse graphics problem of inferring a holistic representation for natural images.
Learning to learn generative programs with Memoised Wake-Sleep
We study a class of neuro-symbolic generative models in which neural networks are used both for inference and as priors over symbolic, data-generating programs.
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation.
Foley Music: Learning to Generate Music from Videos
In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments.
End-to-End Optimization of Scene Layout
Experiments suggest that our model achieves higher accuracy and diversity in conditional scene synthesis and allows exemplar-based scene generation from various input forms.
Noisy Agents: Self-supervised Exploration by Predicting Auditory Events
Humans integrate multiple sensory modalities (e. g. visual and audio) to build a causal understanding of the physical world.
Learning abstract structure for drawing by efficient motor program induction
Humans flexibly solve new problems that differ qualitatively from those they were trained on.
Learning Online Data Association
When an agent interacts with a complex environment, it receives a stream of percepts in which it may detect entities, such as objects or people.
Causal Inductive Synthesis Corpus
We introduce the Causal Inductive Synthesis Corpus (CISC) -- a manually constructed collection of interactive domains.
Measuring few-shot extrapolation with program induction
Program induction lies at the opposite end of the spectrum: programs are capable of extrapolating from very few examples, but we still do not know how to efficiently search for complex programs.
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents.
Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling
Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the (aggregate) posterior to encourage statistical independence of the latent factors.
Data-Efficient Learning for Complex and Real-Time Physical Problem Solving using Augmented Simulation
The physics engine augmented with the residual model is then used to control the marble in the maze environment using a model-predictive feedback over a receding horizon.
Multi-Plane Program Induction with 3D Box Priors
We consider two important aspects in understanding and editing images: modeling regular, program-like texture or patterns in 2D planes, and 3D posing of these planes in the scene.
Neural Radiance Flow for 4D View Synthesis and Video Processing
We present a method, Neural Radiance Flow (NeRFlow), to learn a 4D spatial-temporal representation of a dynamic scene from a set of RGB images.
Object-Centric Diagnosis of Visual Reasoning
In contrast, symbolic and modular models have a relatively better grounding and robustness, though at the cost of accuracy.
Augmenting Policy Learning with Routines Discovered from a Single Demonstration
Humans can abstract prior knowledge from very little data and use it to boost skill learning.
Representing Partial Programs with Blended Abstract Semantics
In this search process, a key challenge is representing the behavior of a partially written program before it can be executed, to judge if it is on the right track and predict where to search next.
Temporal and Object Quantification Nets
We aim to learn generalizable representations for complex activities by quantifying over both entities and time, as in “the kicker is behind all the other players,” or “the player controls the ball until it moves toward the goal.” Such a structural inductive bias of object relations, object quantification, and temporal orders will enable the learned representation to generalize to situations with varying numbers of agents, objects, and time courses.
Unsupervised Discovery of 3D Physical Objects
We study the problem of unsupervised physical object discovery.
Grounding Physical Object and Event Concepts Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos.
A Bayesian-Symbolic Approach to Learning and Reasoning for Intuitive Physics
As such, learning the laws is then reduced to symbolic regression and Bayesian inference methods are used to obtain the distribution of unobserved properties.
Learning Task Decomposition with Order-Memory Policy Network
Many complex real-world tasks are composed of several levels of sub-tasks.
AGENT: A Benchmark for Core Psychological Reasoning
For machine agents to successfully interact with humans in real-world settings, they will need to develop an understanding of human mental life.
Ranked #1 on
Core Psychological Reasoning
on AGENT
PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception
The ability to perceive and reason about social interactions in the context of physical environments is core to human social intelligence and human-machine cooperation.
Learning Task Decomposition with Ordered Memory Policy Network
The discovered subtask hierarchy could be used to perform task decomposition, recovering the subtask boundaries in an unstruc-tured demonstration.
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI
To complete the task, an embodied agent must plan a sequence of actions to change the state of a large number of objects in the face of realistic physical constraints.
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos.
PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics
Experimental results suggest that 1) RL-based approaches struggle to solve most of the tasks efficiently; 2) gradient-based approaches, by optimizing open-loop control sequences with the built-in differentiable physics engine, can rapidly find a solution within tens of iterations, but still fall short on multi-stage tasks that require long-term planning.
Learning Neuro-Symbolic Relational Transition Models for Bilevel Planning
In robotic domains, learning and planning are complicated by continuous state spaces, continuous action spaces, and long task horizons.
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering
In this work, we propose a novel neural scene representation, Light Field Networks or LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, four-dimensional light field parameterized via a neural implicit representation.
Temporal and Object Quantification Networks
We present Temporal and Object Quantification Networks (TOQ-Nets), a new class of neuro-symbolic networks with a structural bias that enables them to learn to recognize complex relational-temporal events.
Communicating Natural Programs to Humans and Machines
We present LARC, the \textit{Language-complete ARC}: a collection of natural language descriptions by a group of human participants who instruct each other on how to solve ARC tasks using language alone, which contains successful instructions for 88\% of the ARC tasks.
Physion: Evaluating Physical Prediction from Vision in Humans and Machines
While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments.
Leveraging Language to Learn Program Abstractions and Search Heuristics
Inductive program synthesis, or inferring programs from examples of desired behavior, offers a general paradigm for building interpretable, robust, and generalizable machine learning systems.
Modeling the Mistakes of Boundedly Rational Agents Within a Bayesian Theory of Mind
Here, we extend the Bayesian Theory of Mind framework to model boundedly rational agents who may have mistaken goals, plans, and actions.
Hybrid Memoised Wake-Sleep: Approximate Inference at the Discrete-Continuous Interface
We build on a recent approach, Memoised Wake-Sleep (MWS), which alleviates part of the problem by memoising discrete variables, and extend it to allow for a principled and effective way to handle continuous variables by learning a separate recognition model used for importance-sampling based approximate inference and marginalization.
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning
Human reasoning can often be understood as an interplay between two systems: the intuitive and associative ("System 1") and the deliberative and logical ("System 2").
Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
Here we propose a new approach to this challenge based on a particularly strong form of model-based RL which we call Theory-Based Reinforcement Learning, because it uses human-like intuitive theories -- rich, abstract, causal models of physical objects, intentional agents, and their interactions -- to explore and model an environment, and plan effectively to achieve task goals.
Learning to solve complex tasks by growing knowledge culturally across generations
The video game paradigm we pioneer here is thus a rich test bed for developing AI systems capable of acquiring and transmitting cultural knowledge.
Dynamic Modeling of Hand-Object Interactions via Tactile Sensing
This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing, which opens the door for future applications in activity learning, human-computer interactions, and imitation learning for robotics.
On the Expressiveness and Learning of Relational Neural Networks on Hypergraphs
Our first contribution is a fine-grained analysis of the expressiveness of these neural networks, that is, the set of functions that they can realize and the set of problems that they can solve.
Inducing Reusable Skills From Demonstrations with Option-Controller Network
With the isolation of information and the synchronous calling mechanism, we can impose a division of works between the controller and options in an end-to-end training regime.
Learning Rational Skills for Planning from Demonstrations and Instructions
We present a framework for learning compositional, rational skill models (RatSkills) that support efficient planning and inverse planning for achieving novel goals and recognizing activities.
AutumnSynth: Synthesis of Reactive Programs with Structured Latent State
The human ability to efficiently discover causal theories of their environments from observations is a feat of nature that remains elusive in machines.
OPEn: An Open-ended Physics Environment for Learning Without a Task
We test several existing RL-based exploration methods on this benchmark and find that an agent using unsupervised contrastive learning for representation learning, and impact-driven learning for exploration, achieved the best results.
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language
This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine.
3DP3: 3D Scene Perception via Probabilistic Programming
We present 3DP3, a framework for inverse graphics that uses inference in a structured generative model of objects, scenes, and images.
MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation
Recent advances in generative adversarial networks (GANs) have led to remarkable achievements in face image synthesis.
Unsupervised Learning of Compositional Energy Concepts
In this work, we propose COMET, which discovers and represents concepts as separate energy functions, enabling us to represent both global concepts as well as objects under a unified framework.
Learning Signal-Agnostic Manifolds of Neural Fields
We leverage neural fields to capture the underlying structure in image, shape, audio and cross-modal audiovisual domains in a modality-independent manner.
Learning to Compose Visual Relations
The visual world around us can be described as a structured set of objects and their associated relations.
STAR: A Benchmark for Situated Reasoning in Real-World Videos
This paper introduces a new benchmark that evaluates the situated reasoning ability via situation abstraction and logic-grounded question answering for real-world videos, called Situated Reasoning in Real-World Videos (STAR).
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierarchies.
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation
Our performance generalizes across both object instances and 6-DoF object poses, and significantly outperforms a recent baseline that relies on 2D descriptors.
Grammar-Based Grounded Lexicon Learning
We present Grammar-Based Grounded Lexicon Learning (G2L2), a lexicalist approach toward learning a compositional and grounded meaning representation of language from grounded data, such as paired images and texts.
Linking Emergent and Natural Languages via Corpus Transfer
In this work, we propose a novel way to establish such a link by corpus transfer, i. e. pretraining on a corpus of emergent language for downstream natural language tasks, which is in contrast to prior work that directly transfers speaker and listener parameters.
FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations
We present a meta-learning framework for learning new visual concepts quickly, from just one or a few examples, guided by multiple naturally occurring data streams: simultaneously looking at images, reading sentences that describe the objects in the scene, and interpreting supplemental sentences that relate the novel concept with other concepts.
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools
We consider the problem of sequential robotic manipulation of deformable objects using tools.
Learning Neural Acoustic Fields
By modeling acoustic propagation in a scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds.
ComPhy: Compositional Physical Reasoning of Objects and Events from Videos
In this paper, we take an initial step to highlight the importance of inferring the hidden physical properties not directly observable from visual appearances, by introducing the Compositional Physical Reasoning (ComPhy) dataset.
Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction
Specifically, FixNet consists of a perception module to extract the structured representation from the 3D point cloud, a physical dynamics prediction module to simulate the results of interactions on 3D objects, and a functionality prediction module to evaluate the functionality and choose the correct fix.
Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics
Extensive experimental results suggest that: 1) on multi-stage tasks that are infeasible for the vanilla differentiable physics solver, our approach discovers contact points that efficiently guide the solver to completion; 2) on tasks where the vanilla solver performs sub-optimally or near-optimally, our contact point discovery method performs better than or on par with the manipulation performance obtained with handcrafted contact points.
Unsupervised Discovery and Composition of Object Light Fields
Neural scene representations, both continuous and discrete, have recently emerged as a powerful new paradigm for 3D scene understanding.
Identifying concept libraries from language about object structure
Our understanding of the visual world goes beyond naming objects, encompassing our ability to parse objects into meaningful parts, attributes, and relations.
RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation
To train this predictor, we formulate a new loss on rendering variances using gradients from differentiable rendering.
Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks
We first contribute a new challenge benchmark for comparing humans and distributional large language models (LLMs).
Unsupervised Segmentation in Real-World Images via Spelke Object Inference
Self-supervised, category-agnostic segmentation of real-world images is a challenging open problem in computer vision.
Planning with Diffusion for Flexible Behavior Synthesis
Model-based reinforcement learning methods often use learning only for the purpose of estimating an approximate dynamics model, offloading the rest of the decision-making work to classical trajectory optimizers.
Compositional Visual Generation with Composable Diffusion Models
Large text-guided diffusion models, such as DALLE-2, are able to generate stunning photorealistic images given natural language descriptions.
Drawing out of Distribution with Neuro-Symbolic Generative Models
We then adopt a subset of the Omniglot challenge tasks, and evaluate its ability to generate new exemplars (both unconditionally and conditionally), and perform one-shot classification, showing that DooD matches the state of the art.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Learning Neuro-Symbolic Skills for Bilevel Planning
Decision-making is challenging in robotics environments with continuous object-centric states, continuous actions, long horizons, and sparse feedback.
Prompting Decision Transformer for Few-Shot Policy Generalization
Humans can leverage prior experience and learn novel tasks from a handful of demonstrations.
Learning Iterative Reasoning through Energy Minimization
Finally, we illustrate that our approach can recursively solve algorithmic problems requiring nested reasoning
Finding Fallen Objects Via Asynchronous Audio-Visual Integration
The way an object looks and sounds provide complementary reflections of its physical properties.
3D Concept Grounding on Neural Fields
Experimental results show that our proposed framework outperforms unsupervised/language-mediated segmentation models on semantic and instance segmentation tasks, as well as outperforms existing models on the challenging 3D aware visual reasoning tasks.
Neural Groundplans: Persistent Neural Scene Representations from a Single Image
We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene.
Robust Change Detection Based on Neural Descriptor Fields
The ability to reason about changes in the environment is crucial for robots operating over extended periods of time.
Solving the Baby Intuitions Benchmark with a Hierarchically Bayesian Theory of Mind
To facilitate the development of new models to bridge the gap between machine and human social intelligence, the recently proposed Baby Intuitions Benchmark (arXiv:2102. 11938) provides a suite of tasks designed to evaluate commonsense reasoning about agents' goals and actions that even young infants exhibit.
Abstract Interpretation for Generalized Heuristic Search in Model-Based Planning
Domain-general model-based planners often derive their generality by constructing search heuristics through the relaxation or abstraction of symbolic world models.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
Learning Physical Dynamics with Subequivariant Graph Neural Networks
Graph Neural Networks (GNNs) have become a prevailing tool for learning physical dynamics.
Revisiting the Roles of "Text" in Text Games
Text games present opportunities for natural language understanding (NLU) methods to tackle reinforcement learning (RL) challenges.
Composing Ensembles of Pre-trained Models via Iterative Consensus
Such closed-loop communication enables models to correct errors caused by other models, significantly boosting performance on downstream tasks, e. g. improving accuracy on grade school math problems by 7. 5%, without requiring any model finetuning.
Ranked #1 on
Video Question Answering
on ActivityNet-QA
H-SAUR: Hypothesize, Simulate, Act, Update, and Repeat for Understanding Object Articulations from Interactions
The world is filled with articulated objects that are difficult to determine how to use from vision alone, e. g., a door might open inwards or outwards.
On the Complexity of Bayesian Generalization
Specifically, at the $representational \ level$, we seek to answer how the complexity varies when a visual concept is mapped to the representation space.
Top-Down Synthesis for Library Learning
This paper introduces corpus-guided top-down synthesis as a mechanism for synthesizing library functions that capture common functionality from a corpus of programs in a domain specific language (DSL).
Are Deep Neural Networks SMARTer than Second Graders?
To answer this question, we propose SMART: a Simple Multimodal Algorithmic Reasoning Task and the associated SMART-101 dataset, for evaluating the abstraction, deduction, and generalization abilities of neural networks in solving visuo-linguistic puzzles designed specifically for children in the 6--8 age group.
3D Shape Perception Integrates Intuitive Physics and Analysis-by-Synthesis
Many surface cues support three-dimensional shape perception, but people can sometimes still see shape when these features are missing -- in extreme cases, even when an object is completely occluded, as when covered with a draped cloth.
NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants
Experiments show that our helper agent robustly updates its goal inference and adapts its helping plans to the changing level of uncertainty.
Dissociating language and thought in large language models
Large Language Models (LLMs) have come closest among all models to date to mastering human language, yet opinions about their linguistic and cognitive capabilities remain split.
3D Neural Embedding Likelihood: Probabilistic Inverse Graphics for Robust 6D Pose Estimation
In this paper, we introduce probabilistic modeling to the inverse graphics framework to quantify uncertainty and achieve robustness in 6D pose estimation tasks.
Ranked #1 on
on YCB-Video
ConceptFusion: Open-set Multimodal 3D Mapping
ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio.
Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC
In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance.
PDSketch: Integrated Planning Domain Programming and Learning
This paper studies a model learning and online planning approach towards building flexible and general robots.
On the Expressiveness and Generalization of Hypergraph Neural Networks
Next, we analyze the learning properties of these neural networks, especially focusing on how they can be trained on a finite set of small graphs and generalize to larger graphs, which we term structural generalization.
Learning Rational Subgoals from Demonstrations and Instructions
We present a framework for learning useful subgoals that support efficient long-term planning to achieve novel goals.
Planning with Large Language Models for Code Generation
Existing large language model-based code generation pipelines typically use beam search or sampling algorithms during the decoding process.
Tactile-Filter: Interactive Tactile Perception for Part Mating
We evaluate our method on several part-mating tasks with novel objects using a robot equipped with a vision-based tactile sensor.
