Search Results for author:
Found 239 papers, 83 papers with code
GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignment
GOMA formulates verbal communication as a planning problem that minimizes the misalignment between the parts of agents' mental states that are relevant to the goals.
Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling
Questions combine our mastery of language with our remarkable facility for reasoning about uncertainty.
Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning
Our agent assists a human by modeling them as a cooperative planner who communicates joint plans to the assistant, then performs multimodal Bayesian inference over the human's goal from actions and language, using large language models (LLMs) to evaluate the likelihood of an instruction given a hypothesized plan.
ContPhy: Continuum Physical Concept Learning and Reasoning from Videos
We evaluated a range of AI models and found that they still struggle to achieve satisfactory performance on ContPhy, which shows that the current AI models still lack physical commonsense for the continuum, especially soft-bodies, and illustrates the value of the proposed dataset.
HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments
Recent advances in high-fidelity virtual environments serve as one of the major driving forces for building intelligent embodied agents to perceive, reason and interact with the physical world.
MMToM-QA: Multimodal Theory of Mind Question Answering
Human ToM, on the other hand, is more than video or text understanding.
Learning adaptive planning representations with natural language guidance
Effective planning in the real world requires not only world knowledge, but the ability to leverage that knowledge to build the right representation of the task at hand.
How to guess a gradient
We study how to narrow the gap in optimization performance between methods that calculate exact gradients and those that use directional derivatives.
What Planning Problems Can A Relational Neural Network Solve?
Goal-conditioned policies are generally understood to be "feed-forward" circuits, in the form of neural networks that map from the current state and the goal specification to the next action to take.
Learning Reusable Manipulation Strategies
Humans demonstrate an impressive ability to acquire and generalize manipulation "tricks."
LILO: Learning Interpretable Libraries by Compressing and Documenting Code
While large language models (LLMs) now excel at code generation, a key aspect of software development is the art of refactoring: consolidating code into libraries of reusable and readable programs.
What's Left? Concept Grounding with Logic-Enhanced Foundation Models
We propose the Logic-Enhanced Foundation Model (LEFT), a unified framework that learns to ground and reason with concepts across domains with a differentiable, domain-independent, first-order logic-based program executor.
Inferring Relational Potentials in Interacting Systems
In this work, we propose Neural Interaction Inference with Potentials (NIIP) as an alternative approach to discover such interactions that enables greater flexibility in trajectory modeling: it discovers a set of relational potentials, represented as energy functions, which when minimized reconstruct the original trajectory.
LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers
Logical reasoning, i. e., deductively inferring the truth value of a conclusion from a set of premises, is an important task for artificial intelligence with wide potential impacts on science, mathematics, and society.
AI for Mathematics: A Cognitive Science Perspective
Mathematics is one of the most powerful conceptual systems developed and used by the human species.
Getting aligned on representational alignment
Finally, we lay out open problems in representational alignment where progress can benefit all three of these fields.
Video Language Planning
We are interested in enabling visual planning for complex long-horizon tasks in the space of generated videos and language, leveraging recent advances in large generative models pretrained on Internet-scale data.
Learning to Act from Actionless Videos through Dense Correspondences
In this work, we present an approach to construct a video-based robot policy capable of reliably executing diverse tasks across different robots and environments from few video demonstrations without using any action annotations.
HandMeThat: Human-Robot Communication in Physical and Social Environments
We introduce HandMeThat, a benchmark for a holistic evaluation of instruction understanding and following in physical and social environments.
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning
We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts.
Tactile Estimation of Extrinsic Contact Patch for Stable Placement
In particular, we estimate the contact patch between a grasped object and its environment using force and tactile observations to estimate the stability of the object during a contact formation.
Compositional Diffusion-Based Continuous Constraint Solvers
This paper introduces an approach for learning to solve continuous constraint satisfaction problems (CCSP) in robotic reasoning and planning.
Neural Amortized Inference for Nested Multi-agent Reasoning
Multi-agent interactions, such as communication, teaching, and bluffing, often rely on higher-order social inference, i. e., understanding how others infer oneself.
Building Cooperative Embodied Agents Modularly with Large Language Models
In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments.
Inferring the Goals of Communicating Agents from Actions and Instructions
When humans cooperate, they frequently coordinate their activity through both verbal communication and non-verbal actions, using this information to infer a shared goal and plan.
The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputs
To test our model, we design and run a human experiment on a linguistic goal inference task.
From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought
Our architecture integrates two computational tools that have not previously come together: we model thinking with probabilistic programs, an expressive representation for commonsense reasoning; and we model meaning construction with large language models (LLMs), which support broad-coverage translation from natural language utterances to code expressions in a probabilistic programming language.
Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models
Text-to-image generative models have enabled high-resolution image synthesis across different domains, but require users to specify the content they wish to generate.
Evaluating Language Models for Mathematics through Interactions
There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants.
Probabilistic Adaptation of Text-to-Video Models
Large text-to-video models trained on internet-scale data have demonstrated exceptional capabilities in generating high-fidelity videos from arbitrary textual descriptions.
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks.
Generalized Planning in PDDL Domains with Pretrained Large Language Models
We investigate whether LLMs can serve as generalized planners: given a domain and training tasks, generate a program that efficiently produces plans for other tasks in the domain.
Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics
Many NN approaches learn an end-to-end model that implicitly models both the governing PDE and constitutive models (or material models).
Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following
ECL consists of: (i) an instruction parser that translates the natural languages into executable programs; (ii) an embodied concept learner that grounds visual concepts based on language descriptions; (iii) a map constructor that estimates depth and constructs semantic maps by leveraging the learned concepts; and (iv) a program executor with deterministic policies to execute each program.
Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention
When looking at an image, we can decompose the scene into entities and their parts as well as obtain the dependencies between them.
DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics
Reinforcement learning approaches for dexterous rigid object manipulation would struggle in this setting due to the complexity of physics interaction with deformable objects.
3D Concept Learning and Reasoning from Multi-View Images
We suggest that a principled approach for 3D reasoning from multi-view images should be to infer a compact 3D representation of the world from the multi-view images, which is further grounded on open-vocabulary semantic concepts, and then to execute reasoning on these 3D representations.
SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environments
Existing work has typically been tailored for particular environments or representations.
NeuSE: Neural SE(3)-Equivariant Embedding for Consistent Spatial Understanding with Objects
We present NeuSE, a novel Neural SE(3)-Equivariant Embedding for objects, and illustrate how it supports object SLAM for consistent spatial understanding with long-term scene changes.
Tactile-Filter: Interactive Tactile Perception for Part Mating
We evaluate our method on several part-mating tasks with novel objects using a robot equipped with a vision-based tactile sensor.
Planning with Large Language Models for Code Generation
Existing large language model-based code generation pipelines typically use beam search or sampling algorithms during the decoding process.
On the Expressiveness and Generalization of Hypergraph Neural Networks
Next, we analyze the learning properties of these neural networks, especially focusing on how they can be trained on a finite set of small graphs and generalize to larger graphs, which we term structural generalization.
PDSketch: Integrated Planning Domain Programming and Learning
This paper studies a model learning and online planning approach towards building flexible and general robots.
Learning Rational Subgoals from Demonstrations and Instructions
We present a framework for learning useful subgoals that support efficient long-term planning to achieve novel goals.
Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC
In this work, we build upon these ideas using the score-based interpretation of diffusion models, and explore alternative ways to condition, modify, and reuse diffusion models for tasks involving compositional generation and guidance.
ConceptFusion: Open-set Multimodal 3D Mapping
ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio.
3D Neural Embedding Likelihood: Probabilistic Inverse Graphics for Robust 6D Pose Estimation
In this paper, we introduce probabilistic modeling to the inverse graphics framework to quantify uncertainty and achieve robustness in 6D pose estimation tasks.
Ranked #1 on
on YCB-Video
Dissociating language and thought in large language models
Large Language Models (LLMs) have come closest among all models to date to mastering human language, yet opinions about their linguistic and cognitive capabilities remain split.
NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants
Experiments show that our helper agent robustly updates its goal inference and adapts its helping plans to the changing level of uncertainty.
3D Shape Perception Integrates Intuitive Physics and Analysis-by-Synthesis
Many surface cues support three-dimensional shape perception, but people can sometimes still see shape when these features are missing -- in extreme cases, even when an object is completely occluded, as when covered with a draped cloth.
Are Deep Neural Networks SMARTer than Second Graders?
To answer this question, we propose SMART: a Simple Multimodal Algorithmic Reasoning Task and the associated SMART-101 dataset, for evaluating the abstraction, deduction, and generalization abilities of neural networks in solving visuo-linguistic puzzles designed specifically for children in the 6--8 age group.
Top-Down Synthesis for Library Learning
This paper introduces corpus-guided top-down synthesis as a mechanism for synthesizing library functions that capture common functionality from a corpus of programs in a domain specific language (DSL).
On the Complexity of Bayesian Generalization
Specifically, at the $representational \ level$, we seek to answer how the complexity varies when a visual concept is mapped to the representation space.
H-SAUR: Hypothesize, Simulate, Act, Update, and Repeat for Understanding Object Articulations from Interactions
The world is filled with articulated objects that are difficult to determine how to use from vision alone, e. g., a door might open inwards or outwards.
Composing Ensembles of Pre-trained Models via Iterative Consensus
Such closed-loop communication enables models to correct errors caused by other models, significantly boosting performance on downstream tasks, e. g. improving accuracy on grade school math problems by 7. 5%, without requiring any model finetuning.
Ranked #1 on
Video Question Answering
on ActivityNet-QA
Revisiting the Roles of "Text" in Text Games
Text games present opportunities for natural language understanding (NLU) methods to tackle reinforcement learning (RL) challenges.
Learning Physical Dynamics with Subequivariant Graph Neural Networks
Graph Neural Networks (GNNs) have become a prevailing tool for learning physical dynamics.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
Abstract Interpretation for Generalized Heuristic Search in Model-Based Planning
Domain-general model-based planners often derive their generality by constructing search heuristics through the relaxation or abstraction of symbolic world models.
Solving the Baby Intuitions Benchmark with a Hierarchically Bayesian Theory of Mind
To facilitate the development of new models to bridge the gap between machine and human social intelligence, the recently proposed Baby Intuitions Benchmark (arXiv:2102. 11938) provides a suite of tasks designed to evaluate commonsense reasoning about agents' goals and actions that even young infants exhibit.
Robust Change Detection Based on Neural Descriptor Fields
The ability to reason about changes in the environment is crucial for robots operating over extended periods of time.
Neural Groundplans: Persistent Neural Scene Representations from a Single Image
We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene.
3D Concept Grounding on Neural Fields
Experimental results show that our proposed framework outperforms unsupervised/language-mediated segmentation models on semantic and instance segmentation tasks, as well as outperforms existing models on the challenging 3D aware visual reasoning tasks.
Finding Fallen Objects Via Asynchronous Audio-Visual Integration
The way an object looks and sounds provide complementary reflections of its physical properties.
Learning Iterative Reasoning through Energy Minimization
Finally, we illustrate that our approach can recursively solve algorithmic problems requiring nested reasoning
Prompting Decision Transformer for Few-Shot Policy Generalization
Humans can leverage prior experience and learn novel tasks from a handful of demonstrations.
Learning Neuro-Symbolic Skills for Bilevel Planning
Decision-making is challenging in robotics environments with continuous object-centric states, continuous actions, long horizons, and sparse feedback.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Compositional Visual Generation with Composable Diffusion Models
Large text-guided diffusion models, such as DALLE-2, are able to generate stunning photorealistic images given natural language descriptions.
Drawing out of Distribution with Neuro-Symbolic Generative Models
We then adopt a subset of the Omniglot challenge tasks, and evaluate its ability to generate new exemplars (both unconditionally and conditionally), and perform one-shot classification, showing that DooD matches the state of the art.
Planning with Diffusion for Flexible Behavior Synthesis
Model-based reinforcement learning methods often use learning only for the purpose of estimating an approximate dynamics model, offloading the rest of the decision-making work to classical trajectory optimizers.
Unsupervised Segmentation in Real-World Images via Spelke Object Inference
Self-supervised, category-agnostic segmentation of real-world images is a challenging open problem in computer vision.
RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation
To train this predictor, we formulate a new loss on rendering variances using gradients from differentiable rendering.
Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks
We first contribute a new challenge benchmark for comparing humans and distributional large language models (LLMs).
Identifying concept libraries from language about object structure
Our understanding of the visual world goes beyond naming objects, encompassing our ability to parse objects into meaningful parts, attributes, and relations.
Unsupervised Discovery and Composition of Object Light Fields
Neural scene representations, both continuous and discrete, have recently emerged as a powerful new paradigm for 3D scene understanding.
Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction
Specifically, FixNet consists of a perception module to extract the structured representation from the 3D point cloud, a physical dynamics prediction module to simulate the results of interactions on 3D objects, and a functionality prediction module to evaluate the functionality and choose the correct fix.
Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics
Extensive experimental results suggest that: 1) on multi-stage tasks that are infeasible for the vanilla differentiable physics solver, our approach discovers contact points that efficiently guide the solver to completion; 2) on tasks where the vanilla solver performs sub-optimally or near-optimally, our contact point discovery method performs better than or on par with the manipulation performance obtained with handcrafted contact points.
ComPhy: Compositional Physical Reasoning of Objects and Events from Videos
In this paper, we take an initial step to highlight the importance of inferring the hidden physical properties not directly observable from visual appearances, by introducing the Compositional Physical Reasoning (ComPhy) dataset.
Learning Neural Acoustic Fields
By modeling acoustic propagation in a scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds.
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools
We consider the problem of sequential robotic manipulation of deformable objects using tools.
FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations
We present a meta-learning framework for learning new visual concepts quickly, from just one or a few examples, guided by multiple naturally occurring data streams: simultaneously looking at images, reading sentences that describe the objects in the scene, and interpreting supplemental sentences that relate the novel concept with other concepts.
Linking Emergent and Natural Languages via Corpus Transfer
In this work, we propose a novel way to establish such a link by corpus transfer, i. e. pretraining on a corpus of emergent language for downstream natural language tasks, which is in contrast to prior work that directly transfers speaker and listener parameters.
Grammar-Based Grounded Lexicon Learning
We present Grammar-Based Grounded Lexicon Learning (G2L2), a lexicalist approach toward learning a compositional and grounded meaning representation of language from grounded data, such as paired images and texts.
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation
Our performance generalizes across both object instances and 6-DoF object poses, and significantly outperforms a recent baseline that relies on 2D descriptors.
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierarchies.
STAR: A Benchmark for Situated Reasoning in Real-World Videos
This paper introduces a new benchmark that evaluates the situated reasoning ability via situation abstraction and logic-grounded question answering for real-world videos, called Situated Reasoning in Real-World Videos (STAR).
Learning to Compose Visual Relations
The visual world around us can be described as a structured set of objects and their associated relations.
Learning Signal-Agnostic Manifolds of Neural Fields
We leverage neural fields to capture the underlying structure in image, shape, audio and cross-modal audiovisual domains in a modality-independent manner.
Unsupervised Learning of Compositional Energy Concepts
In this work, we propose COMET, which discovers and represents concepts as separate energy functions, enabling us to represent both global concepts as well as objects under a unified framework.
MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation
Recent advances in generative adversarial networks (GANs) have led to remarkable achievements in face image synthesis.
3DP3: 3D Scene Perception via Probabilistic Programming
We present 3DP3, a framework for inverse graphics that uses inference in a structured generative model of objects, scenes, and images.
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language
This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine.
OPEn: An Open-ended Physics Environment for Learning Without a Task
We test several existing RL-based exploration methods on this benchmark and find that an agent using unsupervised contrastive learning for representation learning, and impact-driven learning for exploration, achieved the best results.
AutumnSynth: Synthesis of Reactive Programs with Structured Latent State
The human ability to efficiently discover causal theories of their environments from observations is a feat of nature that remains elusive in machines.
On the Expressiveness and Learning of Relational Neural Networks on Hypergraphs
Our first contribution is a fine-grained analysis of the expressiveness of these neural networks, that is, the set of functions that they can realize and the set of problems that they can solve.
Inducing Reusable Skills From Demonstrations with Option-Controller Network
With the isolation of information and the synchronous calling mechanism, we can impose a division of works between the controller and options in an end-to-end training regime.
Learning Rational Skills for Planning from Demonstrations and Instructions
We present a framework for learning compositional, rational skill models (RatSkills) that support efficient planning and inverse planning for achieving novel goals and recognizing activities.
Dynamic Modeling of Hand-Object Interactions via Tactile Sensing
This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing, which opens the door for future applications in activity learning, human-computer interactions, and imitation learning for robotics.
Learning to solve complex tasks by growing knowledge culturally across generations
The video game paradigm we pioneer here is thus a rich test bed for developing AI systems capable of acquiring and transmitting cultural knowledge.
Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
Here we propose a new approach to this challenge based on a particularly strong form of model-based RL which we call Theory-Based Reinforcement Learning, because it uses human-like intuitive theories -- rich, abstract, causal models of physical objects, intentional agents, and their interactions -- to explore and model an environment, and plan effectively to achieve task goals.
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning
Human reasoning can often be understood as an interplay between two systems: the intuitive and associative ("System 1") and the deliberative and logical ("System 2").
Hybrid Memoised Wake-Sleep: Approximate Inference at the Discrete-Continuous Interface
We build on a recent approach, Memoised Wake-Sleep (MWS), which alleviates part of the problem by memoising discrete variables, and extend it to allow for a principled and effective way to handle continuous variables by learning a separate recognition model used for importance-sampling based approximate inference and marginalization.
Modeling the Mistakes of Boundedly Rational Agents Within a Bayesian Theory of Mind
Here, we extend the Bayesian Theory of Mind framework to model boundedly rational agents who may have mistaken goals, plans, and actions.
Leveraging Language to Learn Program Abstractions and Search Heuristics
Inductive program synthesis, or inferring programs from examples of desired behavior, offers a general paradigm for building interpretable, robust, and generalizable machine learning systems.
Physion: Evaluating Physical Prediction from Vision in Humans and Machines
While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments.
Communicating Natural Programs to Humans and Machines
We present LARC, the \textit{Language-complete ARC}: a collection of natural language descriptions by a group of human participants who instruct each other on how to solve ARC tasks using language alone, which contains successful instructions for 88\% of the ARC tasks.
Temporal and Object Quantification Networks
We present Temporal and Object Quantification Networks (TOQ-Nets), a new class of neuro-symbolic networks with a structural bias that enables them to learn to recognize complex relational-temporal events.
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering
In this work, we propose a novel neural scene representation, Light Field Networks or LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, four-dimensional light field parameterized via a neural implicit representation.
Learning Neuro-Symbolic Relational Transition Models for Bilevel Planning
In robotic domains, learning and planning are complicated by continuous state spaces, continuous action spaces, and long task horizons.
PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics
Experimental results suggest that 1) RL-based approaches struggle to solve most of the tasks efficiently; 2) gradient-based approaches, by optimizing open-loop control sequences with the built-in differentiable physics engine, can rapidly find a solution within tens of iterations, but still fall short on multi-stage tasks that require long-term planning.
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos.
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI
To complete the task, an embodied agent must plan a sequence of actions to change the state of a large number of objects in the face of realistic physical constraints.
Learning Task Decomposition with Ordered Memory Policy Network
The discovered subtask hierarchy could be used to perform task decomposition, recovering the subtask boundaries in an unstruc-tured demonstration.
PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception
The ability to perceive and reason about social interactions in the context of physical environments is core to human social intelligence and human-machine cooperation.
AGENT: A Benchmark for Core Psychological Reasoning
For machine agents to successfully interact with humans in real-world settings, they will need to develop an understanding of human mental life.
Ranked #1 on
Core Psychological Reasoning
on AGENT
A Bayesian-Symbolic Approach to Learning and Reasoning for Intuitive Physics
As such, learning the laws is then reduced to symbolic regression and Bayesian inference methods are used to obtain the distribution of unobserved properties.
Learning Task Decomposition with Order-Memory Policy Network
Many complex real-world tasks are composed of several levels of sub-tasks.
Grounding Physical Object and Event Concepts Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos.
Unsupervised Discovery of 3D Physical Objects
We study the problem of unsupervised physical object discovery.
Temporal and Object Quantification Nets
We aim to learn generalizable representations for complex activities by quantifying over both entities and time, as in “the kicker is behind all the other players,” or “the player controls the ball until it moves toward the goal.” Such a structural inductive bias of object relations, object quantification, and temporal orders will enable the learned representation to generalize to situations with varying numbers of agents, objects, and time courses.
Augmenting Policy Learning with Routines Discovered from a Single Demonstration
Humans can abstract prior knowledge from very little data and use it to boost skill learning.
Representing Partial Programs with Blended Abstract Semantics
In this search process, a key challenge is representing the behavior of a partially written program before it can be executed, to judge if it is on the right track and predict where to search next.
Object-Centric Diagnosis of Visual Reasoning
In contrast, symbolic and modular models have a relatively better grounding and robustness, though at the cost of accuracy.
Neural Radiance Flow for 4D View Synthesis and Video Processing
We present a method, Neural Radiance Flow (NeRFlow), to learn a 4D spatial-temporal representation of a dynamic scene from a set of RGB images.
Multi-Plane Program Induction with 3D Box Priors
We consider two important aspects in understanding and editing images: modeling regular, program-like texture or patterns in 2D planes, and 3D posing of these planes in the scene.
Data-Efficient Learning for Complex and Real-Time Physical Problem Solving using Augmented Simulation
The physics engine augmented with the residual model is then used to control the marble in the maze environment using a model-predictive feedback over a receding horizon.
Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling
Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the (aggregate) posterior to encourage statistical independence of the latent factors.
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents.
Causal Inductive Synthesis Corpus
We introduce the Causal Inductive Synthesis Corpus (CISC) -- a manually constructed collection of interactive domains.
Measuring few-shot extrapolation with program induction
Program induction lies at the opposite end of the spectrum: programs are capable of extrapolating from very few examples, but we still do not know how to efficiently search for complex programs.
Learning Online Data Association
When an agent interacts with a complex environment, it receives a stream of percepts in which it may detect entities, such as objects or people.
Learning abstract structure for drawing by efficient motor program induction
Humans flexibly solve new problems that differ qualitatively from those they were trained on.
Noisy Agents: Self-supervised Exploration by Predicting Auditory Events
Humans integrate multiple sensory modalities (e. g. visual and audio) to build a causal understanding of the physical world.
End-to-End Optimization of Scene Layout
Experiments suggest that our model achieves higher accuracy and diversity in conditional scene synthesis and allows exemplar-based scene generation from various input forms.
Foley Music: Learning to Generate Music from Videos
In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments.
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation.
Learning to learn generative programs with Memoised Wake-Sleep
We study a class of neuro-symbolic generative models in which neural networks are used both for inference and as priors over symbolic, data-generating programs.
Perspective Plane Program Induction from a Single Image
We study the inverse graphics problem of inferring a holistic representation for natural images.
Learning Physical Graph Representations from Visual Scenes
To overcome these limitations, we introduce the idea of Physical Scene Graphs (PSGs), which represent scenes as hierarchical graphs, with nodes in the hierarchy corresponding intuitively to object parts at different scales, and edges to physical connections between parts.
Learning with AMIGo: Adversarially Motivated Intrinsic Goals
A key challenge for reinforcement learning (RL) consists of learning in environments with sparse extrinsic rewards.
DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning
It builds expertise by creating programming languages for expressing domain concepts, together with neural networks to guide the search for programs within these languages.
Online Bayesian Goal Inference for Boundedly-Rational Planning Agents
These models are specified as probabilistic programs, allowing us to represent and perform efficient Bayesian inference over an agent's goals and internal planning processes.
Deep Audio Priors Emerge From Harmonic Convolutional Networks
We show that networks using Harmonic Convolution can reliably model audio priors and achieve high performance in unsupervised audio restoration tasks.
Visual Grounding of Learned Physical Models
The abilities to perform physical reasoning and to adapt to new environments, while intrinsic to humans, remain challenging to state-of-the-art computational models.
Music Gesture for Visual Sound Separation
Recent deep learning approaches have achieved impressive performance on visual sound separation tasks.
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility inference, and social learning.
Too many cooks: Bayesian inference for coordinating multi-agent collaboration
Underlying the human ability to collaborate is theory-of-mind, the ability to infer the hidden mental states that drive others to act.
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
The focus of recent meta-learning research has been on the development of learning algorithms that can quickly adapt to test time tasks with limited data and low computational cost.
Learning Compositional Rules via Neural Program Synthesis
Many aspects of human reasoning, including language, require learning rules from very little data.
Visual Concept-Metaconcept Learning
Humans reason with concepts and metaconcepts: we recognize red and green from visual input; we also understand that they describe the same property of objects (i. e., the color).
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation
In this paper, we attempt to approach the problem of Audio-Visual Embodied Navigation, the task of planning the shortest path from a random starting location in a scene to the sound source in an indoor environment, given only raw egocentric visual and audio sensory data.
Accurate Vision-based Manipulation through Contact Reasoning
Planning contact interactions is one of the core challenges of many robotic tasks.
Entity Abstraction in Visual Model-Based Reinforcement Learning
This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before.
CLEVRER: CoLlision Events for Video REpresentation and Reasoning
While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations.
DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs
A major difficulty of solving continuous POMDPs is to infer the multi-modal distribution of the unobserved true states and to make the planning algorithm dependent on the perceived uncertainty.
Program-Guided Image Manipulators
Humans are capable of building holistic representations for images at various levels, from local objects, to pairwise relations, to global structures.
Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
But human beings remain distinctive in their capacity for flexible, creative tool use -- using objects in new ways to act on the world, achieve a goal, or solve a problem.
Neurally-Guided Structure Inference
In this paper, we propose a hybrid inference algorithm, the Neurally-Guided Structure Inference (NG-SI), keeping the advantages of both search-based and data-driven methods.
Finding Friend and Foe in Multi-Agent Games
Here we develop the DeepRole algorithm, a multi-agent reinforcement learning agent that we test on The Resistance: Avalon, the most popular hidden role game.
Predicting the Present and Future States of Multi-agent Systems from Partially-observed Visual Data
We present a method which learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents.
Learning to Describe Scenes with Programs
We are able to understand the higher-level, abstract regularities within the scene such as symmetry and repetition.
Modeling Parts, Structure, and System Dynamics via Predictive Learning
Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future.
Reasoning About Physical Interactions with Object-Centric Models
Object-based factorizations provide a useful level of abstraction for interacting with the world.
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation.
Ranked #5 on
Visual Question Answering (VQA)
on CLEVR
Combining Physical Simulators and Object-Based Networks for Control
Physics engines play an important role in robot planning and control; however, many real-world control problems involve complex contact dynamics that cannot be characterized analytically.
Visualizing and Understanding GANs
We present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level.
Unsupervised Discovery of Parts, Structure, and Dynamics
Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future.
Stochastic Prediction of Multi-Agent Interactions from Partial Observations
We present a method that learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents.
Infinite Mixture Prototypes for Few-Shot Learning
We propose infinite mixture prototypes to adaptively represent both simple and complex data distributions for few-shot learning.
The Omniglot challenge: a 3-year progress report
Three years ago, we released the Omniglot dataset for one-shot learning, along with five challenge tasks and a computational model that addresses these tasks.
On the Units of GANs (Extended Abstract)
We quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.
Theory of Minds: Understanding Behavior in Groups Through Inverse Planning
This representation is grounded in the formalism of stochastic games and multi-agent reinforcement learning.
Learning to Infer and Execute 3D Shape Programs
Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higher-level shape structure such as the repetition and reflective symmetry of object parts.
Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
Object-based factorizations provide a useful level of abstraction for interacting with the world.
Learning to Reconstruct Shapes from Unseen Classes
From a single image, humans are able to perceive the full 3D shape of an object by exploiting learned shape priors from everyday life.
Visual Object Networks: Image Generation with Disentangled 3D Representation
Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes.
Library Learning for Neurally-Guided Bayesian Program Induction
Successful approaches to program induction require a hand-engineered domain-specific language (DSL), constraining the space of allowed programs and imparting prior knowledge of the domain.
GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output.
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
Second, the model is more data- and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering.
Ranked #1 on
Visual Question Answering (VQA)
on CLEVR
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
In this paper, we propose to learn a particle-based simulator for complex control tasks.
ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Robotics
The underlying physical laws of deformable objects are more complex, and the resulting systems have orders of magnitude more degrees of freedom and therefore they are significantly more computationally expensive to simulate.
Propagation Networks for Model-Based Control Under Partial Observation
There has been an increasing interest in learning dynamics simulators for model-based control.
Physical Primitive Decomposition
As annotated data for object parts and physics are rare, we propose a novel formulation that learns physical primitives by explaining both an object's appearance and its behaviors in physical events.
Learning Shape Priors for Single-View 3D Completion and Reconstruction
The problem of single-view 3D shape completion or reconstruction is challenging, because among the many possible shapes that explain an observation, most are implausible and do not correspond to natural objects.
Seeing Tree Structure from Vibration
Humans recognize object structure from both their appearance and motion; often, motion helps to resolve ambiguities in object structure that arise when we observe object appearance only.
Modeling human intuitions about liquid flow with particle-based simulation
Humans can easily describe, imagine, and, crucially, predict a wide variety of behaviors of liquids--splashing, squirting, gushing, sloshing, soaking, dripping, draining, trickling, pooling, and pouring--despite tremendous variability in their material and dynamical properties.
3D-Aware Scene Manipulation via Inverse Graphics
In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model.
3D Shape Perception from Monocular Vision, Touch, and Shape Priors
Perceiving accurate 3D object shape is important for robots to interact with the physical world.
Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncing
An efficient, generalizable physical simulator with universal uncertainty estimates has wide applications in robot state estimation, planning, and control.
The Variational Homoencoder: Learning to learn high capacity generative models from few examples
However, when this generative model is expressed as a powerful neural network such as a PixelCNN, we show that existing learning techniques typically fail to effectively use latent variables.
Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networks
We propose a framework for the completely unsupervised learning of latent object properties from their interactions: the perception-prediction network (PPN).
Flexible Neural Representation for Physics Prediction
Humans have a remarkable capacity to understand the physical dynamics of objects in their environment, flexibly capturing complex structures and interactions at multiple levels of detail.
Relational inductive bias for physical construction in humans and machines
While current deep learning systems excel at tasks such as object classification, language processing, and gameplay, few can construct or modify a complex system such as a tower of blocks.
Word learning and the acquisition of syntactic--semantic overhypotheses
Children learning their first language face multiple problems of induction: how to learn the meanings of words, and how to build meaningful phrases from those words according to syntactic rules.
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
We study 3D shape modeling from a single image and make contributions to it in three aspects.
Ranked #1 on
3D Shape Classification
on Pix3D
Discovery and usage of joint attention in images
Second, we use a human study to demonstrate the sensitivity of humans to joint attention, suggesting that the detection of such a configuration in an image can be useful for understanding the image, including the goals of the agents and their joint activity, and therefore can contribute to image captioning and related tasks.
3D Interpreter Networks for Viewer-Centered Wireframe Modeling
3D-INN is trained on real images to estimate 2D keypoint heatmaps from an input image; it then predicts 3D object structure from heatmaps using knowledge learned from synthetic 3D shapes.
The Three Pillars of Machine Programming
In this position paper, we describe our vision of the future of machine programming through a categorical examination of three pillars of research.
