Search Results for author:
Found 57 papers, 13 papers with code
Explicit Recall for Efficient Exploration
In this paper, we advocate the use of explicit memory for efficient exploration in reinforcement learning.
One-Shot Manipulation Strategy Learning by Making Contact Analogies
We present a novel approach, MAGIC (manipulation analogies for generalizable intelligent contacts), for one-shot learning of manipulation strategies with fast and extensive generalization to novel objects.
Keypoint Abstraction using Large Models for Object-Relative Imitation Learning
In this paper, we propose KALM, a framework that leverages large pre-trained vision-language models (LMs) to automatically generate task-relevant and cross-instance consistent keypoints.
Learning Linear Attention in Polynomial Time
Our findings bridge a critical gap between theoretical expressivity and learnability of Transformers, and show that flexible and general models of computation are efficiently learnable.
Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
We aim to evaluate Large Language Models (LLMs) for embodied decision making.
Infer Human's Intentions Before Following Natural Language Instructions
We propose a new framework, Follow Instructions with Social and Embodied Reasoning (FISER), aiming for better natural language instruction following in collaborative embodied tasks.
What Makes a Maze Look Like a Maze?
A unique aspect of human visual understanding is the ability to flexibly interpret abstract concepts: acquiring lifted rules explaining what they symbolize, grounding them across familiar and unfamiliar contexts, and making predictions or reasoning about them.
Agent Workflow Memory
Despite the potential of language model-based agents to solve real-world tasks such as web navigation, current methods still struggle with long-horizon tasks with complex action trajectories.
Learning Iterative Reasoning through Energy Diffusion
We introduce iterative reasoning through energy diffusion (IRED), a novel framework for learning to reason for a variety of tasks by formulating reasoning and decision-making problems with energy-based optimization.
"Set It Up!": Functional Object Arrangement with Compositional Generative Models
This paper studies the challenge of developing robots capable of understanding under-specified instructions for creating functional object arrangements, such as "set up a dining table for two"; previous arrangement approaches have focused on much more explicit instructions, such as "put object A on the table."
Finding structure in logographic writing with library learning
Built on top of state-of-the-art library learning and program synthesis techniques, our computational framework discovers known linguistic structures in the Chinese writing system and reveals how the system evolves towards simplification under pressures for representational efficiency.
Composable Part-Based Manipulation
In this paper, we propose composable part-based manipulation (CPM), a novel approach that leverages object-part decomposition and part-part correspondences to improve learning and generalization of robotic manipulation skills.
Learning Planning Abstractions from Language
This paper presents a framework for learning state and action abstractions in sequential decision-making domains.
Grounding Language Plans in Demonstrations Through Counterfactual Perturbations
Grounding the common-sense reasoning of Large Language Models (LLMs) in physical domains remains a pivotal yet unsolved problem for embodied AI.
Learning adaptive planning representations with natural language guidance
Effective planning in the real world requires not only world knowledge, but the ability to leverage that knowledge to build the right representation of the task at hand.
What Planning Problems Can A Relational Neural Network Solve?
Goal-conditioned policies are generally understood to be "feed-forward" circuits, in the form of neural networks that map from the current state and the goal specification to the next action to take.
Learning Reusable Manipulation Strategies
Humans demonstrate an impressive ability to acquire and generalize manipulation "tricks."
What's Left? Concept Grounding with Logic-Enhanced Foundation Models
We propose the Logic-Enhanced Foundation Model (LEFT), a unified framework that learns to ground and reason with concepts across domains with a differentiable, domain-independent, first-order logic-based program executor.
Visual Question Answering (VQA) Split A
Visual Question Answering (VQA) Split B
+1
Learning to Act from Actionless Videos through Dense Correspondences
In this work, we present an approach to construct a video-based robot policy capable of reliably executing diverse tasks across different robots and environments from few video demonstrations without using any action annotations.
HandMeThat: Human-Robot Communication in Physical and Social Environments
We introduce HandMeThat, a benchmark for a holistic evaluation of instruction understanding and following in physical and social environments.
CLEVRER-Humans: Describing Physical and Causal Events the Human Way
First, there is a lack of diversity in both event types and natural language descriptions; second, causal relationships based on manually-defined heuristics are different from human judgments.
Compositional Diffusion-Based Continuous Constraint Solvers
This paper introduces an approach for learning to solve continuous constraint satisfaction problems (CCSP) in robotic reasoning and planning.
Programmatically Grounded, Compositionally Generalizable Robotic Manipulation
Robots operating in the real world require both rich manipulation skills as well as the ability to semantically reason about when to apply those skills.
NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations
Different functional modules in the programs are implemented as neural networks.
Sparse and Local Networks for Hypergraph Reasoning
Reasoning about the relationships between entities from input facts (e. g., whether Ari is a grandparent of Charlie) generally requires explicit consideration of other entities that are not mentioned in the query (e. g., the parents of Charlie).
PDSketch: Integrated Planning Domain Programming and Learning
This paper studies a model learning and online planning approach towards building flexible and general robots.
On the Expressiveness and Generalization of Hypergraph Neural Networks
Next, we analyze the learning properties of these neural networks, especially focusing on how they can be trained on a finite set of small graphs and generalize to larger graphs, which we term structural generalization.
Learning Rational Subgoals from Demonstrations and Instructions
We present a framework for learning useful subgoals that support efficient long-term planning to achieve novel goals.
IKEA-Manual: Seeing Shape Assembly Step by Step
Human-designed visual manuals are crucial components in shape assembly activities.
Translating a Visual LEGO Manual to a Machine-Executable Plan
We study the problem of translating an image-based, step-by-step assembly manual created by human designers into machine-interpretable instructions.
Programmatic Concept Learning for Human Motion Description and Synthesis
We introduce Programmatic Motion Concepts, a hierarchical motion representation for human actions that captures both low-level motion and high-level description as motion concepts.
FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations
We present a meta-learning framework for learning new visual concepts quickly, from just one or a few examples, guided by multiple naturally occurring data streams: simultaneously looking at images, reading sentences that describe the objects in the scene, and interpreting supplemental sentences that relate the novel concept with other concepts.
Grammar-Based Grounded Lexicon Learning
We present Grammar-Based Grounded Lexicon Learning (G2L2), a lexicalist approach toward learning a compositional and grounded meaning representation of language from grounded data, such as paired images and texts.
Efficient Training and Inference of Hypergraph Reasoning Networks
To leverage the sparsity in hypergraph neural networks, SpaLoc represents the grounding of relationships such as parent and grandparent as sparse tensors and uses neural networks and finite-domain quantification operations to infer new facts based on the input.
On the Expressiveness and Learning of Relational Neural Networks on Hypergraphs
Our first contribution is a fine-grained analysis of the expressiveness of these neural networks, that is, the set of functions that they can realize and the set of problems that they can solve.
Learning Rational Skills for Planning from Demonstrations and Instructions
We present a framework for learning compositional, rational skill models (RatSkills) that support efficient planning and inverse planning for achieving novel goals and recognizing activities.
Temporal and Object Quantification Networks
We present Temporal and Object Quantification Networks (TOQ-Nets), a new class of neuro-symbolic networks with a structural bias that enables them to learn to recognize complex relational-temporal events.
Hierarchical Motion Understanding via Motion Programs
We posit that adding higher-level motion primitives, which can capture natural coarser units of motion such as backswing or follow-through, can be used to improve downstream analysis tasks.
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos.
Temporal and Object Quantification Nets
We aim to learn generalizable representations for complex activities by quantifying over both entities and time, as in “the kicker is behind all the other players,” or “the player controls the ball until it moves toward the goal.” Such a structural inductive bias of object relations, object quantification, and temporal orders will enable the learned representation to generalize to situations with varying numbers of agents, objects, and time courses.
Grounding Physical Object and Event Concepts Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos.
Language-Mediated, Object-Centric Representation Learning
These object-centric concepts derived from language facilitate the learning of object-centric representations.
Object-Centric Diagnosis of Visual Reasoning
In contrast, symbolic and modular models have a relatively better grounding and robustness, though at the cost of accuracy.
Multi-Plane Program Induction with 3D Box Priors
We consider two important aspects in understanding and editing images: modeling regular, program-like texture or patterns in 2D planes, and 3D posing of these planes in the scene.
Perspective Plane Program Induction from a Single Image
We study the inverse graphics problem of inferring a holistic representation for natural images.
Visual Concept-Metaconcept Learning
Humans reason with concepts and metaconcepts: we recognize red and green from visual input; we also understand that they describe the same property of objects (i. e., the color).
Program-Guided Image Manipulators
Humans are capable of building holistic representations for images at various levels, from local objects, to pairwise relations, to global structures.
Neurally-Guided Structure Inference
In this paper, we propose a hybrid inference algorithm, the Neurally-Guided Structure Inference (NG-SI), keeping the advantages of both search-based and data-driven methods.
Visually Grounded Neural Syntax Acquisition
We define concreteness of constituents by their matching scores with images, and use it to guide the parsing of text.
Neural Logic Machines
We propose the Neural Logic Machine (NLM), a neural-symbolic architecture for both inductive learning and logic reasoning.
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation.
Ranked #6 on
Visual Question Answering (VQA)
on CLEVR
UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations
We propose Unified Visual-Semantic Embeddings (UniVSE) for learning a joint space of visual and textual concepts.
Neural Phrase-to-Phrase Machine Translation
We also design an efficient dynamic programming algorithm to decode segments that allows the model to be trained faster than the existing neural phrase-based machine translation method by Huang et al. (2018).
Acquisition of Localization Confidence for Accurate Object Detection
The network acquires this confidence of localization, which improves the NMS procedure by preserving accurately localized bounding boxes.
Ranked #185 on
Object Detection
on COCO test-dev
Learning Visually-Grounded Semantics from Contrastive Adversarial Samples
Begin with an insightful adversarial attack on VSE embeddings, we show the limitation of current frameworks and image-text datasets (e. g., MS-COCO) both quantitatively and qualitatively.
Universal Agent for Disentangling Environments and Tasks
Recent state-of-the-art reinforcement learning algorithms are trained under the goal of excelling in one specific task.
Hierarchical Reinforcement Learning
reinforcement-learning
+2
What Can Help Pedestrian Detection?
Aggregating extra features has been considered as an effective approach to boost traditional pedestrian detection methods.
Ranked #15 on
Pedestrian Detection
on Caltech
