Search Results for author:
Found 65 papers, 28 papers with code
PreAfford: Universal Affordance-Based Pre-Grasping for Diverse Objects and Environments
Robotic manipulation of ungraspable objects with two-finger grippers presents significant challenges due to the paucity of graspable features, while traditional pre-grasping techniques, which rely on repositioning objects and leveraging external aids like table edges, lack the adaptability across object categories and scenes.
Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance
Despite significant advancements in text-to-motion synthesis, generating language-guided human motion within 3D environments poses substantial challenges.
AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents
Traditional approaches in physics-based motion generation, centered around imitation learning and reward shaping, often struggle to adapt to new scenarios.
Zero-Shot Image Feature Consensus with Deep Functional Maps
We demonstrate that our technique yields correspondences that are not only smoother but also more accurate, with the possibility of better reflecting the knowledge embedded in the large-scale vision models that we are studying.
Scaling Up Dynamic Human-Scene Interaction Modeling
Confronting the challenges of data scarcity and advanced motion synthesis in human-scene interaction modeling, we introduce the TRUMANS dataset alongside a novel HSI motion synthesis method.
I-PHYRE: Interactive Physical Reasoning
Current evaluation protocols predominantly assess physical reasoning in stationary scenes, creating a gap in evaluating agents' abilities to interact with dynamic events.
Single-view 3D Scene Reconstruction with High-fidelity Shape and Texture
Reconstructing detailed 3D scenes from single-view images remains a challenging task due to limitations in existing approaches, which primarily focus on geometric shape recovery, overlooking object appearances and fine shape details.
SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation
Central to SparseDFF is a feature refinement network, optimized with a contrastive loss between views and a point-pruning mechanism for feature continuity.
ChimpACT: A Longitudinal Dataset for Understanding Chimpanzee Behaviors
ChimpACT is both comprehensive and challenging, consisting of 163 videos with a cumulative 160, 500 frames, each richly annotated with detection, identification, pose estimation, and fine-grained spatiotemporal behavior labels.
Grasp Multiple Objects with One Hand
The intricate kinematics of the human hand enable simultaneous grasping and manipulation of multiple objects, essential for tasks such as object transfer and in-hand manipulation.
X-VoE: Measuring eXplanatory Violation of Expectation in Physical Events
Intuitive physics is pivotal for human understanding of the physical world, enabling prediction and interpretation of events even in infancy.
Making the Most Out of the Limited Context Length: Predictive Power Varies with Clinical Note Type and Note Section
Recent advances in large language models have led to renewed interest in natural language processing in healthcare using the free text of clinical notes.
Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs).
MEWL: Few-shot multimodal word learning with referential uncertainty
To fill in this gap, we introduce the MachinE Word Learning (MEWL) benchmark to assess how machines learn word meaning in grounded visual scenes.
STRAP: Structured Object Affordance Segmentation with Point Supervision
By label affinity, we refer to affordance segmentation as a multi-label prediction problem: A plate can be both holdable and containable.
Rearrange Indoor Scenes for Human-Robot Co-Activity
We present an optimization-based framework for rearranging indoor furniture to accommodate human-robot co-activities better.
Diffusion-based Generation, Optimization, and Planning in 3D Scenes
SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization, and planning.
A Reconfigurable Data Glove for Reconstructing Physical and Virtual Grasps
In this work, we present a reconfigurable data glove design to capture different modes of human hand-object interactions, which are critical in training embodied artificial intelligence (AI) agents for fine manipulation tasks.
Full-Body Articulated Human-Object Interaction
By learning the geometrical relationships in HOI, we devise the very first model that leverage human pose estimation to tackle the estimation of articulated object poses and shapes during whole-body interactions.
To think inside the box, or to think out of the box? Scientific discovery via the reciprocation of insights and concepts
If scientific discovery is one of the main driving forces of human progress, insight is the fuel for the engine, which has long attracted behavior-level research to understand and model its underlying cognitive process.
On the Complexity of Bayesian Generalization
Specifically, at the $representational \ level$, we seek to answer how the complexity varies when a visual concept is mapped to the representation space.
HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes
Learning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characteristics of the existing datasets on Human-Scene Interaction (HSI); they only have limited scale/quality and lack semantics.
Understanding Embodied Reference with Touch-Line Transformer
Human studies have revealed that objects referred to or pointed to do not lie on the elbow-wrist line, a common misconception; instead, they lie on the so-called virtual touch line.
On the Learning Mechanisms in Physical Reasoning
Taken together, the results on the challenging benchmark of PHYRE show that LfI is, if not better, as good as LfD for dynamics prediction.
Neural-Symbolic Recursive Machine for Systematic Generalization
In experiments, NSR achieves state-of-the-art performance in three benchmarks from different domains: SCAN for semantic parsing, PCFG for string manipulation, and HINT for arithmetic reasoning.
GenDexGrasp: Generalizable Dexterous Grasping
By leveraging the contact map as a hand-agnostic intermediate representation, GenDexGrasp efficiently generates diverse and plausible grasping poses with a high success rate and can transfer among diverse multi-fingered robotic hands.
Sequential Manipulation Planning on Scene Graph
We devise a 3D scene graph representation, contact graph+ (cg+), for efficient sequential task planning.
Understanding Physical Effects for Effective Tool-use
We present a robot learning and planning framework that produces an effective tool-use strategy with the least joint efforts, capable of handling objects different from training.
Latent Diffusion Energy-Based Model for Interpretable Text Modeling
Latent space Energy-Based Models (EBMs), also known as energy-based priors, have drawn growing interests in generative modeling.
PartAfford: Part-level Affordance Discovery from 3D Objects
We propose a novel learning framework for PartAfford, which discovers part-level representations by leveraging only the affordance set supervision and geometric primitive regularization, without dense supervision.
Emergent Graphical Conventions in a Visual Communication Game
Humans communicate with graphical sketches apart from symbolic languages.
Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning
Extensive experiments show that by incorporating an algebraic treatment, the ALANS learner outperforms various pure connectionist models in domains requiring systematic generalization.
Unsupervised Foreground Extraction via Deep Region Competition
Foreground extraction can be viewed as a special case of generic image segmentation that focuses on identifying and disentangling objects from the background.
YouRefIt: Embodied Reference Understanding with Language and Gesture
To the best of our knowledge, this is the first embodied reference dataset that allows us to study referring expressions in daily physical scenes to understand referential behavior, human communication, and human-robot interaction.
Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds
To date, various 3D scene understanding tasks still lack practical and generalizable pre-trained models, primarily due to the intricate nature of 3D scene understanding tasks and their immense variations introduced by camera views, lighting, occlusions, etc.
Ranked #4 on
3D Object Detection
on SUN-RGBD
Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene
In a series of experiments, we demonstrate that human gesture cues, even without predefined semantics, improve the object-goal navigation for an embodied agent, outperforming various state-of-the-art methods.
Individual vs. Joint Perception: a Pragmatic Model of Pointing as Communicative Smithian Helping
We add another agent as a guide who can only help by marking an observation already perceived by the hunter with a pointing or not, without providing new observations or offering any instrumental help.
Learning Triadic Belief Dynamics in Nonverbal Communication from Videos
By aggregating different beliefs and true world states, our model essentially forms "five minds" during the interactions between two agents.
Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments
In this paper, we rethink the problem of scene reconstruction from an embodied agent's perspective: While the classic view focuses on the reconstruction accuracy, our new perspective emphasizes the underlying functions and constraints such that the reconstructed scenes provide \em{actionable} information for simulating \em{interactions} with agents.
ACRE: Abstract Causal REasoning Beyond Covariation
Causal induction, i. e., identifying unobservable mechanisms that lead to the observable relations among variables, has played a pivotal role in modern scientific discovery, especially in scenarios with only sparse and limited data.
Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution
To fill in this gap, we propose a neuro-symbolic Probabilistic Abduction and Execution (PrAE) learner; central to the PrAE learner is the process of probabilistic abduction and execution on a probabilistic scene representation, akin to the mental manipulation of objects.
Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance
Predicting agents' future trajectories plays a crucial role in modern AI systems, yet it is challenging due to intricate interactions exhibited in multi-agent systems, especially when it comes to collision avoidance.
A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
We believe the HINT dataset and the experimental findings are of great interest to the learning community on systematic generalization.
HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
Leveraging these concepts, they could understand the internal structure of this task, without seeing all of the problem instances.
Incorporating Vision Bias into Click Models for Image-oriented Search Engine
Most typical click models assume that the probability of a document to be examined by users only depends on position, such as PBM and UBM.
Learning Algebraic Representation for Abstract Spatial-Temporal Reasoning
We further show that the algebraic representation learned can be decoded by isomorphism and used to generate an answer.
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities
Understanding and interpreting human actions is a long-standing challenge and a critical indicator of perception in artificial intelligence.
Congestion-aware Evacuation Routing using Augmented Reality Devices
We present a congestion-aware routing solution for indoor evacuation, which produces real-time individual-customized evacuation routes among multiple destinations while keeping tracks of all evacuees' locations.
Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning
To endow such a crucial cognitive ability to machine intelligence, we propose a dataset, Machine Number Sense (MNS), consisting of visual arithmetic problems automatically generated using a grammar model--And-Or Graph (AOG).
Joint Inference of States, Robot Knowledge, and Human (False-)Beliefs
Aiming to understand how human (false-)belief--a core socio-cognitive ability--would affect human interactions with robots, this paper proposes to adopt a graphical model to unify the representation of object states, robot knowledge, and human (false-)beliefs.
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility inference, and social learning.
Lagrangian-Eulerian Multi-Density Topology Optimization with the Material Point Method
A quadrature-level connectivity graph-based method is adopted to avoid the artificial checkerboard issues commonly existing in multi-resolution topology optimization methods.
Computational Physics Computational Engineering, Finance, and Science Graphics
PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points
Detecting 3D objects from a single RGB image is intrinsically ambiguous, thus requiring appropriate prior knowledge and intermediate representations as constraints to reduce the uncertainties and improve the consistencies between the 2D image plane and the 3D world coordinate.
Ranked #2 on
Monocular 3D Object Detection
on SUN RGB-D
(AP@0.15 (10 / PNet-30) metric)
Learning Perceptual Inference by Contrasting
"Thinking in pictures," [1] i. e., spatial-temporal reasoning, effortless and instantaneous for humans, is believed to be a significant ability to perform logical induction and a crucial factor in the intellectual history of technology development.
Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning
Given these general theories, the goal is to train an agent by interactively exploring the problem space to (i) discover, form, and transfer useful abstract and structural knowledge, and (ii) induce useful knowledge from the instance-level attributes observed in the environment.
Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
We propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction---3D estimations of object bounding boxes, camera pose, and room layout, and (ii) 3D human pose estimation.
3D Human Pose Estimation
Human-Object Interaction Detection
+1
RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
In this work, we propose a new dataset, built in the context of Raven's Progressive Matrices (RPM) and aimed at lifting machine intelligence by associating vision with structural, relational, and analogical reasoning in a hierarchical representation.
MetaStyle: Three-Way Trade-Off Among Speed, Flexibility, and Quality in Neural Style Transfer
An unprecedented booming has been witnessed in the research area of artistic style transfer ever since Gatys et al. introduced the neural method.
Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation
Holistic 3D indoor scene understanding refers to jointly recovering the i) object bounding boxes, ii) room layout, and iii) camera pose, all in 3D.
Ranked #5 on
Monocular 3D Object Detection
on SUN RGB-D
Human-centric Indoor Scene Synthesis Using Stochastic Grammar
We present a human-centric method to sample and synthesize 3D room layouts and 2D images thereof, to obtain large-scale 2D/3D image data with perfect per-pixel ground truth.
Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image
We propose a computational framework to jointly parse a single RGB image and reconstruct a holistic 3D configuration composed by a set of CAD models using a stochastic grammar model.
Ranked #4 on
Monocular 3D Object Detection
on SUN RGB-D
(AP@0.15 (10 / PNet-30) metric)
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammars
We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algorithms.
Inferring Forces and Learning Human Utilities From Videos
We propose a notion of affordance that takes into account physical quantities generated when the human body interacts with real-world objects, and introduce a learning framework that incorporates the concept of human utilities, which in our opinion provides a deeper and finer-grained account not only of object affordance but also of people's interaction with objects.
Understanding Tools: Task-Oriented Object Modeling, Learning and Recognition
In this paper, we present a new framework - task-oriented modeling, learning and recognition which aims at understanding the underlying functions, physics and causality in using objects as "tools".
