Search Results for author:
Found 17 papers, 5 papers with code
3D-VLA: A 3D Vision-Language-Action Generative World Model
Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world.
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World
Human beings possess the capability to multiply a melange of multisensory cues while actively exploring and interacting with the 3D world.
GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs
If not, we initialize a new module needed by the task and specify the inputs and outputs of this new module.
CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
A communication token is generated by the LLM following a visual entity or a relation, to inform the detection network to propose regions that are relevant to the sentence generated so far.
3D-LLM: Injecting the 3D World into Large Language Models
Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs.
3D Concept Learning and Reasoning from Multi-View Images
We suggest that a principled approach for 3D reasoning from multi-view images should be to infer a compact 3D representation of the world from the multi-view images, which is further grounded on open-vocabulary semantic concepts, and then to execute reasoning on these 3D representations.
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning
The see stage scans the image and grounds the visual concept candidates with a visual perception model.
3D Concept Grounding on Neural Fields
Experimental results show that our proposed framework outperforms unsupervised/language-mediated segmentation models on semantic and instance segmentation tasks, as well as outperforms existing models on the challenging 3D aware visual reasoning tasks.
Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction
Specifically, FixNet consists of a perception module to extract the structured representation from the 3D point cloud, a physical dynamics prediction module to simulate the results of interactions on 3D objects, and a functionality prediction module to evaluate the functionality and choose the correct fix.
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierarchies.
Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks
Previous math word problem solvers following the encoder-decoder paradigm fail to explicitly incorporate essential math symbolic constraints, leading to unexplainable and unreasonable predictions.
VLGrammar: Grounded Grammar Induction of Vision and Language
In this work, we study grounded grammar induction of vision and language in a joint learning framework.
A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
We believe the HINT dataset and the experimental findings are of great interest to the learning community on systematic generalization.
SMART: A Situation Model for Algebra Story Problems via Attributed Grammar
Solving algebra story problems remains a challenging task in artificial intelligence, which requires a detailed understanding of real-world situations and a strong mathematical reasoning capability.
Learning by Fixing: Solving Math Word Problems with Weak Supervision
To generate more diverse solutions, \textit{tree regularization} is applied to guide the efficient shrinkage and exploration of the solution space, and a \textit{memory buffer} is designed to track and save the discovered various fixes for each problem.
Ranked #1 on
Math Word Problem Solving
on Math23K
(weakly-supervised metric)
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
Humans can progressively learn visual concepts from easy to hard questions.
Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning
In this paper, we address these issues and close the loop of neural-symbolic learning by (1) introducing the \textbf{grammar} model as a \textit{symbolic prior} to bridge neural perception and symbolic reasoning, and (2) proposing a novel \textbf{back-search} algorithm which mimics the top-down human-like learning procedure to propagate the error through the symbolic reasoning module efficiently.
