Search Results for author:
Found 22 papers, 7 papers with code
SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models
In this paper, we systematically study the impact of 3D-informed data, architecture, and training setups, introducing SpatialLLM, a large multi-modal model with advanced 3D spatial reasoning abilities.
SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning
Despite recent advances on multi-modal models, 3D spatial reasoning remains a challenging task for state-of-the-art open-source and proprietary models.
DINeMo: Learning Neural Mesh Models with no 3D Annotations
In this work, we present DINeMo, a novel neural mesh model that is trained with no 3D annotations by leveraging pseudo-correspondence obtained from large visual foundation models.
Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models
Although large multimodal models (LMMs) have demonstrated remarkable capabilities in visual scene interpretation and reasoning, their capacity for complex and precise 3-dimensional spatial reasoning remains uncertain.
Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Mutimodal Models
Although large multimodal models (LMMs) have demonstrated remarkable capabilities in visual scene interpretation and reasoning, their capacity for complex and precise 3-dimensional spatial reasoning remains uncertain.
3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark
We benchmark a wide range of open-sourced and proprietary LMMs, uncovering their limitations in various aspects of 3D awareness, such as height, orientation, location, and multi-object reasoning, as well as their degraded performance on images with uncommon camera viewpoints.
Rethinking Video-Text Understanding: Retrieval from Counterfactually Augmented Data
In order to narrow the gap between video-text models and human performance on RCAD, we identify a key limitation of current contrastive approaches on video-text data and introduce LLM-teacher, a more effective approach to learn action semantics by leveraging knowledge obtained from a pretrained large language model.
ImageNet3D: Towards General-Purpose Object-Level 3D Understanding
A vision model with general-purpose object-level 3D understanding should be capable of inferring both 2D (e. g., class name and bounding box) and 3D information (e. g., 3D location and 3D viewpoint) for arbitrary rigid objects in natural images.
Image Captioning
Linear Probing Object-Level 3D Awareness
+2
Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering
To demonstrate the importance of an explicit 4D dynamics representation of the scenes in understanding world dynamics, we further propose NS-4Dynamics, a Neural-Symbolic model for reasoning on 4D Dynamics properties under explicit scene representation from videos.
Uncertainty-Aware Deep Video Compression with Ensembles
Deep learning-based video compression is a challenging task, and many previous state-of-the-art learning-based video codecs use optical flows to exploit the temporal correlation between successive frames and then compress the residual error.
3D-Aware Visual Question Answering about Parts, Poses and Occlusions
In this work, we introduce the task of 3D-aware VQA, which focuses on challenging questions that require a compositional reasoning over the 3D structure of visual scenes.
Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shape
Animal3D consists of 3379 images collected from 40 mammal species, high-quality annotations of 26 keypoints, and importantly the pose and shape parameters of the SMAL model.
Ranked #1 on
Animal Pose Estimation
on Animal3D
Generating Images with 3D Annotations Using Diffusion Models
With explicit 3D geometry control, we can easily change the 3D structures of the objects in the generated images and obtain ground-truth 3D annotations automatically.
Neural Textured Deformable Meshes for Robust Analysis-by-Synthesis
Human vision demonstrates higher robustness than current AI algorithms under out-of-distribution scenarios.
Robust Category-Level 3D Pose Estimation from Synthetic Data
In this work, we aim to narrow the performance gap between models trained on synthetic data and few real images and fully supervised models trained on large-scale data.
NOVUM: Neural Object Volumes for Robust Object Classification
Discriminative models for object classification typically learn image-based representations that do not capture the compositional and 3D nature of objects.
OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images
Enhancing the robustness of vision algorithms in real-world scenarios is challenging.
Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning
Visual Question Answering (VQA) models often perform poorly on out-of-distribution data and struggle on domain generalization.
Robust Category-Level 6D Pose Estimation with Coarse-to-Fine Rendering of Neural Features
We consider the problem of category-level 6D pose estimation from a single RGB image.
OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images
One reason is that existing robustness benchmarks are limited, as they either rely on synthetic data or ignore the effects of individual nuisance factors.
Image-driven discriminative and generative machine learning algorithms for establishing microstructure-processing relationships
A binary alloy (uranium-molybdenum) that is currently under development as a nuclear fuel was studied for the purpose of developing an improved machine learning approach to image recognition, characterization, and building predictive capabilities linking microstructure to processing conditions.
An image-driven machine learning approach to kinetic modeling of a discontinuous precipitation reaction
Here, we apply these well-established methods to develop an approach to microstructure quantification for kinetic modeling of a discontinuous precipitation reaction in a case study on the uranium-molybdenum system.
