Search Results for author:
Found 84 papers, 43 papers with code
MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone's Potential with Masked Autoregressive Pretraining
We found that using the correct autoregressive pretraining can significantly boost the performance of the Mamba architecture.
VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation
VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation.
CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement
With 1K human-object-human motion sequences captured in the real world, we enrich CORE4D by contributing an iterative collaboration retargeting strategy to augment motions to a variety of novel objects.
Human-Object Interaction Detection
Human-Object Interaction Generation
+2
FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models
At the same time, foundation models trained with internet-scale image and text data have demonstrated surprising world knowledge and reasoning ability for various downstream tasks.
4DRecons: 4D Neural Implicit Deformable Objects Reconstruction from a single RGB-D Camera with Geometrical and Topological Regularizations
To this end, we introduce a novel approach to compute correspondences between adjacent textured implicit surfaces, which are used to define the ARAP regularization term.
Physics-aware Hand-object Interaction Denoising
The credibility and practicality of a reconstructed hand-object interaction sequence depend largely on its physical plausibility.
QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfer
We explore the dexterous manipulation transfer problem by designing simulators.
GenN2N: Generative NeRF2NeRF Translation
We present GenN2N, a unified NeRF-to-NeRF translation framework for various NeRF translation tasks such as text-driven NeRF editing, colorization, super-resolution, inpainting, etc.
PhysReaction: Physically Plausible Real-Time Humanoid Reaction Synthesis via Forward Dynamics Guided 4D Imitation
Humanoid Reaction Synthesis is pivotal for creating highly interactive and empathetic robots that can seamlessly integrate into human environments, enhancing the way we live, work, and communicate.
ShapeLLM: Universal 3D Object Understanding for Embodied Interaction
This paper presents ShapeLLM, the first 3D Multimodal Large Language Model (LLM) designed for embodied interaction, exploring a universal 3D object understanding with 3D point clouds and languages.
Ranked #1 on
3D Question Answering (3D-QA)
on 3D MM-Vet
GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion
We tackle those challenges through a novel approach, GeneOH Diffusion, incorporating two key designs: an innovative contact-centric HOI representation named GeneOH and a new domain-generalizable denoising scheme.
LEMMA: Towards LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation
In this paper, we first investigate the potential of LVLM on multimodal misinformation detection.
Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective
In this paper, we use well-designed Body Pose Graph (BPG) to represent the human body and translate the challenge into a prediction problem of graph missing nodes.
CrossVideo: Self-supervised Cross-modal Contrastive Learning for Point Cloud Video Understanding
This paper introduces a novel approach named CrossVideo, which aims to enhance self-supervised cross-modal contrastive learning in the field of point cloud video understanding.
TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding
Humans commonly work with multiple objects in daily life and can intuitively transfer manipulation skills to novel objects by understanding object functional regularities.
GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation, Demonstration, and Imitation
We further introduce a distillation-friendly demonstration generation method that automatically generates a million high-quality demonstrations suitable for learning.
GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation Demonstration and Imitation
This paper presents GenH2R a framework for learning generalizable vision-based human-to-robot (H2R) handover skills.
Interactive Humanoid: Online Full-Body Motion Reaction Synthesis with Social Affordance Canonicalization and Forecasting
To support this task, we construct two datasets named HHI and CoChair and propose a unified method.
Semantic Complete Scene Forecasting from a 4D Dynamic Point Cloud Sequence
To tackle this challenging problem, we properly model the synergetic relationship between future forecasting and semantic scene completion through a novel network named SCSFNet.
Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
In this study, we are interested in imbuing robots with the capability of physically-grounded task planning.
NSM4D: Neural Scene Model Based Online 4D Point Cloud Sequence Understanding
We integrate NSM4D with state-of-the-art 4D perception backbones, demonstrating significant improvements on various online perception benchmarks in indoor and outdoor settings.
DreamLLM: Synergistic Multimodal Comprehension and Creation
This paper presents DreamLLM, a learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently overlooked synergy between multimodal comprehension and creation.
Ranked #5 on
Visual Question Answering
on MMBench
TransTouch: Learning Transparent Objects Depth Sensing Through Sparse Touches
By approximating and optimizing the utility function, we can optimize the probing locations given a fixed touching budget to better improve the network's performance on real objects.
3D Implicit Transporter for Temporally Consistent Keypoint Discovery
To address this issue, the Transporter method was introduced for 2D data, which reconstructs the target frame from the source frame to incorporate both spatial and temporal information.
Few-Shot Physically-Aware Articulated Mesh Generation via Hierarchical Deformation
By observing an articulated object dataset containing only a few examples, we wish to learn a model that can generate diverse meshes with high visual fidelity and physical validity.
UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learning
We propose a novel, object-agnostic method for learning a universal policy for dexterous object grasping from realistic point cloud observations and proprioceptive information under a table-top setting, namely UniDexGrasp++.
JacobiNeRF: NeRF Shaping with Mutual Information Gradients
We propose a method that trains a neural radiance field (NeRF) to encode not only the appearance of the scene but also semantic correlations between scene points, regions, or entities -- aiming to capture their mutual co-variation patterns.
Semi-Weakly Supervised Object Kinematic Motion Prediction
First, although 3D dataset with fully annotated motion labels is limited, there are existing datasets and methods for object part semantic segmentation at large scale.
SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer
High-resolution images enable neural networks to learn richer visual representations.
CAMS: CAnonicalized Manipulation Spaces for Category-Level Functional Hand-Object Manipulation Synthesis
In this work, we focus on a novel task of category-level functional hand-object manipulation synthesis covering both rigid and articulated object categories.
Controllable Ancient Chinese Lyrics Generation Based on Phrase Prototype Retrieving
The phrase connector then selects a series of phrases from the phrase pool that minimizes a multi-term loss function that considers rhyme, song structure, and fluency.
HumanBench: Towards General Human-centric Perception with Projector Assisted Pretraining
Specifically, we propose a \textbf{HumanBench} based on existing datasets to comprehensively evaluate on the common ground the generalization abilities of different pretraining methods on 19 datasets from 6 diverse downstream tasks, including person ReID, pose estimation, human parsing, pedestrian attribute recognition, pedestrian detection, and crowd counting.
Ranked #1 on
Pedestrian Attribute Recognition
on PA-100K
(using extra training data)
UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy
Trained on our synthesized large-scale dexterous grasp dataset, this model enables us to sample diverse and high-quality dexterous grasp poses for the object point cloud. For the second stage, we propose to replace the motion planning used in parallel gripper grasping with a goal-conditioned grasp policy, due to the complexity involved in dexterous grasping execution.
Self-Supervised Category-Level Articulated Object Pose Estimation with Part-Level SE(3) Equivariance
Category-level articulated object pose estimation aims to estimate a hierarchy of articulation-aware object poses of an unseen articulated object from a known category.
Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining
This motivates us to learn 3D representations by sharing the merits of both paradigms, which is non-trivial due to the pattern difference between the two paradigms.
Ranked #1 on
Zero-Shot Transfer 3D Point Cloud Classification
on ModelNet10
(using extra training data)
When Source-Free Domain Adaptation Meets Learning with Noisy Labels
We also prove that such a difference makes existing LLN methods that rely on their distribution assumptions unable to address the label noise in SFDA.
LeaF: Learning Frames for 4D Point Cloud Sequence Understanding
With such frames, we can factorize geometry and motion to facilitate a feature-space geometric reconstruction for more effective 4D learning.
Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?
The success of deep learning heavily relies on large-scale data with comprehensive labels, which is more expensive and time-consuming to fetch in 3D compared to 2D images or natural languages.
Ranked #7 on
Few-Shot 3D Point Cloud Classification
on ModelNet40 10-way (10-shot)
(using extra training data)
Few-Shot 3D Point Cloud Classification
Knowledge Distillation
+1
Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning
Recent work on 4D point cloud sequences has attracted a lot of attention.
Language-Assisted 3D Feature Learning for Semantic Scene Understanding
To guide 3D feature learning toward important geometric attributes and scene context, we explore the help of textual scene descriptions.
GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts
Based on GAPartNet, we investigate three cross-category tasks: part segmentation, part pose estimation, and part-based object manipulation.
Morig: Motion-aware rigging of character meshes from point clouds
We present MoRig, a method that automatically rigs character meshes driven by single-view point cloud streams capturing the motion of performing characters.
Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing
When manipulating an object to accomplish complex tasks, humans rely on both vision and touch to keep track of the object's 6D pose.
Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild
We for the first time propose a point cloud based hand joint tracking network, HandTrackNet, to estimate the inter-frame hand joint motion.
AccoMontage2: A Complete Harmonization and Accompaniment Arrangement System
We propose AccoMontage2, a system capable of doing full-length song harmonization and accompaniment arrangement based on a lead melody.
Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding
This paper proposes a 4D backbone for long-term point cloud video understanding.
BOSS: A Benchmark for Human Belief Prediction in Object-context Scenarios
Humans with an average level of social cognition can infer the beliefs of others based solely on the nonverbal communication signals (e. g. gaze, gesture, pose and contextual information) exhibited during social interactions.
APES: Articulated Part Extraction from Sprite Sheets
Rigged puppets are one of the most prevalent representations to create 2D character animations.
Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction
Specifically, FixNet consists of a perception module to extract the structured representation from the 3D point cloud, a physical dynamics prediction module to simulate the results of interactions on 3D objects, and a functionality prediction module to evaluate the functionality and choose the correct fix.
Rotationally Equivariant 3D Object Detection
To incorporate object-level rotation equivariance into 3D object detectors, we need a mechanism to extract equivariant features with local object-level spatial support while being able to model cross-object context information.
Multi-Robot Active Mapping via Neural Bipartite Graph Matching
Previous approaches either choose the frontier as the goal position via a myopic solution that hinders the time efficiency, or maximize the long-term value via reinforcement learning to directly regress the goal position, but does not guarantee the complete map construction.
CodedVTR: Codebook-based Sparse Voxel Transformer with Geometric Guidance
We propose CodedVTR (Codebook-based Voxel TRansformer), which improves data efficiency and generalization ability for 3D sparse voxel transformers.
AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation
To solve the above issues, we propose AutoGPart, a generic method enabling training generalizable 3D part segmentation networks with the task prior considered.
HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction
We present HOI4D, a large-scale 4D egocentric dataset with rich annotations, to catalyze the research of category-level human-object interaction.
On Learning Contrastive Representations for Learning with Noisy Labels
To address this issue, we focus on learning robust contrastive representations of data on which the classifier is hard to memorize the label noise under the CE loss.
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierarchies.
Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation
Category-level object pose estimation aims to find 6D object poses of previously unseen object instances from known categories without access to object CAD models.
Contrastive Multimodal Fusion with TupleInfoNCE
Another approach is to concatenate all the modalities into a tuple and then contrast positive and negative tuple correspondences.
Ranked #75 on
Semantic Segmentation
on NYU Depth v2
Leveraging SE(3) Equivariance for Self-supervised Category-Level Object Pose Estimation from Point Clouds
To reduce the huge amount of pose annotations needed for category-level learning, we propose for the first time a self-supervised learning framework to estimate category-level 6D object pose from single 3D point clouds.
P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding
Self-supervised representation learning is a critical problem in computer vision, as it provides a way to pretrain feature extractors on large unlabeled datasets that can be used as an initialization for more efficient and effective training on downstream tasks.
Robust Neural Routing Through Space Partitions for Camera Relocalization in Dynamic Indoor Environments
Localizing the camera in a known indoor environment is a key building block for scene mapping, robot navigation, AR, etc.
Compositionally Generalizable 3D Structure Prediction
However, there remains a much more difficult and under-explored issue on how to generalize the learned skills over unseen object categories that have very different shape geometry distributions.
Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Clouds
We study an unsupervised domain adaptation problem for the semantic labeling of 3D point clouds, with a particular focus on domain discrepancies induced by different LiDAR sensors.
Rethinking Sampling in 3D Point Cloud Generative Adversarial Networks
We further study how different evaluation metrics weigh the sampling pattern against the geometry and propose several perceptual metrics forming a sampling spectrum of metrics.
SAPIEN: A SimulAted Part-based Interactive ENvironment
To achieve this task, a simulated environment with physically realistic simulation, sufficient articulated objects, and transferability to the real robot is indispensable.
Curriculum DeepSDF
When learning to sketch, beginners start with simple and flexible shapes, and then gradually strive for more complex and accurate ones in the subsequent training sessions.
Category-Level Articulated Object Pose Estimation
We develop a deep network based on PointNet++ that predicts ANCSH from a single depth point cloud, including part segmentation, normalized coordinates, and joint parameters in the canonical object space.
StructEdit: Learning Structural Shape Variations
Learning to encode differences in the geometry and (topological) structure of the shapes of ordinary objects is key to generating semantically plausible variations of a given shape, transferring edits from one shape to another, and many other applications in 3D content creation.
StructureNet: Hierarchical Graph Networks for 3D Shape Generation
We introduce StructureNet, a hierarchical graph network which (i) can directly encode shapes represented as such n-ary graphs; (ii) can be robustly trained on large and complex shape families; and (iii) can be used to generate a great diversity of realistic structured shape geometries.
AdaCoSeg: Adaptive Shape Co-Segmentation with Group Consistency Loss
While the part prior network can be trained with noisy and inconsistently segmented shapes, the final output of AdaCoSeg is a consistent part labeling for the input set, with each shape segmented into up to (a user-specified) K parts.
GeoNet: Deep Geodesic Networks for Point Cloud Analysis
Surface-based geodesic topology provides strong cues for object semantic analysis and geometric modeling.
GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud
We introduce a novel 3D object proposal approach named Generative Shape Proposal Network (GSPN) for instance segmentation in point cloud data.
Ranked #28 on
3D Object Detection
on ScanNetV2
PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding
We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information.
Ranked #3 on
3D Instance Segmentation
on PartNet
TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes
We introduce, TextureNet, a neural network architecture designed to extract features from high-resolution signals associated with 3D surface meshes (e. g., color texture maps).
Ranked #23 on
Semantic Segmentation
on ScanNet
(test mIoU metric)
Supervised Fitting of Geometric Primitives to 3D Point Clouds
Fitting geometric primitives to 3D point cloud data bridges a gap between low-level digitized 3D data and high-level structural information on the underlying 3D shapes.
Deep Part Induction from Articulated Object Pairs
In this paper, we explore how the observation of different articulation states provides evidence for part structure and motion of 3D objects.
Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
We introduce a large-scale 3D shape understanding benchmark using data and annotation from ShapeNet 3D object database.
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
By exploiting metric space distances, our network is able to learn local features with increasing contextual scales.
Ranked #2 on
Semantic Segmentation
on Toronto-3D L002
Beyond Holistic Object Recognition: Enriching Image Understanding with Part States
Important high-level vision tasks such as human-object interaction, image captioning and robotic manipulation require rich semantic descriptions of objects at part level.
SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation
To enable the prediction of vertex functions on them by convolutional neural networks, we resort to spectral CNN method that enables weight sharing by parameterizing kernels in the spectral domain spanned by graph laplacian eigenbases.
Ranked #55 on
3D Part Segmentation
on ShapeNet-Part
Dialogue Session Segmentation by Embedding-Enhanced TextTiling
In human-computer conversation systems, the context of a user-issued utterance is particularly important because it provides useful background information of the conversation.
ShapeNet: An Information-Rich 3D Model Repository
We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects.
Learning Discriminative Representations for Semantic Cross Media Retrieval
As a result, an efficient linear semantic down mapping is jointly learned for multimodal data, leading to a common space where they can be compared.
3D-Assisted Image Feature Synthesis for Novel Views of an Object
In this paper, given a single input image of an object, we synthesize new features for other views of the same object.
