Search Results for author:
Found 89 papers, 44 papers with code
EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation
We introduce EnerVerse, a comprehensive framework for embodied future space generation specifically designed for robotic manipulation tasks.
A3: Android Agent Arena for Mobile GUI Agents
AI agents have become increasingly prevalent in recent years, driven by significant advancements in the field of large language models (LLMs).
MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes
Repurposing pre-trained diffusion models has been proven to be effective for NVS.
MMAD-Purify: A Precision-Optimized Framework for Efficient and Scalable Multi-Modal Attacks
With the rise of multimodality, diffusion models have emerged as powerful tools not only for generative tasks but also for various applications such as image editing, inpainting, and super-resolution.
Cluster-wise Graph Transformer with Dual-granularity Kernelized Attention
In the realm of graph learning, there is a category of methods that conceptualize graphs as hierarchical structures, utilizing node clustering to capture broader structural information.
Mirror-Consistency: Harnessing Inconsistency in Majority Voting
Self-Consistency, a widely-used decoding strategy, significantly boosts the reasoning capabilities of Large Language Models (LLMs).
Autonomous Character-Scene Interaction Synthesis from Text Instruction
Synthesizing human motions in 3D environments, particularly those with complex activities such as locomotion, hand-reaching, and human-object interaction, presents substantial demands for user-defined waypoints and stage transitions.
Effective Tuning Strategies for Generalist Robot Manipulation Policies
While fine-tuning offers a practical way to quickly adapt a GMPs to novel domains and tasks with limited samples, we observe that the performance of the resulting GMPs differs significantly with respect to the design choices of fine-tuning strategies.
SKT: Integrating State-Aware Keypoint Trajectories with Vision-Language Models for Robotic Garment Manipulation
In addition, this research also underscores the potential of VLMs to unify various garment manipulation tasks within a single framework, paving the way for broader applications in home automation and assistive robotics for future.
PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions
Furthermore, we adopt Diffusion Transformers (DiT) as our foundation model and extend its capabilities with a flexible any resolution mechanism, enabling the model to dynamically process images based on the aspect ratio of the input, closely aligning with human perceptual processes.
Multi-modal Situated Reasoning in 3D Scenes
Comprehensive evaluations on MSQA and MSNN highlight the limitations of existing vision-language models and underscore the importance of handling multi-modal interleaved inputs and situation modeling.
PhysPart: Physically Plausible Part Completion for Interactable Objects
We also demonstrate our applications in 3D printing, robot manipulation, and sequential part generation, showing our strength in realistic tasks with the demand for high physical plausibility.
SlotLifter: Slot-guided Feature Lifting for Learning Object-centric Radiance Fields
The ability to distill object-centric abstractions from intricate visual scenes underpins human-level generalization.
Task-oriented Sequential Grounding in 3D Scenes
Grounding natural language in physical 3D environments is essential for the advancement of embodied artificial intelligence.
F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions
Existing 3D human object interaction (HOI) datasets and models simply align global descriptions with the long HOI sequence, while lacking a detailed understanding of intermediate states and the transitions between states.
3D Vision and Language Pretraining with Large-Scale Synthetic Data
3D Vision-Language Pre-training (3D-VLP) aims to provide a pre-train model which can bridge 3D scenes with natural language, which is an important technique for embodied intelligence.
AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
To advance research on AI agents in mobile scenarios, we introduce the Android Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for generalist mobile GUI-control agents.
GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices
Smartphone users often navigate across multiple applications (apps) to complete tasks such as sharing content between social media platforms.
Unifying 3D Vision-Language Understanding via Promptable Queries
A unified model for 3D vision-language (3D-VL) understanding is expected to take various scene representations and perform a wide range of tasks in a 3D scene.
Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations
Autonomous robotic systems capable of learning novel manipulation tasks are poised to transform industries from manufacturing to service automation.
PhyRecon: Physically Plausible Neural Scene Reconstruction
In this paper, we introduce PHYRECON, the first approach to leverage both differentiable rendering and differentiable physics simulation to learn implicit surface representations.
Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V
Autonomous robot navigation and manipulation in open environments require reasoning and replanning with closed-loop feedback.
PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI
With recent developments in Embodied Artificial Intelligence (EAI) research, there has been a growing demand for high-quality, large-scale interactive scene generation.
Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
In this paper, we introduce the Draw-and-Understand project: a new model, a multi-domain dataset, and a challenging benchmark for visual prompting.
Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance
Despite significant advancements in text-to-motion synthesis, generating language-guided human motion within 3D environments poses substantial challenges.
AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents
Traditional approaches in physics-based motion generation, centered around imitation learning and reward shaping, often struggle to adapt to new scenarios.
Scaling Up Dynamic Human-Scene Interaction Modeling
Confronting the challenges of data scarcity and advanced motion synthesis in human-scene interaction modeling, we introduce the TRUMANS dataset alongside a novel HSI motion synthesis method.
Graph Parsing Networks
GPN benefits from the discrete assignments generated by the graph parsing algorithm, allowing good memory efficiency while preserving node information intact.
Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
Specifically, we propose, for the first time to our best knowledge, post-training approaches for task-agnostic and task-specific expert pruning and skipping of MoE LLMs, tailored to improve deployment efficiency while maintaining model performance across a wide range of tasks.
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM) series developed upon SPHINX.
Ranked #14 on
Video Question Answering
on MVBench
SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding
In comparison to recent advancements in the 2D domain, grounding language in 3D scenes faces several significant challenges: (i) the inherent complexity of 3D scenes due to the diverse object configurations, their rich attributes, and intricate relationships; (ii) the scarcity of paired 3D vision-language data to support grounded learning; and (iii) the absence of a unified learning framework to distill knowledge from grounded 3D data.
Move as You Say Interact as You Can: Language-guided Human Motion Generation with Scene Affordance
Despite significant advancements in text-to-motion synthesis generating language-guided human motion within 3D environments poses substantial challenges.
VILLS -- Video-Image Learning to Learn Semantics for Person Re-Identification
Person Re-identification is a research area with significant real world applications.
An Embodied Generalist Agent in 3D World
However, several significant challenges remain: (i) most of these models rely on 2D images yet exhibit a limited capacity for 3D input; (ii) these models rarely explore the tasks inherently defined in 3D world, e. g., 3D grounding, embodied reasoning and acting.
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models
We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, tuning tasks, and visual embeddings.
Ranked #2 on
Visual Question Answering
on BenchLMM
(using extra training data)
Whole-body Detection, Recognition and Identification at Altitude and Range
In this paper, we address the challenging task of whole-body biometric detection, recognition, and identification at distances of up to 500m and large pitch angles of up to 50 degree.
Single-view 3D Scene Reconstruction with High-fidelity Shape and Texture
Reconstructing detailed 3D scenes from single-view images remains a challenging task due to limitations in existing approaches, which primarily focus on geometric shape recovery, overlooking object appearances and fine shape details.
Grasp Multiple Objects with One Hand
The intricate kinematics of the human hand enable simultaneous grasping and manipulation of multiple objects, essential for tasks such as object transfer and in-hand manipulation.
Tailoring Self-Attention for Graph via Rooted Subtrees
Attention mechanisms have made significant strides in graph learning, yet they still exhibit notable limitations: local attention faces challenges in capturing long-range information due to the inherent problems of the message-passing scheme, while global attention cannot reflect the hierarchical neighborhood structure and fails to capture fine-grained local information.
MLLM-DataEngine: An Iterative Refinement Approach for MLLM
Despite the great advance of Multimodal Large Language Models (MLLMs) in both instruction dataset building and benchmarking, the independence of training and evaluation makes current MLLMs hard to further improve their capability under the guidance of evaluation results with a relatively low human cost.
3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
3D vision-language grounding (3D-VL) is an emerging field that aims to connect the 3D physical world with natural language, which is crucial for achieving embodied intelligence.
TinyLVLM-eHub: Towards Comprehensive and Efficient Evaluation for Large Vision-Language Models
Secondly, it conducts an in-depth analysis of LVLMs' predictions using the ChatGPT Ensemble Evaluation (CEE), which leads to a robust and accurate evaluation and exhibits improved alignment with human evaluation compared to the word matching approach.
Distillation-guided Representation Learning for Unconstrained Gait Recognition
Gait recognition holds the promise of robustly identifying subjects based on walking patterns instead of appearance information.
LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Large Vision-Language Models (LVLMs) have recently played a dominant role in multimodal vision-language learning.
Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model
This paper presents Instruct2Act, a framework that utilizes Large Language Models to map multi-modal instructions to sequential actions for robotic manipulation tasks.
SUG: Single-dataset Unified Generalization for 3D Point Cloud Classification
In this paper, different from previous 2D DG works, we focus on the 3D DG problem and propose a Single-dataset Unified Generalization (SUG) framework that only leverages a single source dataset to alleviate the unforeseen domain differences faced by a well-trained source model.
ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes
To tackle these challenges, we present ARNOLD, a benchmark that evaluates language-grounded task learning with continuous states in realistic 3D scenes.
Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
Our CaFo incorporates CLIP's language-contrastive knowledge, DINO's vision-contrastive knowledge, DALL-E's vision-generative knowledge, and GPT-3's language-generative knowledge.
Diffusion-based Generation, Optimization, and Planning in 3D Scenes
SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization, and planning.
Full-Body Articulated Human-Object Interaction
By learning the geometrical relationships in HOI, we devise the very first model that leverage human pose estimation to tackle the estimation of articulated object poses and shapes during whole-body interactions.
UniDA3D: Unified Domain Adaptive 3D Semantic Segmentation Pipeline
Different from previous studies that only focus on a single adaptation task, UniDA3D can tackle several adaptation tasks in 3D segmentation field, by designing a unified source-and-target active sampling strategy, which selects a maximally-informative subset from both source and target domains for effective model adaptation.
Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representation
Current computer vision models, unlike the human visual system, cannot yet achieve general-purpose visual understanding.
GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts
Based on GAPartNet, we investigate three cross-category tasks: part segmentation, part pose estimation, and part-based object manipulation.
HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes
Learning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characteristics of the existing datasets on Human-Scene Interaction (HSI); they only have limited scale/quality and lack semantics.
Improving Object-centric Learning with Query Optimization
The ability to decompose complex natural scenes into meaningful object-centric abstractions lies at the core of human perception and reasoning.
SQA3D: Situated Question Answering in 3D Scenes
We propose a new task to benchmark scene understanding of embodied agents: Situated Question Answering in 3D Scenes (SQA3D).
Ranked #1 on
Referring Expression
on SQA3D
EgoTaskQA: Understanding Human Tasks in Egocentric Videos
The challenges of such capability lie in the difficulty of generating a detailed understanding of situated actions, their effects on object states (i. e., state changes), and their causal dependencies.
Neural-Symbolic Recursive Machine for Systematic Generalization
We evaluate NSR's efficacy across four challenging benchmarks designed to probe systematic generalization capabilities: SCAN for semantic parsing, PCFG for string manipulation, HINT for arithmetic reasoning, and a compositional machine translation task.
GenDexGrasp: Generalizable Dexterous Grasping
By leveraging the contact map as a hand-agnostic intermediate representation, GenDexGrasp efficiently generates diverse and plausible grasping poses with a high success rate and can transfer among diverse multi-fingered robotic hands.
Infrared Invisible Clothing:Hiding from Infrared Detectors at Multiple Angles in Real World
We simulated the process from cloth to clothing in the digital world and then designed the adversarial "QR code" pattern.
Adversarial Texture for Fooling Person Detectors in the Physical World
Experiments showed that these clothes could fool person detectors in the physical world.
PartAfford: Part-level Affordance Discovery from 3D Objects
We propose a novel learning framework for PartAfford, which discovers part-level representations by leveraging only the affordance set supervision and geometric primitive regularization, without dense supervision.
Multi-modal Sensor Fusion for Auto Driving Perception: A Survey
Multi-modal fusion is a fundamental task for the perception of an autonomous driving system, which has recently intrigued many researchers.
Infrared Invisible Clothing: Hiding From Infrared Detectors at Multiple Angles in Real World
We simulated the process from cloth to clothing in the digital world and then designed the adversarial "QR code" pattern.
YouRefIt: Embodied Reference Understanding with Language and Gesture
To the best of our knowledge, this is the first embodied reference dataset that allows us to study referring expressions in daily physical scenes to understand referential behavior, human communication, and human-robot interaction.
Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds
To date, various 3D scene understanding tasks still lack practical and generalizable pre-trained models, primarily due to the intricate nature of 3D scene understanding tasks and their immense variations introduced by camera views, lighting, occlusions, etc.
Ranked #4 on
3D Object Detection
on SUN-RGBD
Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
We further propose a novel geometry solving approach with formal language and symbolic reasoning, called Interpretable Geometry Problem Solver (Inter-GPS).
Ranked #1 on
Mathematical Question Answering
on GeoS
Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis
Specifically, the camera pose and 3D scene are represented as vectors and the local camera movement is represented as a matrix operating on the vector of the camera pose.
VLGrammar: Grounded Grammar Induction of Vision and Language
In this work, we study grounded grammar induction of vision and language in a joint learning framework.
A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics
We believe the HINT dataset and the experimental findings are of great interest to the learning community on systematic generalization.
SMART: A Situation Model for Algebra Story Problems via Attributed Grammar
Solving algebra story problems remains a challenging task in artificial intelligence, which requires a detailed understanding of real-world situations and a strong mathematical reasoning capability.
Learning by Fixing: Solving Math Word Problems with Weak Supervision
To generate more diverse solutions, \textit{tree regularization} is applied to guide the efficient shrinkage and exploration of the solution space, and a \textit{memory buffer} is designed to track and save the discovered various fixes for each problem.
Ranked #1 on
Math Word Problem Solving
on Math23K
(weakly-supervised metric)
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities
Understanding and interpreting human actions is a long-standing challenge and a critical indicator of perception in artificial intelligence.
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
Humans can progressively learn visual concepts from easy to hard questions.
Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning
In this paper, we address these issues and close the loop of neural-symbolic learning by (1) introducing the \textbf{grammar} model as a \textit{symbolic prior} to bridge neural perception and symbolic reasoning, and (2) proposing a novel \textbf{back-search} algorithm which mimics the top-down human-like learning procedure to propagate the error through the symbolic reasoning module efficiently.
Memory-efficient training with streaming dimensionality reduction
The movement of large quantities of data during the training of a Deep Neural Network presents immense challenges for machine learning workloads.
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility inference, and social learning.
PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points
Detecting 3D objects from a single RGB image is intrinsically ambiguous, thus requiring appropriate prior knowledge and intermediate representations as constraints to reduce the uncertainties and improve the consistencies between the 2D image plane and the 3D world coordinate.
Ranked #2 on
Monocular 3D Object Detection
on SUN RGB-D
(AP@0.15 (10 / PNet-30) metric)
Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
We propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction---3D estimations of object bounding boxes, camera pose, and room layout, and (ii) 3D human pose estimation.
3D Human Pose Estimation
Human-Object Interaction Detection
+1
Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
This paper addresses a new problem of understanding human gaze communication in social videos from both atomic-level and event-level, which is significant for studying human social interactions.
Streaming Batch Eigenupdates for Hardware Neuromorphic Networks
Neuromorphic networks based on nanodevices, such as metal oxide memristors, phase change memories, and flash memory cells, have generated considerable interest for their increased energy efficiency and density in comparison to graphics processing units (GPUs) and central processing units (CPUs).
Learning V1 Simple Cells with Vector Representation of Local Content and Matrix Representation of Local Motion
This paper proposes a representational model for image pairs such as consecutive video frames that are related by local pixel displacements, in the hope that the model may shed light on motion perception in primary visual cortex (V1).
Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation
Holistic 3D indoor scene understanding refers to jointly recovering the i) object bounding boxes, ii) room layout, and iii) camera pose, all in 3D.
Ranked #5 on
Monocular 3D Object Detection
on SUN RGB-D
Human-centric Indoor Scene Synthesis Using Stochastic Grammar
We present a human-centric method to sample and synthesize 3D room layouts and 2D images thereof, to obtain large-scale 2D/3D image data with perfect per-pixel ground truth.
Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image
We propose a computational framework to jointly parse a single RGB image and reconstruct a holistic 3D configuration composed by a set of CAD models using a stochastic grammar model.
Ranked #4 on
Monocular 3D Object Detection
on SUN RGB-D
(AP@0.15 (10 / PNet-30) metric)
Predicting Human Activities Using Stochastic Grammar
This paper presents a novel method to predict future human activities from partially observed RGB-D videos.
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammars
We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algorithms.
Nonlinear Local Metric Learning for Person Re-identification
In this paper, we propose a nonlinear local metric learning (NLML) method to improve the state-of-the-art performance of person re-identification on public datasets.
Image Set Querying Based Localization
Conventional single image based localization methods usually fail to localize a querying image when there exist large variations between the querying image and the pre-built scene.
