Search Results for author:
Found 25 papers, 7 papers with code
3D-LLM: Injecting the 3D World into Large Language Models
Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs.
Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties
Specifically, we test scenarios where accurate prediction relies on estimates of properties such as mass, friction, elasticity, and deformability, and where the values of those properties can only be inferred by observing how objects move and interact with other objects or fluids.
ModuleFormer: Modularity Emerges from Mixture-of-Experts
In our experiment, we found that the modular architecture enables three important abilities for large pre-trained language models: 1) Efficiency, since ModuleFormer only activates a subset of its modules for each input token, thus it could achieve the same performance as dense LLMs with more than two times throughput; 2) Extendability, ModuleFormer is more immune to catastrophic forgetting than dense LLMs and can be easily extended with new modules to learn new knowledge that is not included in the training data; 3) Specialisation, finetuning ModuleFormer could specialize a subset of modules to the finetuning task and the task-unrelated modules could be easily pruned for a lightweight deployment.
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to align the output of large language models (LLMs) with human intentions, ensuring they are helpful, ethical, and reliable.
Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following
ECL consists of: (i) an instruction parser that translates the natural languages into executable programs; (ii) an embodied concept learner that grounds visual concepts based on language descriptions; (iii) a map constructor that estimates depth and constructs semantic maps by leveraging the learned concepts; and (iv) a program executor with deterministic policies to execute each program.
Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention
When looking at an image, we can decompose the scene into entities and their parts as well as obtain the dependencies between them.
3D Concept Learning and Reasoning from Multi-View Images
We suggest that a principled approach for 3D reasoning from multi-view images should be to infer a compact 3D representation of the world from the multi-view images, which is further grounded on open-vocabulary semantic concepts, and then to execute reasoning on these 3D representations.
Planning with Large Language Models for Code Generation
Existing large language model-based code generation pipelines typically use beam search or sampling algorithms during the decoding process.
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning
The see stage scans the image and grounds the visual concept candidates with a visual perception model.
Mod-Squad: Designing Mixtures of Experts As Modular Multi-Task Learners
To address the MTL challenge, we propose Mod-Squad, a new model that is Modularized into groups of experts (a 'Squad').
Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners
To address the MTL challenge, we propose Mod-Squad, a new model that is Modularized into groups of experts (a 'Squad').
S$^3$-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint
Different from existing single-view methods which can only recover a 2. 5D scene representation (i. e., a normal / depth map for the visible surface), our method learns a neural reflectance field to represent the 3D geometry and BRDFs of a scene.
PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo
It then jointly optimizes the surface normals, spatially-varying BRDFs, and lights based on a shadow-aware differentiable rendering layer.
ComPhy: Compositional Physical Reasoning of Objects and Events from Videos
In this paper, we take an initial step to highlight the importance of inferring the hidden physical properties not directly observable from visual appearances, by introducing the Compositional Physical Reasoning (ComPhy) dataset.
A Unified Framework for Masked and Mask-Free Face Recognition via Feature Rectification
We introduce rectification blocks to rectify features extracted by a state-of-the-art recognition model, in both spatial and channel dimensions, to minimize the distance between a masked face and its mask-free counterpart in the rectified feature space.
STAR: A Benchmark for Situated Reasoning in Real-World Videos
This paper introduces a new benchmark that evaluates the situated reasoning ability via situation abstraction and logic-grounded question answering for real-world videos, called Situated Reasoning in Real-World Videos (STAR).
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language
This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine.
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos.
The Blessings of Unlabeled Background in Untrimmed Videos
The key challenge is how to distinguish the action of interest segments from the background, which is unlabelled even on the video-level.
Weakly-supervised Temporal Action Localization
Weakly Supervised Temporal Action Localization
Grounding Physical Object and Event Concepts Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos.
Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehension
To bridge the gap, we propose a new dataset for visual reasoning in context of referring expression comprehension with two main features.
Look Closer to Ground Better: Weakly-Supervised Temporal Grounding of Sentence in Video
In this paper, we study the problem of weakly-supervised temporal grounding of sentence in video.
Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video
In this paper, we address a novel task, namely weakly-supervised spatio-temporally grounding natural sentence in video.
Boosting up Scene Text Detectors with Guided CNN
Deep CNNs have achieved great success in text detection.
