Search Results for author:
Found 50 papers, 36 papers with code
ChatIllusion: Efficient-Aligning Interleaved Generation ability with Visual Instruction Model
As the capabilities of Large-Language Models (LLMs) become widely recognized, there is an increasing demand for human-machine chat applications.
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models
We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, tuning tasks, and visual embeddings.
Ranked #8 on
Visual Question Answering
on MM-Vet
Improving Compositional Text-to-image Generation with Large Vision-Language Models
However, compositional text-to-image models frequently encounter difficulties in generating high-quality images that accurately align with input texts describing multiple objects, variable attributes, and intricate spatial relationships.
MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning
In this paper, we present a method to fine-tune open-source language models, enabling them to use code for modeling and deriving math equations and, consequently, enhancing their mathematical reasoning abilities.
NOC: High-Quality Neural Object Cloning with 3D Lifting of Segment Anything
Firstly, to separate the target object from the scene, we propose a novel strategy to lift the multi-view 2D segmentation masks of SAM into a unified 3D variation field.
PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation
Our dataset poses new challenges in panoramic VOS and we hope that our PanoVOS can advance the development of panoramic segmentation/tracking.
RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision
3D occupancy prediction holds significant promise in the fields of robot perception and autonomous driving, which quantifies 3D scenes into grid cells with semantic labels.
ImageBind-LLM: Multi-modality Instruction Tuning
During training, we adopt a learnable bind network to align the embedding space between LLaMA and ImageBind's image encoder.
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following
We introduce Point-Bind, a 3D multi-modality model aligning point clouds with 2D image, language, audio, and video.
Less is More: Towards Efficient Few-shot 3D Semantic Segmentation via Training-free Networks
However, the prior pre-training stage not only introduces excessive time overhead, but also incurs a significant domain gap on `unseen' classes.
3D Semantic Segmentation
Few-shot 3D semantic segmentation
+2
Dynamic Embedding Size Search with Minimum Regret for Streaming Recommender System
The high-throughput data requires the model to be updated in a timely manner for capturing the user interest dynamics, which leads to the emergence of streaming recommender systems.
Revisiting Event-based Video Frame Interpolation
We therefore propose to incorporate RGB information in an event-guided optical flow refinement strategy.
Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models
Video Question Answering (VideoQA) has been significantly advanced from the scaling of recent Large Language Models (LLMs).
Ranked #3 on
Temporal/Casual QA
on NExT-QA
(using extra training data)
ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation
Note that, our method can be regarded as a novel transfer paradigm for large-scale models, delivering promising results in adaptation to continually changing distributions.
Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation
In this paper, we propose MUTR, a Multi-modal Unified Temporal transformer for Referring video object segmentation.
Referring Expression Segmentation
Referring Video Object Segmentation
+2
Personalize Segment Anything Model with One Shot
Driven by large-data pre-training, Segment Anything Model (SAM) has been demonstrated as a powerful and promptable framework, revolutionizing the segmentation models.
Ranked #1 on
Personalized Segmentation
on PerSeg
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
This strategy effectively alleviates the interference between the two tasks of image-text alignment and instruction following and achieves strong multi-modal reasoning with only a small-scale image-text and instruction dataset.
Ranked #5 on
Visual Question Answering (VQA)
on CORE-MM
Instruction Following
Optical Character Recognition (OCR)
+7
Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement
The popularity of Contrastive Language-Image Pre-training (CLIP) has propelled its application to diverse downstream vision tasks.
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model.
Ranked #2 on
Music Question Answering
on MusicQA Dataset
PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection
Masked Autoencoders learn strong visual representations and achieve state-of-the-art results in several independent modalities, yet very few works have addressed their capabilities in multi-modality settings.
Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysis
We present a Non-parametric Network for 3D point cloud analysis, Point-NN, which consists of purely non-learnable components: farthest point sampling (FPS), k-nearest neighbors (k-NN), and pooling operations, with trigonometric functions.
Ranked #1 on
Training-free 3D Part Segmentation
on ShapeNet-Part
3D Point Cloud Classification
Training-free 3D Part Segmentation
+1
Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimicking
To alleviate this, previous methods simply replace the pixel reconstruction targets of 75% masked tokens by encoded features from pre-trained image-image (DINO) or image-language (CLIP) contrastive learning.
Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
Our CaFo incorporates CLIP's language-contrastive knowledge, DINO's vision-contrastive knowledge, DALL-E's vision-generative knowledge, and GPT-3's language-generative knowledge.
Nearest Neighbors Meet Deep Neural Networks for Point Cloud Analysis
Performances on standard 3D point cloud benchmarks have plateaued, resulting in oversized models and complex network design to make a fractional improvement.
Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training
In this paper, we explore how the 2D modality can benefit 3D masked autoencoding, and propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training.
Starting From Non-Parametric Networks for 3D Point Cloud Analysis
We present a Non-parametric Network for 3D point cloud analysis, Point-NN, which consists of purely non-learnable components: farthest point sampling (FPS), k-nearest neighbors (k-NN), and pooling operations, with trigonometric functions.
SparseMAE: Sparse Training Meets Masked Autoencoders
In this paper, we aim to reduce model complexity from large vision transformers pretrained by MAE with assistant of sparse training.
TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning
In this paper, we propose the learning scheme of Target Inner-Geometry from the LiDAR modality into camera-based BEV detectors for both dense depth and BEV features, termed as TiG-BEV.
Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders
Pre-training by numerous image data has become de-facto for robust 2D representations.
Ranked #1 on
3D Point Cloud Linear Classification
on ModelNet40
(using extra training data)
3D Point Cloud Linear Classification
Few-Shot 3D Point Cloud Classification
Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation
Since pseudo labels are noisy and unreliable, these methods suffer from catastrophic forgetting and error accumulation when dealing with dynamic data distributions.
iQuery: Instruments as Queries for Audio-Visual Sound Separation
To generalize to a new instrument or event class, drawing inspiration from the text-prompt design, we insert an additional query as an audio prompt while freezing the attention mechanism.
PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning
In this paper, we first collaborate CLIP and GPT to be a unified 3D open-world learner, named as PointCLIP V2, which fully unleashes their potential for zero-shot 3D classification, segmentation, and detection.
Ranked #2 on
Open Vocabulary Object Detection
on STPLS3D
EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding
3D visual grounding aims to find the object within point clouds mentioned by free-form natural language descriptions with rich semantic cues.
CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention
Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual representations with great transferability, which achieves promising accuracy for zero-shot classification.
Ranked #4 on
Training-free 3D Point Cloud Classification
on ScanObjectNN
(using extra training data)
Training-free 3D Point Cloud Classification
Transfer Learning
+1
Collaboration of Pre-trained Models Makes Better Few-shot Learner
In this paper, we propose CoMo, a Collaboration of pre-trained Models that incorporates diverse prior knowledge from various pre-training paradigms for better few-shot learning.
Frozen CLIP Models are Efficient Video Learners
Video recognition has been dominated by the end-to-end learning paradigm -- first initializing a video recognition model with weights of a pretrained image model and then conducting end-to-end training on videos.
Ranked #24 on
Action Classification
on Kinetics-400
(using extra training data)
Exploring Resolution and Degradation Clues as Self-supervised Signal for Low Quality Object Detection
Image restoration algorithms such as super resolution (SR) are indispensable pre-processing modules for object detection in low quality images.
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
On top of that, the performance of Tip-Adapter can be further boosted to be state-of-the-art on ImageNet by fine-tuning the cache model for 10$\times$ fewer epochs than existing methods, which is both effective and efficient.
Can Language Understand Depth?
To our best knowledge, we are the first to conduct zero-shot adaptation from the semantic language knowledge to quantified downstream tasks and perform zero-shot monocular depth estimation.
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
By fine-tuning on downstream tasks, Point-M2AE achieves 86. 43% accuracy on ScanObjectNN, +3. 36% to the second-best, and largely benefits the few-shot classification, part segmentation and 3D object detection with the hierarchical pre-training scheme.
Ranked #3 on
3D Point Cloud Linear Classification
on ModelNet40
(using extra training data)
POS-BERT: Point Cloud One-Stage BERT Pre-Training
We propose to use the dynamically updated momentum encoder as the tokenizer, which is updated and outputs the dynamic supervision signal along with the training process.
MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection
In this paper, we introduce the first DETR framework for Monocular DEtection with a depth-guided TRansformer, named MonoDETR.
Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning
Especially, we develop a variant of ViT for 3D point cloud feature extraction, which also achieves comparable results with existing backbones when combined with our framework, and visualization of the attention maps show that our model does understand the point cloud by combining the global shape information and multiple local structural information, which is consistent with the inspiration of our representation learning method.
VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts
Contrastive Language-Image Pre-training (CLIP) has drawn increasing attention recently for its transferable visual representation learning.
PointCLIP: Point Cloud Understanding by CLIP
On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D.
Ranked #3 on
Open Vocabulary Object Detection
on STPLS3D
DSPoint: Dual-scale Point Cloud Recognition with High-frequency Fusion
We reverse the conventional design of applying convolution on voxels and attention to points.
Ranked #36 on
Semantic Segmentation
on S3DIS
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification.
CLIP-Adapter: Better Vision-Language Models with Feature Adapters
Large-scale contrastive vision-language pre-training has shown significant progress in visual representation learning.
Dual-stream Network for Visual Recognition
Transformers with remarkable global representation capacities achieve competitive results for visual tasks, but fail to consider high-level local pattern information in input images.
End-to-End Object Detection with Adaptive Clustering Transformer
In this paper, a novel variant of transformer named Adaptive Clustering Transformer(ACT) has been proposed to reduce the computation cost for high-resolution input.
