Search Results for author:
Found 46 papers, 31 papers with code
A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models
The rise of foundation models has transformed machine learning research, prompting efforts to uncover their inner workings and develop more efficient and reliable applications for better control.
Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models
Recently, promising progress has been made by open-source vision-language models (VLMs) in bringing their capabilities closer to those of proprietary frontier models.
TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any Point in Long Video
In this paper, we present TAPTRv3, which is built upon TAPTRv2 to improve its point tracking robustness in long videos.
DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding
DINO-X employs the same Transformer-based encoder-decoder architecture as Grounding DINO 1. 5 to pursue an object-level representation for open-world object understanding.
Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspective
First, we present the trend for next generation of vision foundation models, i. e., unifying both understanding and generation in vision tasks.
TAPTRv2: Attention-based Position Update Improves Tracking Any Point
In this paper, we present TAPTRv2, a Transformer-based approach built upon TAPTR for solving the Tracking Any Point (TAP) task.
MMedAgent: Learning to Use Medical Tools with Multi-modal Agent
We curate an instruction-tuning dataset comprising six medical tools solving seven tasks across five modalities, enabling the agent to choose the most suitable tools for a given task.
CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents
Leveraging Crab, we developed a cross-platform Crab Benchmark-v0 comprising 120 tasks in computer desktop and mobile phone environments.
Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detection
Empirical results demonstrate the effectiveness of Grounding DINO 1. 5, with the Grounding DINO 1. 5 Pro model attaining a 54. 3 AP on the COCO detection benchmark and a 55. 7 AP on the LVIS-minival zero-shot transfer benchmark, setting new records for open-set object detection.
Ranked #1 on
Zero-Shot Object Detection
on MSCOCO
(using extra training data)
Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models
We introduce Vidu, a high-performance text-to-video generator that is capable of producing 1080p videos up to 16 seconds in a single generation.
T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy
Recognizing the complementary strengths and weaknesses of both text and visual prompts, we introduce T-Rex2 that synergizes both prompts within a single model through contrastive learning.
TAPTR: Tracking Any Point with Transformers as Detection
Based on the observation that point tracking bears a great resemblance to object detection and tracking, we borrow designs from DETR-like algorithms to address the task of TAP.
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks
We introduce Grounded SAM, which uses Grounding DINO as an open-set object detector to combine with the segment anything model (SAM).
Interfacing Foundation Models' Embeddings
To further unleash the power of foundation models, we present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity.
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
To address this issue, we have created GVC data that allows for the combination of grounding and chat capabilities.
T-Rex: Counting by Visual Prompting
Guided by the visual feedback from T-Rex, users can also interactively refine the counting results by prompting on missing or falsely-detected objects.
Visual In-Context Prompting
In-context prompting in large language models (LLMs) has become a prevalent approach to improve zero-shot capabilities, but this idea is less explored in the vision domain.
LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
LLaVA-Plus is a general-purpose multimodal assistant that expands the capabilities of large multimodal models.
Ranked #1 on
LMM real-life tasks
on Leaderboard
InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image
With the success of Neural Radiance Field (NeRF) in 3D-aware portrait editing, a variety of works have achieved promising results regarding both quality and 3D consistency.
TOSS:High-quality Text-guided Novel View Synthesis from a Single Image
In this paper, we present TOSS, which introduces text to the task of novel view synthesis (NVS) from just a single RGB image.
Neural Interactive Keypoint Detection
Click-Pose explores how user feedback can cooperate with a neural keypoint detector to correct the predicted keypoints in an interactive way for a faster and more effective annotation process.
A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision
Existing model evaluation tools mainly focus on evaluating classification models, leaving a gap in evaluating more complex models, such as object detection.
DFA3D: 3D Deformable Attention For 2D-to-3D Feature Lifting
Existing feature lifting approaches, such as Lift-Splat-based and 2D attention-based, either use estimated depth to get pseudo LiDAR features and then splat them to a 3D space, which is a one-pass operation without feature refinement, or ignore depth and lift features by 2D attention mechanisms, which achieve finer semantics while suffering from a depth ambiguity problem.
Semantic-SAM: Segment and Recognize Anything at Any Granularity
In this paper, we introduce Semantic-SAM, a universal image segmentation model to enable segment and recognize anything at any desired granularity.
detrex: Benchmarking Detection Transformers
To address this issue, we develop a unified, highly modular, and lightweight codebase called detrex, which supports a majority of the mainstream DETR-based instance recognition algorithms, covering various fundamental tasks, including object detection, segmentation, and pose estimation.
Recognize Anything: A Strong Image Tagging Model
We are releasing the RAM at \url{https://recognize-anything. github. io/} to foster the advancements of large models in computer vision.
A Strong and Reproducible Object Detector with Only Public Datasets
This work presents Focal-Stable-DINO, a strong and reproducible object detection model which achieves 64. 6 AP on COCO val2017 and 64. 8 AP on COCO test-dev using only 700M parameters without any test time augmentation.
Ranked #4 on
Object Detection
on COCO test-dev
PREIM3D: 3D Consistent Precise Image Attribute Editing from a Single Image
As for imprecise image editing, we attribute the problem to the gap between the latent space of real images and that of generated images.
Detection Transformer with Stable Matching
We point out that the unstable matching in DETR is caused by a multi-optimization path problem, which is highlighted by the one-to-one matching design in DETR.
A Simple Framework for Open-Vocabulary Segmentation and Detection
We present OpenSeeD, a simple Open-vocabulary Segmentation and Detection framework that jointly learns from different segmentation and detection datasets.
Ranked #1 on
Instance Segmentation
on Cityscapes val
(using extra training data)
MP-Former: Mask-Piloted Transformer for Image Segmentation
We present a mask-piloted Transformer which improves masked-attention in Mask2Former for image segmentation.
Lite DETR : An Interleaved Multi-Scale Encoder for Efficient DETR
Recent DEtection TRansformer-based (DETR) models have obtained remarkable performance.
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion.
Ranked #2 on
Zero Shot Segmentation
on Segmentation in the Wild
Introducing Depth into Transformer-based 3D Object Detection
To address the second issue, we introduce an auxiliary learning task called Depth-aware Negative Suppression loss.
Explicit Box Detection Unifies End-to-End Multi-Person Pose Estimation
This paper presents a novel end-to-end framework with Explicit box Detection for multi-person Pose estimation, called ED-Pose, where it unifies the contextual learning between human-level (global) and keypoint-level (local) information.
Ranked #2 on
2D Human Pose Estimation
on Human-Art
Lite DETR: An Interleaved Multi-Scale Encoder for Efficient DETR
Recent DEtection TRansformer-based (DETR) models have obtained remarkable performance.
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding
As phrase extraction can be regarded as a $1$D text segmentation problem, we formulate PEG as a dual detection problem and propose a novel DQ-DETR model, which introduces dual queries to probe different features from image and text for object prediction and phrase mask prediction.
Ranked #11 on
Referring Expression Comprehension
on RefCOCO
A Unified Mutual Supervision Framework for Referring Expression Segmentation and Generation
Our mutual supervision contains two directions.
Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation
In this paper we present Mask DINO, a unified object detection and segmentation framework.
Ranked #1 on
Panoptic Segmentation
on COCO test-dev
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Compared to other models on the leaderboard, DINO significantly reduces its model size and pre-training data size while achieving better results.
Ranked #2 on
Object Detection
on SA-Det-100k
Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
This survey is inspired by the remarkable progress in both computer vision and natural language processing, and recent trends shifting from single modality processing to multiple modality comprehension.
DN-DETR: Accelerate DETR Training by Introducing Query DeNoising
Our method is universal and can be easily plugged into any DETR-like methods by adding dozens of lines of code to achieve a remarkable improvement.
Ranked #84 on
Object Detection
on COCO minival
DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR
We present in this paper a novel query formulation using dynamic anchor boxes for DETR (DEtection TRansformer) and offer a deeper understanding of the role of queries in DETR.
Ranked #102 on
Object Detection
on COCO minival
Unrestricted Adversarial Attacks on ImageNet Competition
Many works have investigated the adversarial attacks or defenses under the settings where a bounded and imperceptible perturbation can be added to the input.
Query2Label: A Simple Transformer Way to Multi-Label Classification
The use of Transformer is rooted in the need of extracting local discriminative features adaptively for different labels, which is a strongly desired property due to the existence of multiple objects in one image.
Ranked #1 on
Multi-Label Classification
on PASCAL VOC 2012
Unsupervised Part Segmentation through Disentangling Appearance and Shape
We study the problem of unsupervised discovery and segmentation of object parts, which, as an intermediate local representation, are capable of finding intrinsic object structure and providing more explainable recognition results.
