Search Results for author:
Found 29 papers, 25 papers with code
LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
To approach this evaluation trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that emphasizes both coverage and efficiency.
Long Context Transfer from Language to Vision
By simply extrapolating the context length of the language backbone, we enable LMMs to comprehend orders of magnitude more visual tokens without any video training.
4D Panoptic Scene Graph Generation
To facilitate research in this new area, we build a richly annotated PSG-4D dataset consisting of 3K RGB-D videos with a total of 1M frames, each of which is labeled with 4D panoptic segmentation masks as well as fine-grained, dynamic scene graphs.
WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoning
Multimodal information, together with our knowledge, help us to understand the complex and dynamic world.
Unsolvable Problem Detection: Evaluating Trustworthiness of Vision Language Models
This paper introduces a novel and significant challenge for Vision Language Models (VLMs), termed Unsolvable Problem Detection (UPD).
Towards Language-Driven Video Inpainting via Multimodal Large Language Models
We introduce a new task -- language-driven video inpainting, which uses natural language instructions to guide the inpainting process.
Panoptic Video Scene Graph Generation
PVSG relates to the existing video scene graph generation (VidSGG) problem, which focuses on temporal interactions between humans and objects grounded with bounding boxes in videos.
OtterHD: A High-Resolution Multi-modality Model
In this paper, we present OtterHD-8B, an innovative multimodal model evolved from Fuyu-8B, specifically engineered to interpret high-resolution visual inputs with granular precision.
Ranked #124 on
Visual Question Answering
on MM-Vet
Octopus: Embodied Vision-Language Programmer from Environmental Feedback
Large vision-language models (VLMs) have achieved substantial progress in multimodal perception and reasoning.
Pair then Relation: Pair-Net for Panoptic Scene Graph Generation
Panoptic Scene Graph (PSG) is a challenging task in Scene Graph Generation (SGG) that aims to create a more comprehensive scene graph representation using panoptic segmentation instead of boxes.
FunQA: Towards Surprising Video Comprehension
Surprising videos, such as funny clips, creative performances, or visual illusions, attract significant attention.
OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection
Out-of-Distribution (OOD) detection is critical for the reliable operation of open-world intelligent systems.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
MIMIC-IT: Multi-Modal In-Context Instruction Tuning
We release the MIMIC-IT dataset, instruction-response collection pipeline, benchmarks, and the Otter model.
Ranked #19 on
Visual Question Answering
on MM-Vet v2
SAD: Segment Any RGBD
The Segment Anything Model (SAM) has demonstrated its effectiveness in segmenting any part of 2D RGB images.
3D Panoptic Segmentation
Open Vocabulary Semantic Segmentation
+2
Otter: A Multi-Modal Model with In-Context Instruction Tuning
Large language models (LLMs) have demonstrated significant universal capabilities as few/zero-shot learners in various tasks due to their pre-training on vast amounts of text data, as exemplified by GPT-3, which boosted to InstrctGPT and ChatGPT, effectively following natural language instructions to accomplish real-world tasks.
Ranked #8 on
Visual Question Answering
on BenchLMM
OpenOOD: Benchmarking Generalized Out-of-Distribution Detection
Out-of-distribution (OOD) detection is vital to safety-critical machine learning applications and has thus been extensively studied, with a plethora of methods developed in the literature.
On-Device Domain Generalization
Another interesting observation is that the teacher-student gap on out-of-distribution data is bigger than that on in-distribution data, which highlights the capacity mismatch issue as well as the shortcoming of KD.
Panoptic Scene Graph Generation
Existing research addresses scene graph generation (SGG) -- a critical technology for scene understanding in images -- from a detection perspective, i. e., objects are detected using bounding boxes followed by prediction of their pairwise relationships.
Ranked #5 on
Panoptic Scene Graph Generation
on PSG Dataset
Sparse Mixture-of-Experts are Domain Generalizable Learners
It is motivated by an empirical finding that transformer-based models trained with empirical risk minimization (ERM) outperform CNN-based models employing state-of-the-art (SOTA) DG algorithms on multiple DG datasets.
Ranked #17 on
Domain Generalization
on DomainNet
(using extra training data)
Full-Spectrum Out-of-Distribution Detection
In this paper, we take into account both shift types and introduce full-spectrum OOD (FS-OOD) detection, a more realistic problem setting that considers both detecting semantic shift and being tolerant to covariate shift; and designs three benchmarks.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
Conditional Prompt Learning for Vision-Language Models
With the rise of powerful pre-trained vision-language models like CLIP, it becomes essential to investigate ways to adapt these models to downstream datasets.
Ranked #3 on
Prompt Engineering
on ImageNet V2
Generalized Out-of-Distribution Detection: A Survey
In this survey, we first present a unified framework called generalized OOD detection, which encompasses the five aforementioned problems, i. e., AD, ND, OSR, OOD detection, and OD.
Learning to Prompt for Vision-Language Models
Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks.
Ranked #2 on
Few-shot Age Estimation
on MORPH Album2
Semantically Coherent Out-of-Distribution Detection
The proposed UDG can not only enrich the semantic knowledge of the model by exploiting unlabeled data in an unsupervised manner, but also distinguish ID/OOD samples to enhance ID classification and OOD detection tasks simultaneously.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
Progressive Representative Labeling for Deep Semi-Supervised Learning
In this paper, we propose to label only the most representative samples to expand the labeled set.
GIST: Distributed Training for Large-Scale Graph Convolutional Networks
The graph convolutional network (GCN) is a go-to solution for machine learning on graphs, but its training is notoriously difficult to scale both in terms of graph size and the number of model parameters.
Webly Supervised Image Classification with Metadata: Automatic Noisy Label Correction via Visual-Semantic Graph
VSGraph-LC starts from anchor selection referring to the semantic similarity between metadata and correct label concepts, and then propagates correct labels from anchors on a visual graph using graph neural network (GNN).
Ranked #9 on
Image Classification
on WebVision-1000
Webly Supervised Image Classification with Self-Contained Confidence
Therefore, a simple yet effective WSL framework is proposed.
Ranked #7 on
Image Classification
on WebVision-1000
SeDMiD for Confusion Detection: Uncovering Mind State from Time Series Brain Wave Data
In this paper, we propose an extension of State Space Model to work with different sources of information together with its learning and inference algorithms.
