Search Results for author:
Found 11 papers, 9 papers with code
Beyond Embeddings: The Promise of Visual Table in Multi-Modal Models
When visual tables serve as standalone visual representations, our model can closely match or even beat the SOTA MLLMs that are built on CLIP visual embeddings.
Ranked #34 on
Visual Question Answering
on MM-Vet
Towards Learning a Generalist Model for Embodied Navigation
We conduct extensive experiments to evaluate the performance and generalizability of our model.
3D Question Answering (3D-QA)
Embodied Question Answering
+3
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
We first benchmark MM-Navigator on our collected iOS screen dataset.
Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models
Medical image classification is a critical problem for healthcare, with the potential to alleviate the workload of doctors and facilitate diagnoses of patients.
Learning Concise and Descriptive Attributes for Visual Recognition
Recent advances in foundation models present new opportunities for interpretable visual recognition -- one can first query Large Language Models (LLMs) to obtain a set of attributes that describe each class, then apply vision-language models to classify images via these attributes.
Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations
In this work, we propose to learn video representation that encodes both action steps and their temporal ordering, based on a large-scale dataset of web instructional videos and their narrations, without using human annotations.
RegionCLIP: Region-based Language-Image Pretraining
However, we show that directly applying such models to recognize image regions for object detection leads to poor performance due to a domain shift: CLIP was trained to match an image as a whole to a text description, without capturing the fine-grained alignment between image regions and text spans.
Ranked #11 on
Open Vocabulary Object Detection
on MSCOCO
(using extra training data)
Grounded Language-Image Pre-training
The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representation semantic-rich.
Ranked #1 on
2D Object Detection
on RF100
Learning to Generate Scene Graph from Natural Language Supervision
To bridge the gap between images and texts, we leverage an off-the-shelf object detector to identify and localize object instances, match labels of detected regions to concepts parsed from captions, and thus create "pseudo" labels for learning scene graph.
A Simple Baseline for Weakly-Supervised Scene Graph Generation
We investigate the weakly-supervised scene graph generation, which is a challenging task since no correspondence of label and object is provided.
Comprehensive Image Captioning via Scene Graph Decomposition
We address the challenging problem of image captioning by revisiting the representation of image scene graph.
