Search Results for author:
Found 33 papers, 16 papers with code
Bridging the Gap between Recognition-level Pre-training and Commonsensical Vision-language Tasks
Large-scale visual-linguistic pre-training aims to capture the generic representations from multimodal features, which are essential for downstream vision-language tasks.
DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models
Video large language models (VLLMs) have significantly advanced recently in processing complex video content, yet their inference efficiency remains constrained because of the high computational cost stemming from the thousands of visual tokens generated from the video inputs.
MM-Ego: Towards Building Egocentric Multimodal LLMs
First, as there is a lack of QA data for egocentric video understanding, we develop a data engine that efficiently generates 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long, based on human-annotated data.
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
We present MM1. 5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning.
Ranked #55 on
Visual Question Answering
on MM-Vet
JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
In this paper, we release JourneyBench, a comprehensive human-annotated benchmark of generated images designed to assess the model's fine-grained multimodal reasoning abilities across five tasks: complementary multimodal chain of thought, multi-image VQA, imaginary image captioning, VQA with hallucination triggers, and fine-grained retrieval with sample-specific distractors.
Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions
Despite the widespread adoption of Vision-Language Understanding (VLU) benchmarks such as VQA v2, OKVQA, A-OKVQA, GQA, VCR, SWAG, and VisualCOMET, our analysis reveals a pervasive issue affecting their integrity: these benchmarks contain samples where answers rely on assumptions unsupported by the provided context.
Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual encoder and failed to perform well on broader tasks.
Ranked #150 on
Visual Question Answering
on MM-Vet
LLM-based Conversational AI Therapist for Daily Functioning Screening and Psychotherapeutic Intervention via Everyday Smart Devices
Despite the global mental health crisis, access to screenings, professionals, and treatments remains high.
Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyond
Vision-language (VL) understanding tasks evaluate models' comprehension of complex visual scenes through multiple-choice questions.
Ferret: Refer and Ground Anything Anywhere at Any Granularity
We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions.
UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding
However, we find visual and textual fine-grained information, e. g., keywords in the sentence and objects in the image, can be fairly informative for semantics understanding.
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models
Specifically, IdealGPT utilizes an LLM to generate sub-questions, a VLM to provide corresponding sub-answers, and another LLM to reason to achieve the final answer.
CoBIT: A Contrastive Bi-directional Image-Text Generation Model
The field of vision and language has witnessed a proliferation of pre-trained foundation models.
Find Someone Who: Visual Commonsense Understanding in Human-Centric Grounding
We present a new commonsense task, Human-centric Commonsense Grounding, that tests the models' ability to ground individuals given the context descriptions about what happened before, and their mental/physical states or intentions.
Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense
Visual commonsense understanding requires Vision Language (VL) models to not only understand image and text but also cross-reference in-between to fully integrate and achieve comprehension of the visual scene described.
Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training
Large-scale multi-modal contrastive pre-training has demonstrated great utility to learn transferable features for a range of downstream tasks by mapping multiple modalities into a shared embedding space.
Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks
Experiments demonstrate that MAD leads to consistent gains in the low-shot, domain-shifted, and fully-supervised conditions on VCR, SNLI-VE, and VQA, achieving SOTA performance on VCR compared to other single models pretrained with image-text data.
Ranked #4 on
Visual Question Answering (VQA)
on VCR (Q-A) test
Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework
We notice that detailed local geometrical information probably is not the key to point cloud analysis -- we introduce a pure residual MLP network, called PointMLP, which integrates no sophisticated local geometrical extractors but still performs very competitively.
Ranked #4 on
Point Cloud Segmentation
on PointCloud-C
Point Cloud Segmentation
Supervised Only 3D Point Cloud Classification
CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
Experiments demonstrate that our proposed CLIP-TD leads to exceptional gains in the low-shot (up to 51. 9%) and domain-shifted (up to 71. 3%) conditions of VCR, while simultaneously improving performance under standard fully-supervised conditions (up to 2%), achieving state-of-art performance on VCR compared to other single models that are pretrained with image-text data only.
SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning
As for pre-training, a scene-graph-aware pre-training method is proposed to leverage structure knowledge extracted in the visual scene graph.
MA-CLIP: Towards Modality-Agnostic Contrastive Language-Image Pre-training
Large-scale multimodal contrastive pretraining has demonstrated great utility to support high performance in a range of downstream tasks by mapping multiple modalities into a shared embedding space.
Graph-MLP: Node Classification without Message Passing in Graph
Graph Neural Network (GNN) has been demonstrated its effectiveness in dealing with non-Euclidean structural data.
Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
Pre-trained contextual vision-and-language (V&L) models have achieved impressive performance on various benchmarks.
Learning Visual Commonsense for Robust Scene Graph Generation
Scene graph generation models understand the scene through object and predicate recognition, but are prone to mistakes due to the challenges of perception in the wild.
PointDAN: A Multi-Scale 3D Domain Adaption Network for Point Cloud Representation
Specifically, most general-purpose DA methods that struggle for global feature alignment and ignore local geometric information are not suitable for 3D domain alignment.
Ranked #1 on
Unsupervised Domain Adaptation
on PreSIL to KITTI
Multi-modality Latent Interaction Network for Visual Question Answering
The proposed module learns the cross-modality relationships between latent visual and language summarizations, which summarize visual regions and question into a small number of latent representations to avoid modeling uninformative individual region-word relations.
PointHop: An Explainable Machine Learning Method for Point Cloud Classification
In the attribute building stage, we address the problem of unordered point cloud data using a space partitioning procedure and developing a robust descriptor that characterizes the relationship between a point and its one-hop neighbor in a PointHop unit.
Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
It can robustly capture the high-level interactions between language and vision domains, thus significantly improves the performance of visual question answering.
PVRNet: Point-View Relation Neural Network for 3D Shape Recognition
More specifically, based on the relation score module, the point-single-view fusion feature is first extracted by fusing the point cloud feature and each single view feature with point-singe-view relation, then the point-multi-view fusion feature is extracted by fusing the point cloud feature and the features of different number of views with point-multi-view relation.
MeshNet: Mesh Neural Network for 3D Shape Representation
However, there is little effort on using mesh data in recent years, due to the complexity and irregularity of mesh data.
Hypergraph Neural Networks
In this paper, we present a hypergraph neural networks (HGNN) framework for data representation learning, which can encode high-order data correlation in a hypergraph structure.
PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition
With the recent proliferation of deep learning, various deep models with different representations have achieved the state-of-the-art performance.
Restricting Greed in Training of Generative Adversarial Network
Generative adversarial network (GAN) has gotten wide re-search interest in the field of deep learning.
