Search Results for author:
Found 24 papers, 15 papers with code
Sparse Attention Vectors: Generative Multimodal Model Features Are Discriminative Vision-Language Classifiers
Generative Large Multimodal Models (LMMs) like LLaVA and Qwen-VL excel at a wide variety of vision-language (VL) tasks such as image captioning or visual question answering.
Teaching VLMs to Localize Specific Objects from In-context Examples
In this work, we focus on the task of few-shot personalized localization, where a model is given a small set of annotated images (in-context examples) -- each with a category label and bounding box -- and is tasked with localizing the same object type in a query image.
LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content
Moreover, we introduce an efficient evaluation approach that estimates the performance of all models on the evolving benchmark using evaluations of only a subset of models.
Augmenting In-Context-Learning in LLMs via Automatic Data Labeling and Refinement
In this work, we propose Automatic Data Labeling and Refinement (ADLR), a method to automatically generate and filter demonstrations which include the above intermediate steps, starting from a small seed of manually crafted examples.
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
The recent success of interleaved Large Multimodal Models (LMMs) in few-shot learning suggests that in-context learning (ICL) with many examples can be promising for learning new tasks.
ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs
Compositional Reasoning (CR) entails grasping the significance of attributes, relations, and word order.
NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning
Language models struggle with handling numerical data and performing arithmetic operations.
Towards Multimodal In-Context Learning for Vision & Language Models
State-of-the-art Vision-Language Models (VLMs) ground the vision and the language modality primarily via projecting the vision tokens from the encoder to language-like tokens, which are directly fed to the Large Language Model (LLM) decoder.
Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs
For the visual side, we incorporate a special "SG Component" in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions that highlight different compositional aspects of the scene.
Ranked #24 on
Visual Reasoning
on Winoground
Teaching Structured Vision & Language Concepts to Vision & Language Models
Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks.
PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data
In this work, we propose an approach to leverage synthetic scene data for improving video understanding.
MAEDAY: MAE for few and zero shot AnomalY-Detection
We propose using Masked Auto-Encoder (MAE), a transformer model self-supervisedly trained on image inpainting, for anomaly detection (AD).
CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning
Our experiments show that we outperform the current SOTA method DualPrompt on established benchmarks by as much as 4. 5% in average final accuracy.
Teaching Structured Vision&Language Concepts to Vision&Language Models
Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks.
ConStruct-VL: Data-Free Continual Structured VL Concepts Learning
This leads to reasoning mistakes, which need to be corrected as they occur by teaching VL models the missing SVLC skills; often this must be done using private data where the issue was found, which naturally leads to a data-free continual (no task-id) VL learning setting.
FETA: Towards Specializing Foundation Models for Expert Task Applications
However, as we show in this paper, FMs still have poor out-of-the-box performance on expert tasks (e. g. retrieval of car manuals technical illustrations from language queries), data for which is either unseen or belonging to a long-tail part of the data distribution of the huge datasets used for FM pre-training.
Ranked #1 on
Image-to-Text Retrieval
on FETA Car-Manuals
Unsupervised Domain Generalization by Learning a Bridge Across Domains
The ability to generalize learned representations across significantly different visual domains, such as between real photos, clipart, paintings, and sketches, is a fundamental capacity of the human visual system.
CHARTER: heatmap-based multi-type chart data extraction
The digital conversion of information stored in documents is a great source of knowledge.
Detector-Free Weakly Supervised Grounding by Separation
In this work, we focus on the task of Detector-Free WSG (DF-WSG) to solve WSG without relying on a pre-trained detector.
Ranked #1 on
Phrase Grounding
on Visual Genome
DeepHist: Differentiable Joint and Color Histogram Layers for Image-to-Image Translation
Promising results are shown for the tasks of color transfer, image colorization and edges $\rightarrow$ photo, where the color distribution of the output image is controlled.
Hue-Net: Intensity-based Image-to-Image Translation with Differentiable Histogram Loss Functions
To enforce color-free similarity between the source and the output images, we define a semantic-based loss by a differentiable approximation of the MI of these images.
QANet -- Quality Assurance Network for Image Segmentation
We refer to this method as a Quality Assurance Network -- QANet.
Microscopy Cell Segmentation via Convolutional LSTM Networks
Live cell microscopy sequences exhibit complex spatial structures and complicated temporal behaviour, making their analysis a challenging task.
Ranked #1 on
Cell Segmentation
on Fluo-N2DH-SIM+
Microscopy Cell Segmentation via Adversarial Neural Networks
We present a novel method for cell segmentation in microscopy images which is inspired by the Generative Adversarial Neural Network (GAN) approach.
