Search Results for author:
Found 20 papers, 10 papers with code
DocVLM: Make Your VLM an Efficient Reader
Vision-Language Models (VLMs) excel in diverse visual tasks but face challenges in document understanding, which requires fine-grained text processing.
TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models
In this paper, we focus on enhancing the first strategy by introducing a novel method, named TAP-VL, which treats OCR information as a distinct modality and seamlessly integrates it into any VL model.
Optical Character Recognition
Optical Character Recognition (OCR)
Text-to-Image Generation Via Energy-Based CLIP
For the discriminative objective, we employ contrastive adversarial loss, extending the adversarial training objective to the multimodal domain.
Adversaries With Incentives: A Strategic Alternative to Adversarial Robustness
We conduct a series of experiments that show how even mild knowledge regarding the opponent's incentives can be useful, and that the degree of potential gains depends on how these incentives relate to the structure of the learning task.
Enhancing Consistency-Based Image Generation via Adversarialy-Trained Classification and Energy-Based Discrimination
While the classifier aims to grade an image based on its assignment to a designated class, the discriminator portion of the very same network leverages the softmax values to assess the proximity of the input image to the targeted data manifold, thereby serving as an Energy-based Model.
Paint by Inpaint: Learning to Add Image Objects by Removing Them First
We address this by leveraging the insight that removing objects (Inpaint) is significantly simpler than its inverse process of adding them (Paint), attributed to the utilization of segmentation mask datasets alongside inpainting models that inpaint within these masks.
Question Aware Vision Transformer for Multimodal Reasoning
This integration results in dynamic visual features focusing on relevant image aspects to the posed question.
GRAM: Global Reasoning for Multi-Page VQA
The increasing use of transformer-based large language models brings forward the challenge of processing long sequences.
CLIPAG: Towards Generator-Free Text-to-Image Generation
Perceptually Aligned Gradients (PAG) refer to an intriguing property observed in robust image classification models, wherein their input gradients align with human perception and pose semantic meanings.
FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
Our proposed method, FuseCap, fuses the outputs of such vision experts with the original captions using a large language model (LLM), yielding comprehensive image descriptions.
Ranked #1 on
Image Captioning
on COCO Captions
(CLIPScore metric)
Class-Conditioned Transformation for Enhanced Robust Image Classification
Robust classification methods predominantly concentrate on algorithms that address a specific threat model, resulting in ineffective defenses against other threat models.
Towards Models that Can See and Read
Visual Question Answering (VQA) and Image Captioning (CAP), which are among the most popular vision-language tasks, have analogous scene-text versions that require reasoning from the text in the image.
CLIPTER: Looking at the Bigger Picture in Scene Text Recognition
Reading text in real-world scenarios often requires understanding the context surrounding it, especially when dealing with poor-quality text.
Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance
In order to obtain class-conditional generation, it was suggested to guide the diffusion process by gradients from a time-dependent classifier.
Do Perceptually Aligned Gradients Imply Adversarial Robustness?
In this work, we focus on this trait and test whether \emph{Perceptually Aligned Gradients imply Robustness}.
Threat Model-Agnostic Adversarial Defense using Diffusion Models
Deep Neural Networks (DNNs) are highly sensitive to imperceptible malicious perturbations, known as adversarial attacks.
Multimodal Semi-Supervised Learning for Text Recognition
In a novel setup, consistency is enforced on each modality separately.
Improved Image Generation via Sparsity
The interest of the deep learning community in image synthesis has grown massively in recent years.
BIGRoC: Boosting Image Generation via a Robust Classifier
The interest of the machine learning community in image synthesis has grown significantly in recent years, with the introduction of a wide range of deep generative models and means for training them.
Ranked #7 on
Image Generation
on ImageNet 128x128
Improved Image Generation via Sparse Modeling
The interest of the deep learning community in image synthesis has grown massively in recent years.
