Search Results for author:
Found 66 papers, 38 papers with code
LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
Recent progress in Multimodal Large Language Models (MLLMs) has highlighted the critical roles of both the visual backbone and the underlying language model.
Hyperbolic Safety-Aware Vision-Language Models
We propose to encode safe and unsafe content as an entailment hierarchy, where both are placed in different regions of hyperbolic space.
Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval
Cross-modal retrieval is gaining increasing efficacy and interest from the research community, thanks to large-scale training, novel architectural and learning designs, and its application in LLMs and multimodal LLMs.
Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries
This progress is largely driven by the effective alignment between visual data and the language space of MLLMs.
Causal Graphical Models for Vision-Language Compositional Understanding
Recent work has empirically shown that Vision-Language Models (VLMs) struggle to fully understand the compositional properties of the human language, usually modeling an image caption as a "bag of words".
Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis
The task of image captioning demands an algorithm to generate natural language descriptions of visual inputs.
Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation
At training time, we exploit the attention maps of DINOv2 to selectively align local visual patches with textual embeddings.
Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering
Multimodal LLMs (MLLMs) are the natural extension of large language models to handle multimodal inputs, combining text and image data.
Personalized Instance-based Navigation Toward User-Specific Objects in Realistic Environments
In the last years, the research interest in visual navigation towards objects in indoor environments has grown significantly.
Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training
Metrics can indeed play a key role in the fine-tuning stage of captioning models, ultimately enhancing the quality of the generated captions.
Optimizing Resource Consumption in Diffusion Models through Hallucination Early Detection
Diffusion models have significantly advanced generative AI, but they encounter difficulties when generating complex combinations of multiple objects.
Fluent and Accurate Image Captioning with a Self-Trained Reward Model
Fine-tuning image captioning models with hand-crafted rewards like the CIDEr metric has been a classical strategy for promoting caption quality at the sequence level.
Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization
However, when attempting to optimize modern and higher-quality metrics like CLIP-Score and PAC-Score, this training method often encounters instability and fails to acquire the genuine descriptive capabilities needed to produce fluent and informative captions.
BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues
Effectively aligning with human judgment when evaluating machine-generated image captions represents a complex yet intriguing challenge.
Contrasting Deepfakes Diffusion via Contrastive Learning and Global-Local Similarities
To sustain the training of our model, we generate a comprehensive dataset that focuses on images generated by diffusion models and encompasses a collection of 9. 2 million images produced by using four different generators.
Towards Retrieval-Augmented Architectures for Image Captioning
The objective of image captioning models is to bridge the gap between the visual and linguistic modalities by generating natural language descriptions that accurately reflect the content of input images.
Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
Multimodal LLMs are the natural evolution of LLMs, and enlarge their capabilities so as to work beyond the pure textual modality.
AIGeN: An Adversarial Approach for Instruction Generation in VLN
VLN is a challenging task that involves an agent following human instructions and navigating in a previously unknown environment to reach a specified goal.
Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation
Open-vocabulary semantic segmentation aims at segmenting arbitrary categories expressed in textual form.
Open Vocabulary Semantic Segmentation
Open-Vocabulary Semantic Segmentation
+1
Mapping High-level Semantic Regions in Indoor Environments without Object Recognition
Robots require a semantic understanding of their surroundings to operate in an efficient and explainable way in human environments.
The Revolution of Multimodal Large Language Models: A Survey
Connecting text and visual modalities plays an essential role in generative intelligence.
Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models
We show how this can be done by fine-tuning a CLIP model on synthetic data obtained from a large language model trained to convert between safe and unsafe sentences, and a text-to-image generator.
With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning
Image captioning, like many tasks involving vision and language, currently relies on Transformer-based architectures for extracting the semantics in an image and translating it into linguistically coherent descriptions.
Let's ViCE! Mimicking Human Cognitive Behavior in Image Generation Evaluation
Research in Image Generation has recently made significant progress, particularly boosted by the introduction of Vision-Language models which are able to produce high-quality visual content based on textual inputs.
Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training
The use of self-supervised pre-training has emerged as a promising approach to enhance the performance of many different visual tasks.
Multi-Class Unlearning for Image Classification via Weight Filtering
Machine Unlearning is an emerging paradigm for selectively removing the impact of training datapoints from a network.
Evaluating Synthetic Pre-Training for Handwriting Processing Tasks
In this work, we explore massive pre-training on synthetic word images for enhancing the performance on four benchmark downstream handwriting analysis tasks.
Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images
Recent advancements in diffusion models have enabled the generation of realistic deepfakes from textual prompts in natural language.
Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation
The CLIP model has been recently proven to be very effective for a variety of cross-modal tasks, including the evaluation of captions generated from vision-and-language architectures.
Embodied Agents for Efficient Exploration and Smart Scene Description
The development of embodied agents that can communicate with humans in natural language has gained increasing interest over the last years, as it facilitates the diffusion of robotic platforms in human-populated environments.
Boosting Modern and Historical Handwritten Text Recognition with Deformable Convolutions
Handwritten Text Recognition (HTR) in free-layout pages is a challenging image understanding task that can provide a relevant boost to the digitization of handwritten documents and reuse of their content.
The LAM Dataset: A Novel Benchmark for Line-Level Handwritten Text Recognition
With the aim of fostering the research on this topic, in this paper we present the Ludovico Antonio Muratori (LAM) dataset, a large line-level HTR dataset of Italian ancient manuscripts edited by a single author over 60 years.
ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval
In literature, this task is often used as a pre-training objective to forge architectures able to jointly deal with images and texts.
Ranked #22 on
Cross-Modal Retrieval
on COCO 2014
(using extra training data)
Retrieval-Augmented Transformer for Image Captioning
In this paper, we investigate the development of an image captioning approach with a kNN memory, with which knowledge can be retrieved from an external corpus to aid the generation process.
Embodied Navigation at the Art Gallery
This feature is challenging for occupancy-based agents which are usually trained in crowded domestic environments with plenty of occupancy information.
Spot the Difference: A Novel Task for Embodied Agents in Changing Environments
To make a step towards this setting, we propose Spot the Difference: a novel task for Embodied AI where the agent has access to an outdated map of the environment and needs to recover the correct layout in a fixed time budget.
CaMEL: Mean Teacher Learning for Image Captioning
Describing images in natural language is a fundamental step towards the automatic modeling of connections between the visual and textual modalities.
Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets
This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources, containing both human-annotated and web-collected captions.
Focus on Impact: Indoor Exploration with Intrinsic Motivation
The proposed exploration approach outperforms DRL-based competitors relying on intrinsic rewards and surpasses the agents trained with a dense extrinsic reward computed with the environment layouts.
Working Memory Connections for LSTM
Numerical results suggest that the cell state contains useful information that is worth including in the gate structure.
From Show to Tell: A Survey on Deep Learning-based Image Captioning
Starting from 2015 the task has generally been addressed with pipelines composed of a visual encoder and a language model for text generation.
Learning to Select: A Fully Attentive Approach for Novel Object Captioning
In this paper, we present a novel approach for NOC that learns to select the most relevant objects of an image, regardless of their adherence to the training set, and to constrain the generative process of a language model accordingly.
Out of the Box: Embodied Navigation in the Real World
In this work, we detail how to transfer the knowledge acquired in simulation into the real world.
Revisiting The Evaluation of Class Activation Mapping for Explainability: A Novel Metric and Experimental Analysis
As the request for deep learning solutions increases, the need for explainability is even more fundamental.
RMS-Net: Regression and Masking for Soccer Event Spotting
The recently proposed action spotting task consists in finding the exact timestamp in which an event occurs.
Ranked #1 on
Action Spotting
on SoccerNet
Inter-Homines: Distance-Based Risk Estimation for Human Safety
In this document, we report our proposal for modeling the risk of possible contagiousity in a given area monitored by RGB cameras where people freely move and interact.
Explore and Explain: Self-supervised Navigation and Recounting
In this paper, we devise a novel embodied setting in which an agent needs to explore a previously unknown environment while recounting what it sees during the path.
A Novel Attention-based Aggregation Function to Combine Vision and Language
The joint understanding of vision and language has been recently gaining a lot of attention in both the Computer Vision and Natural Language Processing communities, with the emergence of tasks such as image captioning, image-text matching, and visual question answering.
Meshed-Memory Transformer for Image Captioning
Transformer-based architectures represent the state of the art in sequence modeling tasks like machine translation and language understanding.
Ranked #2 on
Image Captioning
on MS COCO
Video action detection by learning graph-based spatio-temporal interactions
Action Detection is a complex task that aims to detect and classify human actions in video clips.
Multimodal Attention Networks for Low-Level Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) is a challenging task in which an agent needs to follow a language-specified path to reach a target destination.
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability
The ability to generate natural language explanations conditioned on the visual perception is a crucial step towards autonomous agents which can explain themselves and communicate with humans.
Artpedia
As vision and language techniques are widely applied to realistic images , there is a growing interest in designing visual-semantic models suitable for more complex and challenging scenarios.
Embodied Vision-and-Language Navigation with Dynamic Convolutional Filters
In Vision-and-Language Navigation (VLN), an embodied agent needs to reach a target destination with the only guidance of a natural language instruction.
A Deep Learning based approach to VM behavior identification in cloud systems
The experiments compare our proposal with state-of-the-art solutions available in literature, demonstrating that our proposal achieve better performance.
M-VAD Names: a Dataset for Video Captioning with Naming
Current movie captioning architectures are not capable of mentioning characters with their proper name, replacing them with a generic "someone" tag.
Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions
Current captioning approaches can describe images using black-box architectures whose behavior is hardly controllable and explainable from the exterior.
Art2Real: Unfolding the Reality of Artworks via Semantically-Aware Image-to-Image Translation
The applicability of computer vision to real paintings and artworks has been rarely investigated, even though a vast heritage would greatly benefit from techniques which can understand and process data from the artistic domain.
LAMV: Learning to Align and Match Videos With Kernelized Temporal Layers
This paper considers a learnable approach for comparing and aligning videos.
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention
Image captioning has been recently gaining a lot of attention thanks to the impressive achievements shown by deep captioning architectures, which combine Convolutional Neural Networks to extract image representations, and Recurrent Neural Networks to generate the corresponding captions.
Ranked #2 on
Image Captioning
on Flickr30k Captions test
(using extra training data)
Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model
Data-driven saliency has recently gained a lot of attention thanks to the use of Convolutional Neural Networks for predicting gaze fixations.
Hierarchical Boundary-Aware Neural Encoder for Video Captioning
The use of Recurrent Neural Networks for video captioning has recently gained a lot of attention, since they can be used both to encode the input video and to generate the corresponding description.
Recognizing and Presenting the Storytelling Video Structure with Deep Multimodal Networks
This paper presents a novel approach for temporal and semantic segmentation of edited videos into meaningful segments, from the point of view of the storytelling structure.
A Deep Multi-Level Network for Saliency Prediction
Current state of the art models for saliency prediction employ Fully Convolutional networks that perform a non-linear combination of features extracted from the last convolutional layer to predict saliency maps.
Scene-driven Retrieval in Edited Videos using Aesthetic and Semantic Deep Features
This paper presents a novel retrieval pipeline for video collections, which aims to retrieve the most significant parts of an edited video for a given query, and represent them with thumbnails which are at the same time semantically meaningful and aesthetically remarkable.
A Deep Siamese Network for Scene Detection in Broadcast Videos
We present a model that automatically divides broadcast videos into coherent scenes by learning a distance measure between shots.
