Search Results for author:
Found 23 papers, 17 papers with code
Scaling Laws for Native Multimodal Models Scaling Laws for Native Multimodal Models
Building general-purpose models that can effectively perceive the world through multimodal signals has been a long-standing goal.
On Large Multimodal Models as Open-World Image Classifiers
Despite this remarkable capability, most existing studies on LMM classification performance are surprisingly limited in scope, often assuming a closed-world setting with a predefined set of categories.
FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
A key finding is that FlexTok enables next-token prediction to describe images in a coarse-to-fine "visual vocabulary", and that the number of tokens to generate depends on the complexity of the generation task.
Multimodal Autoregressive Pre-training of Large Vision Encoders
We introduce a novel method for pre-training of large-scale vision encoders.
Ranked #1 on
Image Classification
on iNaturalist
Retrieval-enriched zero-shot image classification in low-resource domains
Low-resource domains, characterized by scarce data and annotations, present significant challenges for language and visual understanding tasks, with the latter much under-explored in the literature.
Visual Scratchpads: Enabling Global Reasoning in Vision
Modern vision models have achieved remarkable success in benchmarks where local features provide critical information about the target.
Automatic benchmarking of large multimodal models via iterative experiment programming
Finally, the LLM refines the report, presenting the results to the user in natural language.
Vocabulary-free Image Classification and Semantic Segmentation
To address VIC, we propose Category Search from External Databases (CaSED), a training-free method that leverages a pre-trained vision-language model and an external database.
Continual Contrastive Spoken Language Understanding
In this paper, we investigate the problem of learning sequence-to-sequence models for spoken language understanding in a class-incremental learning (CIL) setting and we propose COCONUT, a CIL method that relies on the combination of experience replay and contrastive learning.
Semi-supervised learning made simple with self-supervised clustering
Self-supervised learning models have been shown to learn rich visual representations without requiring human annotations.
Vocabulary-free Image Classification
We thus formalize a novel task, termed as Vocabulary-free Image Classification (VIC), where we aim to assign to an input image a class that resides in an unconstrained language-induced semantic space, without the prerequisite of a known vocabulary.
Improved baselines for vision-language pre-training
Indeed, we find that a simple CLIP baseline can also be improved substantially, up to a 25% relative improvement on downstream zero-shot tasks, by using well-known training techniques that are popular in other subfields.
StyLIP: Multi-Scale Style-Conditioned Prompt Learning for CLIP-based Domain Generalization
Our method focuses on a domain-agnostic prompt learning strategy, aiming to disentangle the visual style and content information embedded in CLIP's pre-trained vision encoder, enabling effortless adaptation to novel domains during inference.
A soft nearest-neighbor framework for continual semi-supervised learning
Despite significant advances, the performance of state-of-the-art continual learning approaches hinges on the unrealistic scenario of fully labeled data.
Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss
Our method, as being based on contrastive loss between pairwise modalities, is the first attempt in MER literature.
Cultural Vocal Bursts Intensity Prediction
Multimodal Emotion Recognition
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation
This problem has been widely investigated in the research community and several Incremental Learning (IL) approaches have been proposed in the past years.
Continual Attentive Fusion for Incremental Learning in Semantic Segmentation
To fill this gap, in this paper we introduce a novel attentive feature distillation approach to mitigate catastrophic forgetting while accounting for semantic spatial- and channel-level dependencies.
Self-Supervised Models are Continual Learners
Self-supervised models have been shown to produce comparable or better visual representations than their supervised counterparts when trained offline on unlabeled data at scale.
A Unified Objective for Novel Class Discovery
In this paper, we study the problem of Novel Class Discovery (NCD).
Ranked #3 on
Novel Object Detection
on LVIS v1.0 val
Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning
This paper presents solo-learn, a library of self-supervised methods for visual representation learning.
Neighborhood Contrastive Learning for Novel Class Discovery
In this paper, we address Novel Class Discovery (NCD), the task of unveiling new classes in a set of unlabeled samples given a labeled dataset with known classes.
Online Continual Learning under Extreme Memory Constraints
Continual Learning (CL) aims to develop agents emulating the human ability to sequentially learn new tasks while being able to retain knowledge obtained from past experiences.
Supervised online diarization with sample mean loss for multi-domain data
Recently, a fully supervised speaker diarization approach was proposed (UIS-RNN) which models speakers using multiple instances of a parameter-sharing recurrent neural network.
Ranked #1 on
Speaker Diarization
on DIHARD II
