Search Results for author:
Found 12 papers, 3 papers with code
Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models
With models getting stronger, evaluations have grown more complex, testing multiple skills in one benchmark and even in the same instance at once.
Understanding and Mitigating Compositional Issues in Text-to-Image Generative Models
We also show that re-weighting the erroneous attention contributions in CLIP can also lead to improved compositional performances, however these improvements are often less significant than those achieved by solely learning a linear projection head, highlighting erroneous attentions to be only a minor error source.
An Introduction to Vision-Language Modeling
Then, we present and discuss approaches to evaluate VLMs.
Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class
We propose a method to encode and account for diversity within a class using inferred attributes, still in the zero-shot setting without retraining.
Rethinking Artistic Copyright Infringements in the Era of Text-to-Image Generative Models
Recent text-to-image generative models such as Stable Diffusion are extremely adept at mimicking and generating copyrighted content, raising concerns amongst artists that their unique styles may be improperly copied.
PRIME: Prioritizing Interpretability in Failure Mode Extraction
To improve on these shortcomings, we propose a novel approach that prioritizes interpretability in this problem: we start by obtaining human-understandable concepts (tags) of images in the dataset and then analyze the model's behavior based on the presence or absence of combinations of these tags.
Text-To-Concept (and Back) via Cross-Model Alignment
We observe that the mapping between an image's representation in one model to its representation in another can be learned surprisingly well with just a linear layer, even across diverse models.
Data-Centric Debugging: mitigating model failures via targeted data collection
Our Data-Centric Debugging (DCD) framework carefully creates a debug-train set by selecting images from $\mathcal{F}$ that are perceptually similar to the images in $\mathcal{E}_{sample}$.
Explicit Tradeoffs between Adversarial and Natural Distributional Robustness
In this setting, through theoretical and empirical analysis, we show that (i) adversarial training with $\ell_1$ and $\ell_2$ norms increases the model reliance on spurious features; (ii) For $\ell_\infty$ adversarial training, spurious reliance only occurs when the scale of the spurious features is larger than that of the core features; (iii) adversarial training can have an unintended consequence in reducing distributional robustness, specifically when spurious correlations are changed in the new test domain.
Core Risk Minimization using Salient ImageNet
Deep neural networks can be unreliable in the real world especially when they heavily use spurious features for their predictions.
A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes
While datasets with single-label supervision have propelled rapid advances in image classification, additional annotations are necessary in order to quantitatively assess how models make predictions.
Sample Efficient Detection and Classification of Adversarial Attacks via Self-Supervised Embeddings
In this paper, we propose a self-supervised method to detect adversarial attacks and classify them to their respective threat models, based on a linear model operating on the embeddings from a pre-trained self-supervised encoder.
