Search Results for author:
Found 58 papers, 31 papers with code
Self-supervised Normality Learning and Divergence Vector-guided Model Merging for Zero-shot Congenital Heart Disease Detection in Fetal Ultrasound Videos
To address these challenges, we introduce, for the first time, a novel privacy-preserving, zero-shot CHD detection framework that formulates CHD detection as a normality modeling problem integrated with model merging.
Text-guided Synthetic Geometric Augmentation for Zero-shot 3D Understanding
Zero-shot recognition models require extensive training data for generalization.
Redefining Normal: A Novel Object-Level Approach for Multi-Object Novelty Detection
Adapting to our object-level definition of `normal', we modify knowledge distillation frameworks, where a student network learns from a pre-trained teacher network.
Learning to Ground VLMs without Forgetting
Yet, without vast amounts of spatial supervision, current Visual Language Models (VLMs) struggle at this task.
TULIP: Token-length Upgraded CLIP
By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such as retrieval and text-to-image generation.
TVBench: Redesigning Video-Language Evaluation
Large language models have demonstrated impressive performance when integrated with vision models even enabling video understanding.
Do better language models have crisper vision?
To this end, we propose the Visual Text Representation Benchmark (ViTeRB) to isolate key properties that make language models well-aligned with the visual world.
Self-Masking Networks for Unsupervised Adaptation
We validate the usefulness of learning binary masks as a fine-tuning method on 8 datasets and 3 model architectures, and we demonstrate the effectiveness of SMNs in 3 label-efficient settings.
Foundation Model or Finetune? Evaluation of few-shot semantic segmentation for river pollution
In this work, we compare the performance of FMs to finetuned pre-trained supervised models in the task of semantic segmentation on an entirely new dataset.
Rethinking Image Super-Resolution from Training Data Perspectives
In this work, we investigate the understudied effect of the training data used for image super-resolution (SR).
SelEx: Self-Expertise in Fine-Grained Generalized Category Discovery
In this paper, we address Generalized Category Discovery, aiming to simultaneously uncover novel categories and accurately classify known ones.
Near, far: Patch-ordering enhances vision foundation models' scene understanding
We introduce NeCo: Patch Neighbor Consistency, a novel self-supervised training loss that enforces patch-level nearest neighbor consistency across a student and teacher model.
Scaling Backwards: Minimal Synthetic Pre-training?
To this end, we search for a minimal, purely synthetic pre-training dataset that allows us to achieve performance similar to the 1 million images of ImageNet-1k.
SIGMA:Sinkhorn-Guided Masked Video Modeling
To tackle this, we present Sinkhorn-guided Masked Video Modelling (SIGMA), a novel video pretraining method that jointly learns the video model in addition to a target feature space using a projection network.
GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Features
These features are fed to an attention-based discriminator, which is trained to score every patch in the image.
Ranked #1 on
Anomaly Detection
on One-class CIFAR-100
No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations
This paper introduces FUNGI, Features from UNsupervised GradIents, a method to enhance the features of transformer encoders by leveraging self-supervised gradients.
Federated Learning with a Single Shared Image
To this end, in this paper, we introduce a new method that improves this knowledge distillation method to only rely on a single shared image between clients and server.
Privacy-Aware Visual Language Models
This paper aims to advance our understanding of how Visual Language Models (VLMs) handle privacy-sensitive information, a crucial concern as these technologies become integral to everyday life.
Bitune: Bidirectional Instruction-Tuning
We introduce Bitune, a method that improves instruction-tuning of pretrained decoder-only large language models, leading to consistent gains on downstream tasks.
Self-supervised visual learning in the low-data regime: a comparative evaluation
Self-Supervised Learning (SSL) is a valuable and robust training methodology for contemporary Deep Neural Networks (DNNs), enabling unsupervised pretraining on a 'pretext task' that does not require ground-truth labels/annotation.
DNA: Differentially private Neural Augmentation for contact tracing
The COVID19 pandemic had enormous economic and societal consequences.
The Common Stability Mechanism behind most Self-Supervised Learning Approaches
Last couple of years have witnessed a tremendous progress in self-supervised learning (SSL), the success of which can be attributed to the introduction of useful inductive biases in the learning process to learn meaningful visual representations while avoiding collapse.
PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs
Vision-Language Models (VLMs), such as Flamingo and GPT-4V, have shown immense potential by integrating large language models with vision systems.
Object-Centric Diffusion for Efficient Video Editing
Both techniques are readily applicable to a given video editing model without retraining, and can drastically reduce its memory and computational cost.
The LLM Surgeon
Experimentally, our method can prune rows and columns from a range of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance, and achieve state-of-the-art results in unstructured and semi-structured pruning of large language models.
Protect Your Score: Contact Tracing With Differential Privacy Guarantees
The pandemic in 2020 and 2021 had enormous economic and societal consequences, and studies show that contact tracing algorithms can be key in the early containment of the virus.
VaLID: Variable-Length Input Diffusion for Novel View Synthesis
This is because the source-view images and corresponding poses are processed separately and injected into the model at different stages.
Guided Diffusion from Self-Supervised Diffusion Features
However, recent studies have revealed that the feature representation derived from diffusion model itself is discriminative for numerous downstream tasks as well, which prompts us to propose a framework to extract guidance from, and specifically for, diffusion models.
VeRA: Vector-based Random Matrix Adaptation
Low-rank adapation (LoRA) is a popular method that reduces the number of trainable parameters when finetuning large language models, but still faces acute storage challenges when scaling to even larger models or deploying numerous per-user or per-task adapted models.
Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video
But are we making the best use of data?
Self-Supervised Open-Ended Classification with Small Visual Language Models
We present Self-Context Adaptation (SeCAt), a self-supervised approach that unlocks few-shot abilities for open-ended classification with small visual language models.
Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations
Our paper aims to address this gap by proposing a novel approach that incorporates temporal consistency in dense self-supervised learning.
Efficient Neural PDE-Solvers using Quantization Aware Training
In the past years, the application of neural networks as an alternative to classical numerical methods to solve Partial Differential Equations has emerged as a potential paradigm shift in this century-old mathematical field.
Learning to Count without Annotations
While recent supervised methods for reference-based object counting continue to improve the performance on benchmark datasets, they have to rely on small datasets due to the cost associated with manually annotating dozens of objects in images.
BISCUIT: Causal Representation Learning from Binary Interactions
Identifying the causal variables of an environment and how to intervene on them is of core value in applications such as robotics and embodied AI.
Self-Ordering Point Clouds
In this paper we address the task of finding representative subsets of points in a 3D point cloud by means of a point-wise ordering.
Towards Label-Efficient Incremental Learning: A Survey
The current dominant paradigm when building a machine learning model is to iterate over a dataset over and over until convergence.
Skip-Attention: Improving Vision Transformers by Paying Less Attention
This work aims to improve the efficiency of vision transformers (ViT).
VTC: Improving Video-Text Retrieval with User Comments
In this paper, we a) introduce a new dataset of videos, titles and comments; b) present an attention-based mechanism that allows the model to learn from sometimes irrelevant data such as comments; c) show that by using comments, our method is able to learn better, more contextualised, representations for image, video and audio representations.
Prompt Generation Networks for Input-Space Adaptation of Frozen Vision Transformers
With the introduction of the transformer architecture in computer vision, increasing model scale has been demonstrated as a clear path to achieving performance and robustness gains.
Self-Guided Diffusion Models
Diffusion models have demonstrated remarkable progress in image generation quality, especially when guidance is used to control the generative process.
Measuring the Interpretability of Unsupervised Representations via Quantized Reverse Probing
Self-supervised visual representation learning has recently attracted significant research interest.
Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systems
To address this issue, we propose iCITRIS, a causal representation learning method that allows for instantaneous effects in intervened temporal sequences when intervention targets can be observed, e. g., as actions of an agent.
Looking for a Handsome Carpenter! Debiasing GPT-3 Job Advertisements
The growing capability and availability of generative language models has enabled a wide range of new downstream tasks.
Self-Supervised Learning of Object Parts for Semantic Segmentation
However, learning dense representations is challenging, as in the unsupervised context it is not clear how to guide the model to learn representations that correspond to various potential object categories.
Ranked #5 on
Unsupervised Semantic Segmentation
on PASCAL VOC 2012 val
(using extra training data)
Less than Few: Self-Shot Video Instance Segmentation
The goal of this paper is to bypass the need for labelled examples in few-shot video understanding at run time.
CITRIS: Causal Identifiability from Temporal Intervened Sequences
Understanding the latent causal factors of a dynamical system from visual observations is considered a crucial step towards agents reasoning in complex environments.
The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Image
What can neural networks learn about the visual world when provided with only a single image as input?
PASS: An ImageNet replacement for self-supervised pretraining without humans
On the other hand, state-of-the-art pretraining is nowadays obtained with unsupervised methods, meaning that labelled datasets such as ImageNet may not be necessary, or perhaps not even optimal, for model pretraining.
Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset
In this paper, we collect hateful and non-hateful memes from Pinterest to evaluate out-of-sample performance on models pre-trained on the Facebook dataset.
Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers
In video transformers, the time dimension is often treated in the same way as the two spatial dimensions.
Ranked #16 on
Action Recognition
on EPIC-KITCHENS-100
(using extra training data)
Self-supervised object detection from audio-visual correspondence
We tackle the problem of learning object detectors without supervision.
Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning
First, for space, we show that spatial augmentations such as cropping do work well for videos too, but that previous implementations, due to the high processing and memory cost, could not do this at a scale sufficient for it to work well.
Privacy-preserving Object Detection
Privacy considerations and bias in datasets are quickly becoming high-priority issues that the computer vision community needs to face.
Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models
Using a template-based data collection pipeline, we collect 396K sentence completions made by GPT-2 and find: (i) The machine-predicted jobs are less diverse and more stereotypical for women than for men, especially for intersections; (ii) Intersectional interactions are highly relevant for occupational associations, which we quantify by fitting 262 logistic models; (iii) For most occupations, GPT-2 reflects the skewed gender and ethnicity distribution found in US Labor Bureau data, and even pulls the societally-skewed distribution towards gender parity in cases where its predictions deviate from real labor market observations.
Labelling unlabelled videos from scratch with multi-modal self-supervision
A large part of the current success of deep learning lies in the effectiveness of data -- more precisely: labelled data.
On Compositions of Transformations in Contrastive Self-Supervised Learning
In the image domain, excellent representations can be learned by inducing invariance to content-preserving transformations via noise contrastive learning.
A critical analysis of self-supervision, or what we can learn from a single image
We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels.
