Search Results for author:
Found 76 papers, 43 papers with code
Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Models
Multilingual vision-language models promise universal image-text retrieval, yet their social biases remain under-explored.
Temporal Score Analysis for Understanding and Correcting Diffusion Artifacts
Visual artifacts remain a persistent challenge in diffusion models, even with training on massive datasets.
AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data
Two key challenges are identified in this fine-tuning paradigm, 1) the low quality of synthetic data, which can still happen even with advanced generative models, and 2) the domain and bias gap between real and synthetic data.
P-TAME: Explain Any Image Classifier with Trained Perturbations
The adoption of Deep Neural Networks (DNNs) in critical fields where predictions need to be accompanied by justifications is hindered by their inherent black-box nature.
A Comprehensive Social Bias Audit of Contrastive Vision Language Models
In the domain of text-to-image generative models, biases inherent in training datasets often propagate into generated content, posing significant ethical challenges, particularly in socially sensitive contexts.
VidCtx: Context-aware Video Question Answering with Image Models
To address those shortcomings, in this paper, we introduce VidCtx, a novel training-free VideoQA framework which integrates both modalities, i. e. both visual information from input frames and textual descriptions of others frames that give the appropriate context.
Ranked #2 on
Zero-Shot Video Question Answer
on STAR Benchmark
Multimodal Outer Arithmetic Block Dual Fusion of Whole Slide Images and Omics Data for Precision Oncology
In this paper, we propose the use of omic embeddings during early and late fusion to capture complementary information from local (patch-level) to global (slide-level) interactions, boosting performance through multimodal integration.
ReWind: Understanding Long Videos with Instructed Learnable Memory
We empirically demonstrate ReWind's superior performance in visual question answering (VQA) and temporal grounding tasks, surpassing previous methods on long video benchmarks.
CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition
Privacy issue is a main concern in developing face recognition techniques.
Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkit
In this paper, we introduce Behavior4All, a comprehensive, open-source toolkit for in-the-wild facial behavior analysis, integrating Face Localization, Valence-Arousal Estimation, Basic Expression Recognition and Action Unit Detection, all within a single framework.
MM2Latent: Text-to-facial image generation and editing in GANs with multimodal assistance
In this paper, we propose a practical framework - MM2Latent - for multimodal image generation and editing.
CLIPCleaner: Cleaning Noisy Labels with CLIP
This has the advantage that the sample selection is decoupled from the in-training model and that the sample selection is aware of the semantic and visual similarities between the classes due to the way that CLIP is trained.
Are CLIP features all you need for Universal Synthetic Image Origin Attribution?
The steady improvement of Diffusion Models for visual synthesis has given rise to many new and interesting use cases of synthetic images but also has raised concerns about their potential abuse, which poses significant societal threats.
Get Confused Cautiously: Textual Sequence Memorization Erasure with Selective Entropy Maximization
To achieve a better trade-off between the effectiveness of TSM erasure and model utility in LLMs, our paper proposes a new framework based on Entropy Maximization with Selective Optimization (EMSO), where the updated weights are chosen with a novel contrastive gradient metric without any participation of additional model or data.
Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges
Multimodal machine learning (MML) is rapidly reshaping the way mental-health disorders are detected, characterized, and longitudinally monitored.
Efficient Unsupervised Visual Representation Learning with Explicit Cluster Balancing
We conduct extensive experiments to evaluate our approach and demonstrate that ExCB: a) achieves state-of-the-art results with significantly reduced resource requirements compared to previous works, b) is fully online, and therefore scalable to large datasets, and c) is stable and effective even with very small batch sizes.
FairCoT: Enhancing Fairness in Diffusion Models via Chain of Thought Reasoning of Multimodal Language Models
In the domain of text-to-image generative models, biases inherent in training datasets often propagate into generated content, posing significant ethical challenges, particularly in socially sensitive contexts.
Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfer
Current facial expression recognition (FER) models are often designed in a supervised learning manner and thus are constrained by the lack of large-scale facial expression images with high-quality annotations.
FashionSD-X: Multimodal Fashion Garment Synthesis using Latent Diffusion
The rapid evolution of the fashion industry increasingly intersects with technological advancements, particularly through the integration of generative AI.
VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning
In the first stage, we propose prompting VLLMs to generate descriptions in natural language of the subject's apparent emotion relative to the visual context.
Ranked #1 on
Emotion Recognition in Context
on EMOTIC
DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactment
To this end, in this paper we present DiffusionAct, a novel method that leverages the photo-realistic image generation of diffusion models to perform neural face reenactment.
LAFS: Landmark-based Facial Self-supervised Learning for Face Recognition
This enables our method - namely LAndmark-based Facial Self-supervised learning LAFS), to learn key representation that is more critical for face recognition.
MOAB: Multi-Modal Outer Arithmetic Block For Fusion Of Histopathological Images And Genetic Data For Brain Tumor Grading
We propose a novel Multi-modal Outer Arithmetic Block (MOAB) based on arithmetic operations to combine latent representations of the different modalities for predicting the tumor grade (Grade \rom{2}, \rom{3} and \rom{4}).
FOAA: Flattened Outer Arithmetic Attention For Multimodal Tumor Classification
Fusion of multimodal healthcare data holds great promise to provide a holistic view of a patient's health, taking advantage of the complementarity of different modalities while leveraging their correlation.
Self-Supervised Facial Representation Learning with Facial Region Awareness
Recent efforts toward this goal are limited to treating each face image as a whole, i. e., learning consistent facial representations at the image-level, which overlooks the consistency of local facial representations (i. e., facial regions like eyes, nose, etc).
Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
The Mixture of Experts (MoE) paradigm provides a powerful way to decompose dense layers into smaller, modular computations often more amenable to human interpretation, debugging, and editability.
One-shot Neural Face Reenactment via Finding Directions in GAN's Latent Space
Moreover, we show that by embedding real images in the GAN latent space, our method can be successfully used for the reenactment of real-world faces.
Improving Fairness using Vision-Language Driven Image Augmentation
These paths are then applied to augment images to improve the fairness of a given dataset.
EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognition
To test this, we evaluate using zero-shot classification of the model trained on sample-level descriptions on four popular dynamic FER datasets.
Ranked #1 on
Zero-Shot Facial Expression Recognition
on MAFW
A Simple Baseline for Knowledge-Based Visual Question Answering
This paper is on the problem of Knowledge-Based Visual Question Answering (KB-VQA).
Ranked #5 on
Visual Question Answering (VQA)
on A-OKVQA
(DA VQA Score metric)
Prompting Visual-Language Models for Dynamic Facial Expression Recognition
For the visual part, based on the CLIP image encoder, a temporal model consisting of several Transformer encoders is introduced for extracting temporal facial expression features, and the final feature embedding is obtained as a learnable "class" token.
Dynamic Facial Expression Recognition
Facial Expression Recognition
+1
Self-Supervised Representation Learning with Cross-Context Learning between Global and Hypercolumn Features
Inspired by cross-modality learning, we extend this existing framework that only learns from global features by encouraging the global features and intermediate layer features to learn from each other.
Aligned Unsupervised Pretraining of Object Detectors with Self-training
This results in complex pipelines and a task gap between the pretraining and the downstream task.
HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces
In this paper, we present our method for neural face reenactment, called HyperReenact, that aims to generate realistic talking head images of a source identity, driven by a target facial pose.
Parts of Speech-Grounded Subspaces in Vision-Language Models
Latent image representations arising from vision-language models have proved immensely useful for a variety of downstream tasks.
Self-Supervised Video Similarity Learning
We introduce S$^2$VS, a video similarity learning approach with self-supervision.
Ranked #1 on
Video Retrieval
on FIVR-200K
DivClust: Controlling Diversity in Deep Clustering
Clustering has been a major research topic in the field of machine learning, one to which Deep Learning has recently been applied with significant success.
MaskCon: Masked Contrastive Learning for Coarse-Labelled Dataset
More specifically, within the contrastive learning framework, for each sample our method generates soft-labels with the aid of coarse labels against other samples and another augmented view of the sample in question.
Ranked #1 on
Learning with coarse labels
on cifar100
Attribute-preserving Face Dataset Anonymization via Latent Code Optimization
By optimizing the latent codes directly, we ensure both that the identity is of a desired distance away from the original (with an identity obfuscation loss), whilst preserving the facial attributes (using a novel feature-matching loss in FaRL's deep feature space).
MMA-MRNNet: Harnessing Multiple Models of Affect and Dynamic Masked RNN for Precise Facial Expression Intensity Estimation
This paper presents MMA-MRNNet, a novel deep learning architecture for dynamic multi-output Facial Expression Intensity Estimation (FEIE) from video data.
Motor Imagery Decoding Using Ensemble Curriculum Learning and Collaborative Training
The first loss applies curriculum learning, forcing each feature extractor to specialize to a subset of the training subjects and promoting feature diversity.
StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment
In this paper we address the problem of neural face reenactment, where, given a pair of a source and a target facial image, we need to transfer the target's pose (defined as the head pose and its facial expressions) to the source image, by preserving at the same time the source's identity characteristics (e. g., facial shape, hair style, etc), even in the challenging case where the source and the target faces belong to different identities.
Capsule Network based Contrastive Learning of Unsupervised Visual Representations
Capsule Networks have shown tremendous advancement in the past decade, outperforming the traditional CNNs in various task due to it's equivariant properties.
Adaptive Soft Contrastive Learning
Self-supervised learning has recently achieved great success in representation learning without human annotations.
Learning from Label Relationships in Human Affect
In the case of affect recognition, we outperform previous vision-based methods in terms of CCC on both the OMG and the AMIGOS datasets.
Ranked #1 on
Continuous Affect Estimation
on AMIGOS
Summarizing Videos using Concentrated Attention and Considering the Uniqueness and Diversity of the Video Frames
Instead of simply modeling the frames' dependencies based on global attention, our method integrates a concentrated attention mechanism that is able to focus on non-overlapping blocks in the main diagonal of the attention matrix, and to enrich the existing information by extracting and exploiting knowledge about the uniqueness and diversity of the associated frames of the video.
Ranked #5 on
Unsupervised Video Summarization
on SumMe
ContraCLIP: Interpretable GAN generation driven by pairs of contrasting sentences
This work addresses the problem of discovering non-linear interpretable paths in the latent space of pre-trained GANs in a model-agnostic manner.
PandA: Unsupervised Learning of Parts and Appearances in the Feature Maps of GANs
Recent advances in the understanding of Generative Adversarial Networks (GANs) have led to remarkable progress in visual editing and synthesis tasks, capitalizing on the rich semantics that are embedded in the latent spaces of pre-trained GANs.
Combining Global and Local Attention with Positional Encoding for Video Summarization
This paper presents a new method for supervised video summarization.
Ranked #1 on
Video Summarization
on SumMe
Tensor Component Analysis for Interpreting the Latent Space of GANs
This paper addresses the problem of finding interpretable directions in the latent space of pre-trained Generative Adversarial Networks (GANs) to facilitate controllable image synthesis.
SSR: An Efficient and Robust Framework for Learning with Unknown Label Noise
Under this setting, unlike previous methods that often introduce multiple assumptions and lead to complex solutions, we propose a simple, efficient and robust framework named Sample Selection and Relabelling(SSR), that with a minimal number of hyperparameters achieves SOTA results in various conditions.
Ranked #1 on
Image Classification
on CIFAR-10 (with noisy labels)
WarpedGANSpace: Finding non-linear RBF paths in GAN latent space
This work addresses the problem of discovering, in an unsupervised manner, interpretable paths in the latent space of pretrained GANs, so as to provide an intuitive and easy way of controlling the underlying generative factors.
DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval
In this work, we propose a Knowledge Distillation framework, called Distill-and-Select (DnS), that starting from a well-performing fine-grained Teacher Network learns: a) Student Networks at different retrieval performance and computational efficiency trade-offs and b) a Selector Network that at test time rapidly directs samples to the appropriate student to maintain both high retrieval performance and high computational efficiency.
Ranked #2 on
Video Retrieval
on FIVR-200K
Few-Shot Action Localization without Knowing Boundaries
Learning to localize actions in long, cluttered, and untrimmed videos is a hard task, that in the literature has typically been addressed assuming the availability of large amounts of annotated training samples for each class -- either in a fully-supervised setting, where action boundaries are known, or in a weakly-supervised setting, where only class labels are known for each video.
Relationship-based Neural Baby Talk
Understanding interactions between objects in an image is an important element for generating captions.
Uncertainty Propagation in Convolutional Neural Networks: Technical Report
In this technical report we study the problem of propagation of uncertainty (in terms of variances of given uni-variate normal random variables) through typical building blocks of a Convolutional Neural Network (CNN).
Video Summarization Using Deep Neural Networks: A Survey
Video summarization technologies aim to create a concise and complete synopsis by selecting the most informative parts of the video content.
AC-SUM-GAN: Connecting Actor-Critic and Generative Adversarial Networks for Unsupervised Video Summarization
This paper presents a new method for unsupervised video summarization.
Ranked #3 on
Unsupervised Video Summarization
on TvSum
Generative Adversarial Network
Unsupervised Video Summarization
Temporal Action Localization with Variance-Aware Networks
Results in the action localization problem show that the incorporation of second order statistics improves over the baseline network, and that VANp surpasses the accuracy of virtually all other two-stage networks without involving any additional parameters.
Boundary Uncertainty in a Single-Stage Temporal Action Localization Network
We use two uncertainty-aware boundary regression losses: first, the Kullback-Leibler divergence between the ground truth location of the boundary and the Gaussian modeling the prediction of the boundary and second, the expectation of the $\ell_1$ loss under the same Gaussian.
Unsupervised Video Summarization via Attention-Driven Adversarial Learning
Experimental evaluation on two datasets (SumMe and TVSum) documents the contribution of the attention auto-encoder to faster and more stable training of the model, resulting in a significant performance improvement with respect to the original model and demonstrating the competitiveness of the proposed SUM-GAN-AAE against the state of the art.
Ranked #6 on
Unsupervised Video Summarization
on TvSum
A Stepwise, Label-based Approach for Improving the Adversarial Training in Unsupervised Video Summarization
In this paper we present our work on improving the efficiency of adversarial training for unsupervised video summarization.
Ranked #5 on
Unsupervised Video Summarization
on TvSum
ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning
Subsequently, the similarity matrix between all video frames is fed to a four-layer CNN, and then summarized using Chamfer Similarity (CS) into a video-to-video similarity score -- this avoids feature aggregation before the similarity calculation between videos and captures the temporal similarity patterns between matching frame sequences.
Ranked #5 on
Video Retrieval
on FIVR-200K
TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition
At the heart of our network is a meta-learning approach that learns to compare representations of variable temporal length, that is, either two videos of different length (in the case of few-shot action recognition) or a video and a semantic representation such as word vector (in the case of zero-shot action recognition).
Ranked #8 on
Few Shot Action Recognition
on Kinetics-100
Exploring Feature Representation and Training strategies in Temporal Action Localization
Temporal action localization has recently attracted significant interest in the Computer Vision community.
Registration-free Face-SSD: Single shot analysis of smiles, facial attributes, and affect in the wild
In this paper, we present a novel single shot face-related task analysis method, called Face-SSD, for detecting faces and for performing various face-related (classification/regression) tasks including smile recognition, face attribute prediction and valence-arousal estimation in the wild.
FIVR: Fine-grained Incident Video Retrieval
To create the dataset, we devise a process for the collection of YouTube videos based on major news events from recent years crawled from Wikipedia and deploy a retrieval pipeline for the automatic selection of query videos based on their estimated suitability as benchmarks.
SchiNet: Automatic Estimation of Symptoms of Schizophrenia from Facial Behaviour Analysis
Patients with schizophrenia often display impairments in the expression of emotion and speech and those are observed in their facial behaviour.
Semi-supervised Fisher vector network
In this work we explore how the architecture proposed in [8], which expresses the processing steps of the classical Fisher vector pipeline approaches, i. e. dimensionality reduction by principal component analysis (PCA) projection, Gaussian mixture model (GMM) and Fisher vector descriptor extraction as network layers, can be modified into a hybrid network that combines the benefits of both unsupervised and supervised training methods, resulting in a model that learns a semi-supervised Fisher vector descriptor of the input data.
Deep Globally Constrained MRFs for Human Pose Estimation
We overcome this by introducing a Markov Random Field (MRF)-based spatial model network between the coarse and the refinement model that introduces geometric constraints on the relative locations of the body joints.
Discriminative convolutional Fisher vector network for action recognition
In this work we propose a novel neural network architecture for the problem of human action recognition in videos.
Unsupervised convolutional neural networks for motion estimation
In this paper, we propose a direct method and train a Convolutional Neural Network (CNN) that when, at test time, is given a pair of images as input it produces a dense motion field F at its output layer.
Learning to detect video events from zero or very few video examples
In this work we deal with the problem of high-level event detection in video.
Face Alignment Assisted by Head Pose Estimation
In this paper we propose a supervised initialization scheme for cascaded face alignment based on explicit head pose estimation.
Linear Maximum Margin Classifier for Learning from Uncertain Data
In this paper, we propose a maximum margin classifier that deals with uncertainty in data input.
Mirror, mirror on the wall, tell me, is the error small?
Our experiments lead to several interesting findings: 1) Surprisingly, most of state of the art methods struggle to preserve the mirror symmetry, despite the fact that they do have very similar overall performance on the original and mirror images; 2) the low mirrorability is not caused by training or testing sample bias - all algorithms are trained on both the original images and their mirrored versions; 3) the mirror error is strongly correlated to the localization/alignment error (with correlation coefficients around 0. 7).
