Search Results for author:
Found 56 papers, 35 papers with code
VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning
In the first stage, we propose prompting VLLMs to generate descriptions in natural language of the subject's apparent emotion relative to the visual context.
Ranked #1 on
Emotion Recognition in Context
on EMOTIC
DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactment
To this end, in this paper we present DiffusionAct, a novel method that leverages the photo-realistic image generation of diffusion models to perform neural face reenactment.
LAFS: Landmark-based Facial Self-supervised Learning for Face Recognition
This enables our method - namely LAndmark-based Facial Self-supervised learning LAFS), to learn key representation that is more critical for face recognition.
MOAB: Multi-Modal Outer Arithmetic Block For Fusion Of Histopathological Images And Genetic Data For Brain Tumor Grading
We propose a novel Multi-modal Outer Arithmetic Block (MOAB) based on arithmetic operations to combine latent representations of the different modalities for predicting the tumor grade (Grade \rom{2}, \rom{3} and \rom{4}).
FOAA: Flattened Outer Arithmetic Attention For Multimodal Tumor Classification
Fusion of multimodal healthcare data holds great promise to provide a holistic view of a patient's health, taking advantage of the complementarity of different modalities while leveraging their correlation.
Self-Supervised Facial Representation Learning with Facial Region Awareness
Recent efforts toward this goal are limited to treating each face image as a whole, i. e., learning consistent facial representations at the image-level, which overlooks the consistency of local facial representations (i. e., facial regions like eyes, nose, etc).
Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
The Mixture of Experts (MoE) paradigm provides a powerful way to decompose inscrutable dense layers into smaller, modular computations often more amenable to human interpretation, debugging, and editability.
One-shot Neural Face Reenactment via Finding Directions in GAN's Latent Space
Moreover, we show that by embedding real images in the GAN latent space, our method can be successfully used for the reenactment of real-world faces.
Improving Fairness using Vision-Language Driven Image Augmentation
These paths are then applied to augment images to improve the fairness of a given dataset.
EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognition
To test this, we evaluate using zero-shot classification of the model trained on sample-level descriptions on four popular dynamic FER datasets.
Ranked #1 on
Zero-Shot Facial Expression Recognition
on MAFW
A Simple Baseline for Knowledge-Based Visual Question Answering
This paper is on the problem of Knowledge-Based Visual Question Answering (KB-VQA).
Ranked #5 on
Visual Question Answering (VQA)
on A-OKVQA
(DA VQA Score metric)
Prompting Visual-Language Models for Dynamic Facial Expression Recognition
For the visual part, based on the CLIP image encoder, a temporal model consisting of several Transformer encoders is introduced for extracting temporal facial expression features, and the final feature embedding is obtained as a learnable "class" token.
Dynamic Facial Expression Recognition
Facial Expression Recognition
+1
Self-Supervised Representation Learning with Cross-Context Learning between Global and Hypercolumn Features
Inspired by cross-modality learning, we extend this existing framework that only learns from global features by encouraging the global features and intermediate layer features to learn from each other.
SimDETR: Simplifying self-supervised pretraining for DETR
DETR-based object detectors have achieved remarkable performance but are sample-inefficient and exhibit slow convergence.
HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces
In this paper, we present our method for neural face reenactment, called HyperReenact, that aims to generate realistic talking head images of a source identity, driven by a target facial pose.
Parts of Speech-Grounded Subspaces in Vision-Language Models
Latent image representations arising from vision-language models have proved immensely useful for a variety of downstream tasks.
Self-Supervised Video Similarity Learning
We introduce S$^2$VS, a video similarity learning approach with self-supervision.
Ranked #1 on
Video Retrieval
on FIVR-200K
DivClust: Controlling Diversity in Deep Clustering
Clustering has been a major research topic in the field of machine learning, one to which Deep Learning has recently been applied with significant success.
MaskCon: Masked Contrastive Learning for Coarse-Labelled Dataset
More specifically, within the contrastive learning framework, for each sample our method generates soft-labels with the aid of coarse labels against other samples and another augmented view of the sample in question.
Ranked #1 on
Learning with coarse labels
on cifar100
Attribute-preserving Face Dataset Anonymization via Latent Code Optimization
By optimizing the latent codes directly, we ensure both that the identity is of a desired distance away from the original (with an identity obfuscation loss), whilst preserving the facial attributes (using a novel feature-matching loss in FaRL's deep feature space).
Motor Imagery Decoding Using Ensemble Curriculum Learning and Collaborative Training
The first loss applies curriculum learning, forcing each feature extractor to specialize to a subset of the training subjects and promoting feature diversity.
StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment
In this paper we address the problem of neural face reenactment, where, given a pair of a source and a target facial image, we need to transfer the target's pose (defined as the head pose and its facial expressions) to the source image, by preserving at the same time the source's identity characteristics (e. g., facial shape, hair style, etc), even in the challenging case where the source and the target faces belong to different identities.
Capsule Network based Contrastive Learning of Unsupervised Visual Representations
Capsule Networks have shown tremendous advancement in the past decade, outperforming the traditional CNNs in various task due to it's equivariant properties.
Adaptive Soft Contrastive Learning
Self-supervised learning has recently achieved great success in representation learning without human annotations.
Learning from Label Relationships in Human Affect
In the case of affect recognition, we outperform previous vision-based methods in terms of CCC on both the OMG and the AMIGOS datasets.
Ranked #1 on
Continuous Affect Estimation
on AMIGOS
Summarizing Videos using Concentrated Attention and Considering the Uniqueness and Diversity of the Video Frames
Instead of simply modeling the frames' dependencies based on global attention, our method integrates a concentrated attention mechanism that is able to focus on non-overlapping blocks in the main diagonal of the attention matrix, and to enrich the existing information by extracting and exploiting knowledge about the uniqueness and diversity of the associated frames of the video.
Ranked #1 on
Unsupervised Video Summarization
on TvSum
ContraCLIP: Interpretable GAN generation driven by pairs of contrasting sentences
This work addresses the problem of discovering non-linear interpretable paths in the latent space of pre-trained GANs in a model-agnostic manner.
PandA: Unsupervised Learning of Parts and Appearances in the Feature Maps of GANs
Recent advances in the understanding of Generative Adversarial Networks (GANs) have led to remarkable progress in visual editing and synthesis tasks, capitalizing on the rich semantics that are embedded in the latent spaces of pre-trained GANs.
Combining Global and Local Attention with Positional Encoding for Video Summarization
This paper presents a new method for supervised video summarization.
Ranked #1 on
Video Summarization
on SumMe
Tensor Component Analysis for Interpreting the Latent Space of GANs
This paper addresses the problem of finding interpretable directions in the latent space of pre-trained Generative Adversarial Networks (GANs) to facilitate controllable image synthesis.
SSR: An Efficient and Robust Framework for Learning with Unknown Label Noise
Under this setting, unlike previous methods that often introduce multiple assumptions and lead to complex solutions, we propose a simple, efficient and robust framework named Sample Selection and Relabelling(SSR), that with a minimal number of hyperparameters achieves SOTA results in various conditions.
Ranked #1 on
Image Classification
on CIFAR-10 (with noisy labels)
WarpedGANSpace: Finding non-linear RBF paths in GAN latent space
This work addresses the problem of discovering, in an unsupervised manner, interpretable paths in the latent space of pretrained GANs, so as to provide an intuitive and easy way of controlling the underlying generative factors.
DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval
In this work, we propose a Knowledge Distillation framework, called Distill-and-Select (DnS), that starting from a well-performing fine-grained Teacher Network learns: a) Student Networks at different retrieval performance and computational efficiency trade-offs and b) a Selector Network that at test time rapidly directs samples to the appropriate student to maintain both high retrieval performance and high computational efficiency.
Ranked #2 on
Video Retrieval
on FIVR-200K
Few-Shot Action Localization without Knowing Boundaries
Learning to localize actions in long, cluttered, and untrimmed videos is a hard task, that in the literature has typically been addressed assuming the availability of large amounts of annotated training samples for each class -- either in a fully-supervised setting, where action boundaries are known, or in a weakly-supervised setting, where only class labels are known for each video.
Relationship-based Neural Baby Talk
Understanding interactions between objects in an image is an important element for generating captions.
Uncertainty Propagation in Convolutional Neural Networks: Technical Report
In this technical report we study the problem of propagation of uncertainty (in terms of variances of given uni-variate normal random variables) through typical building blocks of a Convolutional Neural Network (CNN).
Video Summarization Using Deep Neural Networks: A Survey
Video summarization technologies aim to create a concise and complete synopsis by selecting the most informative parts of the video content.
AC-SUM-GAN: Connecting Actor-Critic and Generative Adversarial Networks for Unsupervised Video Summarization
This paper presents a new method for unsupervised video summarization.
Ranked #3 on
Unsupervised Video Summarization
on TvSum
Generative Adversarial Network
Unsupervised Video Summarization
Temporal Action Localization with Variance-Aware Networks
Results in the action localization problem show that the incorporation of second order statistics improves over the baseline network, and that VANp surpasses the accuracy of virtually all other two-stage networks without involving any additional parameters.
Boundary Uncertainty in a Single-Stage Temporal Action Localization Network
We use two uncertainty-aware boundary regression losses: first, the Kullback-Leibler divergence between the ground truth location of the boundary and the Gaussian modeling the prediction of the boundary and second, the expectation of the $\ell_1$ loss under the same Gaussian.
Unsupervised Video Summarization via Attention-Driven Adversarial Learning
Experimental evaluation on two datasets (SumMe and TVSum) documents the contribution of the attention auto-encoder to faster and more stable training of the model, resulting in a significant performance improvement with respect to the original model and demonstrating the competitiveness of the proposed SUM-GAN-AAE against the state of the art.
Ranked #6 on
Unsupervised Video Summarization
on SumMe
A Stepwise, Label-based Approach for Improving the Adversarial Training in Unsupervised Video Summarization
In this paper we present our work on improving the efficiency of adversarial training for unsupervised video summarization.
Ranked #5 on
Unsupervised Video Summarization
on TvSum
ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning
Subsequently, the similarity matrix between all video frames is fed to a four-layer CNN, and then summarized using Chamfer Similarity (CS) into a video-to-video similarity score -- this avoids feature aggregation before the similarity calculation between videos and captures the temporal similarity patterns between matching frame sequences.
Ranked #5 on
Video Retrieval
on FIVR-200K
TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition
At the heart of our network is a meta-learning approach that learns to compare representations of variable temporal length, that is, either two videos of different length (in the case of few-shot action recognition) or a video and a semantic representation such as word vector (in the case of zero-shot action recognition).
Ranked #7 on
Few Shot Action Recognition
on Kinetics-100
Exploring Feature Representation and Training strategies in Temporal Action Localization
Temporal action localization has recently attracted significant interest in the Computer Vision community.
Registration-free Face-SSD: Single shot analysis of smiles, facial attributes, and affect in the wild
In this paper, we present a novel single shot face-related task analysis method, called Face-SSD, for detecting faces and for performing various face-related (classification/regression) tasks including smile recognition, face attribute prediction and valence-arousal estimation in the wild.
FIVR: Fine-grained Incident Video Retrieval
To create the dataset, we devise a process for the collection of YouTube videos based on major news events from recent years crawled from Wikipedia and deploy a retrieval pipeline for the automatic selection of query videos based on their estimated suitability as benchmarks.
SchiNet: Automatic Estimation of Symptoms of Schizophrenia from Facial Behaviour Analysis
Patients with schizophrenia often display impairments in the expression of emotion and speech and those are observed in their facial behaviour.
Semi-supervised Fisher vector network
In this work we explore how the architecture proposed in [8], which expresses the processing steps of the classical Fisher vector pipeline approaches, i. e. dimensionality reduction by principal component analysis (PCA) projection, Gaussian mixture model (GMM) and Fisher vector descriptor extraction as network layers, can be modified into a hybrid network that combines the benefits of both unsupervised and supervised training methods, resulting in a model that learns a semi-supervised Fisher vector descriptor of the input data.
Deep Globally Constrained MRFs for Human Pose Estimation
We overcome this by introducing a Markov Random Field (MRF)-based spatial model network between the coarse and the refinement model that introduces geometric constraints on the relative locations of the body joints.
Discriminative convolutional Fisher vector network for action recognition
In this work we propose a novel neural network architecture for the problem of human action recognition in videos.
Unsupervised convolutional neural networks for motion estimation
In this paper, we propose a direct method and train a Convolutional Neural Network (CNN) that when, at test time, is given a pair of images as input it produces a dense motion field F at its output layer.
Learning to detect video events from zero or very few video examples
In this work we deal with the problem of high-level event detection in video.
Face Alignment Assisted by Head Pose Estimation
In this paper we propose a supervised initialization scheme for cascaded face alignment based on explicit head pose estimation.
Linear Maximum Margin Classifier for Learning from Uncertain Data
In this paper, we propose a maximum margin classifier that deals with uncertainty in data input.
Mirror, mirror on the wall, tell me, is the error small?
Our experiments lead to several interesting findings: 1) Surprisingly, most of state of the art methods struggle to preserve the mirror symmetry, despite the fact that they do have very similar overall performance on the original and mirror images; 2) the low mirrorability is not caused by training or testing sample bias - all algorithms are trained on both the original images and their mirrored versions; 3) the mirror error is strongly correlated to the localization/alignment error (with correlation coefficients around 0. 7).
