Search Results for author: Zeynep Akata

Found 76 papers, 45 papers with code

KG-SP: Knowledge Guided Simple Primitives for Open World Compositional Zero-Shot Learning

1 code implementation13 May 2022 Shyamgopal Karthik, Massimiliano Mancini, Zeynep Akata

The goal of open-world compositional zero-shot learning (OW-CZSL) is to recognize compositions of state and objects in images, given only a subset of them during training and no prior on the unseen compositions.

Compositional Zero-Shot Learning

Attention Consistency on Visual Corruptions for Single-Source Domain Generalization

1 code implementation27 Apr 2022 Ilke Cugu, Massimiliano Mancini, Yanbei Chen, Zeynep Akata

Generalizing visual recognition models trained on a single distribution to unseen input distributions (i. e. domains) requires making them robust to superfluous correlations in the training set.

Domain Generalization

Probabilistic Compositional Embeddings for Multimodal Image Retrieval

1 code implementation12 Apr 2022 Andrei Neculai, Yanbei Chen, Zeynep Akata

Without bells and whistles, we show that our probabilistic model formulation significantly outperforms existing related methods on multimodal image retrieval while generalizing well to query with different amounts of inputs given in arbitrary visual and (or) textual modalities.

Image Retrieval

CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations

1 code implementation5 Apr 2022 Leonard Salewski, A. Sophia Koepke, Hendrik P. A. Lensch, Zeynep Akata

We present baseline results for generating natural language explanations in the context of VQA using two state-of-the-art frameworks on the CLEVR-X dataset.

Explanation Generation Question Answering +4

Attribute Prototype Network for Any-Shot Learning

no code implementations4 Apr 2022 Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, Zeynep Akata

While a visual-semantic embedding layer learns global features, local features are learned through an attribute prototype network that simultaneously regresses and decorrelates attributes from intermediate features.

Few-Shot Image Classification Representation Learning +1

VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning

1 code implementation20 Mar 2022 Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, Zeynep Akata

Our model visually divides a set of images from seen classes into clusters of local image regions according to their visual similarity, and further imposes their class discrimination and semantic relatedness.

Transfer Learning Word Embeddings +1

Integrating Language Guidance into Vision-based Deep Metric Learning

1 code implementation16 Mar 2022 Karsten Roth, Oriol Vinyals, Zeynep Akata

This causes learned embedding spaces to encode incomplete semantic context and misrepresent the semantic relation between classes, impacting the generalizability of the learned metric space.

Ranked #4 on Metric Learning on CARS196 (using extra training data)

Metric Learning

Non-isotropy Regularization for Proxy-based Deep Metric Learning

1 code implementation16 Mar 2022 Karsten Roth, Oriol Vinyals, Zeynep Akata

Deep Metric Learning (DML) aims to learn representation spaces on which semantic relations can simply be expressed through predefined distance metrics.

Ranked #7 on Metric Learning on CUB-200-2011 (using extra training data)

Metric Learning

Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language

1 code implementation7 Mar 2022 Otniel-Bogdan Mercea, Lukas Riesch, A. Sophia Koepke, Zeynep Akata

Focusing on the relatively underexplored task of audio-visual zero-shot learning, we propose to learn multi-modal representations from audio-visual data using cross-modal attention and exploit textual label embeddings for transferring knowledge from seen classes to unseen classes.

GZSL Video Classification Zero-Shot Learning +1

Audio Retrieval with Natural Language Queries: A Benchmark Study

1 code implementation17 Dec 2021 A. Sophia Koepke, Andreea-Maria Oncescu, João F. Henriques, Zeynep Akata, Samuel Albanie

Additionally, we introduce the SoundDescs benchmark, which consists of paired audio and natural language descriptions for a diverse collection of sounds that are complementary to those found in AudioCaps and Clotho.

Audio captioning Audio to Text Retrieval +1

Human Attention in Fine-grained Classification

1 code implementation2 Nov 2021 Yao Rong, Wenjia Xu, Zeynep Akata, Enkelejda Kasneci

The way humans attend to, process and classify a given image has the potential to vastly benefit the performance of deep learning models.

Classification Decision Making +1

Robustness via Uncertainty-aware Cycle Consistency

1 code implementation NeurIPS 2021 Uddeshya Upadhyay, Yanbei Chen, Zeynep Akata

Unpaired image-to-image translation refers to learning inter-image-domain mapping without corresponding image pairs.

Autonomous Driving Image-to-Image Translation +1

Conditional De-Identification of 3D Magnetic Resonance Images

no code implementations18 Oct 2021 Lennart Alexander Van der Goten, Tobias Hepp, Zeynep Akata, Kevin Smith

Solutions have been developed to de-identify diagnostic scans by obfuscating or removing parts of the face.

De-identification

Variational Perturbations for Visual Feature Attribution

no code implementations29 Sep 2021 Jae Myung Kim, Eunji Kim, Sungroh Yoon, Jungwoo Lee, Cordelia Schmid, Zeynep Akata

Explaining a complex black-box system in a post-hoc manner is important to understand its predictions.

Fine-Grained Zero-Shot Learning with DNA as Side Information

1 code implementation NeurIPS 2021 Sarkhan Badirli, Zeynep Akata, George Mohler, Christine Picard, Murat Dundar

Fine-grained zero-shot learning task requires some form of side-information to transfer discriminative information from seen to unseen classes.

Zero-Shot Learning

The Manifold Hypothesis for Gradient-Based Explanations

no code implementations29 Sep 2021 Sebastian Bordt, Uddeshya Upadhyay, Zeynep Akata, Ulrike Von Luxburg

Across a range of different datasets -- MNIST, EMNIST, CIFAR10, X-ray pneumonia and Diabetic Retinopathy detection -- we demonstrate empirically that the more an explanation is aligned with the tangent space of the data, the more interpretable it tends to be.

Diabetic Retinopathy Detection

Concurrent Discrimination and Alignment for Self-Supervised Feature Learning

no code implementations19 Aug 2021 Anjan Dutta, Massimiliano Mancini, Zeynep Akata

Existing self-supervised learning methods learn representation by means of pretext tasks which are either (1) discriminating that explicitly specify which features should be separated or (2) aligning that precisely indicate which features should be closed together, but ignore the fact how to jointly and principally define which features to be repelled and which ones to be attracted.

Self-Supervised Learning Semantic Segmentation +1

Uncertainty-Guided Progressive GANs for Medical Image Translation

1 code implementation29 Jun 2021 Uddeshya Upadhyay, Yanbei Chen, Tobias Hepp, Sergios Gatidis, Zeynep Akata

However, the state-of-the-art GAN-based frameworks do not estimate the uncertainty in the predictions made by the network that is essential for making informed medical decisions and subsequent revision by medical experts and has recently been shown to improve the performance and interpretability of the model.

Denoising Image-to-Image Translation +2

Keep CALM and Improve Visual Feature Attribution

1 code implementation ICCV 2021 Jae Myung Kim, Junsuk Choe, Zeynep Akata, Seong Joon Oh

The class activation mapping, or CAM, has been the cornerstone of feature attribution methods for multiple vision tasks.

Weakly-Supervised Object Localization

e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks

2 code implementations ICCV 2021 Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, Thomas Lukasiewicz

e-ViL is a benchmark for explainable vision-language tasks that establishes a unified evaluation framework and provides the first comprehensive comparison of existing approaches that generate NLEs for VL tasks.

Language Modelling Text Generation

Learning Graph Embeddings for Open World Compositional Zero-Shot Learning

2 code implementations3 May 2021 Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, Zeynep Akata

In this work, we overcome this assumption operating on the open world setting, where no limit is imposed on the compositional space at test time, and the search space contains a large number of unseen compositions.

Compositional Zero-Shot Learning

Distilling Audio-Visual Knowledge by Compositional Contrastive Learning

1 code implementation CVPR 2021 Yanbei Chen, Yongqin Xian, A. Sophia Koepke, Ying Shan, Zeynep Akata

Having access to multi-modal cues (e. g. vision and audio) empowers some cognitive tasks to be done faster compared to learning from a single modality.

Audio Tagging audio-visual learning +5

A Closer Look at Self-training for Zero-Label Semantic Segmentation

1 code implementation21 Apr 2021 Giuseppe Pastore, Fabio Cermelli, Yongqin Xian, Massimiliano Mancini, Zeynep Akata, Barbara Caputo

Being able to segment unseen classes not observed during training is an important technical challenge in deep learning, because of its potential to reduce the expensive annotation required for semantic segmentation.

Semantic Segmentation

Uncertainty-aware Generalized Adaptive CycleGAN

1 code implementation23 Feb 2021 Uddeshya Upadhyay, Yanbei Chen, Zeynep Akata

Unpaired image-to-image translation refers to learning inter-image-domain mapping in an unsupervised manner.

Image Denoising Image-to-Image Translation +1

Learning Graph Embeddings for Compositional Zero-shot Learning

1 code implementation CVPR 2021 Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, Zeynep Akata

In compositional zero-shot learning, the goal is to recognize unseen compositions (e. g. old dog) of observed visual primitives states (e. g. old, cute) and objects (e. g. car, dog) in the training set.

Compositional Zero-Shot Learning Graph Embedding +1

Open World Compositional Zero-Shot Learning

2 code implementations CVPR 2021 Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, Zeynep Akata

After estimating the feasibility score of each composition, we use these scores to either directly mask the output space or as a margin for the cosine similarity between visual features and compositional embeddings during training.

Compositional Zero-Shot Learning

Adversarial Privacy Preservation in MRI Scans of the Brain

no code implementations1 Jan 2021 Lennart Alexander Van der Goten, Tobias Hepp, Zeynep Akata, Kevin Smith

De-identification of magnetic resonance imagery (MRI) is intrinsically difficult since, even with all metadata removed, a person's face can easily be rendered and matched against a database.

De-identification

Prototype-based Incremental Few-Shot Semantic Segmentation

1 code implementation30 Nov 2020 Fabio Cermelli, Massimiliano Mancini, Yongqin Xian, Zeynep Akata, Barbara Caputo

Semantic segmentation models have two fundamental weaknesses: i) they require large training sets with costly pixel-level annotations, and ii) they have a static output space, constrained to the classes of the training set.

Few-Shot Semantic Segmentation Incremental Learning +2

Attribute Prototype Network for Zero-Shot Learning

no code implementations NeurIPS 2020 Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, Zeynep Akata

As an additional benefit, our model points to the visual evidence of the attributes in an image, e. g. for the CUB dataset, confirming the improved attribute localization ability of our image representation.

Representation Learning Zero-Shot Learning

Towards Recognizing Unseen Categories in Unseen Domains

1 code implementation ECCV 2020 Massimiliano Mancini, Zeynep Akata, Elisa Ricci, Barbara Caputo

The key idea of CuMix is to simulate the test-time domain and semantic shift using images and features from unseen domains and categories generated by mixing up the multiple source domains and categories available during training.

Domain Generalization Zero-Shot Learning +1

Evaluation for Weakly Supervised Object Localization: Protocol, Metrics, and Datasets

2 code implementations8 Jul 2020 Junsuk Choe, Seong Joon Oh, Sanghyuk Chun, Seungho Lee, Zeynep Akata, Hyunjung Shim

In this paper, we argue that WSOL task is ill-posed with only image-level labels, and propose a new evaluation protocol where full supervision is limited to only a small held-out set not overlapping with the test set.

Few-Shot Learning Model Selection +1

Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-based Image Retrieval

no code implementations20 Jun 2020 Anjan Dutta, Zeynep Akata

Low-shot sketch-based image retrieval is an emerging task in computer vision, allowing to retrieve natural images relevant to hand-drawn sketch queries that are rarely seen during the training phase.

Sketch-Based Image Retrieval

e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations

3 code implementations7 Apr 2020 Virginie Do, Oana-Maria Camburu, Zeynep Akata, Thomas Lukasiewicz

The recently proposed SNLI-VE corpus for recognising visual-textual entailment is a large, real-world dataset for fine-grained multimodal reasoning.

Natural Language Inference

Evaluating Weakly Supervised Object Localization Methods Right

2 code implementations CVPR 2020 Junsuk Choe, Seong Joon Oh, Seungho Lee, Sanghyuk Chun, Zeynep Akata, Hyunjung Shim

In this paper, we argue that WSOL task is ill-posed with only image-level labels, and propose a new evaluation protocol where full supervision is limited to only a small held-out set not overlapping with the test set.

Few-Shot Learning Model Selection +1

Understanding Misclassifications by Attributes

1 code implementation15 Oct 2019 Sadaf Gulshad, Zeynep Akata, Jan Hendrik Metzen, Arnold Smeulders

We study the changes in attributes for clean as well as adversarial images in both standard and adversarially robust networks.

Modeling Conceptual Understanding in Image Reference Games

1 code implementation NeurIPS 2019 Rodolfo Corona, Stephan Alaniz, Zeynep Akata

An agent who interacts with a wide population of other agents needs to be aware that there may be variations in their understanding of the world.

Relational Generalized Few-Shot Learning

no code implementations22 Jul 2019 Xiahan Shi, Leonard Salewski, Martin Schiegg, Zeynep Akata, Max Welling

Instead, we consider the extended setup of generalized few-shot learning (GFSL), where the model is required to perform classification on the joint label space consisting of both previously seen and novel classes.

Few-Shot Learning Generalized Few-Shot Learning

Bayesian Zero-Shot Learning

1 code implementation22 Jul 2019 Sarkhan Badirli, Zeynep Akata, Murat Dundar

Object classes that surround us have a natural tendency to emerge at varying levels of abstraction.

Zero-Shot Learning

Interpreting Adversarial Examples with Attributes

1 code implementation17 Apr 2019 Sadaf Gulshad, Jan Hendrik Metzen, Arnold Smeulders, Zeynep Akata

Deep computer vision systems being vulnerable to imperceptible and carefully crafted noise have raised questions regarding the robustness of their decisions.

General Classification

f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning

no code implementations CVPR 2019 Yongqin Xian, Saurabh Sharma, Bernt Schiele, Zeynep Akata

When labeled training data is scarce, a promising data augmentation approach is to generate visual features of unknown classes using their attributes.

Data Augmentation Few-Shot Learning +1

Cross-Linked Variational Autoencoders for Generalized Zero-Shot Learning

no code implementations ICLR Workshop LLD 2019 Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, Zeynep Akata

While following the same direction, we also take artificial feature generation one step further and propose a model where a shared latent space of image features and class embeddings is learned by aligned variational autoencoders, for the purpose of generating latent features to train a softmax classifier.

Few-Shot Learning Generalized Zero-Shot Learning

Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrieval

1 code implementation CVPR 2019 Anjan Dutta, Zeynep Akata

Existing works either require aligned sketch-image pairs or inefficient memory fusion layer for mapping the visual information to a semantic space.

Sketch-Based Image Retrieval

Learning Decision Trees Recurrently Through Communication

no code implementations CVPR 2021 Stephan Alaniz, Diego Marcos, Bernt Schiele, Zeynep Akata

Integrated interpretability without sacrificing the prediction accuracy of decision making algorithms has the potential of greatly improving their value to the user.

Decision Making Image Classification

Visual Rationalizations in Deep Reinforcement Learning for Atari Games

no code implementations1 Feb 2019 Laurens Weitkamp, Elise van der Pol, Zeynep Akata

Due to the capability of deep learning to perform well in high dimensional problems, deep reinforcement learning agents perform well in challenging tasks such as Atari 2600 games.

Atari Games Decision Making +1

Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders

2 code implementations5 Dec 2018 Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, Zeynep Akata

Many approaches in generalized zero-shot learning rely on cross-modal mapping between the image feature space and the class embedding space.

Few-Shot Learning Generalized Zero-Shot Learning

Manipulating Attributes of Natural Scenes via Hallucination

no code implementations22 Aug 2018 Levent Karacan, Zeynep Akata, Aykut Erdem, Erkut Erdem

In this study, we explore building a two-stage framework for enabling users to directly manipulate high-level attributes of a natural scene.

Style Transfer Translation

Textual Explanations for Self-Driving Vehicles

1 code implementation ECCV 2018 Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, Zeynep Akata

Finally, we explore a version of our model that generates rationalizations, and compare with introspective explanations on the same video segments.

Grounding Visual Explanations

no code implementations ECCV 2018 Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, Zeynep Akata

Our model improves the textual explanation quality of fine-grained classification decisions on the CUB dataset by mentioning phrases that are grounded in the image.

General Classification

Generating Counterfactual Explanations with Natural Language

no code implementations26 Jun 2018 Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, Zeynep Akata

We call such textual explanations counterfactual explanations, and propose an intuitive method to generate counterfactual explanations by inspecting which evidence in an input is missing, but might contribute to a different classification decision if present in the image.

Classification Fine-Grained Image Classification +1

Primal-Dual Wasserstein GAN

no code implementations24 May 2018 Mevlana Gemici, Zeynep Akata, Max Welling

We introduce Primal-Dual Wasserstein GAN, a new learning algorithm for building latent variable models of the data distribution based on the primal and the dual formulations of the optimal transport (OT) problem.

Feature Generating Networks for Zero-Shot Learning

4 code implementations CVPR 2018 Yongqin Xian, Tobias Lorenz, Bernt Schiele, Zeynep Akata

Suffering from the extreme training data imbalance between seen and unseen classes, most of existing state-of-the-art approaches fail to achieve satisfactory results for the challenging generalized zero-shot learning task.

Generalized Zero-Shot Learning

Grounding Visual Explanations (Extended Abstract)

no code implementations17 Nov 2017 Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, Zeynep Akata

Existing models which generate textual explanations enforce task relevance through a discriminative term loss function, but such mechanisms only weakly constrain mentioned object parts to actually be present in the image.

Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly

9 code implementations3 Jul 2017 Yongqin Xian, Christoph H. Lampert, Bernt Schiele, Zeynep Akata

Due to the importance of zero-shot learning, i. e. classifying images where there is a lack of labeled training data, the number of proposed approaches has recently increased steadily.

Zero-Shot Learning

Zero-Shot Learning -- The Good, the Bad and the Ugly

no code implementations CVPR 2017 Yongqin Xian, Bernt Schiele, Zeynep Akata

Due to the importance of zero-shot learning, the number of proposed approaches has increased steadily recently.

Zero-Shot Learning

Exploiting saliency for object segmentation from image level labels

no code implementations CVPR 2017 Seong Joon Oh, Rodrigo Benenson, Anna Khoreva, Zeynep Akata, Mario Fritz, Bernt Schiele

We show how to combine both information sources in order to recover 80% of the fully supervised performance - which is the new state of the art in weakly supervised training for pixel-wise semantic labelling.

Semantic Segmentation

Attentive Explanations: Justifying Decisions and Pointing to the Evidence

no code implementations14 Dec 2016 Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Bernt Schiele, Trevor Darrell, Marcus Rohrbach

In contrast, humans can justify their decisions with natural language and point to the evidence in the visual world which led to their decisions.

Decision Making Question Answering +1

Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts

no code implementations1 Dec 2016 Levent Karacan, Zeynep Akata, Aykut Erdem, Erkut Erdem

Automatic image synthesis research has been rapidly growing with deep networks getting more and more expressive.

Image Generation

Gaze Embeddings for Zero-Shot Image Classification

no code implementations CVPR 2017 Nour Karessli, Zeynep Akata, Bernt Schiele, Andreas Bulling

Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts.

Classification Fine-Grained Image Classification +2

Learning What and Where to Draw

no code implementations NeurIPS 2016 Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, Honglak Lee

Generative Adversarial Networks (GANs) have recently demonstrated the capability to synthesize compelling real-world images, such as room interiors, album covers, manga, faces, birds, and flowers.

Ranked #14 on Text-to-Image Generation on CUB (using extra training data)

Text-to-Image Generation

Generative Adversarial Text to Image Synthesis

38 code implementations17 May 2016 Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee

Automatic synthesis of realistic images from text would be interesting and useful, but current AI systems are still far from this goal.

Adversarial Text Text-to-Image Generation

Latent Embeddings for Zero-shot Classification

no code implementations CVPR 2016 Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, Bernt Schiele

We train the model with a ranking based objective function which penalizes incorrect rankings of the true class for a given image.

Classification General Classification +1

Multi-Cue Zero-Shot Learning with Strong Supervision

no code implementations CVPR 2016 Zeynep Akata, Mateusz Malinowski, Mario Fritz, Bernt Schiele

A promising research direction is zero-shot learning, which does not require any training data to recognize new classes, but rather relies on some form of auxiliary information describing the new classes.

Zero-Shot Learning

Generating Visual Explanations

no code implementations28 Mar 2016 Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, Trevor Darrell

Clearly explaining a rationale for a classification decision to an end-user can be as important as the decision itself.

General Classification

Label-Embedding for Image Classification

2 code implementations30 Mar 2015 Zeynep Akata, Florent Perronnin, Zaid Harchaoui, Cordelia Schmid

Attributes act as intermediate representations that enable parameter sharing between classes, a must when training data is scarce.

Classification General Classification +3

Label-Embedding for Attribute-Based Classification

no code implementations CVPR 2013 Zeynep Akata, Florent Perronnin, Zaid Harchaoui, Cordelia Schmid

The label embedding framework offers other advantages such as the ability to leverage alternative sources of information in addition to attributes (e. g. class hierarchies) or to transition smoothly from zero-shot learning to learning with large quantities of data.

Classification General Classification +2

Cannot find the paper you are looking for? You can Submit a new open access paper.