TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Model Description Based Search	Generative Models	Image (Gen.) CLIP+Gaussian Density (ours)	Top-1 Accuracy	0.81	# 1
Model Description Based Search	Generative Models	Image (Gen.) CLIP+Gaussian Density (ours)	Top-5 Accuracy	0.75	# 2
Model Description Based Search	Generative Models	Image (Gen.) CLIP+Gaussian Density (ours)	Top-10 Accuracy	0.76	# 2
Model Description Based Search	Generative Models	Sketch CLIP+Gaussian Density (ours)	Top-1 Accuracy	0.68	# 3
Model Description Based Search	Generative Models	Sketch CLIP+Gaussian Density (ours)	Top-5 Accuracy	0.64	# 3
Model Description Based Search	Generative Models	Sketch CLIP+Gaussian Density (ours)	Top-10 Accuracy	0.66	# 3
Image and Sketch based Model Retrieval	Generative Models	Sketch CLIP+Gaussian Density (ours)	Top-5 Accuracy	0.67	# 3
Image and Sketch based Model Retrieval	Generative Models	Sketch CLIP+Gaussian Density (ours)	Top-10 Accuracy	0.86	# 3
Image and Sketch based Model Retrieval	Generative Models	Sketch CLIP+Gaussian Density (ours)	MAP@5	0.68	# 4
Image and Sketch based Model Retrieval	Generative Models	Sketch CLIP+Gaussian Density (ours)	MAP@10	0.64	# 3
Image and Sketch based Model Retrieval	Generative Models	Sketch CLIP+Gaussian Density (ours)	mAP	0.66	# 3
Model Description Based Search	Generative Models	CLIP+Gaussian Density (ours)	Top-1 Accuracy	0.77	# 2
Model Description Based Search	Generative Models	CLIP+Gaussian Density (ours)	Top-5 Accuracy	0.95	# 1
Model Description Based Search	Generative Models	CLIP+Gaussian Density (ours)	Top-10 Accuracy	1	# 1
Image and Sketch based Model Retrieval	Generative Models	Sketch CLIP+1st Moment (ours)	Top-10 Accuracy	0.84	# 4
Image and Sketch based Model Retrieval	Generative Models	Sketch CLIP+1st Moment (ours)	MAP@5	0.7	# 3
Image and Sketch based Model Retrieval	Generative Models	Sketch CLIP+1st Moment (ours)	MAP@10	0.64	# 3
Image and Sketch based Model Retrieval	Generative Models	Sketch CLIP+1st Moment (ours)	mAP	0.64	# 4
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+Gaussian Density (ours)	Top-1 Accuracy	0.77	# 1
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+Gaussian Density (ours)	Top-10 Accuracy	1	# 1
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+Gaussian Density (ours)	MAP@5	0.81	# 1
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+Gaussian Density (ours)	MAP@10	0.75	# 1
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+Gaussian Density (ours)	mAP	0.76	# 1
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+1st Moment (ours)	Top-1 Accuracy	0.75	# 2
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+1st Moment (ours)	Top-5 Accuracy	0.95	# 1
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+1st Moment (ours)	Top-10 Accuracy	0.99	# 2
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+1st Moment (ours)	MAP@5	0.79	# 2
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+1st Moment (ours)	MAP@10	0.74	# 2
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+1st Moment (ours)	mAP	0.74	# 2
Text based Model Retrieval	Generative Models	Augmented 1st + 2nd Moment (ours)	MAP@5	0.77	# 3
Text based Model Retrieval	Generative Models	Augmented 1st + 2nd Moment (ours)	mAP	0.72	# 1
Text based Model Retrieval	Generative Models	Augmented 1st Moment (ours)	MAP@5	0.79	# 1
Text based Model Retrieval	Generative Models	Augmented 1st Moment (ours)	MAP@10	0.72	# 1
Text based Model Retrieval	Generative Models	Augmented 1st Moment (ours)	mAP	0.69	# 3
Text based Model Retrieval	Generative Models	Original 1st + 2nd Moment (ours)	MAP@5	0.74	# 4
Text based Model Retrieval	Generative Models	Original 1st + 2nd Moment (ours)	MAP@10	0.7	# 3
Text based Model Retrieval	Generative Models	Original 1st + 2nd Moment (ours)	mAP	0.70	# 2
Text based Model Retrieval	Generative Models	Original 1st Moment (ours)	MAP@5	0.78	# 2
Text based Model Retrieval	Generative Models	Original 1st Moment (ours)	MAP@10	0.71	# 2
Text based Model Retrieval	Generative Models	Original 1st Moment (ours)	mAP	0.68	# 4
Image and Sketch based Model Retrieval	Generative Models	CLIP+Gaussian Density (ours)	Top-1 Accuracy	0.33	# 4
Image and Sketch based Model Retrieval	Generative Models	CLIP+1st Moment (ours)	Top-1 Accuracy	0.35	# 3
Image and Sketch based Model Retrieval	Generative Models	CLIP+1st Moment (ours)	Top-5 Accuracy	0.7	# 2

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/content-based-search-for-deep-generative/model-description-based-search-on-generative)](https://paperswithcode.com/sota/model-description-based-search-on-generative?p=content-based-search-for-deep-generative)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/content-based-search-for-deep-generative/image-and-sketch-based-model-retrieval-on)](https://paperswithcode.com/sota/image-and-sketch-based-model-retrieval-on?p=content-based-search-for-deep-generative)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/content-based-search-for-deep-generative/text-based-model-retrieval-on-generative)](https://paperswithcode.com/sota/text-based-model-retrieval-on-generative?p=content-based-search-for-deep-generative)`

Content-Based Search for Deep Generative Models

6 Oct 2022 · Daohan Lu, Sheng-Yu Wang, Nupur Kumari, Rohan Agarwal, Mia Tang, David Bau, Jun-Yan Zhu ·

The growing proliferation of customized and pretrained generative models has made it infeasible for a user to be fully cognizant of every model in existence. To address this need, we introduce the task of content-based model search: given a query and a large set of generative models, finding the models that best match the query. As each generative model produces a distribution of images, we formulate the search task as an optimization problem to select the model with the highest probability of generating similar content as the query. We introduce a formulation to approximate this probability given the query from different modalities, e.g., image, sketch, and text. Furthermore, we propose a contrastive learning framework for model retrieval, which learns to adapt features for various query modalities. We demonstrate that our method outperforms several baselines on Generative Model Zoo, a new benchmark we create for the model retrieval task.

PDF Abstract

Code

Add Remove Mark official

generative-intelligence-lab/modelve… official

216

Tasks

Add Remove

Contrastive Learning

Image and Sketch based Model Retrieval

Model Description Based Search

Retrieval

Text based Model Retrieval

Transfer Learning

Datasets

Add Datasets introduced or used in this paper

Results from the Paper

Edit

Ranked #1 on Model Description Based Search on Generative Models

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Model Description Based Search	Generative Models	Image (Gen.) CLIP+Gaussian Density (ours)	Top-1 Accuracy	0.81	# 1	Compare
			Top-5 Accuracy	0.75	# 2	Compare
			Top-10 Accuracy	0.76	# 2	Compare
Model Description Based Search	Generative Models	Sketch CLIP+Gaussian Density (ours)	Top-1 Accuracy	0.68	# 3	Compare
			Top-5 Accuracy	0.64	# 3	Compare
			Top-10 Accuracy	0.66	# 3	Compare
Image and Sketch based Model Retrieval	Generative Models	Sketch CLIP+Gaussian Density (ours)	Top-5 Accuracy	0.67	# 3	Compare
			Top-10 Accuracy	0.86	# 3	Compare
			MAP@5	0.68	# 4	Compare
			MAP@10	0.64	# 3	Compare
			mAP	0.66	# 3	Compare
Model Description Based Search	Generative Models	CLIP+Gaussian Density (ours)	Top-1 Accuracy	0.77	# 2	Compare
			Top-5 Accuracy	0.95	# 1	Compare
			Top-10 Accuracy	1	# 1	Compare
Image and Sketch based Model Retrieval	Generative Models	Sketch CLIP+1st Moment (ours)	Top-10 Accuracy	0.84	# 4	Compare
			MAP@5	0.7	# 3	Compare
			MAP@10	0.64	# 3	Compare
			mAP	0.64	# 4	Compare
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+Gaussian Density (ours)	Top-1 Accuracy	0.77	# 1	Compare
			Top-10 Accuracy	1	# 1	Compare
			MAP@5	0.81	# 1	Compare
			MAP@10	0.75	# 1	Compare
			mAP	0.76	# 1	Compare
Image and Sketch based Model Retrieval	Generative Models	Image (Gen.) CLIP+1st Moment (ours)	Top-1 Accuracy	0.75	# 2	Compare
			Top-5 Accuracy	0.95	# 1	Compare
			Top-10 Accuracy	0.99	# 2	Compare
			MAP@5	0.79	# 2	Compare
			MAP@10	0.74	# 2	Compare
			mAP	0.74	# 2	Compare
Text based Model Retrieval	Generative Models	Augmented 1st + 2nd Moment (ours)	MAP@5	0.77	# 3	Compare
Text based Model Retrieval	Generative Models	Augmented 1st + 2nd Moment (ours)	mAP	0.72	# 1	Compare
Text based Model Retrieval	Generative Models	Augmented 1st Moment (ours)	MAP@5	0.79	# 1	Compare
			MAP@10	0.72	# 1	Compare
			mAP	0.69	# 3	Compare
Text based Model Retrieval	Generative Models	Original 1st + 2nd Moment (ours)	MAP@5	0.74	# 4	Compare
			MAP@10	0.7	# 3	Compare
			mAP	0.70	# 2	Compare
Text based Model Retrieval	Generative Models	Original 1st Moment (ours)	MAP@5	0.78	# 2	Compare
			MAP@10	0.71	# 2	Compare
			mAP	0.68	# 4	Compare
Image and Sketch based Model Retrieval	Generative Models	CLIP+Gaussian Density (ours)	Top-1 Accuracy	0.33	# 4	Compare
Image and Sketch based Model Retrieval	Generative Models	CLIP+1st Moment (ours)	Top-1 Accuracy	0.35	# 3	Compare
Image and Sketch based Model Retrieval	Generative Models	CLIP+1st Moment (ours)	Top-5 Accuracy	0.7	# 2	Compare

Methods

Add Remove

SPEED

Edit Social Preview

Content-Based Search for Deep Generative Models

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove