Referring Image Matting

CVPR 2023  ยท  Jizhizi Li, Jing Zhang, DaCheng Tao ยท

Different from conventional image matting, which either requires user-defined scribbles/trimap to extract a specific foreground object or directly extracts all the foreground objects in the image indiscriminately, we introduce a new task named Referring Image Matting (RIM) in this paper, which aims to extract the meticulous alpha matte of the specific object that best matches the given natural language description, thus enabling a more natural and simpler instruction for image matting. First, we establish a large-scale challenging dataset RefMatte by designing a comprehensive image composition and expression generation engine to automatically produce high-quality images along with diverse text attributes based on public datasets. RefMatte consists of 230 object categories, 47,500 images, 118,749 expression-region entities, and 474,996 expressions. Additionally, we construct a real-world test set with 100 high-resolution natural images and manually annotate complex phrases to evaluate the out-of-domain generalization abilities of RIM methods. Furthermore, we present a novel baseline method CLIPMat for RIM, including a context-embedded prompt, a text-driven semantic pop-up, and a multi-level details extractor. Extensive experiments on RefMatte in both keyword and expression settings validate the superiority of CLIPMat over representative methods. We hope this work could provide novel insights into image matting and encourage more follow-up studies. The dataset, code and models are available at https://github.com/JizhiziLi/RIM.

PDF Abstract CVPR 2023 PDF CVPR 2023 Abstract

Datasets


Introduced in the Paper:

RefMatte

Used in the Paper:

AIM-500 BG-20k P3M-10k AM-2k
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Referring Image Matting (RefMatte-RW100) RefMatte CLIPMat (ViT-L/14) SAD 88.52 # 1
MSE 0.0488 # 1
MAD 0.0510 # 1
SAD(E) 87.92 # 1
MSE(E) 0.0483 # 1
MAD(E) 0.0505 # 1
Referring Image Matting (RefMatte-RW100) RefMatte CLIPMat (ViT-B/16) SAD 110.66 # 2
MSE 0.0614 # 2
MAD 0.0636 # 2
SAD(E) 110.63 # 2
MSE(E) 0.0612 # 2
MAD(E) 0.0635 # 2
Referring Image Matting (Keyword-based) RefMatte CLIPMat (ViT-B/16) SAD 9.91 # 2
MSE 0.0028 # 2
MAD 0.0057 # 2
SAD(E) 10.41 # 2
MSE(E) 0.0029 # 2
MAD(E) 0.0059 # 2
Referring Image Matting (Keyword-based) RefMatte CLIPMat (ViT-L/14) SAD 8.51 # 1
MSE 0.0022 # 1
MAD 0.0049 # 1
SAD(E) 8.98 # 1
MSE(E) 0.0023 # 1
MAD(E) 0.0051 # 1
Referring Image Matting (Expression-based) RefMatte CLIPMat (ViT-L/14) SAD 42.05 # 1
MSE 0.0212 # 1
MAD 0.0238 # 1
SAD(E) 44.77 # 1
MSE(E) 0.0226 # 1
MAD(E) 0.0254 # 1
Referring Image Matting (Expression-based) RefMatte CLIPMat (ViT-B/16) SAD 47.97 # 2
MSE 0.0245 # 2
MAD 0.0273 # 2
SAD(E) 50.84 # 2
MSE(E) 0.0260 # 2
MAD(E) 0.0273 # 2

Methods