ProMISe: Promptable Medical Image Segmentation using SAM

7 Mar 2024  ·  Jinfeng Wang, Sifan Song, Xinkun Wang, Yiyi Wang, Yiyi Miao, Jionglong Su, S. Kevin Zhou ·

With the proposal of the Segment Anything Model (SAM), fine-tuning SAM for medical image segmentation (MIS) has become popular. However, due to the large size of the SAM model and the significant domain gap between natural and medical images, fine-tuning-based strategies are costly with potential risk of instability, feature damage and catastrophic forgetting. Furthermore, some methods of transferring SAM to a domain-specific MIS through fine-tuning strategies disable the model's prompting capability, severely limiting its utilization scenarios. In this paper, we propose an Auto-Prompting Module (APM), which provides SAM-based foundation model with Euclidean adaptive prompts in the target domain. Our experiments demonstrate that such adaptive prompts significantly improve SAM's non-fine-tuned performance in MIS. In addition, we propose a novel non-invasive method called Incremental Pattern Shifting (IPS) to adapt SAM to specific medical domains. Experimental results show that the IPS enables SAM to achieve state-of-the-art or competitive performance in MIS without the need for fine-tuning. By coupling these two methods, we propose ProMISe, an end-to-end non-fine-tuned framework for Promptable Medical Image Segmentation. Our experiments demonstrate that both using our methods individually or in combination achieves satisfactory performance in low-cost pattern shifting, with all of SAM's parameters frozen.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Medical Image Segmentation CVC-ColonDB ProMISe mean Dice 0.874 # 4
mIoU 0.789 # 5
Medical Image Segmentation ETIS-LARIBPOLYPDB ProMISe mIoU 0.750 # 4
mean Dice 0.840 # 4
Lesion Segmentation ISIC 2018 ProMISe mean Dice 0.921 # 3
Mean IoU 0.850 # 2
Medical Image Segmentation ISIC 2018 ProMISe DSC 92.10 # 1
mIoU 85.00 # 2
Medical Image Segmentation Kvasir-SEG ProMISe mean Dice 0.911 # 31
mIoU 0.851 # 35

Methods