Short-term and long-term memory self-attention network for segmentation of tumours in 3D medical images

Tumour segmentation in medical images (especially 3D tumour segmentation) is highly challenging due to the possible similarity between tumours and adjacent tissues, occurrence of multiple tumours and variable tumour shapes and sizes. The popular deep learning-based segmentation algorithms generally rely on the convolutional neural network (CNN) and Transformer. The former cannot extract the global image features effectively while the latter lacks the inductive bias and involves the complicated computation for 3D volume data. The existing hybrid CNN-Transformer network can only provide the limited performance improvement or even poorer segmentation performance than the pure CNN. To address these issues, a short-term and long-term memory self-attention network is proposed. Firstly, a distinctive self-attention block uses the Transformer to explore the correlation among the region features at different levels extracted by the CNN. Then, the memory structure filters and combines the above information to exclude the similar regions and detect the multiple tumours. Finally, the multi-layer reconstruction blocks will predict the tumour boundaries. Experimental results demonstrate that our method outperforms other methods in terms of subjective visual and quantitative evaluation. Compared with the most competitive method, the proposed method provides Dice (82.4% vs. 76.6%) and Hausdorff distance 95% (HD95) (10.66 vs. 11.54 mm) on the KiTS19 as well as Dice (80.2% vs. 78.4%) and HD95 (9.632 vs. 12.17 mm) on the LiTS.