Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets

11 Apr 2022  ·  Chinedu Innocent Nwoye, Nicolas Padoy ·

In addition to generating data and annotations, devising sensible data splitting strategies and evaluation metrics is essential for the creation of a benchmark dataset. This practice ensures consensus on the usage of the data, homogeneous assessment, and uniform comparison of research methods on the dataset. This study focuses on CholecT50, which is a 50 video surgical dataset that formalizes surgical activities as triplets of <instrument, verb, target>. In this paper, we introduce the standard splits for the CholecT50 and CholecT45 datasets and show how they compare with existing use of the dataset. CholecT45 is the first public release of 45 videos of CholecT50 dataset. We also develop a metrics library, ivtmetrics, for model evaluation on surgical triplets. Furthermore, we conduct a benchmark study by reproducing baseline methods in the most predominantly used deep learning frameworks (PyTorch and TensorFlow) to evaluate them using the proposed data splits and metrics and release them publicly to support future research. The proposed data splits and evaluation metrics will enable global tracking of research progress on the dataset and facilitate optimal model selection for further deployment.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Action Triplet Recognition CholecT45 Rendezvous mAP 29.4±2.8 # 1
Action Triplet Recognition CholecT45 Attention Tripnet mAP 27.2±2.7 # 2
Action Triplet Recognition CholecT45 Tripnet mAP 24.4±4.7 # 3
Action Triplet Recognition CholecT45 (cross-val) Rendezvous mAP 29.4±2.8 # 1
Action Triplet Recognition CholecT45 (cross-val) Tripnet mAP 24.4±4.7 # 3
Action Triplet Recognition CholecT45 (cross-val) Attention Tripnet mAP 27.2±2.7 # 2
Action Triplet Recognition CholecT50 Rendezvous (PyTorch) Mean AP 29.5 # 2
Action Triplet Recognition CholecT50 Tripnet (PyTorch) Mean AP 21.6 # 5
Action Triplet Recognition CholecT50 Attention Tripnet (PyTorch) Mean AP 23.3 # 4
Action Triplet Recognition CholecT50 (Challenge) Tripnet (PyTorch) mAP 27.4 # 11
Action Triplet Recognition CholecT50 (Challenge) Rendezvous (PyTorch) mAP 32.8 # 5
Action Triplet Recognition CholecT50 (Challenge) Attention Tripnet (PyTorch) mAP 27.7 # 10
Action Triplet Recognition CholecT50 (cross-val) Tripnet mAP 25.3±2.4 # 3
Action Triplet Recognition CholecT50 (cross-val) Attention Tripnet mAP 27.2±2.9 # 2
Action Triplet Recognition CholecT50 (cross-val) Rendezvous mAP 29.4±2.5 # 1

Methods


No methods listed for this paper. Add relevant methods here