Context-aware decision support in the operating room can foster surgical safety and efficiency by leveraging real-time feedback from surgical workflow analysis. Most existing works recognize surgical activities at a coarse-grained level, such as phases, steps or events, leaving out fine-grained interaction details about the surgical activity; yet those are needed for more helpful AI assistance in the operating room. Recognizing surgical actions as triplets of <instrument, verb, target> combination delivers comprehensive details about the activities taking place in surgical videos. This paper presents CholecTriplet2021: an endoscopic vision challenge organized at MICCAI 2021 for the recognition of surgical action triplets in laparoscopic videos. The challenge granted private access to the large-scale CholecT50 dataset, which is annotated with action triplet information. In this paper, we present the challenge setup and assessment of the state-of-the-art deep learning methods proposed by the participants during the challenge. A total of 4 baseline methods from the challenge organizers and 19 new deep learning algorithms by competing teams are presented to recognize surgical action triplets directly from surgical videos, achieving mean average precision (mAP) ranging from 4.2% to 38.1%. This study also analyzes the significance of the results obtained by the presented approaches, performs a thorough methodological comparison between them, in-depth result analysis, and proposes a novel ensemble method for enhanced recognition. Our analysis shows that surgical workflow analysis is not yet solved, and also highlights interesting directions for future research on fine-grained surgical activity recognition which is of utmost importance for the development of AI in surgery.

PDF Abstract

Results from the Paper


 Ranked #1 on Action Triplet Recognition on CholecT50 (Challenge) (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Benchmark
Action Triplet Recognition CholecT50 (Challenge) Team Trequartista mAP 38.1 # 1
Action Triplet Recognition CholecT50 (Challenge) Team HFUT-NUS mAP 9.8 # 23
Action Triplet Recognition CholecT50 (Challenge) Team SJTU-IMR mAP 24.8 # 17
Action Triplet Recognition CholecT50 (Challenge) Team Casia Robotics mAP 26.7 # 12
Action Triplet Recognition CholecT50 (Challenge) Team Med Recognizer mAP 4.2 # 25
Action Triplet Recognition CholecT50 (Challenge) Team CAMP mAP 9.3 # 24
Action Triplet Recognition CholecT50 (Challenge) Team NCT-TSO mAP 10.4 # 22
Action Triplet Recognition CholecT50 (Challenge) Team Band of Broeders mAP 16.0 # 21
Action Triplet Recognition CholecT50 (Challenge) Team MMLAB mAP 18.1 # 20
Action Triplet Recognition CholecT50 (Challenge) Team SK mAP 18.4 # 19
Action Triplet Recognition CholecT50 (Challenge) Team Ceaiik mAP 25.2 # 16
Action Triplet Recognition CholecT50 (Challenge) Team J&M mAP 25.6 # 14
Action Triplet Recognition CholecT50 (Challenge) Team Lsgroup mAP 26.3 # 13
Action Triplet Recognition CholecT50 (Challenge) Team Digital Surgery mAP 31.7 # 9
Action Triplet Recognition CholecT50 (Challenge) Team ANL mAP 31.9 # 8
Action Triplet Recognition CholecT50 (Challenge) Team CITI SJTU mAP 32.0 # 7
Action Triplet Recognition CholecT50 (Challenge) Team 2Ai mAP 36.9 # 2
Action Triplet Recognition CholecT50 (Challenge) Team SIAT CAMI mAP 35.8 # 3
Action Triplet Recognition CholecT50 (Challenge) Team HFUT-MedIA mAP 32.9 # 4
Action Triplet Recognition CholecT50 (Challenge) Rendezvous (TensorFlow v1) mAP 32.7 # 6
Action Triplet Recognition CholecT50 (Challenge) Attention Tripnet (TensorFlow v1) mAP 25.5 # 15

Methods


No methods listed for this paper. Add relevant methods here