Search Results for author:
Found 71 papers, 34 papers with code
Clinnova Federated Learning Proof of Concept: Key Takeaways from a Cross-border Collaboration
The IHU Strasbourg (Institute of Minimal-invasive Surgery) has the lead in this initiative to develop the federated learning (FL) proof of concept (POC) that will serve as a foundation for advancing AI in healthcare.
Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation
Surgical video-language pretraining (VLP) faces unique challenges due to the knowledge domain gap and the scarcity of multi-modal data.
Surgical Text-to-Image Generation
Human expert survey shows that participants were highly challenged by the realistic characteristics of the generated samples, demonstrating Surgical Imagen's effectiveness as a practical alternative to real data collection.
CycleSAM: One-Shot Surgical Scene Segmentation using Cycle-Consistent Feature Matching to Prompt SAM
The recently introduced Segment-Anything Model (SAM) has the potential to greatly accelerate the development of segmentation models.
Self-supervised Learning via Cluster Distance Prediction for Operating Room Context Awareness
Semantic segmentation and activity classification are key components to creating intelligent surgical systems able to understand and assist clinical workflow.
SurgiTrack: Fine-Grained Multi-Class Multi-Tool Tracking in Surgical Videos
This work sets a new standard in surgical tool tracking, providing dynamic trajectories for more adaptable and precise assistance in minimally invasive surgeries.
HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognition
By disentangling embedding spaces of different hierarchical levels, the learned multi-modal representations encode short-term and long-term surgical concepts in the same model.
On-the-Fly Point Annotation for Fast Medical Video Labeling
In this approach, a continuous single-point annotation is maintained by keeping the cursor on the object in a live video, mitigating the need for tedious pausing and repetitive navigation inherent in traditional annotation methods.
SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation
Unlike current state-of-the-art fully-supervised methods, our approach does not require any 2d or 3d ground-truth poses and uses only the multi-view input images from a calibrated camera setup and 2d pseudo poses generated from an off-the-shelf 2d human pose estimator.
Enhancing Gait Video Analysis in Neurodegenerative Diseases by Knowledge Augmentation in Vision Language Model
Based on a large-scale pre-trained Vision Language Model (VLM), our model learns and improves visual, textual, and numerical representations of patient gait videos, through a collective learning across three distinct modalities: gait videos, class-specific descriptions, and numerical gait parameters.
Optimizing Latent Graph Representations of Surgical Scenes for Zero-Shot Domain Transfer
Purpose: Advances in deep learning have resulted in effective models for surgical video analysis; however, these models often fail to generalize across medical centers due to domain shift caused by variations in surgical workflow, camera setups, and patient demographics.
Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentation
In this work, we investigate the application of contrastive learning to the domain of medical image analysis.
ST(OR)2: Spatio-Temporal Object Level Reasoning for Activity Recognition in the Operating Room
Surgical robotics holds much promise for improving patient safety and clinician experience in the Operating Room (OR).
The Endoscapes Dataset for Surgical Scene Segmentation, Object Detection, and Critical View of Safety Assessment: Official Splits and Benchmark
This technical report provides a detailed overview of Endoscapes, a dataset of laparoscopic cholecystectomy (LC) videos with highly intricate annotations targeted at automated assessment of the Critical View of Safety (CVS).
Challenges in Multi-centric Generalization: Phase and Step Recognition in Roux-en-Y Gastric Bypass Surgery
The use of multi-centric training data, experiments 6) and 7), improves the generalization capabilities of the models, bringing them beyond the level of independent mono-centric training and validation (experiments 1) and 2)).
Advancing Surgical VQA with Scene Graph Knowledge
We highlight that the primary limitation in the current surgical VQA systems is the lack of scene knowledge to answer complex queries.
MOSaiC: a Web-based Platform for Collaborative Medical Video Assessment and Annotation
This technical report presents MOSaiC 3. 6. 2, a web-based collaborative platform designed for the annotation and evaluation of medical videos.
CholecTrack20: A Dataset for Multi-Class Multiple Tool Tracking in Laparoscopic Surgery
Tool tracking in surgical videos is vital in computer-assisted intervention for tasks like surgeon skill assessment, safety zone estimation, and human-machine collaboration during minimally invasive procedures.
Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning
Recently, spatiotemporal graphs have emerged as a concise and elegant manner of representing video clips in an object-centric fashion, and have shown to be useful for downstream tasks such as action recognition.
Jumpstarting Surgical Computer Vision
Methods: In this work, we employ self-supervised learning to flexibly leverage diverse surgical datasets, thereby learning taskagnostic representations that can be used for various surgical downstream tasks.
SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of Surgical Tools
In this work, we develop a framework for instance segmentation not relying on spatial annotations for training.
Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures
We demonstrate the representational capability of this space through several vision-and-language surgical tasks and vision-only tasks specific to surgery.
Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions
We analyze how the amount of instrument spatial annotations affects triplet detection and observe that accurate instrument localization does not guarantee better triplet detection due to the risk of erroneous associations with the verbs and targets.
INDEXITY: a web-based collaborative tool for medical video annotation
This technical report presents Indexity 1. 4. 0, a web-based tool designed for medical video annotation in surgical data science projects.
Surgical tool classification and localization: results and methods from the MICCAI 2022 SurgToolLoc challenge
Unfortunately, obtaining the annotations needed to train machine learning models to identify and localize surgical tools is a difficult task.
Why is the winner the best?
The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning.
Self-distillation for surgical action recognition
Surgical scene understanding is a key prerequisite for contextaware decision support in the operating room.
Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition
In this work, we propose to use coarser and easier-to-annotate activity labels, namely phases, as weak supervision to learn step recognition with fewer step annotated videos.
CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection
This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection.
Ranked #1 on
Action Triplet Detection
on CholecT50 (Challenge)
Preserving Privacy in Surgical Video Analysis Using Artificial Intelligence: A Deep Learning Classifier to Identify Out-of-Body Scenes in Endoscopic Videos
Results: The internal dataset consisting of 356, 267 images from 48 videos and the two multicentric test datasets consisting of 54, 385 and 58, 349 images from 10 and 20 videos, respectively, were annotated.
Biomedical image analysis competitions: The state of current participation practice
Of these, 84% were based on standard architectures.
Real-Time Artificial Intelligence Assistance for Safe Laparoscopic Cholecystectomy: Early-Stage Clinical Evaluation
Artificial intelligence is set to be deployed in operating rooms to improve surgical care.
Latent Graph Representations for Critical View of Safety Assessment
Assessing the critical view of safety in laparoscopic cholecystectomy requires accurate identification and localization of key anatomical structures, reasoning about their geometric relationships to one another, and determining the quality of their exposure.
Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognition
Focusing more on the verbs, our RiT explores the connectedness of current and past frames to learn temporal attention-based features for enhanced triplet recognition.
Ranked #2 on
Action Triplet Recognition
on CholecT45 (cross-val)
Dissecting Self-Supervised Learning Methods for Surgical Computer Vision
Correct transfer of these methods to surgery, as described and conducted in this work, leads to substantial performance gains over generic uses of SSL - up to 7. 4% on phase recognition and 20% on tool presence detection - as well as state-of-the-art semi-supervised phase recognition approaches by up to 14%.
Ranked #1 on
Semantic Segmentation
on Endoscapes
Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets
We also develop a metrics library, ivtmetrics, for model evaluation on surgical triplets.
Ranked #1 on
Action Triplet Recognition
on CholecT45
CholecTriplet2021: A benchmark challenge for surgical action triplet recognition
In this paper, we present the challenge setup and assessment of the state-of-the-art deep learning methods proposed by the participants during the challenge.
Ranked #1 on
Action Triplet Recognition
on CholecT50 (Challenge)
(using extra training data)
Federated Cycling (FedCy): Semi-supervised Federated Learning of Surgical Phases
With these constraints in mind, we propose FedCy, a federated semi-supervised learning (FSSL) method that combines FL and self-supervised learning to exploit a decentralized dataset of both labeled and unlabeled videos, thereby improving performance on the task of surgical phase recognition.
Live Laparoscopic Video Retrieval with Compressed Uncertainty
Searching through large volumes of medical data to retrieve relevant information is a challenging yet crucial task for clinical care.
FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation
We then use the obtained instrument masks as pseudo-labels in order to train a per-frame segmentation model; to this aim, we develop a learning-from-noisy-labels architecture, designed to extract a clean supervision signal from these pseudo-labels, leveraging their peculiar noise properties.
Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentation
A major obstacle to building models for effective semantic segmentation, and particularly video semantic segmentation, is a lack of large and well annotated datasets.
Ranked #2 on
Semantic Segmentation
on Endoscapes
2020 CATARACTS Semantic Segmentation Challenge
In 2020, we released pixel-wise semantic annotations for anatomy and instruments for 4670 images sampled from 25 videos of the CATARACTS training set.
Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark
PURPOSE: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems.
Ranked #1 on
Surgical phase recognition
on HeiChole Benchmark
MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation
Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience.
Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos
To achieve this task, we introduce our new model, the Rendezvous (RDV), which recognizes triplets directly from surgical videos by leveraging attention at two different levels.
Ranked #1 on
Action Triplet Recognition
on CholecT50
Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating room
Second, to address the domain shift and the lack of annotations, we propose a novel unsupervised domain adaptation method, called AdaptOR, to adapt a model from an in-the-wild labeled source domain to a statistically different unlabeled target domain.
Surgical data science for safe cholecystectomy: a protocol for segmentation of hepatocystic anatomy and assessment of the critical view of safety
Minimally invasive image-guided surgery heavily relies on vision.
Intraoperative time out to promote the implementation of the critical view of safety in laparoscopic cholecystectomy: a video-based assessment of 343 procedures
The primary endpoint was to compare the rate of CVS achievement between LCs performed in the year before and the year after the 5-second rule.
A Kinematic Bottleneck Approach For Pose Regression of Flexible Surgical Instruments directly from Images
3-D pose estimation of instruments is a crucial step towards automatic scene understanding in robotic minimally invasive surgery.
Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures
Conclusion: In this work, we present a multi-task multi-stage temporal convolutional network for surgical activity recognition, which shows improved results compared to single-task models on the Bypass40 gastric bypass dataset with multi-level annotations.
Surgical Data Science -- from Concepts toward Clinical Translation
We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.
Encode the Unseen: Predictive Video Hashing for Scalable Mid-Stream Retrieval
This paper tackles a new problem in computer vision: mid-stream video-to-video retrieval.
Artificial Intelligence in Surgery: Neural Networks and Deep Learning
Deep neural networks power most recent successes of artificial intelligence, spanning from self-driving cars to computer aided diagnosis in radiology and pathology.
Human Pose Estimation on Privacy-Preserving Low-Resolution Depth Images
Human pose estimation (HPE) is a key building block for developing AI-based context-aware systems inside the operating room (OR).
Self-supervision on Unlabelled OR Data for Multi-person 2D/3D Human Pose Estimation
2D/3D human pose estimation is needed to develop novel intelligent tools for the operating room that can analyze and support the clinical activities.
Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets
Recognition of surgical activity is an essential component to develop context-aware decision support for the operating room.
Ranked #1 on
Action Triplet Recognition
on CholecT40
CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions
Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice.
Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videos
Results: We build a baseline tracker on top of the CNN model and demonstrate that our approach based on the ConvLSTM outperforms the baseline in tool presence detection, spatial localization, and motion tracking by over 5. 0%, 13. 9%, and 12. 6%, respectively.
Ranked #2 on
Surgical tool detection
on Cholec80
Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition
Vision algorithms capable of interpreting scenes from a real-time video stream are necessary for computer-assisted surgery systems to achieve context-aware behavior.
Face Detection in the Operating Room: Comparison of State-of-the-art Methods and a Self-supervised Approach
Methods: We propose a comparison of 6 state-of-the-art face detectors on clinical data using Multi-View Operating Room Faces (MVOR-Faces), a dataset of operating room images capturing real surgical activities.
Future-State Predicting LSTM for Early Surgery Type Recognition
This work presents a novel approach for the early recognition of the type of a laparoscopic surgery from its video.
MVOR: A Multi-view RGB-D Operating Room Dataset for 2D and 3D Human Pose Estimation
In this paper, we present the dataset, its annotations, as well as baseline results from several recent person detection and 2D/3D pose estimation methods.
Weakly-Supervised Learning for Tool Localization in Laparoscopic Videos
We propose a deep architecture, trained solely on image level annotations, that can be used for both tool presence detection and localization in surgical videos.
Ranked #4 on
Surgical tool detection
on Cholec80
Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networks
In this work, we propose a new self-supervised pre-training approach based on the prediction of remaining surgery duration (RSD) from laparoscopic videos.
A generalizable approach for multi-view 3D human pose regression
As 2D poses are collected at test time using a SV pose detector, which might generate inaccurate detections, we model its characteristics and incorporate this information during training.
RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations
In this paper, we propose a deep learning pipeline, referred to as RSDNet, which automatically estimates the remaining surgery duration (RSD) intraoperatively by using only visual information from laparoscopic videos.
A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms
In this paper, we propose an approach for multi-view 3D human pose estimation from RGB-D images and demonstrate the benefits of using the additional depth channel for pose refinement beyond its use for the generation of improved features.
3D Human Pose Estimation
Multi-view 3D Human Pose Estimation
Single- and Multi-Task Architectures for Tool Presence Detection Challenge at M2CAI 2016
The tool presence detection challenge at M2CAI 2016 consists of identifying the presence/absence of seven surgical tools in the images of cholecystectomy videos.
Single- and Multi-Task Architectures for Surgical Workflow Challenge at M2CAI 2016
On top of these architectures we propose to use two different approaches to enforce the temporal constraints of the surgical workflow: (1) HMM-based and (2) LSTM-based pipelines.
Articulated Clinician Detection Using 3D Pictorial Structures on RGB-D Data
Proposed methods for the operating room (OR) rely either on foreground estimation using a multi-camera system, which is a challenge in real ORs due to color similarities and frequent illumination changes, or on wearable sensors or markers, which are invasive and therefore difficult to introduce in the room.
EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos
In the literature, two types of features are typically used to perform this task: visual features and tool usage signals.
Ranked #5 on
Surgical tool detection
on Cholec80
Offline surgical phase recognition
Online surgical phase recognition
+2
