Search Results for author:
Found 79 papers, 29 papers with code
An Intrinsically Explainable Approach to Detecting Vertebral Compression Fractures in CT Scans via Neurosymbolic Modeling
Evaluation of VerSe19 dataset shows that our method achieves an accuracy of 96% and a sensitivity of 91% in VCF detection.
Memorizing SAM: 3D Medical Segment Anything Model with Memorizing Transformer
In this work, we investigate whether introducing the memory mechanism as a plug-in, specifically the ability to memorize and recall internal representations of past inputs, can improve the performance of SAM with limited computation cost.
Intelligent Control of Robotic X-ray Devices using a Language-promptable Digital Twin
The fixed outputs of such AI models limit the functionality of language controls.
Neural Finite-State Machines for Surgical Phase Recognition
Surgical phase recognition is essential for analyzing procedure-specific surgical videos.
Causality-Driven Audits of Model Robustness
Robustness audits of deep neural networks (DNN) provide a means to uncover model sensitivities to the challenging real-world imaging conditions that significantly degrade DNN performance in-the-wild.
Towards Robust Algorithms for Surgical Phase Recognition via Digital Twin-based Scene Representation
This approach takes advantage of the recent vision foundation models that ensure reliable low-level scene understanding to craft DT-based scene representations that support various high-level tasks.
ConceptAgent: LLM-Driven Precondition Grounding and Tree Search for Robust Task Planning and Execution
Robotic planning and execution in open-world environments is a complex problem due to the vast state spaces and high variability of task embodiment.
Seamless Augmented Reality Integration in Arthroscopy: A Pipeline for Articular Reconstruction and Guidance
Extending 3D reconstruction to Augmented Reality (AR) applications, our solution offers AR assistance for articular notch measurement and annotation anchoring in a human-in-the-loop manner.
Performance and Non-adversarial Robustness of the Segment Anything Model 2 in Surgical Video Segmentation
Fully supervised deep learning (DL) models for surgical video segmentation have been shown to struggle with non-adversarial, real-world corruptions of image quality including smoke, bleeding, and low illumination.
FastSAM-3DSlicer: A 3D-Slicer Extension for 3D Volumetric Segment Anything Model with Uncertainty Quantification
Accurate segmentation of anatomical structures and pathological regions in medical images is crucial for diagnosis, treatment planning, and disease monitoring.
Promptable Counterfactual Diffusion Model for Unified Brain Tumor Segmentation and Generation with MRIs
We introduce a novel promptable counterfactual diffusion model as a unified solution for brain tumor segmentation and generation in MRI.
SegSTRONG-C: Segmenting Surgical Tools Robustly On Non-adversarial Generated Corruptions -- An EndoVis'24 Challenge
To address this limitation, we introduce the SegSTRONG-C challenge that aims to promote the development of algorithms robust to unforeseen but plausible image corruptions of surgery, like smoke, bleeding, and low brightness.
FastSAM3D: An Efficient Segment Anything Model for 3D Volumetric Medical Images
Segment anything models (SAMs) are gaining attention for their zero-shot generalization capability in segmenting objects of unseen classes and in unseen domains when properly prompted.
FluoroSAM: A Language-aligned Foundation Model for X-ray Image Segmentation
Recently, foundation models (FMs) -- machine learning models trained on large amounts of highly variable data thus enabling broad applicability -- have emerged as promising tools for automated image analysis.
From Generalization to Precision: Exploring SAM for Tool Segmentation in Surgical Environments
We argue that SAM drastically over-segment images with high corruption levels, resulting in degraded performance when only a single segmentation mask is considered, while the combination of the masks overlapping the object of interest generates an accurate prediction.
An Endoscopic Chisel: Intraoperative Imaging Carves 3D Anatomical Models
Purpose: Preoperative imaging plays a pivotal role in sinus surgery where CTs offer patient-specific insights of complex anatomy, enabling real-time intraoperative navigation to complement endoscopy imaging.
Human-AI collaboration is not very collaborative yet: A taxonomy of interaction patterns in AI-assisted decision making from a systematic review
However, how the information is presented, e. g., the sequence of recommendations and solicitation of interpretations, is equally crucial as complex interactions may emerge between humans and AI.
A Quantitative Evaluation of Dense 3D Reconstruction of Sinus Anatomy from Monocular Endoscopic Video
In this work, we perform a quantitative analysis of a self-supervised approach for sinus reconstruction using endoscopic sequences paired with optical tracking and high-resolution computed tomography acquired from nine ex-vivo specimens.
RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in Object-centric Learning
Lastly, while conventional robustness evaluations view corruptions as out-of-distribution, we use our causal framework to show that even training on in-distribution image corruptions does not guarantee increased model robustness.
TransNuSeg: A Lightweight Multi-Task Transformer for Nuclei Segmentation
In conclusion, TransNuSeg confirms the strength of Transformer in the context of nuclei segmentation, which thus can serve as an efficient solution for real clinical practice.
Automated Artifact Detection in Ultra-widefield Fundus Photography of Patients with Sickle Cell Disease
A pre-trained ResNet-50 neural network was trained on a subset of the images and the classification accuracy, sensitivity, and specificity were quantified on the hold out test set.
Neuralangelo: High-Fidelity Neural Surface Reconstruction
Neural surface reconstruction has been shown to be powerful for recovering dense 3D surfaces via image-based neural rendering.
Pelphix: Surgical Phase Recognition from X-ray Images in Percutaneous Pelvic Fixation
Surgical phase recognition (SPR) is a crucial element in the digital transformation of the modern operating theater.
Data AUDIT: Identifying Attribute Utility- and Detectability-Induced Bias in Task Models
To safely deploy deep learning-based computer vision models for computer-aided detection and diagnosis, we must ensure that they are robust and reliable.
MoViT: Memorizing Vision Transformers for Medical Image Analysis
We evaluate our method on a public histology image dataset and an in-house MRI dataset, demonstrating that MoViT applied to varied medical image analysis tasks, can outperform vanilla transformer models across varied data regimes, especially in cases where only a small amount of annotated data is available.
Task-based Generation of Optimized Projection Sets using Differentiable Ranking
We present a method for selecting valuable projections in computed tomography (CT) scans to enhance image reconstruction and diagnosis.
TAToo: Vision-based Joint Tracking of Anatomy and Tool for Skull-base Surgery
TAToo jointly tracks the rigid 3D motion of patient skull and surgical drill from stereo microscopic videos.
Rethinking Causality-driven Robot Tool Segmentation with Temporal Constraints
Method: To address the above limitations, we take temporal relation into consideration and propose a temporal causal model for robot tool segmentation on video sequences.
Twin-S: A Digital Twin for Skull-base Surgery
Twin-S tracks and updates the virtual model in real-time given measurements from modern tracking technologies.
Context-Enhanced Stereo Transformer
We construct our stereo depth estimation model, Context Enhanced Stereo Transformer (CSTR), by plugging CEP into the state-of-the-art stereo depth estimation method Stereo Transformer.
SyntheX: Scaling Up Learning-based X-ray Image Analysis Through In Silico Experiments
Here, we demonstrate that creating realistic simulated images from human models is a viable alternative and complement to large-scale in situ data collection.
CaRTS: Causality-driven Robot Tool Segmentation from Vision and Kinematics Data
Vision-based segmentation of the robotic tool during robot-assisted surgery enables downstream applications, such as augmented reality feedback, while allowing for inaccuracies in robot kinematics.
SAGE: SLAM with Appearance and Geometry Prior for Endoscopy
In endoscopy, many applications (e. g., surgical navigation) would benefit from a real-time method that can simultaneously track the endoscope and reconstruct the dense 3D geometry of the observed anatomy from a monocular endoscopic video.
Mapping DNN Embedding Manifolds for Network Generalization Prediction
Understanding Deep Neural Network (DNN) performance in changing conditions is essential for deploying DNNs in safety critical applications with unconstrained environments, e. g., perception for self-driving vehicles or medical image analysis.
Explainable Medical Imaging AI Needs Human-Centered Design: Guidelines and Evidence from a Systematic Review
To alleviate these shortcomings in forthcoming research while acknowledging the challenges of human-centered design in healthcare, we introduce the INTRPRT guideline, a systematic design directive for transparent ML systems in medical image analysis.
A Systematic Review of Robustness in Deep Learning for Computer Vision: Mind the gap?
We find this area of research has received disproportionately less attention relative to adversarial machine learning, yet a significant robustness gap exists that manifests in performance degradation similar in magnitude to adversarial conditions.
Temporally Consistent Online Depth Estimation in Dynamic Scenes
We present a framework named Consistent Online Dynamic Depth (CODD) to produce temporally consistent depth estimates in dynamic scenes in an online setting.
Pediatric Otoscopy Video Screening with Shift Contrastive Anomaly Detection
Our method achieves an AUROC of 88. 0% on the patient-level and also outperforms the average of a group of 25 clinicians in a comparative study, which is the largest of such published to date.
On the Sins of Image Synthesis Loss for Self-supervised Depth Estimation
It is based on the idea that observed frames can be synthesized from neighboring frames if accurate depth of the scene is known - or in this case, estimated.
An Interpretable Algorithm for Uveal Melanoma Subtyping from Whole Slide Cytology Images
Algorithmic decision support is rapidly becoming a staple of personalized medicine, especially for high-stakes recommendations in which access to certain information can drastically alter the course of treatment, and thus, patient outcome; a prominent example is radiomics for cancer subtyping.
The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions: A Systematic Review and Perspective
Image-based navigation is widely considered the next frontier of minimally invasive surgery.
E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception
After that, a dynamic reconstruction algorithm which can estimate the tissue deformation and camera movement, and aggregate the information over time is proposed for surgical scene reconstruction.
Neighborhood Normalization for Robust Geometric Feature Learning
Extracting geometric features from 3D models is a common first step in applications such as 3D registration, tracking, and scene flow estimation.
An Interpretable Approach to Automated Severity Scoring in Pelvic Trauma
The method operates similarly to human interpretation of CT scans and first detects distinct pelvic fractures on CT with high specificity using a Faster-RCNN model that are then interpreted using a structural causal model based on clinical best practices to infer an initial Tile grade.
Pose-dependent weights and Domain Randomization for fully automatic X-ray to CT Registration
First, a neural network is trained once to detect a set of anatomical landmarks on simulated X-rays.
Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers
Stereo depth estimation relies on optimal correspondence matching between pixels on epipolar lines in the left and right images to infer depth.
Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery
Automatic surgical gesture recognition is fundamentally important to enable intelligent cognitive assistance in robotic surgery.
Ranked #1 on
Action Segmentation
on JIGSAWS
Multimodal and self-supervised representation learning for automatic gesture recognition in surgical robotics
We develop a self-supervised, multi-modal representation learning paradigm that learns representations for surgical gestures from video and kinematics.
A Learning-based Method for Online Adjustment of C-arm Cone-Beam CT Source Trajectories for Artifact Avoidance
We propose to adjust the C-arm CBCT source trajectory during the scan to optimize reconstruction quality with respect to a certain task, i. e. verification of screw placement.
Augment Yourself: Mixed Reality Self-Augmentation Using Optical See-through Head-mounted Displays and Physical Mirrors
Consequently, most MR applications that are centered around the user, such as virtual dressing rooms or learning of body movements, cannot be realized with HMDs.
Artificial Intelligence-based Clinical Decision Support for COVID-19 -- Where Art Thou?
The COVID-19 crisis has brought about new clinical questions, new workflows, and accelerated distributed healthcare needs.
A County-level Dataset for Informing the United States' Response to COVID-19
As the coronavirus disease 2019 (COVID-19) becomes a global pandemic, policy makers must enact interventions to stop its spread.
Computers and Society Databases Physics and Society Populations and Evolution
Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration
We propose a novel Projective Spatial Transformer module that generalizes spatial transformers to projective geometry, thus enabling differentiable volume rendering.
Reconstructing Sinus Anatomy from Endoscopic Video -- Towards a Radiation-free Approach for Quantitative Longitudinal Assessment
Reconstructing accurate 3D surface models of sinus anatomy directly from an endoscopic video is a promising avenue for cross-sectional and longitudinal analysis to better understand the relationship between sinus anatomy and surgical outcomes.
Leveraging Vision and Kinematics Data to Improve Realism of Biomechanic Soft-tissue Simulation for Robotic Surgery
We train a network to predict this correction factor.
From Perspective X-ray Imaging to Parallax-Robust Orthographic Stitching
Stitching images acquired under perspective projective geometry is a relevant topic in computer vision with multiple applications ranging from smartphone panoramas to the construction of digital maps.
Exploring Partial Intrinsic and Extrinsic Symmetry in 3D Medical Imaging
We present a novel methodology to detect imperfect bilateral symmetry in CT of human anatomy.
Spatiotemporal-Aware Augmented Reality: Redefining HCI in Image-Guided Therapy
Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies.
Extremely Dense Point Correspondences using a Learned Feature Descriptor
In direct comparison to recent local and dense descriptors on an in-house sinus endoscopy dataset, we demonstrate that our proposed dense descriptor can generalize to unseen patients and scopes, thereby largely improving the performance of Structure from Motion (SfM) in terms of model density and completeness.
2018 Robotic Scene Segmentation Challenge
In 2015 we began a sub-challenge at the EndoVis workshop at MICCAI in Munich using endoscope images of ex-vivo tissue with automatically generated annotations from robot forward kinematics and instrument CAD models.
Automatic Annotation of Hip Anatomy in Fluoroscopy for Robust and Efficient 2D/3D Registration
By using these annotations as training data for neural networks, state of the art performance in fluoroscopic segmentation and landmark localization was achieved.
Fast and Automatic Periacetabular Osteotomy Fragment Pose Estimation Using Intraoperatively Implanted Fiducials and Single-View Fluoroscopy
The relative pose of the fragment is established by estimating the movement of the two BB constellations using a single fluoroscopic view taken after osteotomy and fragment relocation.
CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions
Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice.
Learning to Avoid Poor Images: Towards Task-aware C-arm Cone-beam CT Trajectories
Metal artifacts in computed tomography (CT) arise from a mismatch between physics of image formation and idealized assumptions during tomographic reconstruction.
Self-supervised Dense 3D Reconstruction from Monocular Endoscopic Video
We present a self-supervised learning-based pipeline for dense 3D reconstruction from full-length monocular endoscopic videos without a priori modeling of anatomy or shading.
Reflective-AR Display: An Interaction Methodology for Virtual-Real Alignment in Medical Robotics
To overcome this challenge, we introduce a novel registration concept for intuitive alignment of AR content to its physical counterpart by providing a multi-view AR experience via reflective-AR displays that simultaneously show the augmentations from multiple viewpoints.
LumiPath -- Towards Real-time Physically-based Rendering on Embedded Devices
With the increasing computational power of today's workstations, real-time physically-based rendering is within reach, rapidly gaining attention across a variety of domains.
Dense Depth Estimation in Monocular Endoscopy with Self-supervised Learning Methods
We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading.
Localizing dexterous surgical tools in X-ray for image-based navigation
This manuscript describes a first step towards leveraging semantic information of the imaged object to initialize 2D/3D registration within the capture range of image-based registration by performing concurrent segmentation and localization of dexterous surgical tools in X-ray images.
Learning to See Forces: Surgical Force Prediction with RGB-Point Cloud Temporal Convolutional Networks
Our method results in a mean absolute error of 0. 814 N in the ex vivo study, suggesting that it may be a promising alternative to hardware based surgical force feedback in endoscopic procedures.
Self-supervised Learning for Dense Depth Estimation in Monocular Endoscopy
We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading.
Augmented Reality-based Feedback for Technician-in-the-loop C-arm Repositioning
For C-arm repositioning to a particular target view, the recorded C-arm pose is restored as a virtual object and visualized in an AR environment, serving as a perceptual reference for the technician.
Exploiting Partial Structural Symmetry For Patient-Specific Image Augmentation in Trauma Interventions
The main challenge is to automatically estimate the desired plane of symmetry within the patient's pre-operative CT. We propose to estimate this plane using a non-linear optimization strategy, by minimizing Tukey's biweight robust estimator, relying on the partial symmetry of the anatomy.
Closing the Calibration Loop: An Inside-out-tracking Paradigm for Augmented Reality in Orthopedic Surgery
In percutaneous orthopedic interventions the surgeon attempts to reduce and fixate fractures in bony structures.
DeepDRR -- A Catalyst for Machine Learning in Fluoroscopy-guided Procedures
Machine learning-based approaches outperform competing methods in most disciplines relevant to diagnostic radiology.
X-ray-transform Invariant Anatomical Landmark Detection for Pelvic Trauma Surgery
In this work, we present a method to automatically detect anatomical landmarks in X-ray images independent of the viewing direction.
On-the-fly Augmented Reality for Orthopaedic Surgery Using a Multi-Modal Fiducial
Then, annotations on the 2D X-ray images can be rendered as virtual objects in 3D providing surgical guidance.
Plan in 2D, execute in 3D: An augmented reality solution for cup placement in total hip arthroplasty
Reproducibly achieving proper implant alignment is a critical step in total hip arthroplasty (THA) procedures that has been shown to substantially affect patient outcome.
UI-Net: Interactive Artificial Neural Networks for Iterative Image Segmentation Based on a User Model
Our system builds upon a state-of-the-art fully convolutional artificial neural network (FCN) as well as an active user model for training.
