Medical Report Generation
45 papers with code • 3 benchmarks • 8 datasets
Medical report generation (MRG) is a task which focus on training AI to automatically generate professional report according the input image data. This can help clinicians make faster and more accurate decision since the task itself is both time consuming and error prone even for experienced doctors.
Aggfgg
Deep neural network and transformer based architecture are currently the most popular methods for this certain task, however, when we try to transfer out pre-trained model into this certain domain, their performance always degrade.
The following are some of the reasons why RSG is hard for pre-trained models:
- Language datasets in a particular domain can sometimes be quite different from the large number of datasets available on the Internet
- During the fine-tuning phase, datasets in the medical field are often unevenly distributed
More recently, multi-modal learning and contrastive learning have shown some inspiring results in this field, but it's still challenging and requires further attention.
Here are some additional readings to go deeper on the task:
- On the Automatic Generation of Medical Imaging Reports
https://doi.org/10.48550/arXiv.1711.08195
- A scoping review of transfer learning research on medical image analysis using ImageNet
https://arxiv.org/abs/2004.13175
- A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis
https://arxiv.org/abs/2004.12150
(Image credit : Transformers in Medical Imaging: A Survey)
Libraries
Use these libraries to find Medical Report Generation models and implementationsDatasets
MIMIC-CXR
PathVQA
IU X-Ray
CASIA-CXR
Most implemented papers
On the Automatic Generation of Medical Imaging Reports
ZexinYan/Medical-Report-Generation
•
•
ACL 2018
To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the pre- diction of tags and the generation of para- graphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs.
Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation
mlii0117/COV-CTR
•
•
Firstly, the regions of primary interest to radiologists are usually located in a small area of the global image, meaning that the remainder parts of the image could be considered as irrelevant noise in the training procedure.
Cross-Modal Causal Intervention for Medical Report Generation
Medical report generation (MRG) is essential for computer-aided diagnosis and medication guidance, which can relieve the heavy burden of radiologists by automatically generating the corresponding medical reports according to the given radiology image.
Act Like a Radiologist: Radiology Report Generation across Anatomical Regions
ytongxie/x-rgen
•
•
To address these issues, we propose X-RGen, a radiologist-minded report generation framework across six anatomical regions.
CausalVLR: A Toolbox and Benchmark for Visual-Linguistic Causal Reasoning
We present CausalVLR (Causal Visual-Linguistic Reasoning), an open-source toolbox containing a rich set of state-of-the-art causal relation discovery and causal inference methods for various visual-linguistic reasoning tasks, such as VQA, image/video captioning, medical report generation, model generalization and robustness, etc.
S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenes
Grasping is among the most fundamental and long-lasting problems in robotics study.
DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation
To train and validate the effectiveness of our DNN-based module, we propose a large-scale retinal disease image dataset.
Inspecting state of the art performance and NLP metrics in image-based medical report generation
Several deep learning architectures have been proposed over the last years to deal with the problem of generating a written report given an imaging exam as input.
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
Vision-CAIR/VisualGPT
•
•
CVPR 2022
To the best of our knowledge, this is the first work that improves data efficiency of image captioning by utilizing LM pretrained on unimodal data.
Automated radiology report generation using conditioned transformers
omar-mohamed/GPT2-Chest-X-Ray-Report-Generation
•
•
We represent the first work to condition a pre-trained transformer on visual and semantic features to generate medical reports and to include semantic similarity metrics in the quantitative analysis of the generated reports.
