Medical Report Generation

30 papers with code • 2 benchmarks • 3 datasets

Medical report generation (MRG) is a task which focus on training AI to automatically generate professional report according the input image data. This can help clinicians make faster and more accurate decision since the task itself is both time consuming and error prone even for experienced doctors.

Deep neural network and transformer based architecture are currently the most popular methods for this certain task, however, when we try to transfer out pre-trained model into this certain domain, their performance always degrade.

The following are some of the reasons why RSG is hard for pre-trained models:

  • Language datasets in a particular domain can sometimes be quite different from the large number of datasets available on the Internet
  • During the fine-tuning phase, datasets in the medical field are often unevenly distributed

More recently, multi-modal learning and contrastive learning have shown some inspiring results in this field, but it's still challenging and requires further attention.

Here are some additional readings to go deeper on the task:

  • On the Automatic Generation of Medical Imaging Reports

  • A scoping review of transfer learning research on medical image analysis using ImageNet

  • A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis

(Image credit : Transformers in Medical Imaging: A Survey)


Use these libraries to find Medical Report Generation models and implementations

Most implemented papers

On the Automatic Generation of Medical Imaging Reports

ZexinYan/Medical-Report-Generation ACL 2018

To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the pre- diction of tags and the generation of para- graphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs.

Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation

mlii0117/COV-CTR 6 Jun 2020

Firstly, the regions of primary interest to radiologists are usually located in a small area of the global image, meaning that the remainder parts of the image could be considered as irrelevant noise in the training procedure.

Cross-Modal Causal Intervention for Medical Report Generation

hcplab-sysu/causalvlr 16 Mar 2023

Medical report generation (MRG) is essential for computer-aided diagnosis and medication guidance, which can relieve the heavy burden of radiologists by automatically generating the corresponding medical reports according to the given radiology image.

CausalVLR: A Toolbox and Benchmark for Visual-Linguistic Causal Reasoning

hcplab-sysu/causalvlr 30 Jun 2023

We present CausalVLR (Causal Visual-Linguistic Reasoning), an open-source toolbox containing a rich set of state-of-the-art causal relation discovery and causal inference methods for various visual-linguistic reasoning tasks, such as VQA, image/video captioning, medical report generation, model generalization and robustness, etc.

Inspecting state of the art performance and NLP metrics in image-based medical report generation

pdpino/medical-ai 18 Nov 2020

Several deep learning architectures have been proposed over the last years to deal with the problem of generating a written report given an imaging exam as input.

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

Vision-CAIR/VisualGPT CVPR 2022

To the best of our knowledge, this is the first work that improves data efficiency of image captioning by utilizing LM pretrained on unimodal data.

Automated radiology report generation using conditioned transformers

omar-mohamed/GPT2-Chest-X-Ray-Report-Generation 26 Mar 2021

We represent the first work to condition a pre-trained transformer on visual and semantic features to generate medical reports and to include semantic similarity metrics in the quantitative analysis of the generated reports.

FFA-IR: Towards an Explainable and Reliable Medical Report Generation Benchmark

mlii0117/FFA-IR Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) 2021

Researchers have explored advanced methods from computer vision and natural language processing to incorporate medical domain knowledge for the generation of readable medical reports.

Weakly Supervised Contrastive Learning for Chest X-Ray Report Generation

zzxslp/wcl Findings (EMNLP) 2021

Radiology report generation aims at generating descriptive text from radiology images automatically, which may present an opportunity to improve radiology reporting and interpretation.