Learning to Read Chest X-Rays:Recurrent Neural Cascade Model for Automated Image Annotation (CVPR 2016)
Goals:
-Learn to read chest x-rays from an existing dataset of images and text with minimal human effort
-To generate text description about disease in image as well as their context (with pre-defined grammar, thus not multiple-instance-learning)
Approach
-Text-mining based image labeling;train CNN for image, RNN for text
-Extensive regularization (e.g.,batch-normalization, data dropout) to deal with data bias(normal vs. diseased)
-Joint image/text context vector for more composite image labeling
The above picture is an illustration of how joint image/text context vector is obtained. RNN's state vector (h) is initialized with the CNN image embedding (CNN(I)), and it's unrolled over the annotation sequences with the words as input. Mean-pooling is applied over the state vectors in each word of the sequence, to obtain the joint image/text vector. All RNNs share the same parameters, which are trained in the first round.
MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network (CVPR 2017)
MDNet can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize network attention.
TandemNet: Distilling Knowledge from Medical Images Using Diagnostic Reports as Optional Semantic References (MICCAI 2017)
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation (NIPS 2018)
On the Automatic Generation of Medical Imaging Reports (ACL 2018)
Datasets: IU X-Ray , PEIR Gross
ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases (CVPR 2017) Xiaosong Wang
从标题就可以看到这篇论文和Medical Image Report没啥关系, 为了便于继续学习后面的TieNet，还是将它放在这里。
TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays (CVPR 2018) Xiaosong Wang
Reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence ,due to
(1).shortage of large-scale machine-learnable medical image datasets
(2).lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training.
Contributions:
(1).proposed the Text-Image Embedding Network, which is a multi-purpose end-to-end trainable multi-task CNN-RNN framework
(2).show how raw report data, together with paired image, can be utilized to produce meaningful attention-based image and text representations using the proposed TieNet.
(3).outline how the developed text and image embeddings are able to boost the auto-annotation framework and achieve extremely high accuracy for chest x-ray labeling
(4).present a novel image classification framework which takes images as the sole input, but uses the paired text-image representations from training as a prior knowledge injection, in order to produce improved classification scores and preliminary report generations.
Datasets: ChestX-ray14, Hand-labeled, OpenI
The CNN component additionally includes a convolutional layer(transition layer) to manipulate the spatial grid size and feature dimension.
To obtain an interpretable global text and visual embedding for the purpose of classification, introduce two key enhancements in the form of the AETE and SW-GAP
AETE: Attention Encoded Text Embedding
SW-GAP: Saliecny Weighted Global Average Pooling
Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation (AAAI 2019)
Christy Y. Li, Xiaodan Liang**, Zhiting Hu, Eric Xing.
End-to-End Knowledge-Routed Relational Dialogue System for Automatic Diagnosis (AAAI 2019)
Lin Xu, Qixian Zhou, Ke Gong, Xiaodan Liang**, Jianheng Tang, Liang Lin.