In total 806 CAD marks with a high score of suspicion were analyzed for underlying pathology. In 68 cases (8.4%) the clinical patient record did not give clear proof of the underlying pathology and the underlying pathology was determined by an experienced radiologist. The radiologist determined that out of the 68 CAD marks with unclear under- lying pathology, 5 were considered as indicating tumors (2 with benign characteristics), 8 were considered pointing to other significant findings, 7 were classified as non significant findings and 48 were classified as non obvious false positives. CAD marks that were considered as obvious false positives were not checked for underlying pathology information from the clinical patient records.
A minority of the 806 CAD marks were located on abnormalities (n=331). CAD identified 148 tumors, 97 other significant findings, and 86 non significant abnormal findings. Of the 806 CAD marks 475 could be classified as false positives, of which 341 were rated as obvious false positives and 134 were considered to be non-obvious false positives. Detailed information of the structures highlighted by CAD can be found in Table 2.
Figure 3 Distribution of CAD score among the CAD marks. The bars in dark grey indicate the portion of CAD marks that received a high score of suspicion (≥80), and that were analyzed in this study. 0 100 200 300 400 500 600 700 800 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99
Number of CAD marks
Two tumors that were identified by CAD were not mentioned in the radiological report (Figure 4). Three other malignant lesions were not reported as a significant finding in the report (1 scarring, 1 infiltrate, 1 as old rib fracture). Two benign nodules were not mentioned in the report. Four other significant findings were also not reported (3 infiltrates, and 1 lymphadenopathy). All above abnormalities that were not mentioned in the radiological report, the underlying pathology was based on additional information from the patient records, and not determined by the experienced radiologist alone.
Negative Predictive Value of CAD
In total 1381 images did not receive any CAD mark, compared with 1018 radiographs that were rated normal based on the radiological report. From both, the CAD identified normal radiographs and normal radiographs based on the radiological report, a random sample of 500 cases was analyzed. Results of this analysis are shown in Table 3.
Table 2 Underlying pathology of the suspicious areas detected by CAD.
number Abnormal findings 331 Tumor - Lung tumor - Metastasis - Benign 148 54 65 29 Other significant - Infiltrate - Lymphadenopathy - Pleural fluid - Other 97 81 4 10 2 Non significant
- Abnormal rib structures - Pleural lesions - Other 86 33 34 19 Normal findings 475 Obvious - Foreign bodies - Diaphragm - Other 341 287 48 6 Non-obvious
- Hilar vascular shadow - Rib crossing - Retrocardiac - Nipple shadow
- Normal vascular structure
134 54 35 18 17 10
Figure 4 Two cases with a tumor that was not reported by the radiologist, but detected by CAD. Top: 90 year old male with a 22 mm malignant lesion in the left lung. Bottom: 77 year old male with a 10 mm metastasis of a sigmoidcarcinoma.
Table 3 Comparison of cases that were called negative by the radiologist and the CAD system.
Called negative by CAD Called negative by radiologist Total 1381 1018 Unique patients 1339 980 Average age 60.7 (±11.8) 54.1 (±9.5) Male:Female 406:933 491:489 Number analyzed 500 500 True negative 492 495
False negative 8 (4 visible) 5 (4 visible)
Eight cases were falsely reported negative by CAD. Four of these cases were considered obviously visible cancers (Figure 5). All of them had been reported by the radiologist in the radiological report. One was a very large tumor (over 15 cm), one was a pleural tumor with cavitation, and two were tumors in the aortic pulmonary window. For the remaining 4 cases a tumor was seen on CT, but was invisible on the radiograph.
In the 500 analyzed cases that were called normal by the radiologists, five tumors were found. Four of these were considered visible in the images with knowledge of the CT. All of them were subtle lesions.
The negative predictive value of the CAD system was 98.4%, which was comparable with the negative predictive value of 99% of the radiological report.
Figure 5 These four radiographs show cancers in cases that did not receive any mark by the CAD system. Cases that were idenitified as “normal” by the CAD system can therefore not be excluded by the CAD system. The four cancers, a very large tumor (upper left), a pleural tumor (upper right), and two cancers located in the aorticpulmonary window (lower images), were reported by the radiologist.
Discussion
In this study we retrospectively analyzed CAD marks with a high score of suspicion generated by commercial CAD system on a large clinical data set of 11.109 chest radiographs. Of the 806 analyzed CAD marks with a high score of suspicion, 2 CAD marks identified tumors that were not reported by the radiologist, and another 3 CAD marks pointed to tumors that were interpreted as another abnormality than a tumor by the radiologist in the report. Secondly, we analyzed cases that did not receive any CAD mark. We found a high negative predictive value for the presence of lung cancer when the CAD system doesn’t produce a CAD mark at all. This NPV was comparable to the NPV of radiologists that reported cases normal.
In clinical practice the CAD system could be used to reduce overlooked lesions, by only reviewing CAD marks that were identified as very suspicious by the CAD system. In our study group of 11.109 cases, only 806 CAD marks with a high score of suspicion (7.3 percent of all cases) would have to be reviewed by the radiologist, still providing the opportunity to pick up missed lung cancers.
The number of CAD marks actually to be reviewed could be further lowered if the system would be used in an intelligent manner. We propose that the CAD system should only become active (e.g. by a blinking light) in chest radiographs that have been reported normal or non-significant by the radiologist but yielded a high
suspicious (≥ 80) CAD mark. Applying this, in our study group the radiologists would
have to assess only 663 CAD marks each year (6.0% of all cases), with the potential to find five additional cancers and four other significant abnormalities (Figure 6).
By only reviewing CAD marks with a CAD score of 80 or higher, the risk of accepting a false positive CAD mark is considerably decreased, reducing the risk of unnecessary work up examinations. It has to be mentioned that in our study group still a substantial amount of CAD marks with a high score of suspicion were classified as false positive (n=475). The vast majority of these CAD marks were obvious false positives (n=341), which could be easily and quickly discarded by the radiologist. These obvious false positives could also be easily excluded by a new version of the CAD system, further lowering the number of CAD marks that would have to be reviewed to 322 in total (2.9% percent of all cases). Considering that only 134 of these CAD marks were classified as false positives in 11,109 chest radiographs, the number of false positives per image on this data set was very low.
We only analyzed CAD marks that received a score of suspicion of 80 or higher by the CAD system. This threshold was based on our experience with the CAD system
in a previous study for the detection of lung cancer3. Lowering the threshold and
analyzing CAD marks with a lower score of suspicion would improve the detection of missed lesions, but at the expense of a substantial increase of false positive CAD marks. Thus this threshold is crucial for the absolute number of CAD marks that has
to be reviewed by the radiologist and the chance to detect lesions that originally were missed. The optimal threshold is dependent on the population, disease prevalence, the total number of radiographs to be reviewed and last not least the reading skills of the radiologist.
To find 2 additional cancers in a set of 11,000 cases may seem very little. On the other hand, it has to be considered that this additional information comes with minor additional effort: In our case set, every one in 161 (5 of 806) CAD marks was a missed malignant lesion. As discussed earlier, if the software would be upgraded in a way that those most-obvious false lesions would be eliminated and radiologists do not have to secondary review examinations that they already reported with a tumor, every 1 in 64 CAD marks would represent a missed malignant lesion. This compares favorably with the use of CAD in breast cancer screening with mammography, where radiologists encounter around 400 false positive CAD marks, before seeing one true
positive CAD mark6. Thus, although application of CAD as proposed in this article will
not help the radiologist to detect all cancers, it has the potential to help the radiologist to detect lesions that originally were missed, and the odds to pick up lung cancers are higher than for example in mammography screening.
When interpreting these results it has to be noted that the cases were derived from an academic hospital. In this hospital most chest radiographs are double read (resident and radiologist). Therefore the number of missed lesions is likely to be lower than in hospitals where chest radiographs are single read, as proven by several studies
for the detection of breast and lung cancer7,8. The benefit of CAD may therefore be
even larger in a less well controlled environment than demonstrated in this study. Figure 6 Algorithm for use of the CAD system. Only CAD marks with a high score of suspicion have to be reviewed by the radiologist. Cases that already are reported to contain a significant nodule, do not have to be secondly reviewed by the CAD system. According to our study with 11,109 chest radiographs, only 663 CAD marks have to be reviewed by the radiologists, with the potential to find 5 more tumors, and 4 other significant abnormalities.
11.109 CXR Already marked as tumor 143 CAD: 806 suspicious marks Marked non tumor 663 Evaluation of CAD marks Potential gain: 5 cancers 4 other significant abnormalities
Besides alerting the radiologist to very suspicious areas in a radiograph, the CAD system could also be used to reassure the radiologist that a case is negative. The negative predictive value of the CAD system in our study was 98.4%. Disappointing is the fact that in the cases without a CAD mark still eight cancer cases were present. Absence of a CAD mark can therefore not be used to exclude normal cases, without visual control by the radiologist at all. However, four of the eight cancers were found to be invisible on the chest radiographs. The other four lung cancers not indicated by CAD were rather obvious, and therefore had been picked up by the radiologist himself. In that respect, the rate of 4 missed and visible cancers in a case set of 500 without a CAD mark, is similar to the rate of missed cancers in cases that were called normal by the radiologist. The fundamental difference however, is that the missed cancers by the radiologists were very subtle compared with the missed cancers by CAD. Another thing to bear in mind is that the CAD system is not trained to detect normal cases, but to detect cancer cases. A system dedicated to detect normal cases may perform superior to the CAD system we used in this study.
Our study has some limitations. First of all, the CAD system we used is not designed to be used as selection tool for highly suspicious cases. A system that would be specifically trained at a high specificity has the potential to improve outcome. However, this study reflects the possibilities with a currently available commercial CAD system. A prospective assessment of chest radiographs with this CAD system would be needed, to evaluate the added value in clinical practice. Furthermore, we believe that discrimination between TP and FP marks that are indicated to be very suspicious by the CAD system is more straightforward than in CAD marks of lower suspicion. However, also in our scenario, acceptance of FP CAD marks is possible, and would result in unnecessary follow up of patients.
In conclusion, the CAD system that produced only 806 CAD marks in 11,109 chest radiographs, detected 2 cancers that were not reported, and 3 cancers that were noted, but not reported as significant findings. The system could be used in clinical practice reducing the number of CAD marks that have to be reviewed substantially compared with the traditional CAD system that would be used as second reader. Thus with minor interference with the workflow and very little additional effort for the radiologist, the CAD system has the potential to reduce oversight errors of lung malignancies on chest radiographs.
References
1. Quekel LG, Kessels AG, Goei R, et al. Miss rate of lung cancer on the chest radiograph in clinical practice.
Chest 1999;115:720–724.
2. de Hoop B, Schaefer-Prokop CM, Gietema HA, et al. Screening for lung cancer with digital chest radiography: sensitivity and number of secondary work-up CT examinations. Radiology 2010;255:629–637. 3. Schalekamp S, van Ginneken B, Koedam E, et al. Computer aided detection improves detection of
pulmonary nodules in chest radiographs beyond the support by bone suppressed images. Radiology 2014;. 4. de Boo DW, Uffmann M, Weber M, et al. Computer-aided detection of small pulmonary nodules in chest
radiographs an observer study. Academic Radiology 2011;18:1507–1514.
5. de Hoop B, de Boo DW, Gietema HA, et al. Computer-aided detection of lung cancer on chest radiographs: Effect on observer performance. Radiology 2010;257:532–540.
6. Houssami N, Given-Wilson R, Ciatto S. Early detection of breast cancer: overview of the evidence on computer-aided detection in mammography screening. Journal of Medical Imaging and Radiation
Oncology 2009;53:171–176.
7. Thurfjell EL, Lernevall KA, Taube AA. Benefit of independent double reading in a population-based mammography screening program. Radiology 1994;191:241–244.
8. Hessel SJ, Herman PG, Swensson RG. Improving performance by multiple interpretations of chest radiographs: effectiveness and cost. Radiology 1978;127:589–594.