Notes
Dreiseitl 2009
Patient Selection
A. Risk of Bias Patient SamplingStudy design: Case series Data collection: Prospective
Period of data collection Test set: Feb-Nov 2004
Country Austria
Test set derived Study focuses on test set but gives detail of separate study in which classifier was trained
Was a consecutive or random sample of patients enrolled? Yes
Was a case-control design avoided? Yes
Did the study avoid inappropriate exclusions? Yes Could the selection of patients have introduced bias? Low risk B. Concerns regarding applicability
Patient characteristics and setting
Inclusion criteria: Patients presenting at pigmented skin lesion clinic at Dept Dermatology which serves as a secondary and tertiary referral centre
Setting: Specialist unit (skin cancer/pigmented lesions clinic) The pigmented skin lesion unit of the Department of Dermatology at the Medical University of Vienna serves as a secondary and tertiary referral center. Prior testing: Not reported
Setting for prior testing: Specialist unit (skin cancer/pigmented lesions clinic)
Exclusion criteria: None reported
Sample size (patients): No. eligible: 511; No. included: 458 with complete information
Sample size (lesions): No. eligible: 3827; No. included: 3021
Participant characteristics: None reported Lesion characteristics: None reported Are the included patients and chosen study setting appropriate? No
Did the study avoid including participants with multiple lesions? No Was an adequate spectrum of cases used to train the algorithm? Yes Are there concerns that the included patients and setting do not
match the review question? High
Index tests
Dermoscopy No algorithm
Method of diagnosis: In person diagnosis
Prior test data: Clinical examination and/or case notes Diagnostic threshold: Not reported
Diagnosis based on: Single observer (n=1) Observer qualifications: Expert dermatologist Experience in practice: High experience or ‘Expert’
Experience with index test: High experience /‘Expert’ users 3. Computer Assisted Diagnosis - Dermoscopy based
Derm-CAD system: Image J (NIH, Bethesda, USA) (ANN classifier) System details:
image analysis coupled with dermoscope MoleMax II No derivation aspect (external validation)
Described in prior study Hable 2004 (PhD thesis) Lesion characteristics assessed:
29 Features analysed from 38 extracted features describing shape, form and colour. Approach to feature selection Prior study: A stepwise feature selection method used to identify 29 features relevant for the classification process.
Additional predictors included: - No further information used Method of diagnosis: - In person diagnosis - CAD-based diagnosis – CAD–aided diagnosis Test observers: - Single observer Number of examiners 6 Observer qualifications:
- Other (describe) The educational training of the 6 participating physicians ranged from no training in dermatology to 4 years training in dermatology.
Experience in practice:
- Mixed experience (low and high experience combined) educational training of the 6 participating physicians ranged from no training in dermatology to 4 years training in dermatology.
Experience with index test:
- Mixed experience (low and high experience combined) No physician was specifically trained in dermatoscopy.
Prior/other test data:
B. Concerns regarding applicability
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication? No Was the test interpretation carried out by an experienced examiner? No Was the diagnostic threshold to determine presence or absence of disease established in a previously published
study? Yes
Are there concerns that the index test, its conduct, or interpretation differ from the review question? High
Visual inspection
A. Risk of Bias
B. Concerns regarding applicability
Dermoscopy
A. Risk of Bias
Were the index test results interpreted without knowledge of the results of the reference standard? Yes
If a threshold was used, was it pre-specified? Unclear
Was the CAD model evaluated in an independent study population? Was model overfitting accounted for during model development?
Could the conduct or interpretation of the index test have introduced bias? Unclearrisk B. Concerns regarding applicability
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication? No Was the test interpretation carried out by an experienced examiner? Yes Was the diagnostic threshold to determine presence or absence of disease established in a previously published study?
Are there concerns that the index test, its conduct, or interpretation differ from the review question? High
Reference Standard
A. Risk of Bias
Target condition and reference standard(s)
Reference standard Histological diagnosis plus follow up
Histology (excision); No. patient/lesions: Not reported
Clinical FU plus histology of suspicious lesions Length of FU: 6 months; No. patients: Not reported TARGET CONDITION (Final diagnoses)
Melanoma (in situ and invasive, or not reported): 27 patients; 31 lesions
'Benign' diagnoses: 431 patients; 2990 lesions Is the reference standards likely to correctly classify the target
condition? Yes
Were the reference standard results interpreted without knowledge of
the results of the index tests? Yes
Were the reference standard results likely to correctly classify the target
condition (disease negative)? Yes
Could the reference standard, its conduct, or its interpretation have
introduced bias? Low risk
B. Concerns regarding applicability
Was the use of expert opinion (with no histological confirmation) avoided as the reference standard? Yes Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist? Unclear Are there concerns that the target condition as defined by the reference standard does not match the question? Unclear
Flow and Timing
A. Risk of Bias Flow and timing
Excluded participants: 806 lesions (53 patients) with inadequate follow-up Intervals between tests: not reported Was there an appropriate interval between index test and reference standard?Unclear
Did all patients receive the same reference standard? No Were all patients included in the analysis? No Could the patient flow have introduced bias? High risk
Notes
NotesFerris 2015
Patient Selection
A. Risk of Bias Patient SamplingStudy design: Unclear Some dermoscopic images were collected prospectively and some were obtained from collection of existing images; selection process not described.
Data collection: Retrospective image selection / Prospective interpretation
Period of data collection not reported Country USA
Test set derived Some dermoscopic images used to train the classifier were obtained from publicly available or purchased image libraries, these were not included in the reader study or used to test the performance of the classifier. The image set was randomly divided into 2 by diagnosis, with half used for training and half used for testing, with the exception that all high-grade dysplastic nevi were exclusively assigned to the training set to increase the representation of dermoscopic features that could be present in melanoma. Results are presented only for the test set. Was a consecutive or random sample of patients enrolled? No
Was a case-control design avoided? No
Did the study avoid inappropriate exclusions? No Could the selection of patients have introduced bias? High risk
B. Concerns regarding applicability
Patient characteristics and setting
Inclusion criteria: Dermoscopic images of skin lesions excised on the basis of clinical suspicion of malignancy, with available histologic diagnoses Setting: Secondary (general dermatology) Prior testing: Clinical and/or dermatoscopic suspicion; Selected for excision (no further detail) Setting for prior testing: Secondary (general dermatology)
Exclusion criteria: high-grade dysplastic nevi were not included in the test set
Sample size (patients): No. eligible: not reported; No. included: not reported
Sample size (lesions): No. eligible: 473 (includes 273 randomised to training set and 27 non-biopsied lesions); No. included: CAD- Derm- test set 173 lesions; Dermscopy- 65 lesions
Participant characteristics: None reported Lesion characteristics: Test set: mean lesion thickness 0.76 mm, median 0.5 mm, range 0.2-2.98 mm); Reader study: mean 0.93 mm, median 0.74 mm, range 0.2 to 2.98 mm.
Are the included patients and chosen study setting appropriate? No Did the study avoid including participants with multiple lesions? Yes Was an adequate spectrum of cases used to train the algorithm? Unclear Are there concerns that the included patients and setting do not match
the review question? High
Index Test
Index tests