pymia: A Python package for data handling and evaluation in deep learning-based medical image analysis

Contents lists available at ScienceDirect

Computer

Methods

and

Programs

in

Biomedicine

journal homepage: www.elsevier.com/locate/cmpb

pymia:

A

Python

package

for

data

handling

and

evaluation

in

deep

learning-base

d

me

dical

image

analysis

Alain Jungo

a,1,∗

, Olivier Scheidegger

b,c

, Mauricio Reyes

a

, Fabian Balsiger

a,1,∗ a_{ARTORGCenterforBiomedicalEngineeringResearch,UniversityofBern,Bern,Switzerland}

b_{DepartmentofNeurology,Inselspital,BernUniversityHospital,UniversityofBern,Bern,Switzerland}

c_{SupportCenterforAdvancedNeuroimaging(SCAN),InstituteforDiagnosticandInterventionalNeuroradiology,Inselspital,BernUniversityHospital,}

UniversityofBern,Bern,Switzerland

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received15July2020 Accepted8October2020

Keywords:

Medicalimageanalysis Deeplearning Datahandling Evaluation Metrics

a

b

s

t

r

a

c

t

Background andObjective: Deep learningenablestremendous progressinmedical image analysis.One drivingforceofthisprogressareopen-sourceframeworkslikeTensorFlowandPyTorch.However,these frameworksrarelyaddressissuesspecifictothedomainofmedicalimageanalysis,suchas3-Ddata han-dlinganddistancemetricsforevaluation.pymia,anopen-sourcePythonpackage,triestoaddressthese issuesbyprovidingflexibledatahandlingandevaluationindependentofthedeeplearningframework.

Methods: Thepymiapackage providesdatahandlingand evaluationfunctionalities. Thedatahandling allowsflexiblemedicalimagehandlingineverycommonlyusedformat(e.g.,2-D,2.5-D,and3-D; full-orpatch-wise). Evendatabeyondimageslikedemographicsorclinicalreportscaneasilybeintegrated intodeeplearningpipelines.Theevaluationallowsstand-aloneresultcalculationandreporting,aswellas performancemonitoringduringtrainingusingavastamountofdomain-specificmetricsforsegmentation, reconstruction,andregression.

Results:Thepymiapackageishighlyflexible,allowsforfastprototyping,andreducestheburdenof im-plementingdatahandlingroutinesandevaluationmethods.Whiledatahandlingandevaluationare inde-pendentofthedeeplearningframeworkused,theycaneasilybeintegratedintoTensorFlowandPyTorch pipelines.Thedevelopedpackagewassuccessfullyusedinavarietyofresearchprojectsforsegmentation, reconstruction,andregression.

Conclusions:Thepymiapackagefillsthegapofcurrentdeeplearningframeworksregardingdatahandling andevaluationinmedicalimageanalysis.Itisavailableat https://github.com/rundherum/pymia andcan directlybeinstalledfromthePythonPackageIndexusing

pip

install

pymia

.

© 2020TheAuthor(s).PublishedbyElsevierB.V. ThisisanopenaccessarticleundertheCCBY-NC-NDlicense (http://creativecommons.org/licenses/by-nc-nd/4.0/ )

1. Introduction

Deeplearninghasatremendousimpactonmedicalimage anal-ysistaskslikeclassification,segmentation,andreconstructionfrom 2015 onwards [1–4]. This impact is mainly due to methodologi-cal developmentsliketheAlexNet [5]ortheU-Net[6],dedicated hardware (graphics processingunits,GPUs), increaseddata avail-ability, andopen-source deeplearningframeworks.Infact, open-source deeplearningframeworkscanbe seenasoneofthemain

∗_{Correspondingauthor.}

E-mail addresses: [email protected] (A. Jungo), fabian.balsiger@ artorg.unibe.ch(F.Balsiger).

1 _{Equalcontributionandcorrespondingauthors}

driving forces leadingto the wider adoption of deeplearning in the medical image analysis community [1]. Current frameworks likeTensorFlow[7]andPyTorch[8]allowresearchestoimplement methodsratherthanimplementinglow-levelGPUoperations. Nev-ertheless,theadoptionofdeeplearningmethods,usually originat-ingfromthecomputervisioncommunity,isoftenhinderedbythe 3-Dnatureofmedicalimages,making,inparticular,thedata han-dlingandevaluationverydomain-specificandcumbersome.

A few open-source projects addressing medical image analy-siswithdeeplearningexist.Themostprominentprojectislikely NiftyNet [9], which enables fast development of medical image analysismethodsbasedontheTensorFlowframework.Among oth-ers, it provides implementations of training routines, neural net-work architectures, andloss functions. Unfortunately, the project

https://doi.org/10.1016/j.cmpb.2020.105796

0169-2607/© 2020 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)

source:

https://doi.org/10.7892/boris.147477

Fig.1. Thepymiapackageinthedeeplearningenvironment.Thedatapackageenablescreationofadatasetfromrawdata.Extractionofthedatafromthisdatasetis possibleinnearlyeverydesiredformat(2-D,3-D;full-orpatch-wise)forfeedingtoaneuralnetwork.Thepredictionoftheneuralnetworkcan,ifnecessary,beassembled backtotheoriginalsizebeforetheevaluation.Theevaluationpackageallowstheevaluationofpredictionsagainstreferencesusingavastamountofmetrics.Itcanbeused stand-alone(solid)orforperformancemonitoringduringtraining(dashed).

isnot activelymaintainedanymoreasofApril20202_.Similarlyto NiftyNet, the deep learningtoolkit (DLTK) [10] also provides im-plementations ofcommonneural networkarchitectures basedon TensorFlow. But the last updates to the project dateover a year back andit is incompatiblewith version 2 ofTensorFlow, which suggestsreducedornoactivedevelopment.APyTorch-based pack-ageisMedicalTorch[11]withoverlappingbutreduced functional-ityasNiftyNet andDLTK. Amorerecent packageis TorchIO[12], whichprovidespre-processinganddataaugmentationroutinesfor medicalimages,aswell as3-D patch-baseddatahandlingwithin the scope ofthe PyTorchframework. MONAI(Medical Open Net-work for AI)3 _{is a PyTorch-based framework fordeeplearning in} healthcare imaging. It is the predecessor of NiftyNet, and simi-larly, MONAI provides trainingroutines, neural network architec-tures, and loss functionsenabling entire deep learning pipelines fromdataloadingtosaving.AnotherframeworkisDeepNeuro[13], which provides a templating language for designing medial im-age analysispipelines and a modeldeployment system basedon TensorFlow. Insummary,multipleopen-source projectsaimat fa-cilitating deep learning-based medical image analysis by provid-ing out-of-the-boxtrainingroutinesandneural network architec-tures. To date, TorchIO, MONAI, and DeepNeuro seem to be ac-tively developed andthe mostprominentprojects.Unfortunately, all projects relyon oneparticular deeplearningframework (Ten-sorFloworPyTorch),makingitpotentiallyinflexibleforfastswitch toanotherframework.

The evaluationof results in medicalimage analysisis depen-dent ondomain-specific metrics,alsodueto thephysical proper-tiesofmedicalimagessuchasthespacingbetweenpixels. Promi-nentmetricsare,forinstance,theDicecoefficient[14]orthe Haus-dorff distance[15]forsegmentation,andthepeak signal-to-noise ratioorthestructuralsimilarityindexmeasure [16]forimage re-construction. Suchmetrics arerarelyfound tobeimplemented in open-source deep learning frameworks, nor do the projects in-troduced in the last paragraph provide (exhaustive) implementa-tions ofmetrics. Therefore,metricsare oftentakenfrommultiple independent projects. Notable projects covering metrics are cer-tainly the Insight Toolkit (ITK) [17] with its Python variant

Sim-2_{https://github.com/NifTK/NiftyNet} 3_{https://monai.io/}

pleITK[18]covering commonsegmentation metrics.Furthermore, the evaluatesegmentation tool [19] provides an extensive imple-mentationofsegmentation metrics4_{.However,the projectis} C++-based,makingitimpracticaltousewiththecurrentPython-based deeplearning.APython-basedpackageismedpy5_{,whichfeatures} a small set of segmentation metrics. And, metrics beyond seg-mentationcanbefound inthePythonpackagesscikit-image[20], scikit-learn [21],and SciPy[22].Overall, a singlePython package coveringanexhaustiveamountofmetricsforsegmentation, recon-struction,andregressioninmedicalimageanalysisislacking.

We believe that deep learning framework-agnostic data han-dling and evaluation is essential for medical image analysis re-search. Indatahandling,flexibility is highlydesirable,meaning a simpleandfastswitchfrom,e.g.,2-Dto3-Dprocessing,shouldbe possible. For evaluation, performance monitoring during method development,andresultcalculationandreportingforfurther sta-tistical analyses and visualization, encompassing domain-specific metrics withaspects like image spacing,is desirable. Ideally, the evaluationiscompletelydecoupledfromthedeeplearning frame-works such that it can be used for evaluationscripts only. Gen-erallyforprototyping,rewriting code whenmethods are adopted from open-source methods implemented in an arbitrary frame-work shouldnot be necessary. Rather,the relevantcode (i.e.,the model, loss function, and optimizer), should be copied into an existing data handling andevaluation pipeline withminor to no adaptationsoftheexistingcode.

We present pymia, an open-source Python (py) package for deeplearning-basedmedicalimageanalysis(mia).Thepackage ad-dresses twomain parts ofdeeplearningpipelines: datahandling andevaluation. The package is independent ofthe deeplearning frameworkusedbutcaneasily beintegratedintoTensorFlowand PyTorch pipelines. Therefore, pymia is highly flexible, allows for fast prototyping, and facilitates implementing data handling and evaluation.

2. Methods

Theintendeduseofpymiainthedeeplearningenvironmentis depictedinFig.1.Itsmaincomponentsarethedataandthe

eval-4_{https://github.com/Visceral-Project/EvaluateSegmentation} 5_{https://loli.github.io/medpy/}

Fig.2. Overviewofthethreemaincomponentsofthedatapackage.Arrowsrepresentdataflow,andtheboxesrepresentclasssignatures.

uation package.The datapackageisused toextractdata(images, labels,demography, etc.)froma datasetinthedesiredformat (2-D, 3-D; full- orpatch-wise) forfeeding to a neuralnetwork. The output oftheneuralnetworkisthen assembledbackto the orig-inalformatbeforeextraction,ifnecessary.Theevaluationpackage providesbothevaluationroutinesaswellasmetricstoassess pre-dictionsagainstreferences.Thesecanbeusedbothforstand-alone resultcalculationandreporting,andformonitoringofthetraining progress.

2.1. Datapackage

The purposeofthe datapackage isto provideflexible, format independent,andfastaccesstodata.First,flexiblebecausethedata should be accessible in various ways.Meaning that 3-D medical data likemagneticresonance(MR) orcomputedtomography (CT) images could be processed in 2-D, 3-D, or 2.5-D (i.e., the three anatomical planes axial, coronal, and sagittal) and further in its full or reduced spatial extent, i.e., as so-called patches6 _Second, themoreformat-independentthedataaccess,theeasierbecomes prototyping andexperimentingwithclinicaldata beyondmedical images. Meaning that demographic information, patient records, or even more uncommon formats such aselectroencephalogram (EEG) data,laboratory results,point clouds,or meshesshould be accessible. Third, fast because the data access should not slow down thetrainingoftheneuralnetwork,i.e.,notresultinginidle GPUtime.Thethreemaincomponentsofthedatapackageare

cre-6_{Althoughin3-Da(sub)volumewouldbeamoreappropriateterm,itisoften} referredasa(3-D)patchintheliterature.

ation,extraction,andassembly(Fig.2),whicharedescribed here-after.

Creation.Adatasetisfirstcreatedfromtherawdata,whichcan beseenasadatabaseholdingallinformationavailableorrequired forthe training ofa neural network. This dataset isa HDF5 (hi-erarchicaldataformatversion 5)file. TheHDFformat[23] allows multipledifferentdatatypesinonefileandenablesfastaccessof chunks of datawithout the need to loadthe data inits entirety (e.g.,loadingofa2-Dimageslicefroma3-Dimage).Thecreation ofadatasetismanagedbythe Traverser class,whichprocesses the data of every subject (case) iteratively. It employs

Load

to loadtherawdatafromthefile systemand Callback classesto writetherequiredinformationtothedataset. Transform classes can be used toapply modifications to the data,e.g., an intensity normalization. By separating the concerns of the loading, writ-ing,andtransforming,maximalflexibilityinthedatasetcreationis achieved.Fortheeaseofuse,default Callback and Load classes areimplemented,whichcoverthemostfundamentalcases.By de-sign, the dataset should only be created once and should, thus, contain asmuch information as possible.It might be suitable to createthreedistinct datasetsforthetraining,validation,and test-ingsubjects.

Extraction. Once the dataset is created, it can be used for thetraining(or testing)routine. Dataextraction fromthedataset is managed by the

PymiaDatasource

class, which provides a flexible interface for retrieving data, or chunks of data, to form training samples. An

IndexingStrategy

is used to de-fine how the data is indexed, meaning accessing, for instance, an image slice or a 3-D patch of an 3-D image.

Extractor

classesextractthedatafromthedataset,and Transform classes

Table1

Overviewofusecasesfordatahandlingand thecorrespondingclassestouse.Slice:2-Dimagesliceof a3-Dimage;Slab:Multipleconsecutive2-Dimageslices;2.5-D:2-Dimageslicesinallthreeanatomical planes;Patch(equal):3-Dpatchforaneuralnetworkwithequalinputandoutputsize;Patch(padded): 3-Dpatchforaneuralnetworkwithlargerinputthanoutputsize(overlappinginputs);Rawformat:entire 3-D/2-Dimage.

Classsignaturesandimplementations

Usecase IndexingStrategy Extractor Assembler

Slice SliceIndexing DataExtractor SubjectAssembler

Slab PatchIndexing DataExtractor SubjectAssembler

2.5-D SliceIndexing DataExtractor PlaneSubjectAssembler

Patch(equal) PatchWiseIndexing DataExtractor SubjectAssembler Patch(padded) PatchWiseIndexing PadDataExtractor SubjectAssembler

Rawformat EmptyIndexing DataExtractor

-can be used to alter the extracted data. Processing medical im-ages in chunks is typically required in deep learning due to the size of the images and the limitations in GPU memory. The

IndexingStrategy

providesasignatureforanykindofchunks, e.g., 2-D image slices(

SliceIndexing

class) or3-D patches of arbitrarysize(

PatchWiseIndexing

class).Itissufficientto sim-plyexchangethe IndexingStrategy if,forinstance,another in-dexing is desired.For each type ofdatain thedataset,a specific

Extractor

is used,e.g., a DataExtractor to extract the im-age data ora SubjectExtractor to extract the identification of a subject. In a sense, an Extractor is the reading counter-parttoa Callback forwritingduringthedatasetcreation.Since

Extractor

sarethefirstinstanceinteractingwiththedata,they canalsobeusedtoperformspecificdatahandling,suchaspadding (

PadDataExtractor

class) or selecting specific channels (e.g., different MR images) of the data (

SelectiveDataExtractor

class). Finally,the extracted data can be altered via Transform classes. Often, theseare usedto adapt thedata forusage witha neural network (e.g., channel permutations, dimension modifica-tions,orintensitymodifications)andtoalterthedatafortraining purposes(e.g.,dataaugmentation,masking).

Assembly.The outputofaneuralnetworkusually needstobe assembled back totheoriginal format forevaluationandstorage, especiallyforvalidationandtesting. Forinstance,a3-Dimage in-steadofseparate2-Dimageslicesaredesiredwhenchunksofdata arepredicted.The Assembler classmanagestheassemblyofthe predictedneural network outputs byusing theidenticalindexing thatwasemployedtoextractthedatabythe PymiaDatasource class.

2.1.1. Flexibility&extendability

Themodulardesignofthedatapackageaimsatprovidinghigh flexibility andextendabilitytoasmanyusecasesaspossible.The flexibilityisillustratedinTable1,withusecasesofdatahandling. Well-defined interfacesfacilitatetheextendability ofcreation, ex-traction, andassembly. Forthe creationof the dataset,newdata formats (e.g., EEG, laboratory results) can be handled by a cus-tom Load andmightrequirecustom Callback and Extractor implementations. Further,currentindexingpossibilitiescan easily be extended witha custom IndexingStrategy . Likewise, one canaddcustomizeddatamodificationsbyimplementingaspecific

Transform

.

2.1.2. Metadatadataset

The data is ideally written to a dataset, as described before-hand.However, theremight beuse casessuchasa largeamount ofdataortheuseofverylargepatchsizes(orevenentire3-D im-ages), which might questionthe usefulnessofcreating a dataset, i.e., ultimately only saving the data in another format. Usage of the data package without the creation of a dataset whilehaving

thesameflexibilityaswithadatasetisnotpossible.However,the minimum required information in a dataset is fairly small such thatthedatapackagecanbeusedasintended.Onlythemetadata describing the subject identifiers, the file paths, and the shapes (size)oftheimage dataneedtobesavedintothedataset, result-inginametadatadataset.The PymiaDatasource classcanthen beparametrizedtoloadthedatafromthefilesysteminsteadfrom thedataset.Theshapesarerequiredsuchthat theflexibilitywith the IndexingStrategy classesisretained.

2.1.3. Reproducibility&privacy

Reproducibility and privacy might be two important aspects when creating a dataset. Regarding reproducibility, creating a datasetallows writingthenames andpathsof thefilesstoredin the dataset, which in many cases might be sufficient for repro-ducibility.Foradditional reproducibility,itwouldalsobe possible tostore,forexample,thehashvalueoftherawfiles,whichwould allow toverifyat anytime ifa certain rawfile wasused to cre-ateand/oriscontainedinthedataset.Regardingprivacy,assimple asitistoadd additionalinformationlikethe hashvalue,as sim-ple candata be omittedwhen creatingthe dataset.Forexample, datasetscan becreatedwithimage dataonly,andsubject identi-fierscouldsimplybeanonymized.Additionally,theconceptofthe transformation(

Transform

classes) wouldallowto applyimage anonymizationmethodswhencreatingthedataset,e.g.,adefacing transformforheadimages.

2.2. Evaluationpackage

Thepurposeoftheevaluationpackageisdomain-specific evalu-ationformedicalimageanalysis.Thereforeavarietyofmetricsfor image segmentation,image reconstruction, andregressionare in-cluded.Thefunctionalities oftheevaluationpackageallow stand-alone result calculation and reporting, or performance monitor-ing during the training progress independent of the deep learn-ingframework.Theconceptoftheevaluationpackageisillustrated inFig.3.Themetricsinheritfrom Metric andcanbeusedwith the Evaluator class to evaluate predictions against references. For instance, the SegmentationEvaluator class can be used to compare a prediction with a reference label image by calcu-lating the metric(s) for every label one is interested in. The re-sultscan thenbe passedto a Writer toreport theresults. Cur-rently,a CSVWriter class, writingresults toacomma-separated values (CSV) file, and a

ConsoleWriter

class, writing results to the console,are implemented. Further, statistics over all eval-uatedsubjects(andlabels)canbecalculatedandwrittenbyusing a CSVStatisticsWriter ora ConsoleStatisticsWriter . In both cases, the statisticalfunctions can be arbitrary,with the onlyconditionbeingtotake alistofvaluesandtoreturnascalar value(e.g.,themeanorthestandarddeviation).

Table2

Overviewofthecurrentlyimplementedmetricsinpymia.Categoricalmetricscanbeusedforimagesegmentationandcontinuous metricsforimagereconstructionandregression.Theabbreviationsareusedforreportingandcanbeadapteduponinstantiating themetrics.Areferenceisgivenwhereappropriate.

Category Metric Abbreviation Remarks

Categorical Dicecoefficient[14] DICE

-Jaccardcoefficient[25] JACRD

-Sensitivity SNSVTY

-Specificity SPCFTY

-Fallout FALLOUT

-Falsenegativerate FNR

-Accuracy ACURCY -Precision PRCISON -Truepositive TP -Falsepositive FP -Truenegative TN -Falsenegative FN

-F-measure FMEASR βdefinable

Globalconsistencyerror[26] GCOERR

-Volumesimilarity[27] VOLSMTY

-Randindex[28] RNDIND

-Adjustedrandindex[29] ADJRIND

-Mutualinformation MUTINF

-Variationofinformation[30] VARINFO -Interclasscorrelation[31] ICCORR -Probabilisticdistance[32] PROBDST

-CohenKappacoefficient[33] KAPPA

-Areaundercurve[34] AUC

-Hausdorff distance[15] HDRFDST percentiledefinable

Averagedistance AVGDIST

-Mahalanobisdistance[35] MAHLNBS

-Surfaceoverlap[36] SURFOVLP

-SurfaceDiceoverlap[36] SURFDICE

-Area AREA forreferenceorprediction,imageslicedefinable

Volume VOL forreferenceorprediction

Continuous Coefficientofdetermination R2

-Meanabsoluteerror MAE

-Meansquarederror MSE

-Rootmeansquarederror RMSE

-Normalizedrootmeansquarederror NRMSE

-Peaksignal-to-noiseratio PSNR

-Structuralsimilarityindexmeasure[16] SSIM

-Fig.3. Overviewoftheevaluationpackage.Arrowsrepresentdataflow,andthe boxesrepresentclasssignatures.

Avarietyofmetricsareimplemented(Table2),whichare cat-egorized into categorical, i.e., for image segmentation, and con-tinuous, i.e., for image reconstruction andregression. All metrics are implemented such that they work withatleast 2-D and3-D data,andifappropriate,alsowithlowerorhigherdimensions. Fur-ther,imagespacingisconsideredwheneveradequate(e.g.,for dis-tancemetrics). Thecategoricaldatametricsareselectedbasedon TahaandHanbury [19].The continuousdatametrics areinspired by other Pythonpackages likescikit-image [20], scikit-learn [21], and SciPy [22]. Image reconstruction-specific metrics follow the fastMRI challenge [24]. The readerisreferred tothese references

for metric descriptions, mathematical definitions, and guidelines onhowtoselectappropriatemetrics.

2.3. Platformanddependencies

pymiaisimplementedinPython(Python SoftwareFoundation, Wilmington, DA, U.S.) and requires version 3.6 or higher. It de-pendsonthefollowingpackages:h5py,NumPy,scikit-image,SciPy, and SimpleITK. To use the data package with a deep learning framework,eitherPyTorchorTensorFlow isrequiredfurther.Unit testsare implemented usingpytest. To build the documentation, Sphinx,ReadtheDocsSphinxTheme,Sphinx-copybutton,and nb-sphinxarerequired.

3. Results

pymiaishostedonthePythonPackageIndex(PyPI)foreasy in-stallationofthelatestversion usingthecommand pip

install

pymia

.ThecodeispubliclyavailableonGitHub7_{undertheterms} of the Apache 2.0license. The documentation is hosted on Read the Docs8 _{andcontainsdescriptions ofthe classesandfunctions.} Atthetimeofsubmissionofthisarticle,pymiaisatrelease0.3.1.

Severalcodeexamples demonstratetheindented useofpymia in small parts covering isolated functionalities. All examples

7_{https://github.com/rundherum/pymia} 8_{https://pymia.readthedocs.io/en/latest/}

Fig.4. ExemplaryHDF5datasetwithfoursubjects.Thedatasetconsistsofimagedata(images,labels,andmaskgroups),numericaldata(ageandGPA),andthegender ofthesubjects.ThedimensionoftheimagesgroupisZ×Y×X×C=181×217×181×2,whereC=2representsthechanneldimension,i.e.,theconcatenatedT1-and T2-weightedMRimages.Thelabelsandmaskgroupshavethesamedimensions,butC=1.Alongsidethedata,meta-informationisstoredinthedataset.Theopen-source softwareHDFView3.1.0wasusedtoopenthedataset.

are available on GitHub (https://github.com/rundherum/pymia/ tree/master/examples/) or directly rendered in the documenta-tion (https://pymia.readthedocs.io/en/latest/examples.html). In all examples,MRimagesoftheheadoffoursubjectsfromtheHuman ConnectomeProject (HCP)[37]areused.Eachsubjecthasfour 3-D images(in the MetaImageand Niftyformat) anddemographic informationprovidedasatext file.TheimagesareaT1-weighted MRimage,aT2-weightedMRimage,alabelimage(groundtruth), and a brain mask image. The demographic information is artifi-cially createdage,gender, andgradepointaverage (GPA).The la-belimagescontainannotationsoffivebrainstructures(white mat-ter,graymatter,hippocampus,amygdala,andthalamus), automati-callysegmentedbyFreeSurfer5.3[38,39].Therefore,theexamples mimictheproblemofmedicalimagesegmentationofbraintissues. Thenextsectionsshortlysummarizetheexamplesthatcover dedi-catedfunctionalitiesofpymia.Inaddition,trainingexamplescripts forthesegmentationofbraintissuesusinga U-Net[6]in Tensor-FlowandPyTorch,includingtrainingwithdataaugmentation, eval-uation,andlogging,canbefoundonGitHub.

3.1. Datahandling

The example Creation of a dataset illustrates how to create a HDF5 dataset.Fig. 4 showsthe structure ofthe dataset resulting fromtheexampledata.Therootisseparatedinto data and meta groups. The

data

group contains the concatenated T1- and T2-weighted MRimages (

images

group), the labelimage (

labels

group), the brain mask (

mask

group), the concatenated age and GPA (

numerical

group), andthe gender (

gender

group). Note that each group consistsoffour entries astheexample data has foursubjects.The dimensionofthe images group isZ×Y×X×

C₌181_×217_×181_×2_, where Crepresents the channel dimen-sion,i.e., the concatenated T1-andT2-weighted MRimages. The

labels

group and the mask group have the same dimensions, butC=1.The numerical groupisofdimension2(ageandGPA) andthe gender groupofdimension1.The meta groupcontains an entry with the subjectidentifiers (

subjects

), the file paths (

files

group),thephysicalimageinformationlikedirection, ori-gin,andspacing(

info

group),thefile identifiers(

names

group), andshape information(

shape

group).The file identifiers inthis exampleareT1,T2,GT,MASK,AGE,GPA,andGENDER. Theyallow toassociatethedimensionsinthe data groupwiththedatatype, e.g.,thattheMRimagesareconcatenatedintheorderT1-and T2-weightedandnottheotherwayaround.

TheexampleDataextractionandassemblyillustrateshowtouse pymiainatypicaldeeplearningloopoverthedatasamples.More specifically,itshowsthecasewhere2-Dimageslicesareextracted froma datasetinorderto feedit toa neuralnetwork before as-semblingthepredictionsbackto3-Dimages.Italsocovers extract-ing3-Dpatchesandloadingthedatadirectlyfromthefilesystem insteadfromadataset(usecasedescribedinSection2.1.2).

Using pymia, we benchmarked the performance of different waysof dataloading: i) loadingfroma HDF5dataset, ii)loading compressed MetaImages, iii) loading uncompressed MetaImages, andiv) loading NumPy files. The latter threeways loadthe data directlyfromthefilesystem(Section 2.1.2).We furthercompared threeloadingstrategies:i)entire3-Dimage,ii)3-Dpatchesofsize 84_×84_×84_, and iii) 2-D image slices. An artificial dataset was createdwithn=25subjects,eachwithaT1-andT2-weightedMR image ofthe example data(sizeof 181×217×181). The loading timesforonesample(i.e.,concatenated3-Dimages,concatenated 3-D patches, and concatenated 2-D image slices) were averaged

Fig.5. Benchmarkoftheloadingtimesofonesampleforthreeloadingvariantsand fourmethods.Thebarsrepresentthemeanloadingtime±thestandarddeviation.

Fig.6. CSVoutputoftheevaluationexample.Eachlinerepresentsanevaluation result,heretheDicecoefficient(DICE),95th_{Hausdorff distance(HDRFDST95),and} volumesimilarity(VOLSMTY)ofasubjectandlabel(e.g.,graymatterofSubject_1).

overfiveentirerunsoverthedataset9_{.Themeanandstandard} de-viationoftheloadingtimesareshowninFig.5.Clearly,theHDF5 dataset is thefastestloading method independentof theloading variant,followed by NumPy,uncompressed MetaImage, and com-pressedMetaImage.Forthelatterthreemethods,theloadingtimes arealmostequalforeachloadingstrategybecauseloadingthe en-tire 3-D image isalways necessaryeven ifonly a3-D patchora 2-Dimagesliceneedstobeloaded.

3.2. Evaluation

The example Evaluation of results illustrates how to evaluate segmentationresults.AwrittenCSVfilewiththeevaluationresults isshowninFig.6.

TheexampleLoggingthetrainingprogressillustrateshowtouse theevaluationpackagetologtheperformanceofaneuralnetwork during the training process. The evaluation resultsare passedto theTensorBoardbytheframework-specificfunctionsofTensorFlow andPyTorch.Therefore,theevolutionofthemetrics(e.g.,themean

9_{DesktopcomputerwithUbuntu18.04LTS,3.2GHzIntelCorei7-3930K,64GB} memory,SamsungEVO850500GBSSD

Dicecoefficient)overtheepochsduringthetrainingprocessis eas-ilyobservable.

4. Discussion

We developed pymia, a Python package for deep learning-based research in medical image analysis. pymia addresses flex-ible domain-specific data handling and evaluation, a gap of ex-isting open-source projects, and especially current deep learning frameworks. The development emphasized independence to the deep learningframeworks, whichallows forsimplified adoptions ofopen-sourcemethods(e.g.,anovelmodelpresentedinapaper) independent of the framework without rewriting the entire data handlingandevaluation. Therefore,fastprototypingispossibleas newmethodscaneasilybetestedwithouttheneedtoworryabout theframeworkused.

Thedatapackageenablesveryflexibleandfastaccessto med-icaldata.Theflexibilitymanifests inthesimplechangefrom, e.g., 2-D to 3-D;full- orpatch-wise (Table 1).Even non-imagingdata caneasilybe integrated.Themodulardesignensures flexibilityof thedatapackage,enablingextension andhandlingofcustomdata formats. Empirically,thedata loading,relying ona HDF5dataset, wasmeasured to be faster than other common loading methods (Fig. 5). Therefore,the data package smoothly integratesinto the framework-specifictraining routinesof thecurrent deeplearning environment.

Theevaluationpackage providesasimplewaytoevaluate pre-dictionsagainstreferenceswithaconsiderableamountofmetrics formedicalimage analysiscoveringsegmentation, reconstruction, andregression (Table2). It can eitherbe usedstand-alone orin conjunctionwithadeeplearningframeworkforperformance mon-itoring(e.g.,loggingtotheTensorBoard).Writersallowtosavethe evaluation results in the commonly used CSV format. The saved CSVfilescaneasilybeloadedintocommonstatisticalsoftwarefor statisticalanalysisandvisualization. Forinstance,it couldalsobe usedwiththechallengeRframework[40]for analyzingand visual-izingtheresultsofbiomedicalchallenges.

pymia was successfullyused for multiple research projects in medicalimage analysis, demonstratingits versatility. Formedical image segmentation, pymia was applied to 2-D segmentation of peripheral nerves inthigh MR[41],2-D segmentation of skin le-sions [42], 2.5-D [43] and slab-based segmentation of brain tu-mors[44]fromMRimages,and2.5-D braintumorresection cav-ity segmentation [45–47]. For image reconstruction, pymia was usedforreconstructionofMRfingerprinting[48–50], demonstrat-ingthehandlingoflarge5-Dtensors(350×350×5×175×2).In regression,pymia wasapplied to survival prediction of brain tu-morpatientsinthe2017BRATSchallenge[43](2nd rankinthe2017 BRATSoverallsurvivalpredictionchallenge)and2018BRATS chal-lengewherenon-imagingdatawasusedalongsideMRimages[51]. Lastly, even 3-D point cloud data washandled by pymia for the refinement of peripheral nerve segmentation [52]. Mostof these publicationshavepubliccodeavailable andcanserve asan addi-tionalpointofreferencecomplementingthepymiadocumentation. Duetotheexperiencewiththesediverseprojects,weconsiderthe currentstate of thepymia package asstableand usefulfordeep learning-based research inmedicalimage analysis.Indeed,pymia could also be appliedin other domains such asvideo processing orindustrialmanufacturing.Futureplansincludemainlyextending theexamples,increasingcodecoveragebyunittests,andensuring compatibilitywithfutureversionsofthemostuseddeeplearning frameworks. With a growing user base, however, there will cer-tainly emerge feature requests,but we aim atkeepingsimplicity andmodularity inmindforfuturereleases.Forinstance,itwould be beyondthe scopeofthe projecttoimplementneural network architecturesandlossfunctionsasprojectslikeMONAIand

Deep-Neuro do. However, stronger integration of projects like TorchIO andbatchgenerators[53]fordataaugmentationwouldcertainlybe interestingandvaluablefortheintendeduseofpymia.

Inconclusion,pymia wasdevelopedtofillthegapsofexisting deeplearningframeworkswithregardstomedicalimageanalysis. Thedatapackagefacilitatesthehandlingofmedicaldata indepen-dent of the used deep learningframework. The evaluation pack-ageallowstheevaluationofresultsusingtheprevalentmetricsin medicalimagingorperformancemonitoringduringmethod devel-opment.

Conflictofintereststatement

Theauthorsdeclarenoconflictsofinterest. Acknowledgement

The authorsthank allthe contributorsto pymia and acknowl-edge the valuable feedback by Florian Kofler. This research was partially supported by the Swiss National Science Foundation (SNSF)underthegrantnumbers169607and184273,andtheSwiss FoundationforResearchonMuscleDiseases(ssem).

References

[1] G.Litjens,T.Kooi,B.E.Bejnordi,A.A.A.Setio,F.Ciompi,M.Ghafoorian,J.A.van derLaak,B.vanGinneken,C.I.Sánchez,Asurveyondeeplearninginmedical imageanalysis,Med.ImageAnal.42(2017)60–88,doi:10.1016/j.media.2017.07. 005.

[2] D. Shen, G. Wu, H.-I. Suk, Deep learning in medical image anal-ysis, Annu. Rev. Biomed. Eng. 19 (1) (2017) 221–248, doi:10.1146/ annurev-bioeng-071516-044442.

[3] A.Maier,C.Syben,T.Lasser,C.Riess,Agentleintroductiontodeeplearning inmedicalimageprocessing,ZMedPhys29(2)(2018)86–101,doi:10.1016/j. zemedi.2018.12.003.

[4] A.S.Lundervold,A.Lundervold,Anoverviewofdeeplearninginmedical imag-ingfocusingonMRI,Z.Med.Phys.29(2)(2019)102–127,doi:10.1016/j.zemedi. 2018.11.002.

[5] A.Krizhevsky, I. Sutskever, G.E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks, in: F. Pereira, C.J.C. Burges, L. Bottou, K.Q.Weinberger(Eds.),AdvancesinNeuralInformationProcessingSystems25, CurranAssociates,2012,pp.1097–1105.

[6] O.Ronneberger,P.Fischer,T.Brox,U-Net:ConvolutionalNetworksfor Biomed-icalImageSegmentation,in:N.Navab,J.Hornegger,W.M.Wells,A.F.Frangi (Eds.),MedicalImageComputingandComputer-AssistedIntervention- MIC-CAI2015,LectureNotesinComputerScience,9351,Springer,2015,pp.234– 241,doi:10.1007/978-3-319-24574-4_28.

[7] M. Abadi, A.Agarwal, P.Barham, E.Brevdo, Z.Chen, C. Citro, G. Corrado, A.Davis, J.Dean,M.Devin,S.Ghemawat,I.Goodfellow,A.Harp, G.Irving, M.Isard,Y.Jia,L.Kaiser,M.Kudlur,J.Levenberg,D.Man,R.Monga,S.Moore, D.Murray,J.Shlens,B.Steiner,I.Sutskever,P.Tucker,V.Vanhoucke,V. Va-sudevan,O.Vinyals,P.Warden,M.Wicke,Y.Yu,X.Zheng,Tensorflow: large-S-calemachinelearning onheterogeneousdistributedsystems, arXivpreprint arXiv:1603.044671(2015)19.

[8] A.Paszke,S.Gross,F.Massa,A.Lerer,J.Bradbury,G.Chanan,T.Killeen,Z.Lin, N.Gimelshein,L.Antiga,A.Desmaison,A.Kopf,E.Yang,Z.DeVito,M. Rai-son,A.Tejani,S.Chilamkurthy,B.Steiner,L.Fang,J.Bai,S.Chintala,PyTorch: An ImperativeStyle, High-PerformanceDeep Learning Library, in: H. Wal-lach,H.Larochelle,A.Beygelzimer,F.Alché-Buc,E.Fox,R.Garnett(Eds.), Ad-vancesinNeuralInformationProcessingSystems32,CurranAssociates,2019, pp.8024–8035.

[9] E.Gibson,W.Li,C.Sudre,L.Fidon,D.I.Shakir,G.Wang,Z.Eaton-Rosen,R.Gray, T. Doel, Y. Hu, T. Whyntie, P.Nachev, M. Modat,D.C. Barratt, S. Ourselin, M.J.Cardoso,T.Vercauteren,Niftynet:adeep-learningplatformformedical imaging,Comput.MethodsProgramsBiomed.158(2018)113–122,doi:10.1016/ j.cmpb.2018.01.025.

[10] N.Pawlowski,S.I.Ktena,M.C.H.Lee,B.Kainz,D.Rueckert,B.Glocker,M. Ra-jchl,DLTK:StateoftheArtReferenceImplementationsforDeepLearningon MedicalImages,MedicalImagingMeetsNIPS,2017.

[11]C.S.Perone,C.Clauss,E.Saravia,P.L.Ballester,M.Tare,MedicalTorch:An open-sourcePyTorchmedicalimagingframework,2018,10.5281/ZENODO.1495335 [12] F.Pérez-García,R.Sparks,S.Ourselin,TorchIO:aPythonlibraryforefficient

loading,preprocessing, augmentationand patch-basedsampling ofmedical imagesindeeplearning,arXivpreprintarXiv:2003.04696(2020).

[13] A.Beers, J.Brown, K. Chang,K. Hoebel,E.Gerstner, B.Rosen, J. Kalpathy-Cramer,DeepNeuro:anopen-sourcedeeplearningtoolboxforneuroimaging, Neuroinformatics(2020),doi:10.1007/s12021-020-09477-5.

[14] L.R.Dice,Measuresofthe amountofecologicassociationbetweenspecies, Ecology26(3)(1945)297–302,doi:10.2307/1932409.

[15]D.Huttenlocher,G.Klanderman, W.Rucklidge,Comparing imagesusingthe Hausdorff distance,IEEETrans.PatternAnal.Mach.Intell.15(9)(1993)850– 863,doi:10.1109/34.232073.

[16]Z.Wang,A.Bovik,H. Sheikh,E.Simoncelli,Imagequalityassessment:from errorvisibilitytostructuralsimilarity,IEEETrans.ImageProcess.13(4)(2004) 600–612,doi:10.1109/TIP.2003.819861.

[17]M.McCormick,X.Liu,J.Jomier,C.Marion,L.Ibanez,ITK:Enablingreproducible researchandopenscience,Front.Neuroinform.8(FEB)(2014)13,doi:10.3389/ fninf.2014.00013.

[18]B.C.Lowekamp,D.T.Chen,L.Ibáñez,D.Blezek,TheDesignofSimpleITK,Front. Neuroinform.7(DEC)(2013)45,doi:10.3389/fninf.2013.00045.

[19]A.A.Taha,A.Hanbury,Metricsforevaluating3Dmedicalimagesegmentation: analysis,selection,andtool,BMCMed.Imaging15(1)(2015)29,doi:10.1186/ s12880-015-0068-x.

[20]S.vanderWalt,J.L.Schönberger,J.Nunez-Iglesias,F.Boulogne,J.D.Warner, N.Yager,E.Gouillart,T.Yu,T.scikit-imagecontributors,scikit-image:image processinginPython,PeerJ2(2014)e453,doi:10.7717/peerj.453.

[21]F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel,P.Prettenhofer, R.Weiss, V.Dubourg, J. Vanderplas, A.Passos, D. Cournapeau, M. Brucher,M. Perrot, É.Duchesnay, Scikit-learn: Machine Learning in Python, Journal of Machine Learning Research 12 (85) (2011) 2825–2830.

[22]P.Virtanen,R.Gommers,T.E.Oliphant,M.Haberland,T.Reddy,D.Cournapeau, E.Burovski,P.Peterson,W.Weckesser,J. Bright,S.J.vanderWalt,M.Brett, J.Wilson,K.JarrodMillman,N.Mayorov,A.R.Nelson,E.Jones,R.Kern,E. Lar-son, C.J. Carey, b. Polat, Y. Feng, E.W. Moore, J. Vand erPlas, D. Laxalde, J.Perktold,R.Cimrman,I.Henriksen,E.Quintero,C.R.Harris,A.M.Archibald, A.H.Ribeiro,F.Pedregosa,P.vanMulbregt,S.Contributors,SciPy1.0: funda-mentalalgorithmsforscientificcomputinginPython,Nat.Methods17(2020) 261–272,doi:10.1038/s41592-019-0686-2.

[23]A.Collette,PythonandHDF5,1,O’ReillyMedia,Sebastopol,2013.

[24]J.Zbontar,F.Knoll,A.Sriram,M.J.Muckley,M.Bruno,A.Defazio,M.Parente, K.J.Geras, J.Katsnelson, H. Chandarana, Z.Zhang,M. Drozdzal, A.Romero, M.Rabbat,P.Vincent,J.Pinkerton,D.Wang,N.Yakubova,E.Owens,C.L. Zit-nick, M.P. Recht, D.K. Sodickson, Y.W. Lui, fastMRI: An Open Dataset and BenchmarksforAcceleratedMRI,arXivpreprintarXiv:1811.08839(2018). [25]P.Jaccard,Thedistributionoftheflorainthealpinezone,NewPhytol.11(2)

(1912)37–50,doi:10.1111/j.1469-8137.1912.tb05611.x.

[26]D.Martin,C.Fowlkes,D.Tal,J.Malik,Adatabaseofhumansegmentednatural imagesanditsapplicationtoevaluatingsegmentationalgorithmsand measur-ingecologicalstatistics,in:IEEEInternationalConferenceonComputerVision, 2,2001,pp.416–423,doi:10.1109/ICCV.2001.937655.

[27]R.Cárdenes,R.deLuis-García,M.Bach-Cuadra,Amultidimensional segmen-tationevaluationformedicalimagedata,Comput.MethodsProgramsBiomed. 96(2)(2009)108–124,doi:10.1016/j.cmpb.2009.04.009.

[28]W.M.Rand,Objectivecriteriafortheevaluationofclusteringmethods,J.Am. Stat.Assoc.66(336)(1971)846–850,doi:10.1080/01621459.1971.10482356. [29]L.Hubert,P.Arabie,Comparingpartitions,JournalofClassification2(1)(1985)

193–218,doi:10.1007/BF01908075.

[30]M.Meil,Comparingclusteringsbythevariationofinformation,in:Learning Theoryand KernelMachines, in:LectureNotesinComputerScience, 2777, Springer,2003,pp.173–187,doi:10.1007/978-3-540-45167-9_14.

[31]P.E.Shrout,J.L.Fleiss,Intraclasscorrelations:usesinassessingraterreliability, Psychol.Bull.86(2)(1979)420–428,doi:10.1037/0033-2909.86.2.420. [32]G.Gerig,M.Jomier,M.Chakos,Valmet:Anewvalidationtoolforassessing

and improving 3D object segmentation,in: MedicalImage Computingand Computer-AssistedInterventionMICCAI2001,in:LectureNotesinComputer Science,2208,Springer,2001,pp.516–523,doi:10.1007/3-540-45468-3_62. [33]J.Cohen,Acoefficientofagreementfornominalscales,Educ.Psychol.Meas.

20(1)(1960)37–46,doi:10.1177/001316446002000104.

[34]D.M.W. Powers, Evaluation:from precision, recalland F-factor toROC, in-formedness,markednessandcorrelation, JournalofMachineLearning Tech-nologies2(2011)37–63.

[35]P.C.Mahalanobis,Onthegeneralizeddistanceinstatistics,in:Proceedingsof theNationalInstituteofSciencesofIndia,2,NationalInstituteofScienceof India,1936,pp.49–55.

[36]S. Nikolov, S. Blackwell, R. Mendes, J. De Fauw, C. Meyer, C. Hughes, H. Askham, B. Romera-Paredes, A. Karthikesalingam, C. Chu, D. Carnell, C.Boon,D.D’Souza,S.A.Moinuddin,K.Sullivan,D.R.Consortium,H. Mont-gomery,G.Rees,R.Sharma,M.Suleyman,T.Back,J.R.Ledsam,O.Ronneberger, Deeplearningtoachieveclinicallyapplicablesegmentationofheadandneck anatomyforradiotherapy,arXivpreprintarXiv:1809.04430(2018).

[37]D.C. VanEssen,S.M.Smith, D.M.Barch,T.E.Behrens,E.Yacoub,K. Ugurbil, WU-Minn HCP Consortium, The WU-Minn human connectome project: an overview,Neuroimage80(2013)62–79,doi:10.1016/j.neuroimage.2013.05.041. [38]B. Fischl, FreeSurfer, Neuroimage 62 (2) (2012) 774–781, doi:10.1016/j.

neuroimage.2012.01.021.

[39]B.Fischl,D.H.Salat,E.Busa,M.Albert,M.Dieterich,C.Haselgrove,A.VanDer Kouwe,R.Killiany,D.Kennedy,S.Klaveness,A.Montillo,N.Makris,B.Rosen, A.M.Dale,Wholebrainsegmentation:automatedlabelingofneuroanatomical structures inthe humanbrain,Neuron33(3)(2002)341–355,doi:10.1016/ S0896-6273(02)00569-X.

[40]M.Wiesenfarth,A.Reinke,B.A.Landman,M.J.Cardoso,L.Maier-Hein,A. Kop-p-Schneider,Methodsandopen-sourcetoolkitfor analyzingandvisualizing challengeresults,arXivpreprintarXiv:1910.05121(2019).

Valen-zuela, M. Reyes, O. Scheidegger, Segmentation of peripheral nerves from magneticresonanceneurography:AFully-automatic,deeplearning-based ap-proach,Front.Neurol.9(2018)777,doi:10.3389/fneur.2018.00777.

[42]A.Jungo, M.Reyes, Assessing Reliability andChallenges ofUncertainty Es-timationsfor Medical ImageSegmentation,in: D.Shen,T. Liu,T.M.Peters, L.H.Staib,C.Essert,S.Zhou,P.-T.Yap,A.Khan(Eds.),MedicalImage Com-puting and Computer Assisted Intervention - MICCAI 2019, LectureNotes inComputerScience, 11765,Springer, Cham,2019, pp.48–56,doi:10.1007/ 978-3-030-32245-8_6.

[43]A.Jungo,R.McKinley,R.Meier,U.Knecht,L.Vera,J.Pérez-Beteta,D. Molina-García, V.M.Pérez-García, R.Wiest, M. Reyes,Towards uncertainty-assisted brain tumor segmentation and survival prediction,in: A. Crimi,S. Bakas, H. Kuijf,B.Menze,M. Reyes(Eds.),Brainlesion: Glioma,MultipleSclerosis, StrokeandTraumaticBrainInjuries.BrainLes2017,LectureNotesinComputer Science,10670,Springer,2018,pp.474–485,doi:10.1007/978-3-319-75238-9_ 40.

[44] A.Jungo,F.Balsiger,M.Reyes,Analyzingthequalityandchallengesof uncer-taintyestimationsfor braintumorsegmentation,Front. Neurosci.14(2020) 282,doi:10.3389/FNINS.2020.00282.

[45]A.Jungo,R.Meier,E.Ermis,M.Blatti-Moreno,E.Herrmann,R.Wiest,M.Reyes, Ontheeffectofinter-observervariabilityfor areliableestimationof uncer-taintyofmedicalimagesegmentation,in:MedicalImageComputingand Com-puterAssistedIntervention-MICCAI2018,in:LectureNotesinComputer Sci-ence,11070,Springer,2018,pp.682–690,doi:10.1007/978-3-030-00928-1_77. [46]A.Jungo,R.Meier,E.Ermis,E.Herrmann,M.Reyes,Uncertainty-drivenSanity Check:ApplicationtoPostoperativeBrainTumorCavitySegmentation, Medi-calImagingwithDeepLearning-MIDL2018,2018https://arxiv.org/abs/1806. 03106.

[47]E.Ermi,A.Jungo,R.Poel,M.Blatti-Moreno,R.Meier,U.Knecht,D.M. Aeber-sold,M.K.Fix,P.Manser,M.Reyes,E.Herrmann,Fullyautomatedbrain resec-tioncavitydelineationforradiationtargetvolumedefinitioninglioblastoma patientsusingdeeplearning,RadiationOncology15(1)(2020), doi:10.1186/ s13014-020-01553-z.

[48]F. Balsiger, A. Shridhar Konar, S. Chikop, V. Chandran, O. Scheidegger, S.Geethanath,M.Reyes,MagneticResonanceFingerprintingReconstructionvia SpatiotemporalConvolutionalNeuralNetworks,in:F.Knoll,A.Maier,D. Rueck-ert(Eds.),MachineLearningforMedicalImageReconstruction,LectureNotes in ComputerScience, 11074, Springer,Cham,2018, pp.39–46, doi:10.1007/ 978-3-030-00129-2_5.

[49]F.Balsiger, O. Scheidegger,P.G. Carlier, B.Marty, M.Reyes, Onthe Spatial and Temporal Influencefor the ReconstructionofMagneticResonance Fin-gerprinting, in:M.J. Cardoso,A.Feragen,B. Glocker,E.Konukoglu,I. Oguz, G.Unal,T.Vercauteren(Eds.),ProceedingsofThe2ndInternationalConference onMedicalImagingwithDeepLearning,ProceedingsofMachineLearning Re-search,102,PMLR,London,2019,pp.27–38.

[50]F.Balsiger,A.Jungo,O.Scheidegger,P.G.Carlier,M.Reyes,B.Marty,Spatially regularizedparametricmapreconstructionforfastmagneticresonance finger-printing,Med.ImageAnal.64(2020)101741,doi:10.1016/j.media.2020.101741. [51]Y.Suter,A.Jungo,M.Rebsamen,U.Knecht,E.Herrmann,R.Wiest,M.Reyes, Deeplearningversusclassicalregressionforbraintumorpatientsurvival pre-diction,in:A.Crimi,S.Bakas,H.Kuijf,F.Keyvan,M.Reyes,T.V.Walsum(Eds.), Brainlesion:Glioma,MultipleSclerosis, StrokeandTraumatic BrainInjuries. BrainLes 2018, LectureNotes inComputer Science, 11384, Springer, Cham, 2019,pp.429–440,doi:10.1007/978-3-030-11726-9_38.

[52]F.Balsiger, Y. Soom,O. Scheidegger,M. Reyes, Learning Shape Representa-tiononSparsePointCloudsforVolumetricImageSegmentation,in:D.Shen, T.Liu,T.M.Peters,L.H.Staib,C.Essert,S.Zhou,P.-T.Yap,A.Khan(Eds.), Medi-calImageComputingandComputerAssistedInterventionMICCAI2019, Lec-ture NotesinComputerScience,11765,Springer,Cham,2019,pp.273–281, doi:10.1007/978-3-030-32245-8_31.

[53]F.Isensee,P.Jäger,J.Wasserthal,D.Zimmerer,J.Petersen,S.Kohl,J.Schock, A.Klein,T.Roß,S.Wirkert,P.Neher,S.Dinkelacker,G.Köhler,K.Maier-Hein, batchgenerators-apythonframeworkfor dataaugmentation,2020,doi:10. 5281/ZENODO.3632567.