Hyperspectral imaging combined with data classification techniques as an aid for artwork authentication

(1)

Eastaugh, Francis (2017) Hyperspectral imaging combined with data

classification techniques as an aid for artwork authentication. Journal of

Cultural Heritage. ISSN 1296-2074 ,

This version is available at

http://strathprints.strath.ac.uk/59987/

Strathprints is designed to allow users to access the research output of the University of

Strathclyde. Unless otherwise explicitly stated on the manuscript, Copyright © and Moral Rights

for the papers on this site are retained by the individual authors and/or other copyright owners.

Please check the manuscript for details of any other licences that may have been applied. You

may not engage in further distribution of the material for any profitmaking activities or any

commercial gain. You may freely distribute both the url (

http://strathprints.strath.ac.uk/

) and the

content of this paper for research or private study, educational, or not-for-profit purposes without

prior permission or charge.

Any correspondence concerning this service should be sent to the Strathprints administrator:

[email protected]

The Strathprints institutional repository (http://strathprints.strath.ac.uk) is a digital archive of University of Strathclyde research outputs. It has been developed to disseminate open access research outputs, expose data about those outputs, and enable the

(2)

David

J.M.

Stothard

b

_,

_Nicholas

_Eastaugh

c

_,

_Francis

_Eastaugh

c a_Centre_for_Signal_&_Image_Processing,_University_of_Strathclyde,₂₀₄_George_Street,_Glasgow_G1_1XW,_UK b_Fraunhofer_Centre_for_Applied_Photonics,_Fraunhofer_UK_Research_Ltd,₉₉_George_Street,_Glasgow_G1_1RD,_UK c_Art_Analysis_and_Research_Ltd,_162-164_Abbey_Street,_London_SE1_2AN,_UK

i

n

f

o

a

r

t

i

c

l

e

Historiquedel’article: Rec¸ule18mai2016 Acceptéle30janvier2017 DisponiblesurInternetlexxx Keywords:

Hyperspectralimaging(HSI) Infrared

Artworkauthentication Supportvectormachine(SVM)

A

b

s

t

r

a

c

t

Inrecentyearsvariousscientificpracticeshavebeenadaptedtotheartworkanalysisprocess.Althougha setoftechniquesisavailableforarthistoriansandscientists,thereisaconstantneedforrapidand non-destructivemethodstoempowertheartauthenticationprocess.Inthispaperhyperspectralimaging combinedwithsignalprocessingandclassificationtechniquesareproposedasatooltoenhancethe processforidentificationofartforgeries.Usingbespokepaintingsdesignedforthiswork,aspectrallibrary ofselectedpigmentswasestablishedandtheviabilityoftrainingandtheapplicationofclassification techniquesbasedonthisdatawasdemonstrated.Usingthesetechniquesfortheanalysisofactualforged paintingsresultedintheidentificationofanachronisticpaint,confirmingthefalsityoftheartwork.This paperdemonstratestheapplicabilityofinfrared(IR)hyperspectralimagingforartworkauthentication. ©2017LesAuteurs.Publi ´eparElsevierMassonSAS.Cetarticleestpubli ´eenOpenAccesssouslicence CCBY(http://creativecommons.org/licenses/by/4.0/).

1. Introduction

Accordingtoarecent studies,in2014theglobal artmarket reacheditshighestever-recorded levelof just overD₅₁_billion worldwide[1].Thisrepresentsa7%year-on-yearincreasefrom D_47.4_billion_recorded_in_total_sales_of_art_and_antiques_in_2013, consisting of more than 36 million transactions [2]. The vast majorityofthesehighvaluedealingsweremadewithout scien-tificor forensictesting toassuretheauthenticityof thetraded objects.Non-scientificartexpertise–knownintheartworldas connoisseurship–is acommonpracticetoassessthe authenti-city.Nevertheless,anexperiencedspecialist canonlyevaluatea limitedamountoftheartworkandwhennotsupportedby addi-tionalscientifictests,thatevaluationissubjectiveandassuchit isnotinfallible[3,4].Servicesusingscientificapproachesto deter-minetheauthenticityofartworksareavailable;however,thesecan haveperceivedissues,includingthetimeinvolvedandtheneedto removesamplematerialforanumberofthetechniques[3].There isconsequentlyaneedforefficient,portableandcosteffective non-destructivemethodsofartanalysistoserveabroaderrangeofthe

∗ Correspondingauthorat:CentreforSignal&ImageProcessing,Universityof Strathclyde,204GeorgeStreet,GlasgowG11XW,UK.

Adressee-mail:[email protected](A.Polak).

market.Insomecases,duetothehighvalueanduniquenatureof theobjects,thepaintsamplingrequiredbycertaintypesof exa-minationsmayalsoberestricted.Non-destructivetestsprovide thepossibilitytousecomplementarytechniquesandobtainmore informationfromthesamesample.Severalsuchmethods,for ins-tanceX-rayfluorescenceandFTIR(FourierTransformInfrared)or Ramanspectroscopy,existandareapplicableforstudyingartwork

[5,6].Althoughthesemethodsarecommonly usedforscientific artinvestigationaswellasfor someotherapplications,thereis still aneedfor new,non-invasive techniquesthatcouldextend theamountof informationobtainedfrom theartworkanalyses andlimitthenumberofinvasivetestingrequired.Inthisresearch, HyperspectralImaging(HSI)combinedwithchemometrics algo-rithmsisproposedasanovel,non-invasiveanalysismethodfor classificationandmappingofpaintsandpigments.Theaimisthat thesetoolswillserveasanaidforartworkevaluationand specifi-cally,theidentificationofcounterfeits.

In recent years Hyperspectral Imaginghasundergone signi-ficant development. There is an increasing amount of camera technologies that, with different configurations, provide many waystoobtainhyperspectraldataoverseveralspectralranges.This emerging technologyis rapidlyfindingapplicationsin different fields, including pharmaceuticals [7], agriculture [8] and food qualitycontrol[9–12],aswelltheartworld,formaterial identi-ficationandmappingoftheworksofart[13–18].Todate,most http://dx.doi.org/10.1016/j.culher.2017.01.013

(3)

applicationsofhyperspectral(andmultispectral)technologyare fortherestorationandconservationofpaintings[19–22].Pigment analysisprovidedbyHSIsystemscoupledwithdedicated classi-ficationalgorithmsallows theidentificationof“restored zones” inthepaintinganddifferentiatesthesefromsignificantareasof theoriginalpaintingwhichthesystemthensuggestsforpigment analysis[18,19,23,24].TheuseofHSIintheinfraredspectralrange hasalsohelped toreveal featuresof artists’techniquessuchas theirpreparatory drawing[22,25].Duetotheverybroad range of wavelengths available for hyperspectral systems, the trans-mittanceandreflectanceresponseofdifferentlayersofpaintings anddrawingsisfrequentlyobservedduringdataanalysis.When materialthatis transparentataspecificwavelength range(but opaqueatothers)coversmaterialthatisreflectivewithinthesame spectralrange,theunderlyingmaterialcanbedetectedbytheHSI systemand ishencerevealed inthedataacquired.Empowered bysignalprocessing techniquesthis facilitatesa detailed study oftheartworkcreationprocessandenablesidentificationofthe materialsused[26].SpectralselectivityofHSIdatawasalsoused onvariousoccasions toanalysetexts of historic value[27–29]. Identification of pigments and inks facilitated by HSI has also beenusedtoaidindatingofmanuscripts[27].Furthermore,HSI technology empowers the recovery of erased and overwritten scriptsaswellasallowingthedeterminationofappropriatebands formonitoringlaserandnon-lasercleaningprocesses[28].

Alongsidetheaforementionedbenefitsforartconservation,the potentialof hyperspectralimaging inforgery detectionhasalso beenrecognised.TheapplicationofHSItoaddressthechallenge offorensicanalysisofdocumentswassuccessfullyappliedinthe past[30–32].Classificationofdifferentinksafterobliterationofthe textandthe‘crossinglinesproblem’[30]werestudiedand analy-sedwithchemometrics-basedtools.Thesetechniquesappliedto HSIdatahavehadasignificantimpactonforgeryrecognitionand providedobjectiveresultscomparedtotraditionalvisual inspec-tionbasedjudgments[30].OtherforensicapplicationsofHSIwere alsoreported,suchasfingerprintdetection[33]andforbloodstain datingatcrimescenes[34].

Theimplementationofhyperspectralimaginginthe aforemen-tionedapplicationsgaveaccesstotherichdataset,howeverthe conclusionsdrawnfromthisdatawerebasedonsubsequentsignal processingmakingitdifficulttousebynon-experts.Insomecases theuseofpurespectroscopictechniquesachievedsufficientresults

[25],whileinothers,moreadvancedalgorithmswereappliedin ordertoanalysethedata[23,24,26].Itwasalsorecognisedthatfull diagnosticpotentialofHSImaybeimprovedbyimplementationof robustdataprocessingalgorithms[28].

ItisclearthatHSItechnologycombinedwithadvancedsignal processingtechniqueshavealreadyfoundvariousapplicationin theartworld.However,todatethesehavefocusedonsupporting variousaspectsofconservationandhaveallowedresearchersto betterunderstandpaintingsbyallowingthemtoobserve mate-rialsbelowthesurfaceofthecompletedwork.Inthispaper,we illustratea novelcombination ofnear-and mid-infrared hyper-spectralimagingwithstate-of-the-artsignalprocessingalgorithms andbackgroundinformationfromexpertsinthefieldofart analy-sistoprovideHSIdatabasedclassificationofpaintsandartwork forthepurposesofauthentication.Aslongasnearinfraredrange wasreachedwith widelyknown HSItechnology, access tothe mid-IRregionwasgrantedbythenovelapplicationofanactive, laser-basedmid-IRImager.Althoughsimilarwavelengthrangewas alreadyexploredwithapassivesystem[35],totheauthors’ know-ledge,ourworkpresentsthefirsteverapplicationofthisactive devicefor theartwork analysis. Thistext demonstrates hyper-spectral imaging empowered by automated paint classification techniquesasanon-invasivemethodsupportingtheidentification ofcounterfeitpaintings.Ourworkwasdividedintotwoparts:(1)

algorithmsweredevelopedusingbespokepaintingswhichwere createdforthisstudyand imagedin awell-controlled environ-mentunderlaboratoryconditionsand(2)thetechniquesdeveloped wereappliedtohyperspectralimagesofpaintingsheldbytheBerlin Landeskriminalamtwhichofcomprisedknownandsuspected for-geries,including,forthefirsttimeanalysedwithHSI,paintingsfrom theinfamousBeltracchicase [36–40].Thispapermaintainsthis dualstructureandfocusesinitiallyondescribingsystem develop-mentandtestingbeforeprovidingdetailsandtheresultsachieved duringthiswork.Someaspectsofthisstudywerealsodescribedin

[41]howeverthattextfocusesonthedifficultiesofdataacquisition andmethodsforovercomingtheseproblems.Here,thefocusison thedataanalysisandprocessingaswellastheapplication.

2. Materialsandmethods 2.1. Hyperspectralequipment

Applications of HSI systems operating in the visible–near-infrared (Vis–NIR) spectrum (400–1000nm) have already been presentedintheliteratureandtendtofocusonperformingand supporting various tasks including spectral characterisation of pigments[17,18,20,23,26–28].InGaAsdetectorbased hyperspec-tralimagers arealso reachingfurtherintonear-infraredregion (900–1700nm, and in some cases extended up to 2500nm) and thesealsohavefoundapplication inthestudyof artworks

[16–18,25].However,relevantliteraturedescribingtheuseof sen-sorsoperatinginlongerwavelengths,approachingupto4000nm, whichareknowntocontainrichspectralinformationanduseful chemometricdescriptorsisnotsoreadilyavailable.Inthispaper, wethereforeusetwohyperspectralimagingsystemsoperatingin different(butoverlapping)regionsoftheinfraredportionofthe electromagneticspectrum.Thechoiceofthesetwosystems allo-wedustostudytheimpactoftheimageacquisitiontechniques andilluminationmethods[41]ontheperformanceofourproposed signalandimageprocessingtechniquesdesignedtoautomatically analysethenear-andmid-infraredrangedata.Thisworkisdriven bythemotivationthatinadditiontothecolourinformation contai-ned in thevisible spectrum,often sufficient toidentifyvarious pigments,arangeofpainttypes(includingpigments,bindersand solvents)alsohavespectral featuresin thelongerwavelengths. Hencethisstudyisfocussedonexploringthesefortheaccurate discriminationofpaints.Itshouldbenotedhowever,thatwhile theintentionofthisstudyistoexploretheusefulnessofthese lon-gerwavelengths,manypigmentscanbediscriminatedusingthe Vis–NIRregionandthiscouldbebeneficialforthefinalapplication ofthistechnologybyartscientists.Assuch,wediscussthistopic furtherinSection4.1.

The hyperspectral imaging systems which were employed duringthisstudywere:anactive,laserbased,mid-infrared hyper-spectralimager(FireflyIRImager,MSquaredLasers–seeFig.1a) andapassivehyperspectralcameraoperatinginthenear-infrared wavelengthrange(RedEye1.7,inno-specGmbH–seeFig.1b).

Whilethenear-IRrangecoveredbythepassivesystem(from 900nm upto1700nm)is becomingmore commonandcanbe captured by various systems, devices operating in theinfrared bandwidthbeyond3000nmarestillquiterare.TheFireflyIR Ima-ger is based onOptical Parametric Oscillator (OPO) technology that,withitsinherentnarrowspectrallinewidthandwavelength tunabilitymakesafoundationforanewclassofhyperspectral ima-gingtechnologythatisabletosenseradiationbeyond3000nm. It achieves this byconvertingradiation from a fixedfrequency near-infraredlasersource(1064nm)intobroadlytunable radia-tioninthemid-IRportionofthespectrum(2500–3750nm)where compoundscontainedinpaintsexhibitdistinctopticalabsorption.

(4)

Fig.1.Hyperspectralsystemsusedduringtheproject;(a)FireflyIRImager;(b)RedEye1.7.

Fig.2. Illustrationofpaintingsusedduringthelabstageoftheprojectandtheirgroundtruthdescription;(a)thepigmentgridcanvasservingasatrainingdata;(b)description ofthepigmentsusedinthegridcanvas;(c)Pasticheof“Untitled(SuprematistComposition),”byKazimirMalevichastestingpainting;(d)groundtruthdataofpigments usedforcreationoftheKazimirMalevichpastiche[41].

(5)

Fig.3. Averagespectralresponseofallthegreencolouredpigmentsacquiredbypassive(a)andactive(b)system.

ThankstotheOPOtechnology,theFireflyIRImageralsofeaturesa narrowspectralbandinanear-IRrange(1490–1850nm)thus pro-vidingagoodlinkbetweenbothimagingsystemsusedinthiswork. 2.2. Buildingandvalidatingaspectrallibraryusingbespoke paintings

Twodifferentpaintingswerepreparedspecificallytoallowthe constructionofa “spectrallibrary”fromHSIdatasetscontaining onlyknownpaintsatspecificlocations.Bothpaintingswereused foralgorithmdevelopmentandvalidationafterthespectrallibrary wascreated.Onetypeofpaintingusedmainlyfordevelopment wasconstructedintheformofagridof41oil-basedpaintsbased onaselectionrepresentativeofthematerialscommonlyusedby artistsduringthe20thcentury(seeFig.2a)[42].Exemplarpaints werepurchasedfromanumberofspecialistartists’colourmenwith particularreferencetothoseofferingrangesthatinclude ostensi-bly“historicallyappropriate”materials(MichaelHarding;Rublev; Blockx).Thefulldescriptionofallpaintscontainedbythismatrix

canbefoundinFig.2bandAppendix1.Allpaintsusedwerealso analysed usingother instrumentaltechniques suchasscanning electronmicroscopy-energydispersiveX-rayspectrometry (“SEM-EDX”),FTIRspectroscopyandRamanmicroscopy;identifications weremadewithreferencetospectrallibrariesderivedfromthe PigmentumProjectcollectionofhistoricalpigments[42].

Twoofthese“gridcanvases”asdescribedabovewereusedfor signalprocessingalgorithmdevelopmentandalsoservedas trai-ningdataforclassificationsofotherunseenpaintingsbelievedto containatleastoneormoreofthesepigments.Asecondstyleof canvas,usedforalgorithmvalidation,wasapasticheofa Suprema-tistworkbyKazimirMalevich(seeFig.2c).Thisbespokepainting wascreatedusingaselectivesubsetofthepaintscontainedinthe gridcanvas(Fig.2a),andwasaccompaniedbylabelled “ground-truth”datadescribingeachareapaintedwithdifferentpaint(see

Fig.2d).

Examinationofthesecondpaintingprovidedcontrolled condi-tions for validation of the algorithms developed for automatic paintrecognition.Thesetwoanalysedpaintings(gridcanvasand

(6)

Fig.4.PCAscoresplotoffirstthreeprincipalcomponentsforthegroupofgreenpaintsimagedwithpassive(a)andactive(b)system.

pastiche) constituted not only a very well controlled but also a realisticdatasetof paintswhich arecommonly usedin 20th centuryartworkuponwhichthemethodsdevelopedinthiswork couldbeassessed.

2.3. Dataacquisitionandpre-processing

Due to significantly different mechanisms driving the two hyperspectralimagersusedduringthis project,dataacquisition anditspreparationforanalysisvariedconsiderablybetweeneach case[41].

2.3.1. Passivesystem

ThepassiveRedEye1.7systemacquiresHSIdatacubesusing apushbroomtechnique[43]and requiresrelativemovementof theobjectinfrontofthedetector.AZolixKSA11-200S4Nlinear translationstagewasusedtoscanthepaintings.Illuminationwas providedbyoff-the-shelf12VDChalogenreflectors.Thehalogen lamps,asanincandescentlightsource,emitnotonlyaportionof energyinthevisiblebandoftheelectromagneticspectrum,but alsoprovideexcellentilluminationinthenear-IRrangeasrequired

bythissystem[44].Duringeachdataacquisitionrun,reflectance calibrationwasalsoperformedtoensurethatbackground spec-tralresponsesoftheinstrumentandillumination,aswellasthe ‘dark’currentofthecamerawereaccountedforinthedatasetand thereforedonotaffecttheresultsofanysubsequentanalysis.The relativereflectancefortherawimagescanbecalculatedusing: R=_WI0−D

−D (1)

whereRistherelativereflectanceimage,I0istherawreflectance

image,Disthedarkreferenceimage,andWisthewhitereference image[43].ThespectralbackgroundWwasobtainedbyscanninga whitereferencetilemadefromSpectralon–amaterialofhigh Lam-bertianreflectionoveritsreflectivespectralrangeof250–2500nm

[45]. ThedarkreferenceDwascaptured byfullyobscuring the camera objective using an opaque black cap. All hyperspectral imagesofthepaintingswerepreparedinthesamewayusingEq.(1)

priortotheanalysisandnoadditionalpre-processingwasapplied. 2.3.2. Activesystem

TheactiveFireflyIRImageracquiresHSIdatausingactivelaser illuminationwhichscansoveritsworkingspectralrangeandno

(7)

externalilluminationisrequired.Thisimagerisequippedwitha built-inscannerbasedontwooscillatingmirrorsprovidingspatial scanningofthelaserbeamacrossthetargetobjectandtherefore employingawhiskbroom scanningtechnique[43].Tominimise theeffectofwhiskbroomscanningartefacts(spatialdistortionand intensityvariation),andtoincreasespatialresolutionoftheimage, allanalysedpaintingswerescannedinsectionswhichwerethen stitchedtogethertorecreatewholespatial areaofthepainting inonehyperspectraldatacube[41].Thesizeofthepainting sec-tionswerechosenas50×75mm(thissizewasderivedempirically) andafullpaintgridcanvaswasreproducedfrom9suchsections, whiletheslightlysmallerMalevichpasticheconsistedof 6 sec-tions.Duetothenatureofthehardware(complexityofdetector settings),thereflectancecalibrationwasnotpossibleforthis equip-ment.Torescalethespectraldatatothereflectancerange[0–1], the8-bitintensitydataacquiredoneachwavelengthwasdivided bythemaximumvalue(255).Thankstothecontinuousspectral tunabilityofthelasersource,theacquisitionstepmaybe arbitra-rilysetbytheoperator,withthehardwarelimitationof0.1nm. However,duetothefinitelinewidthofthelasersource(approx. 5cm−1)andshapeofspectralfeaturesofimagedpaints,6nm

spec-tralresolutionwaschosen,providing61spectralimagebandsinthe near-IR(1490–1850nm)and209imagebandsinthemid-IRregion (2500–3750nm).Aftercompletingthedatastitchingandrescaling, nofurtherpre-processingwasapplied.

3. Algorithmdevelopmentandfeatureextraction

Afterthepre-treatmenttonormaliseandcalibratethedata(see Section2.3),subsequentprocessingwasthesamefor bothdata setsandanalgorithmwasdesignedtofacilitatehyperspectral ana-lysisoftheartwork.Sincethisworkaimedattheidentificationof differentpaints,themainaspectofalgorithm developmentwas focusedontheapplicationofrobuststatisticalclassification tech-niques.Bothsupervised(guidedbyhumanprovidedtrainingdata) andunsupervised(fullybasedonsoftwareanalysisoftheimage) techniqueswereconsidered[46].Sincetheobjectiveofworkisthe detectionofcounterfeitpaintingsbytheclassificationofknown paints,supervisedclassificationwaschosenasthemostsuitable forthisapplication.ThegridcanvaspresentedinFig.2a)wasused tobuildaspectral library ofselectedpigments andthis served astraining datafor thealgorithm.Fig.3 illustratestheaverage spectra(acquiredbybothsystems)ofthegroupofgreen paints availableonthegridcanvasthatisshowninFig.2.Allthe ave-ragespectrafromthe41 paints in thelibraryare presented in

Appendices2and3.

Althoughthisfigureillustratestheaveragedspectralresponse fromtheentirepaintregions,thealgorithmistrainedtorecognise allvariationsofthisresponseresultingfromtheunevenpaint sur-faceswhenimaged.Asitisexpectedthatduringtheanalysisof artworkthecaptureddatawillalsocontainvariousartefacts of imagingprocess,allthesevariationswerealsoallowedinthe trai-ningsetclasses. Thetraining datatherefore includedexamples ofspecularreflectionsandintensityvariationscomingfromthe three-dimensionalstructureofthepaintblobsandtheirthickness. APrincipalComponentAnalysis(PCA)wasperformedtoassessthe qualityofthistrainingsetandaPCAscoresplotshowingthefirst threeprincipalcomponents,explainingover95%ofvarianceinthe data,wasplotted.Theexamplesoftheseplotsforthegroupofgreen paintsacquiredbybothimagingsystemsareshowninFig.4.For completenessallPCAscoresplotsforthefullacquiredlibraryare showninAppendices4and5.

Althoughthereisawiderangeofalgorithmswhichcouldbe chosentoperformmultivariateanalysisusingsupervised classifi-cation,aSupportVectorMachine(SVM)wasidentifiedasthemost

Fig.5.IllustrationoftheclassificationalgorithmvalidationbasedonRedEye1.7 data;(a)intensityimageononewavelengthfortwogridcanvas;(b)singlerow withassignedcolourlabelstoeachpaint;(c)classificationresultofthetraining dataset(ontheleftside)andvalidationdataset(ontheright).

suitablemethodforthisprojectduetoitsrobustnessandconsistent classificationaccuracy[47–49].SVMsconsistofafamilyoflearning algorithmsthatcanbeusedfordataclassificationanddemonstrates verygoodperformanceinmanyapplications[47,48,50].Forthis reasonitisoneofthemostpopularmachinelearningalgorithms thatisoftenusedtosolveclassificationproblems.Thespecific algo-rithmusedforthepaintclassificationinthisstudyiscontainedin theLIBSVMpackage[51],whichusesaC-SVMtypeof classifica-tion.AstheSVMisabinaryclassifier,themulticlassproblemwas approachedusinga‘one-against-one’technique.Thequalityofthe modelwasalsoevaluatedusingCrossValidationAccuracyscores acquiredduringmodeltraining andfinalClassificationAccuracy

[51].

Inordertofacilitatetheconstructionofaspectrallibraryand algorithmdevelopment,twoofthe“gridcanvases”(seeSection2.2

andFig.2)wereimagedsidebyside,andonewasusedas trai-ningdatasetwhiletheotherasvalidationset(seeFig.5a).This approach alleviatesthe needfor k-fold crossvalidation, as this introducesnew,‘unseen’datafortheclassifier.Regionsof inter-est(ROI)–subsets oftheimagemanuallyselectedtocontaina groupofpixelscorrespondingtoasinglepaint–weredefinedfor all41paintsamplesofthegridcanvas.Sinceeachrowinthe can-vasrepresentsonegroupofcolours(seeFig.2a),eachrowofthe grid,labelled1–7,wasconsideredseparatelyintheHSIdatafor algorithmdevelopment.Eachpaintisallocatedalabeldenotedby coloursinFig.5.

Theleftmostpainting(Fig.5a)wasusedastrainingdataand withuseofROIs,eachpaintoftheselectedrowwasassigneda colourlabelfor classification(seeFig.5b).Duringthisprojectit waschosennottocompressthespectraldimensionandinstead thefullspectralprofilewasusedasthedatafeaturesforalgorithm training.Aftertraining,theclassificationwasperformedonboth paintingsandthewholeprocesswasrepeatedforeachrow1–7in turn.Satisfactoryclassificationresultswereobtainedforboth–the

(8)

Fig.6.Illustrationoftheanalysisapproachforpigmentclassificationontestedpainting.Foreachcolouronthepaintingsubsetoftrainingpaintswaschosen(firstcolumn– Trainingdata)andclassificationwasperformedonthemaskedareaofthepaintingcorrespondingtothiscolour(secondcolumn–Maskedregions).Classificationresulted inper-pixelclassificationoftheselectedarea(thirdcolumn–Classificationresult)andmajorityvotewasdrawnfortheseregionsresultinginselectionofonepigment correspondingtoonecolour(fourthcolumn–majorityvoteresult).Thisfigureshowsthreeexamplesofthisapproachfortheregionsofbrown(a),green(b)andwhite(c) paint(demonstrationbasedonFireflyIRImagerdata).

datasetthatclassifierwastrainedon,aswellasthenew,previously unseendatasettowhichtheclassifierwasapplied.Agraphical illustrationofthecombinedresultforeachrowwhenapplyingthe proposedtechniquetoeachrowseparatelyispresentedinFig.5c. 4. Classification

4.1. Candidatepaintselection

Inthispaper,adualstageclassificationprocessisproposed.To limittheamountoftrainingdatabuiltintotheclassificationmodel torecognisean individualpaint,thefirst stageof classification aimedatchoosingasubsetof3potentialpaintsfromthe hyperspec-tralimageofthe“gridcanvas”.Thismethodnotonlyaccelerated theclassificationprocessandreducedcomputingpowerrequired fortheproblembycomparingonlylikelycandidates,butalso redu-cedthechanceofmisclassification.Inpractice,itdoesnotmake sensetoattempttorecogniseonesinglepaintbyreferencingand comparingitwiththeentirespectrallibraryofallpaintsavailable –especiallythosewhichareclearlya differentcolour.Itshould benotedthatmanypigmentscanbeaccuratelydistinguishedby analysingthevisibleregionofthespectrumand, forthese,this primaryclassificationmaybesufficient.Forallothers,shortlisting likelycandidatesbasedonthecolourandappearanceisapractical solutiontoreducecomputationaloverheadandthelikelihoodof error.Infact,whilethepresentedconcepthasalready demons-trateditspotentialwithatrainingsetofonly41paints,withthe expectedextensionofthelibrarytocontainhundredsofdifferent pigments, a pre-selection suchas that would be necessary for

efficientperformanceofspectraldatabasedclassification.Todate, theselectionofcandidatepaintshasbeencarried outmanually basedonpaintcolour,howeveranRGBoravisiblerangeHSIsystem couldalsobeusedforthistaskandalgorithmswillbedevelopedto performthisinitialcandidateselectionstepbeforethefinalspectral classificationismade.Furthermore,basedonthegroundtruthdata, thedescribedvisualinspectionalwaysshortlistedthecorrectpaint andthereforeitwasclearlyaviablesolutionandonewhichcould beeasilyperformed–evenbyanon-expertuserofthetechnology. 4.2. Accuratespectralclassification

Fig.6providesexamplesofregionsselectedforclassification andpaintcandidateschosenastrainingdatafortheirclassification. Duetothehighlygeometricalstructureofthepaintinganalysedin

Fig.6(alsoshowninfullandincolourinFig.2c),masks(binary imagesusedtospecifyregionsoftheimageforprocessing)were createdforeachpaintingarea.Eachstructureonthepaintingwas preparedwithasingle,unmixedpaint.Therefore,tomake demons-trationoftheclassificationresultssimpler,regionsofthepainting correspondingtoasinglecolourwereclassifiedseparately,with theremainderoftheimagebeingdisabledfromclassificationby applicationofthemasks.

Afterthecandidatepre-selection,aspectralclassificationstage usingaSVMclassifierandmajorityvotingschemewasappliedto assignthechosenpaintregiontooneoftheselectedclassesbased onthespectralsignatureineach pixel ofthemaskedregionin thetestpainting.Theclassificationwasperformedona pixel-by-pixelbasis,butwiththepriorknowledgethateachsectionofthe

(9)

Fig.7.Graphicalillustrationofclasslabellingforalltheregionsofthetestpaining;(a)thecolourcodingoftheresultdemonstration;(b)Groundtruthdataillustration;(c) theresultbasedondatafromFireflyIRImager;(d)theresultbasedondatafromRedEye1.7system[41].

testpaintingwascreatedwithonetypeofthepigment–i.e.no mixturesorotherimpurities.Fig.6illustratesthisapproachand providestheclassificationandmajorityvoteresultsforselected exampleregions.Thecolourofthelabelassignedtoeachpixeland subsequentlyeachregionofpaintcorrespondstothe“classcolour” assignedtotheROIselectedfromthetrainingdataset.

TheapproachshowninFig.6wasappliedtoallregionsofthe painting,assigningoneclassfromthetrainingsettoeachofthem inturn.Atotalof10differentpaintswereusedtopreparethe pain-tingunderstudy,while9ofthemwereavailableinthe“gridcanvas” andwerethereforecontainedwithinourspectrallibrary.Fromthis perspective,usingtheFireflyIRImagedataitwaspossibleto auto-maticallyclassify67%ofpigments(6outof9)correctly,whilethe correctnessratiofordatafromtheRedEye1.7systemreached78% (7outof9)[41].

Fig.7showstheclassificationresultforbothimagingsystems whencomparedwiththegroundtruthdata.Itisclearthatinsome cases,iftheclassificationofdatafromonedeviceiswrong,itmaybe correctiftheotherisused,andviceversa.Forexample,oneofthe regionspaintedwithIvoryblackpaintisalwaysclassifiedcorrectly bybothsystems,butitexistsinadifferentregionforbothofthem (seeFig.7candd).Thisdemonstratesthecomplementarynatureof

thetwodevicesandalsoshowsthatthefullrangeofwavelengths consideredisusefulfordiscriminatingdifferentpaints.Whilstthe classificationaccuraciesdidnotreach100%,thepotentialof auto-matedclassificationtechniquesbasedonthehyperspectraldata hasbeenclearlydemonstrated.

5. Experimentalresults

Havingdemonstratedtheproposedsystem’sabilitytorecognise thepaintsbasedonspectralsignaturesina well-controlled lab-basedenvironment,thistechnologywasappliedtoassessother paintings,suspectedasor,byothermeans,alreadyidentifiedas forgeries.ThankstothecourtesyoftheBerlinPoliceitwaspossible toimageseveralpaintingsfromtheircollectionofforgedpaintings createdbytheinfamousWolfgangBeltracchi[38].

Duetothepracticalitiesintransportingtheequipmentto Ber-lin,onlytheRedEye1.7camerawasusedduringthisexperiment. Dataacquisition,pre-processingandanalysisstepsfollowedthe approachdescribedinSections2and3.Theonlydifferencewas atthestageofclassificationsincetheanalysedpaintingsdidnot featurethesimplegeometricpropertiesofthepaintingsdesigned

(10)

Fig.8. Illustrationoftwoforgedpaintingswithindicationofanalysisregionandclassificationresultofselectedcolours;(a)intensityimageofgridcanvaswithselectionof paintsusedfortrainingandthelegendidentifyingthecolourcodedpigments;(b)paintingdescribedasSevranckxwithwhite/creamcolourclassificationresult;(c)painting referredasHM501andwhitecolourclassification.

forthisworkasshowninprevioussections.Forthisreason,the applicationofbinarymaskswasnotafeasibleapproachtoallow individualregionsofthepaintingstobeprocessedseparately. Addi-tionally,thesepaintingscontainedavarietyofpigments,bothin pureandmixedform.Toovercomethesechallenges,the propo-sedtechniqueswereappliedtoselectedregionsoftwopaintings, chosenbasedonotherinstrumentaltestingtechniques(see Sec-tion2.2),which contained“Titaniumwhite”paintknowninthis casetobeanachronistic.Fig.8(bandc)showsthetwoselected paintingswherethegreenboxoutlinestheregionchosenforthe analysisalongsideamagnificationofthecolourrepresentationof theclassificationresult.

Fig.8(b)showsapaintingreferredtoas‘Sevranckx’inwhich thewhite/creamregionshavebeenidentifiedasTitaniumWhite inaseparateinvasivetests.Thedevelopedsoftwarewastrained onallofthewhite/creampaintsinthespectrallibrary construc-tedinthisstudyusingpigmentsinthe“gridcanvas”painting(see

Fig.8a)andincluded:Cremnitzwhite,Flakewhite,Baritewhite,Zinc white,FlemishwhiteandTitaniumwhite#3.Therighthandpart ofthefigureshowstheclassificationresultforasmallregionof thepainting.YellowinthisimagecorrespondstoTitaniumWhite inthespectrallibrary.Theblackregionscorrespondtopixelsin theimagewherenoclassificationhasbeenmadebythesystem asitdoesnotrecognisethespectraofthesepixelsasbelongingto anypaintinthespectrallibraryprovided.Furthermore,noother classificationlabels/coloursarevisibleintheimage.Thereasonfor thisisthattheclassifierdidnotidentifyanyotherpartofthis sec-tionofthepaintingascontaininganyoftheotherwhitepaints. Fromthisresult,itisclearthattwoofthewhite/creamregionshave beencorrectlyidentifiedandthiswasvalidatedbycomparingwith previouslycapturedground-truthinformation.Asimilarsituation canbeobservedinFig.8c)demonstratingapaintingreferredtoas ‘HM501’,wheresimilarclassificationstepssuccessfullyidentified TitaniumWhiteinagreementwiththeresultofthe aforementio-nedseparateanalysis.Itmaybenoticedthatseveralpixelswere misclassifiedasFlemishwhite(markedwithmagentainFig.8c).

However,thenumberofpixelsinerrorisverysmallandthisistobe expectedofanyautomatedclassificationscheme.Inboththesetest casescorrectidentificationofTitaniumWhitewasveryimportant fortheartscientists,asthiswastheanachronisticpigmentused inBeltracchiforgeriesthat,combinedwithotherevidence,suchas theuseofanotheranachronisticpaint,PhthalocyanineGreen,and averylimitedpoolofre-usedcanvasesandstretchers,confirmed thefalsityofthesepaintings[36].

6. Conclusions

Hyperspectral Imagingcombined with advanced signal pro-cessingtechniquesisavalidandpotenttechniquewhichcanbe usedasatooltosupporttheprocessofartworkauthenticationby identificationandclassificationofpigments.Inthispaperwehave presentedourworkindevelopingsoftwarebasedsignal proces-singalgorithmstofacilitatefeatureextractionandclassification of paintsfrom hyperspectralimagesofbespoke and fraudulent artworkcapturedintheinfraredspectralregion.Awiderangein theinfraredwasaccessedby useof a conventionalnear-IR HSI cameraaswellasanovellaserbasedimager,extendingthereadily accessiblebandwidthofthisportionofthenear-IRspectrumwith asubsetofthemid-IRregion.Collaborationofarthistoriansand signal processing specialistsmade it possible tocreate a small spectral library of pigments,based onthetailoredgrid canvas, andtoapplytheclassificationtechniquestodistinguishdifferent paints usedona speciallyprepared painting.Thankstounique accesstoanextremelyhigh-profiledataset,developedalgorithms alsoallowedustotest–forthefirsttimewiththistechnique–a setofknownBeltracchiforgedpaintings.Theclassificationresults showmoderate-highcorrectclassificationratealreadyonthefirst approachtothisproblem.Infact,upto78%ofpigmentsusedinthe bespoketestpaintingwerecorrectlyclassifiedusingtheRedEye 1.7HSIimagingsensor.Ultimately,andperhapsmostimportantly an anachronistic paint – Titanium White – wasidentified from realforgedpaintings usingoursysteminrealworldconditions

(11)

using our separately acquired spectral library of pigments as reference.

This initial study demonstrates the effectiveness of hyper-spectralimagingincombination withtheimageprocessingand classificationtechniques.However,itshouldbenotedthatsome pigmentswereundistinguishablefortheSVMclassifierandfurther analysiswouldberequiredtodeterminewhichpaintscanbe relia-blyidentifiedwiththismethod.Additionally,otherspectralbands, featureextractiontechniquesandclassificationalgorithmsshould beconsideredtofurtherexplore theeffectivenessand improve therobustness of thefinal system.A robust spectral library of pigmentsisalsocrucialforsuccessfulclassification.Various thi-cknessesofthepaint,differentbase(e.g.woodenboardorcanvas), backgroundpaintingsurfaceandpresenceofvarnishallmayaffect thespectralprofileandshouldbeconsideredforimplementable system.

Theapplication of scientific methodsto revealforgeries has recently gained significant public profile [4]. With increased demandforsuchanalysis,theintroductionofhyperspectral ima-gingcanenhancetheportfolioofavailabletechniquesandtoolsas arapidandnon-destructivemethodofartworkexamination.Even iftheresultsbasedsolelyonHSIdatacannotprovidefull confir-mationthata paintingisa genuineorforged,current practices ofarthistoriansemployacombinationofbackgroundknowledge andscientificdataintheauthenticationprocess.Byadding hyper-spectralimagingtotheiralreadysophisticatedtoolkit,theanalysis proceduremaybecomefaster,easierandmoreaccessibletoalarger portionofthemarketandtheservicesshouldbecomemore affor-dableasaresult.Ultimately,thetechniquesproposedhereincould limittheamountofdestructivetestsrequiredforfinalvalidation andwillincreasetheamountofanalysedartworkthusmakingthe wholeprocedureevenmorecosteffective.

Acknowledgments

This study was carried out within the framework of an IntelligentHyperspectralImaging(INHERIt) Projectfinanced by InnovateUK,TechnologyStrategyBoardincollaborationbetween DepartmentofElectronic &ElectricalEngineering,University of Strathclyde(Glasgow,UK),MSquaredLasers(Glasgow,UK), Fraun-hofer UK (Glasgow, UK)and Art Analysis& Research (London, UK).AuthorswishtothankBerlinLandeskriminalamtforgranting accesstotheirpaintingsandsupportingthisstudy.Thisprojectwas alsosupportedbytheEPSRCCentreforDoctoralTraininginApplied Photonics,funded bytheUK EngineeringandPhysical Sciences ResearchCouncil(grantEP/G037523/1)andbyanIndustrial Fel-lowshipfromtheRoyalCommissionfortheExhibitionof1851. AppendixA. Supplementarydata

Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/ j.culher.2017.01.013.

Références

[1]C.McAndrew,TEFAFArtMarketReport2015,TheEuropeanFineArtFoundation (TEFAF),Helvoirt,2015.

[2]C.McAndrew,TEFAFArtMarketReport2014:TheGlobalArtMarket,Witha FocusontheUSandChina,TheEuropeanFineArtFoundation(TEFAF),Helvoirt, 2014.

[3]S.Hardach,BBC–Culture–TheSurprisingSecretsofBustingArt Forge-ries, 2015, 19 October, Available at: http://www.bbc.com/culture/story/ 20151015-the-surprising-secrets-of-busting-art-forgeries (accessed 08.01.16).

[4]N.Eastaugh,J.Nadolny,SciencefortheArtMarket:ABuyer’sGuide.ArtBanc MarketIntelligence,No.7,2007,April,pp.33–37.

[5]A.Adriaens,COSTActionG8:non-destructiveanalysisandtestingofmuseum objects,in:BenefitsofNon-DestructiveAnalyticalTechniquesforConservation, Kalkara,2004.

[6]G.D.Smith,J.F.Hamm,D.A.Kushel,C.E.Rogge,What’swrongwiththispicture? Thetechnicalanalysisofaknownforgery,in:CollaborativeEndeavorsinthe ChemicalAnalysisofArtandCulturalHeritageMaterials,AmericanChemical Society,2012,pp.1–21.

[7]Y.Roggo,A.Edmond,P.Chalus,M.Ulmschneider,Infraredhyperspectral ima-gingforqualitativeanalysisofpharmaceuticalsolidforms,Anal.Chim.Acta 535(2005)79–87.

[8]B.D.Jadhav,P.M.Patil,Hyperspectralremotesensingforagricultural manage-ment:asurvey,Int.J.Comput.Appl.106(7)(2014)38–43.

[9]G.K.Naganathan,L.M.Grimes,J.Subbiah,C.R.Calkins,A.Samal,G.E.Meyer, Visible/near-infraredhyperspectralimagingforbeeftendernessprediction, Comput.Electron.Agric.64(2)(2008)225–233.

[10]T.Qiao,J.Ren,C.Craigie,J.Zabalza,C.Maltin,S.Marshall,Comparisonbetween nearinfraredspectroscopyandhyperspectralimaginginpredictingbeefeating quality,in:HyperspectralImagingandApplicationsConference(HSI2014), Coventry,2014.

[11]T.Qiao,J.Ren,C.Craigie,J.Zabalza,C.Maltin,S.Marshall,Quantitative predic-tionofbeefqualityusingvisibleandNIRspectroscopywithlargedatasamples underindustryconditions,J.Appl.Spectrosc.82(1)(2015).

[12]T.Qiao,J.Ren,J.Zabalza,S.Marshall,Predictionoflambeatingqualityusing hyperspectralimaging,in:OCM(OpticalCharacterizationofMaterials)2015, Karlsruhe,2015.

[13]S.Legrand,F.Vanmeert,G.V.d.Snickt,M.Alfeld,W.D.Nolf,J.Dik,K.Janssens, Examinationofhistoricalpaintingsbystate-of-the-arthyperspectralimaging methods:fromscanninginfra-redspectroscopytocomputedX-ray laminogra-phy,Herit.Sci.2(2014)13–24.

[14]C.Fischer,I.Kakoulli,Multispectralandhyperspectralimagingtechnologiesin conservation:currentresearchandpotentialapplications,Stud.Conserv.51 (Suppl.1)(2006)3–16.

[15]F.Rosi,C.Miliani,R.Braun,R.Harig,D.Sali,B.G.Brunetti,A.Sgamellotti, Nonin-vasiveanalysisofpaintingsbymid-infraredhyperspectralimaging,Angew. Chem.Int.Ed.52(20)(2013)5258–5261.

[16]K.A.Dooley,S.Lomax,J.G.Zeibel,C.Miliani,P.Ricciardi,A.Hoenigswald,M. Loew,J.K.Delaney,Mappingofeggyolkandanimalskingluepaintbindersin EarlyRenaissancepaintingsusingnearinfraredreflectanceimaging spectro-scopy,Analyst138(2013)4838–4848.

[17]J.K.Delaney,J.G.Zeibel,M.Thoury,R.Littleton,M.Palmer,K.M.Morales,E.R.d.L. Rie,A.Hoenigswald,VisibleandinfraredimagingspectroscopyofPicasso’s Har-lequinmusician:mappingandidentificationofartistmaterialsinsitu,Appl. Spectrosc.64(6)(2010)584–594.

[18]J.K.Delaney,M.Thoury,J.G.Zeibel,P.Ricciardi,K.M.Morales,K.A.Dooley, Visibleandinfraredimagingspectroscopyofpaintingsandimproved reflec-tography,Herit.Sci.4(6)(2016)1–10.

[19]H.Liang,Advancesinmultispectralandhyperspectralimagingforarchaeology andartconservation,Appl.Phys.A106(2)(2012)309–323.

[20]A.Casini,C.Cucci,M.Picollo,L.Stefani,T.Vitorino,Creationofahyper-spectral imagingreferencedatabaseofredlakepigments,COSCHe-Bulletin(2015).

[21]A.Casini,F.Lotti,M.Picollo,L.Stefani,E.Buzzegoli,Imagespectroscopy map-pingtechniquefornoninvasiveanalysisofpaintings,Stud.Conserv.44(1) (1999)39–48.

[22]M.Kubik,Hyperspectralimaging:anewtechniqueforthenon-invasivestudy ofartworks,physicaltechniquesinthestudyofart,Archaeol.Cult.Herit.2 (2007)199–259.

[23]F.Daniel,A.Mounier,J.Pérez-Arantegui,C.Pardos,N.Prieto-Taboada, S.F.-O.d.Vallejuelo,K.Castroc,Hyperspectralimagingappliedtotheanalysisof GoyapaintingsintheMuseumofZaragoza(Spain),Microchem.J.126(2016) 113–120.

[24]S.Baronti,A.Casini,F.Lotti,S.Porcinai,Multispectralimagingsystemforthe mappingofpigmentsinworksofartbyuseofprincipal-componentanalysis, Appl.Opt.37(8)(1998)1299–1309.

[25]C.Cucci,J.K.Delaney,M.Picollo,Reflectancehyperspectralimagingfor investi-gationofworksofart:oldmasterpaintingsandilluminatedmanuscripts,Acc. Chem.Res.49(10)(2016)2070–2079.

[26]M.Attas,E.Cloutis,C.Collins,D.Goltz,C.Majzels,J.R.Mansfield,H.H.Mantsch, Near-infraredspectroscopicimaginginartconservation:investigationof dra-wingconstituents,J.Cult.Herit.4(2003)127–136.

[27]K.Melessanaki,V.Papadakis,C.Balas,D.Anglos,Laserinducedbreakdown spectroscopyandhyper-spectralimaginganalysisofpigmentsonan illumi-natedmanuscript,Spectrochim.ActaPartB56(2001)2337–2346.

[28]C. Balas,V.Papadakis,N.Papadakis,A.Papadakis,E.Vazgiouraki,G. The-melis, Anovel hyper-spectral imaging apparatusforthe non-destructive analysis of objectsofartistic and historicvalue, J. Cult. Herit.4 (2003) 330–337.

[29]M.E.Klein,B.J.Aalderink,R.Padoan,G.d.Bruin,T.A.Steemers,Quantitative hyperspectralreflectanceimaging,Sensors8(2008)5576–5618.

[30]C.S.Silva,M.F.Pimentel,R.S.Honorato,C.Pasquini,J.M.Prats-Montalban,A. Ferrere,Nearinfraredhyperspectralimagingforforensicanalysisofdocument forgery,Analyst139(2014)5176–5184.

[31]Z.Luo,P.Shafait,A.Mian,Localizedforgerydetectioninhyperspectral docu-mentimages,in:13thInternationalConferenceonDocumentAnalysisand Recognition(ICDAR),Nancy,France,2015.

[32]Z.Khan,F.Shafait,A.Mian,Towardsautomatedhyperspectraldocumentimage analysis,in:Proceedingsofthe2ndInternationalWorkshoponAutomated

(12)

[40]M.Sontheimer,ACheerfulPrisoner:ArtForgerAllSmilesAfterGuiltyPlea SealsDeal,DerSpiegel,2011,27October,http://www.spiegel.de/international/ germany/0,1518,794454,00.html(accessed18.03.16).

[41]A.Polak,T.Kelman,P.Murray,S.Marshall,D.Stothard,N.Eastaughc,F. Eas-taugh,Useofinfraredhyperspectralimagingasanaidforpaintidentification J.SpectralImaging5(1)(2016)1–10.

[42]N.Eastaugh,V.Walsh,T.Chaplin,R.Siddall,ThePigmentCompendium,Elsevier, 2008.

ClassifierSystems:8thInternationalWorkshop,MCS2009,Proceedings, Reyk-javik,Iceland,June10–12,2009.

[50]MathWorks, Support Vector Machines (SVM), MathWorks, Available at:

http://uk.mathworks.com/help/stats/support-vector-machines-svm.html

(accessed19.01.16).

[51]C.-C.Chang,C.-J.Lin,LIBSVM:alibraryforsupportvectormachinesACMTrans. Intell.Syst.Technol.2(3)(2011),27:1–27:27.