Inference of a compact representation of sensor fingerprint for source camera identification

Contents lists available at ScienceDirect

Pattern

Recognition

journal homepage: www.elsevier.com/locate/patcog

Inference

of

a

compact

representation

of

sensor

fingerprint

for

source

camera

identification

Ruizhe

Li

a

_,

_Chang-Tsun

_Li

a ,b ,∗

_,

_Yu

_Guan

c

a_{DepartmentofComputerScience,UniversityofWarwick,UK}

b_{SchoolofComputingandMathematics,CharlesSturtUniversity,Australia} c_{OpenLab,SchoolofComputing,NewcastleUniversity,UK}

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received30November2016 Revised28August2017 Accepted18September2017 Availableonline27September2017

Keywords: Imageforensics

Sourcecameraidentification(SCI) Sensorpatternnoise(SPN) PCAde-noising

a

b

s

t

r

a

c

t

Sensor pattern noise (SPN) is an inherent fingerprint of imaging devices, which provides an effective way for source camera identification (SCI). Although SPNs extracted from large image blocks usually yield high identification accuracy, their high dimensionality would incur a high computational cost in the matching stage, consequently hindering many applications that require efficient camera matchings. In this work, we employ and evaluate the concept of principal component analysis (PCA) de-noising in SCI tasks. Based on this concept, we present a framework that formulates a compact SPN representation. To enhance the de-noising effect, we introduce a training set construction procedure that minimizes the impact of var- ious interfering artifacts, which is especially useful in some challenging cases, e.g., when only textured reference images are available. To further boost the SCI performance, a novel approach based on linear discriminant analysis (LDA) is adopted to extract more discriminant SPN features. To evaluate our meth- ods, extensive experiments are conducted on the Dresden image database. The results indicate that the proposed framework can serve as an effective post-processing procedure, which not only boosts the per- formance, but also greatly reduces the computational cost in the matching phase.

© 2017 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license. ( http://creativecommons.org/licenses/by/4.0/)

1. Introduction

Nowadays, the use of digital images or videosas evidence in thefightagainstphysical crimeandcybercrime isanorm, which makes multimedia forensics crucial. Typically, multimedia foren-sicsincludessourcecameraverificationandidentification, source-orientedimagesclassification,integrity verification,forgery detec-tion,authentication,etc.Sourcecameraidentification,asan impor-tantbranchofmultimediaforensics,isaboutansweringthe ques-tion:Whichoneof themanycamerashastaken theimagein

ques-tion?Thisisactually ataskofmatchingthecamerafingerprintof

animageinquestiontoasetofreferencefingerprints,each repre-sentingadifferentcamera.Thesizeofthereferencefingerprintset canbeintheorderofmillions.Howtodealwithsuchataskmore accuratelyandefficientlyisthefocusofthispaper.

In order to link digital images to the source cameras, many techniques have been proposed in the last two decades. These techniquescanbe broadlydividedintothreecategories.The

sim-∗ _{Correspondingauthor.}

E-mailaddresses:[email protected](R.Li),[email protected](C.-T.Li), [email protected](Y.Guan).

plestwayistousedigital images’metadatathatcontainsthe in-formationofthesourcecamera [1] .

However, due to the wide prevalence and great user-friendlinessofmultimediaprocessingtoolsnowadays(e.g.,Adobe PhotoshopandIrfanView), metadatacanbe easily changedor re-movedbylaymen.Therefore,metadataisnolongerregardedas re-liableforauthenticationpurposes.Anotherpossiblewayistouse thedigital watermark,whichis asignatureembeddedinthe im-age by a certain type ofcameras [2] . This techniqueis usefulin the casesofproving ownershipof copyright. Yet itis only appli-cabletothecamerasthathavewatermarkingmechanism [2] .The thirdcategoryoftechniquesrelyontheintrinsiccharacteristicsof digitalcamerasleftinthecapturedimages.Manytracesleftinthe contentbyvarioushardwareandsoftwarecomponentsinthe im-ageacquisition pipeline canbe exploitedto link theimage to its sourcecamera.Goodexamplesaresensorpatternnoise(SPN) [3– 8] ,lens aberrations [9] ,colorfilterarray (CFA) interpolation arte-facts [10] ,JPEGcompression [1] ,andthecombinationofseveral in-trinsiccharacteristics [11] .Amongthesemodalities,SPNhasbeen provedtobethemosteffectivecamerafingerprintasitiscapable ofdifferentiatingindividualcamerasofthesamemodel.

Sensor pattern noise is produced by the imaging sensor and primarily causedby the manufacturing imperfectionsand the

in-https://doi.org/10.1016/j.patcog.2017.09.027

homogeneity of silicon wafers. It is essentially the slight varia-tions inthe intensity ofindividual pixels. Forinstance, even ifa sensor takesan imageof anevenly litscene,theresulting image willstillexhibitslightchangesinintensitybetweenindividual pix-els [3] .Everyimage taken by thesame sensorwouldexhibit the same SPNpattern, while two sensors,even madefrom thesame siliconwafer,wouldexhibituncorrelatedpatterns [3] .

ThedimensionalityofSPNisaslargeasthatoftheoriginal im-age. As a result, not only each SPN needs a fairly large amount ofspaceforstorage,butmemoryaccesswouldalsotake consider-ableamountoftime.Moreover,SPNmatchinginvolvesvector op-erations and the complexity is proportional to the size of SPNs. Thus,withalargenumberofreferenceSPNinthedatabasetobe matched,thecomplexityofmatchingprocesswouldbecomea crit-icalconcern.

Inordertoaddressthehighcomplexityissue,manyefforts [12– 18] havebeenmadeinrecentyears.In [12] ,Bayrametal. embe-dedreferenceSPNsinabinarysearchtree,wheretheleaf/internal node represents a reference/composite SPN.Based on this struc-ture, thetotal numberofSPNmatchings tobe performedis sub-stantially reduced. However, errors tend to increase significantly whena largenumberofreferenceSPNsare storedinasingle bi-narytree.Ontheotherhand,moremethodsreducethe computa-tionalcomplexityby compressingtheSPN.In [13,14] ,theauthors introduced a SPN digest technique for dimensionality reduction, whichpreservesthelargestelementsandtheircorresponding loca-tions. In [15] ,Bayrametal. binarizedSPN,whichconsiderably re-ducesthestoragerequirementsandspeedsuploadingofSPNinto thememory.However,thebinarizationprocessinevitablydegrades thematchingaccuracyduetoinformationloss.In [16,17] ,Valsesia etal.reducedthedimensionalityofSPNusingrandomprojection. However, since the subspace is randomly selected, the obtained representationisunlikelyto beoptimalandtends tocompromise thematchingaccuracy.

To alleviate the common limitation(i.e., reduced accuracy) of the afore-mentioned SPN compression methods [13–17] , in our previouswork [19,20] ,wepresentedafeatureextractionalgorithm basedontheconceptofPCAde-noising [21,22] ,andpromising re-sults were achieved on a smalldataset. However, this methodis based ontheassumption thatthe trainingset iswell representa-tiveofthepopulationsothataneffectiveSPNfeatureextractorcan be learned.Unfortunately, the noise residuals in the trainingset canbecontaminatedby manysourcesofinterference,makingthe trainingset lessrepresentative.Tolearn arobust SPNfeature ex-tractorfromthenoisytrainingdata,inthiswork,we further pro-poseatrainingsetconstructionprocedureandprovideits theoreti-calbasis.WealsoprovidemoredetaileddiscussionoftheSPN fea-tureextractorsandtreatitasageneralpost-processingframework onotherSPNmethods.Itisevaluatedintermofeffectivenessand efficiencyonamuchlargerdataset.Wealsotestthisframeworkon some challengingcases,e.g., all the reference SPNs are extracted fromimageswithsignificantscenedetails(aformofdistortionto theSPN),whicharescenariosbarelyconsideredbypreviousworks. Therestofthispaperisorganizedasfollows. Section 2 provides a briefreviewon thethreemain stepsofthe SPN-basedSCI sys-tem. In Section 3 ,we presentthe proposed trainingdataset con-structionprocedure andthe feature extractionmethodin details. In Section 4 , the proposed source camera identification method issummarized,whichis thenfollowedby extensiveexperimental evaluationsin Section 5 . Section 6 concludesthework.Notethat, inthismanuscript,weuseboldupper-caseletterstorepresent ma-trices,andboldlower-caseletterstodenotevectors.

2. Background

Inorder to decide whethera query image istaken by one of thecamerasina largedataset,threemainstepsare required,i.e., SPNextraction,referenceSPNestimationandSPNmatching.Inthis section,techniquesforthesethreestepsarebrieflyreviewed.

2.1. SPNextraction

The most important step of the SPN-based SCI framework is to extract the SPNs fromdigital images.In [4] ,Chen et al. mod-eledtheoutputofimagingsensorIandexplainedthegeneralidea abouthowtoextractSPN,suchas

I=

(

1+K

)

I(0)₊

_=I(0)_+I(0)_K+

₍₁₎

In Eq. (1) ,I(0) _{isthe noiselesssensoroutput andI}(0)_{K represents} thediscriminativepartofSPN,i.e., PRNUnoise,which isa multi-plicativenoiseandthesignal ofourinterest.The matrixK isthe PRNU multiplicativefactor,where all theelementsin itare typi-callycloseto0.

isacombinationofrandomnoise,suchasshot noise, read-outnoise, andquantization noise. In orderto extract thesignal ofinterestI(0)_{K fromtheobservationI,thehostsignal}

I(0) _{should be removed. Generally, the noiselessimage I}(0) _is un-known,butwecanestimateitbyde-noisingtheobservationI,i.e.,

ˆI(0)_=F

₍

_I

₎

_{,whereFindicatesade-noisingalgorithmandˆI}(0)_isan estimationofthenoiselessimage I(0)_{.Then, thesignal ofinterest} can be roughly extracted by subtracting the estimationˆI(0) _from theobservationI,suchas

X=I−F

(

I

)

=I−ˆI(0)

=I(0)+I(0)K+

−ˆ_I(0)

=IK+I(0)_−ˆI(0)₊

₍

_I(0)_−I

₎

_K+

=IK+

(2)

whereX isthenoiseresidualwherethetrueSPNispresent,

is thesum of

andtwo additionalnoise terms introduced by the de-noisingfilter.

From Eq. (2) ,onecanseethatthebetterade-noisingalgorithm Fis,thecloserthede-noisedversionˆI(0)_{istothenoiselessimage}

I(0)_{,andthusthelessnoisewouldbeintroducedbythede-noising} filterandleftinthefinaloutput X.Therefore,theperformanceof aSPN extractoris primarily determinedby the choice ofthe de-noisingalgorithmF.In [3] ,Lukasetal.proposed totransformthe noisyimageIintowavelettransformdomainandapplytheMihcak filter [23] toextracttheSPNcomponentsfromthehighfrequency waveletcoefficients of I. In [24] , Chierchiaet al. proposed to re-placetheMihcakfilterwithamorerecenttechnique, namelythe sparse 3D transform-domain collaborative filtering [25] . In [26] , Kang et al. proposed a SPN predictor based on context adaptive interpolation(PCAI),whichis toapplythe contextadaptive inter-polator [27] asthe de-noisingfunction F topredict the noiseless imageI(0)_{andextractSPNinthespatialdomain.}

Alsodemonstratedin Eq. (2) isthefactthatthenoiseresidualX

2.2.ReferenceSPNestimation

This step aims at estimating the referenceSPN for a camera. Typically,thereferenceSPN,R,foracameraisestimatedby aver-agingN (e.g.,N≥20)noiseresiduals extractedfrom flat-field/low-variationimages(e.g.,blueskyimages)takenbythatcamera,such as

R=

N

i=1

Xi/N (3)

Random noise presented in different images are different,while thetrueSPNcomponentswouldbethesameaslongasthese im-agesare takenby the samecamera. Therefore,the randomnoise componentscanbe averagedoutinRwhilethetrueSPN compo-nentsare accumulated.In [4] , Chen et al. proposed a maximum likelihoodestimation(MLE)methodtoestimatethereferenceSPN. Theyalso proposedtwo enhancingoperations, namelyzero-mean (ZM)andWiener filtering(WF) in the discrete Fouriertransform (DFT)domain,to remove theartifactscaused by camera process-ingoperationsfromthereferenceSPN.In [30] ,Lin andLiargued thatthetrueSPNisunlikelytobeperiodicandshouldhaveaflat spectrum. Therefore, they proposed another reference enhancing method,namelyspectrum equalizationalgorithm (SEA),todetect andsuppressthepeaksappearingintheDFT spectrumofthe ref-erenceSPNsoastoremovetheperiodicartifacts.

2.3.SPNmatching

Once both query SPN and reference SPN are obtained, the matchingstepcanbe performed. Sucha taskcanbe treatedasa binaryhypothesistestasfollows

H0:X=Ri(thequeryimageisnottaken bytheithcamera), H1:X=Ri(thequeryimageistaken bytheithcamera).

Here a correlation-based detector is used to make the decision between H0 and H1 by comparing the correlation

ρ

(X, Ri) to a

pre-calculatedthreshold

τ

.ThedetectoracceptsH1when

ρ

≥

τ

or H0 when

ρ

<

τ

.Thenormalizedcross-correlation (NCC)isusually usedto measure the similaritybetweenthe query noiseresidual

X∈RM×M_{andthereferenceSPNR∈R}M×M_{,whichisdefinedas}

ρ

(

X,R

)

=

M

i=1

M

j=1

X[i,j]−X

R[i,j]−R

X−X

·

R−R

, (4)

whereX andRarethemeanvalueofXandR,and

·

istheL2 norm.Givenan upperboundon thefalse positiverate(FPR), the threshold

τ

for the detector can be calculated via the Neyman– Pearsonapproach [31] .In [32] ,GoljanpointedoutthatNCCis sen-sitive to the influence ofperiodic noise, and proposed the peak-to-correlationenergy(PCE)ratioasareplacementtomeasurethe similaritybetween two SPNs. More recently, Kang et al. [6] pro-posedanothermeasurement,namelycorrelationovercircular cor-relationnorm,toreducetheFPRofaSCIsystem.

Theaforementionedmethodscanbecombinedtofurtherboost performance gains. For example, forensic investigators can apply theMihcakfiltertoextractthenoiseresidualsfrombothqueryand referenceimages,andenhance thequerynoise residualswithLi’s enhancingmodels [5] andimprovethereferenceSPNswitheither theZM+WF operations [4] or theSEA algorithm [30] , andfinally apply NCCorPCE asthesimilarity measurementfor SPN match-ing.Moreover,inmanyapplications,suchassource-orientedimage clusteringandSCIinalarge-scalereferenceSPNdatabases,taking the full-sizedimage into account is not computationally feasible anda blocksmaller thanthefull-sizedimage isused.Dueto the vignettingeffectontheperipherals ofimages [33] ,itissuggested

that such a block is better cropped from the center of the full-sizedimage.Thenoiseresiduals extractedfromlargerblocks usu-ally yield higher identification accuracy, but they also have high dimensionality. The complexity of matching a query image with the camerain thedatabase isO(mc),where m isthe dimension-alityofeachnoiseresidualandcisthenumberofcamerasinthe database.Consideringthefactthattheremaybetensofthousands of reference SPNs (each representing a camera) in the database, matching the high dimensional noise residuals may incur exces-sive computational costs. Toaddress this problem, we propose a newapproachinthenextsection.

3. ProposedSPNfeatureextractionandenhancement

Generally speaking, high-dimensional SPNs not only incur a high computational costs but also tend to contain more re-dundancy and interfering components. For simplicity, we write Eq. (2) asthesumofthetrueSPNandunwantednoise,i.e.,

X=X(0)+

, (5)

whereX(0) _{isthetrueSPN,and}

_{representsan additivemixture} of unwanted interferences, which may include scene details and the artifacts introduced by color interpolation, JPEG compression and other camera processing operations [4] . The former can be scene-specific, while the latter can be shared among camerasof thesamemodelorsensordesign.Therefore theyare non-unique, lessdiscriminant andredundant. Inorder to improvethe perfor-manceofSCIsystems,oneintuitivewayistosuppressthese arte-facts

.

PCA [34] isawell-knownunsupervisedlearningmethod,which minimizes the reconstruction error using a linear transforma-tion, and can be used to learn compact representation of high-dimension data. This method has been widely used for the pur-poseof de-noising [21,22] , dimensionality reduction [35] , feature extraction [36] ,etc. Comparedwithdata-independent dimension-alityreduction methods,suchasrandomprojection,the PCA pro-jectionmatrixislearnedbasedonatrainingdata,anditgenerally has higher performance in classification tasks [37] . In this work, weattempttofindaPCAtransformeddomain,wherethetrueSPN iswellrepresented.Ideally,byprojectingtheextractednoise resid-ualsontothisdomain,asmallsetofcoefficientsthatcontainmost oftherepresentativeinformationofthetrueSPNcanbeextracted.

3.1. Trainingsetconstruction

Inorder toidentify such a transformeddomain, a representa-tivetrainingsetneedstobeestablishedinadvance.PCAistofind anoptimaltransformeddomainthatbetterrepresentstheprimary signalsharedamongthetrainingsamples.SoifSPNappearsasthe mostrepresentativesignalamongthetrainingsamples,itwouldbe bettertorepresentitintheobtaineddomain.However,some con-tamination(e.g.,scenedetails)canbemoredominantthanSPNin thenoiseresidual(asshownin Fig. 1 (b)).Withoutremovingthese strongcontaminationsfromthetrainingset,theobtaineddomain ismorelikelytorepresentthesenoisycomponentsratherthanthe trueSPN.Toavoidthissituation,weproposethefollowing strate-giestominimizetheimpactoftheunwantednoiseinthetraining set:

1. Trainingsample selection:To build thetrainingset, ifwe have

Fig.1. (a)AnimagetakenbyOlympus_mju_1050SW.(b)Thenoiseresidualextractedfrom(a)usingtheMihcakde-noisingfilter.(c)Thereconstructedversionof(b).Note theintensityof(b)and(c)hasbeendownscaled5timesandupscaled2times,respectively,forvisualizationpurpose.

thecamerastotakeflat-fieldimages.Insteadonlyimageswith varyingscenedetailstakenbythosecamerasareavailable(e.g., fromsomeone’sFacebookaccount).Inthiscase,ourstrategyis tosuppresstheimpactofscenedetailsthroughaveraging. Con-sideringthe fact that scenedetails presentedin different im-agesare normallydifferent,we cangeneratea smoother sam-plebyaveragingseveralnoiseresidualsoftheimagestakenby the samecamera. By repeating thisprocess several times,we canfinally generatea setoftrainingsamples,whicharemore representative.

We also model the afore-mentioned contamination-removal processbasedon Eq. (5) .Inthiscontext,

θ

representsthescene details,while Xˆ isthe sumofSPNandsomenon-unique arti-facts(e.g.,CFApatternandJPEGblockyartifacts),whichwillnot besuppressedbyaveraginginthisstage.Giventhat,fora cam-era with N referenceimages, each pixel’s mean and variance in the reference SPN can be expressed as

μ

X=Xˆ+N1

N i

θ

i,

and

σ

2

X=E[

(

θ

i−N1 N

i

θ

i

)

2], i=1,2,...,N, respectively. For

acamera,ifwerepeat averagingtheSPNsofa randomsubset ofTout oftheNreferenceimagesforLtimes,then according to Eq. (5) wewillhave

X_l=_Xˆ₊1 T

T

t

θ

lt, l=1,2,...,L. (6)

Thenewmeanandvarianceforeachpixelcanbeexpressedas follows

μ

X=Xˆ+

1 LT L l T t

θ

lt, (7)

σ

2

X=

1 L L l

1 T T t

θ

lt− 1 LT L l T t

θ

lt

2 . (8)

In Eq. (8) , the term _LT1 L_lT_t

θ

lt can be approximated asthe

meanof thescene details

θ

¯= 1

N N

i

θ

i whenthe product ofT

andLislarge.Forsimplicity,inthisworkwesetL=Nto gen-erateasmanysamplesastheoriginaldata.In thiscase,ifwe set T→N, the term 1

T T

t

θ

lt of Eq. (8) also converges to the

meanofthescene details

θ

¯₌ 1

N N

i

θ

i, whichmakes

σ

X2 →0,

hencesuppressingtheinterferenceofscenedetails.

2. Trainingsampleenhancement:Inadditiontoscenedetails,some

non-uniqueartifactssuchasCFApatternsandJPEGblocky arti-factsmayalsolead tounsatisfactorytraining.Sincethese arti-factsin the images takenby the camerasof the samemodel or brand are similar (with small variation), they would sur-vive theaveraging operation. Nevertheless,aswe haveshown in [30] ,theseartifactscausepeaksintheDFTmagnitude spec-trum,whiletheSPNappearsasaflatspectrumwithoutsalient peaks.Therefore,bysuppressingthe peakspresentin theDFT spectrum,theseartifactscanbeeffectivelysuppressedandthe

qualityofthetrueSPNinthenoiseresidualcanbethereby en-hanced.

Assume there are n referenceimages

{{

Ii j

}

iN=1

}

cj=1 taken by c cameras

{

Cj

}

cj=1,each responsible forN imagessuch that n=cN. Accordingtothetwo afore-mentionedstrategiesfortraining sam-pleselection and enhancement,we can summarize theproposed trainingsetconstructionasfollows:

(1)Extract the2D noiseresiduals

{{

Xi j

}

iN=1

}

cj=1 fromtheblocksof W_× Wpixelscropped fromthecenterofthenreference im-ages.

(2)For each camera Cj, randomly select T noise residuals from

{

Xi j

}

Ni=1(belong to cameraCj) for averaging.

(3)Detectandsuppressthepeaksoftheaveragednoiseresidualin the DFTmagnitudespecturmwithSEA [30] .Thenconcatenate the 2D output into a column vector as a training samplexij.

Notethatwe useXij torepresent2Dnoise residualsandxij to

representtheir1-dversion.

(4)RepeattheprocessinSteps(2)and(3)Ltimesforeachcamera toformthetrainingset

{{

xi j

}

Li=1

}

cj=1∈Rm,wherem=W×W. InStep(2),werandomlyselectTimagesfromeachcamerafor averaging.As discussedabove,it ispreferable tosetT toa larger valuesoastobetterattenuatetheimpactofscenedetailsand ran-domnoise. However, sincethe CFApatternandJPEGblocky arti-factsaresharedamongtheimagestakenbythecamera,the aver-agingoperationwouldalsoinevitablyenhance thesetwoartifacts ineach trainingsample.However, thepeakscausedbythese arti-factsaremoredistinctintheDFTspectrumandtheycanbemore easilyandaccuratelydetected.Giventhat,settingTtoalargevalue wouldalsohelpSEAtoachieve amoreaccuratepeakdetectionin Step(3),whichwouldconsequentlyincreasetheeffectof enhance-ment.Moredetails abouthow thesettingofT affectsthe perfor-manceisdiscussedin Section 5.2 .

3.2.SPNfeatureextractionthroughPCA

PCA is performed to seek a set of orthonormal eigenvectors

{

vk

}

mk=1 andtheir associated eigenvalues

{

λ

k

}

mk=1of the covariance matrixSgivenby

S=1

n n

i=1

(

xi−¯x

)(

xi−¯x

)

T=AAT, (9)

whereA=_√ 1

n[x1−¯x,...,xn−¯x]∈R

m×n _{and ¯xistheglobalmean}

estimatedby ¯x= 1

n n i=1

xi.The eigenvectors vk andeigenvalues

λ

k

areobtainedbysolving theeigenvaluedecompositionSv_k=

λ

kvk,

inwhich k=1,...,m. Giventhat thedimensionality ofSPNs can be extremely high(e.g., m>107_{), directly decomposingS∈R}m×m

weapply afastmethod [38] insteadofcomputingthese eigenvec-torswhenmn.

AssumingvkistheuniteigenvectorofATA∈Rn×n with

eigen-value

λ

_k,wecanobtainAT_Av

k=

λ

kvk.Bymultiplyingbothsides

byA,weget

AAT

₍

_Av

k

)

=

λ

k

(

Avk

)

, (10)

whereAvk are the eigenvectorsof AAT=S witheigenvalues

λ

k.

Thus, insteadof decomposingmatrix S directly, we can calculate theeigenvectorsvkbydecomposingasmallermatrixATA∈Rn×n.

Thenvk canbeobtainedviavk=Avk.Computingeigenvectorsin

sucha mannerincursa complexityofO(n3_{). Consideringthefact} that the number of training samples tends to be much smaller than the size of SPNs (i.e., nm), thus computing eigenvectors in such a manner would be more efficient than the traditional one. The obtained

{

v_k

}

n

k=1 are normalized and sorted in the de-scendingorderaccordingtotheir correspondingeigenvalues

λ

1

λ

2· · ·

λ

n. Subsequently, a transformed domain can be built as

Mpca=[v1,...,vn]∈Rm×n.After that,we canapply Mpca to noise

residualx(definedin Eq. (5)) through

y=MTpcax=MTpca

(

x(0)+

)

=MT

pcax(0)+MTpca

=y(0)+

y, (11)

wherey(0) _and

yare thetransformed versionsofthe SPNterm

andthenoiseterm, respectively.Now theproblemisrecastas es-timatingy(0)_{fromthenoisyy.Generallyspeaking,inaPCA} trans-formedvector (i.e., y), most energyof theprimary signal among thetrainingsetwouldconcentrateonthefirstseveralelementsof

y,while theenergyof thenoise wouldbe distributedinymuch more evenly. Therefore, only retaining the first several elements ofy while discarding the rest would preserve the energy of the signalofinterestandsuppresstheenergyofthenoise.Following thisconcept,theeigenvectorswiththedlargesteigenvaluesare se-lectedtoforman SPNfeatureextractorMd

pca=[v1,...,vd]∈Rm×d,

withdsatisfying

d=min

{

d

|

d i=1

λ

i/

n

i=1

λ

i>98%

}

. (12)

WiththisSPNfeatureextractorMd

pca,wecanobtainanew

fea-turewithmuchlowerdimensionalityby

yd₌

₍

_Md

pca

)

Tx=

(

Mdpca

)

Tx(0)+

(

Mdpca

)

T

=y(0)d+

d

y, (13)

whereyd_{isthecompactrepresentationofy.Withthefeature}

vec-toryd _{andSPNfeature extractorM}d

pca,it isreasonabletoassume

thatwecan obtainareconstructed SPNinthe spatialdomainvia theinversePCAtransformasfollow

x=

(

Mdpca

)

yd, (14)

wherex isan approximationoftheoriginal x.Ifourassumption iscorrect, noise

y shouldbe suppressed by thePCA-based SPN

feature extractor. As a consequence, the reconstructed x should contain less noise and have a higher signal-to-noise ratio (SNR) thantheoriginalnoiseresidualx.Tovalidateourassumption,we demonstratethebehaviorofourSPNfeatureextractorwitha sim-pleexample. As shownin Fig. 1 (b),the scenedetails in Fig. 1 (a) propagatesthroughtheWienerfilterintothenoiseresidual.After performingtheproposedSPNfeatureextractionandinvertingthe PCAtransformation,theartifactscausedby thescenedetails have beensignificantlysuppressed inthereconstructed SPN,asshown in Fig. 1 (c).Theeffectoftheproposedmethodcanalsobe quanti-tativelyevaluatedbycomparingtheSignal-to-NoiseRatio(SNR)of thetrueSPN, tothe contaminated version (Fig. 1 (b))and (to the reconstructedSPN(Fig. 1 (c)).First, thetrue SPNx(0) _isestimated by averaging 50 noise residuals extracted from blue sky images.

Accordingto Eq. (5) ,thenoise

inthenoise residual(Fig. 1 (b)) and (the reconstructed SPN (Fig. 1 (c)) can be estimated by sub-tracting the true SPN x(0) _{from the observed data, respectively.} Then, theSNR can be calculated accordingto 10log₁₀v_var_ar(x₍(0)₎). As expected, the reconstructed SPN hasa much higher averageSNR (4.3dB)than theoriginal noise residual(−15.5dB),whichfurther validatesourassumption.

3.3. SPNfeatureenhancementthroughLDA

In the task of SCI, the source cameras of the images in the databaseare usuallyknown, whichmeansthe classlabelof each imageisknown.Ifthisisthecase,bytakingadvantageofthisprior knowledge,wecanfurtherextractamorediscriminantfeatureby usingasupervisedlearningmethod,i.e.,lineardiscriminant analy-sis(LDA) [39] .ThepurposeofusingLDAinthisworkistobuildan enhancerM_lda toenhance theSPNfeatureextractorMpcasoasto

extractmorecompactrepresentationfromtheoriginalnoise resid-ual x. Thisenhancer can be obtainedby maximizing the ratioof thedeterminantofthebetween-classscattermatrixS_b tothe de-terminantofthewithin-classscattermatrixSw

Mlda=ˆJ=argmax

J

JT_S

bJ JT_SwJ

, (15)

where Sw is defined as Sw=cj=1

L

i=1

(

yi−

μ

j

)(

yi−

μ

j

)

T. yi

is the ith sample of class j,

μ

j is the mean of class j, c is

the number of classes, and L is the number of samples in each class. The between-class scatter matrix S_b is defined as

Sb= 1c c

j=1

(

μ

j−

μ

)(

μ

j−

μ

)

T,where

μ

representsthemeansof

allclasses.WiththeobtainedenhancerMlda,a

(

c−1

)

-dimensional

vectorzcanbeobtainedsuchthat

z=MT lday

d_=MT lda[

(

M

d pca

)

Tx]

=

(

Md

pcaMlda

)

Tx=MTex, (16)

wherezisanothercompactversionofthenoiseresidualx;Me=

Md

pcaMldaistherefined SPNextractorwhichisusedforextracting

zdirectlyfromtheoriginalx.Inmostcases,c₋1wouldbemuch smallerthandsothatzwouldbemorecompactthanyd_.

4. Sourcecameraidentification

Thecamera identificationprocess usingtheproposed compact featuresaresummarizedin Algorithms 1 and 2 .Wecallthefeature vectors yd _{and z produced by Algorithms 1 and 2 as“PCA-SPN”}

and“LDA-SPN”,respectively,throughouttherestofthispaper.As mentionedearlier,thecomplexityofcalculatingcorrelationis pro-portionaltothefeaturesize.ConsideringthatthesizeofPCA-SPN (yd_∈Rd_{) andLDA-SPN(z∈R}c−1_{) are bothmuch lower than that} oftheoriginalnoiseresidual(x∈Rm_{),usingeithery}d_orzinplace

oftheoriginal xwouldleadtoapproximatelya m/dorm_/

(

c₋1

)

timesgaininspeedinthematchingphase.

In addition, givena requiredfalse positive rate, the detection thresholds

τ

y and

τ

z for thePCA-SPN (yd) andLDA-SPN (z) can

be determinedbyusingthe Neymann–Pearsoncriterion approach [31] .

5. Experiments

Algorithm1 SPNFeatureExtractionthroughPCA

Symbols:

m:TheoriginaldimensionalityoftheSPN; L:Thenumberoftrainingsamplespercamera; c:Thenumberofcameras;

n:Thenumberoftotaltrainingsamples(n=L×c).

1. Perform the trainingset construction procedure (referto Sec-tion 3.1) to generate a set of training samples

{{

xi j

}

Li=1

}

cj=1∈

Rm_.

2. Usethe fastmethodmentionedin Section3.2toestimate the eigenvectors

{

vk

}

nk=1andtheeigenvalues

{

λ

k

}

nk=1.

3. Usethedeigenvectorscorrespondingtothedlargest eigenval-uestobuildtheSPNfeatureextractorMd

pca=[v1,...,vd]∈Rm×d.

4. Extract PCA-SPNs from all the training samples

{{

x_{i j}

}

L i=1

}

cj=1 and the query noise residual xq using yd_{i j}=

(

Mdpca

)

Txi j, ydq=

(

Md

pca

)

Txq.

5. Estimate the reference PCA-SPN for camera Cj using ydC_j=

1

L L

i=1ydi j.

6. Calculate the NCC value

ρ

(

yd

q,ydC_j

)

between the queryydq and

eachreferenceyd

C_j usingEq.(4).

7. AcceptH0if

ρ

(

ydq,yCdj

)

<

τ

y,otherwiseacceptH1(H0andH1are describedinSection2.3).

Algorithm2 SPNFeatureEnhancementthroughLDA 1.-4.SameasStep1-4ofAlgorithm1.

5. Use thePCA-SPNs

{{

yd i j

}

L

i=1

}

cj=1 to estimate thetransformation matrixMldausingEq.(15).

6. ExtractLDA-SPNsfromallthetrainingsamples

{{

xi j

}

Li=1

}

cj=1and thequeryxq usingzi j=MTexi j,zq=MTexq.

7. Calculatethe NCCvalue

ρ

(

zq,zC_j

)

betweenqueryzq andeach

referencezC_j usingEq.(4).

8. AcceptH0if

ρ

(

zq,zC_j

)

<

τ

z,otherwiseacceptH1.

thehistogramofintra-classandinter-classcorrelationsto demon-strate the effectiveness of the PCA/LDA features. Based on sev-eralpopular SPNalgorithms,we alsouse ourmethodsasa post-processingframework,andwealsocomparethedimensionalityof different features underthe same situationso asto evaluate the compactnessofdifferenttypesoffeatures.Finally,theperformance intermsofcomputationalefficiencyoftheproposed methodsare reported.

5.1. Experimentalsetup

Inthiswork,imagestakenby36camerasfromtheDresden im-agedatabase [40] areused.Aslisted in Table 1 ,we canseethese 36camerasareof12differentmodels,eachhaving1–5devices.

A total of 7200 images from these 36 cameras are involved in ourexperiments.Each camera contributes200 images, includ-ing 150images with varying scene details (i.e., textured images) and50flat-fieldimages.Weconsidertwoscenarioswithdifferent typesofreferenceimages(i.e.,flat-fieldandtextured),asshownin Table 2 .Foreachimage,ablocksof512_×512pixelscroppedfrom thecenterisusedintheexperimentssoastoavoidthevignetting effect [33] .

Foreachimageblock,weextractthenoiseresidualsfromthree color channels (i.e., red, green and blue channel) and combine thembyusingthefollowinglinearcombinationtoformagrayscale

Table1

36camerasinvolvedinourexperiments.

Cameramodels Numberofdevices Resolution

Canon_Ixus55 1 2592×1944

Canon_Ixus70 3 3072×2304

Olympus_mju_1050SW 5 3648×2736 Pentax_OptioA40 4 4000×3000

Pentax_OptioW60 1 3648×2736

Praktica_DCZ5.9 5 2560×1920

Rollei_RCP_7325XS 3 3072×2304

Samsung_L74wide 3 3072×2304

Samsung_NV15 3 3648×2736

Sony_DSC_H50 2 3456×2592

Sony_DSC_T77 4 3648×2736

Sony_DSC_W170 2 3648×2736

Table2

ThesetupoftwoSCIscenarios.

Scenario Referenceimagespercamera Queryimagespercamera

1 50flat-fieldimages 100texturedimages 2 50texturedimages 100texturedimages

Fig.2. TheTPR(withtheFPRfixedat10−3_{)ofthePCA-SPNobtainedfromBM3D} w.r.t.differentsettingofparameterTandanddifferentreferencetypes.

version,suchthat

x=0.299xR+0.587xG+0.114xB, (17)

wherexR,xGandxBarethenoiseresidualsextractedfromthered,

greenandbluechannel,respectively.

In our experiments, the noise residuals extracted with the methods in [3] (Basic), [4] (MLE), [24] (BM3D) and [26] (PCAI8) areservedastheoriginalSPNs.SEA [30] isappliedtoenhancethe referenceSPNsandthetrainingsamplesforPCA-SPNandLDA-SPN. TheresultsarecomparedagainsttheSPNDigestof [14] .

NCCdefinedin Eq. (4) isusedtomeasurethesimilarityinthe SCItasks.

5.2.Parametersettingsanddiscussions

Table3

ThedimensionalitydofPCA-SPNsobtainedfromdifferentSPNmethodsw.r.t. dif-ferentsettingofTanddifferentreferencetypes.

Method Flat-field textured

T=20 T=48 T=20 T=48

Basic 1042 609 1159 867

MLE 1013 605 1138 863

BM3D 1029 598 1148 848

PCAI8 1066 663 1148 860

Table 2 .Wecanseethatgenerallytheperformancebasedon PCA-SPNfromBM3D features isnot very sensitiveto the settingofT (i.e.,its performance isstable ina wide rangeof value of T [20, 48]).It improves slightly withan increasing value of T, reaching thepeaksforbothscenarios(i.e.,withflat-field/texturedreference) whenT=48.Itisworth notingtheresultwithT=1isthecase without applyingthe proposed training set construction process, andthelargeperformancegap(e.g.,whencomparedwithT=48) indicates theeffectiveness of ourproposed training set construc-tionprocess,especiallyfortexturedreferences.Itisalsointeresting toseethattheTPRdropsdramaticallywhenT>=49,sincewhen T→N(N=50),allthetrainingsamplesfromthesamecamera be-comesimilar.Especially,whenT₌Nallthetrainingsamplesfrom thesamecamerawouldbecomeexactlythesame.Inthiscase,we literallyhaveonlyonetrainingsampleper camera,andthe train-ingsetisnotlargeenoughtolearntheeffectivefeature represen-tation [41] .Therefore,to minimizeSPN distortion,weset Tto 48 throughouttherestofthepaper.

Itisalsointerestingtodiscussd,thedimensionalityofPCA-SPN indifferentcases. Clearly, we prefer dto be assmallas possible withoutcompromisingtheidentificationaccuracy.disdetermined bytwo main factors,namelythe percentageofthe totalvariance retainedin Eq. (12) andthequality ofthetrainingset.As shown in Eq. (12) ,thevalueofdisaffectedbythepercentageofthe to-talvariancethatweaimstopreserve(i.e.,98%inthispaper).The lesspercentagethat isretained, thesmallervalue ofdwouldbe. Table 3 showsthedimensionalitydofthePCA-SPNsobtainedfrom differentSPNextractionmethodswithrespecttodifferentsettings ofT (i.e., T₌20 andT₌48), for two typesof referenceimages. Inboth flat-field andtextured trainingsets, we can seethat the dimensionalitydofPCA-SPNsdecreasewhenTislarger.One rea-sonis that, witha largerT, accordingto Eq. (8) we can seethat thequality ofthe training settend to be better (i.e.,lower

σ

2

X),

thustheenergyofthetrueSPNismoreconcentratedinthe trans-formeddomain.Asaresult,theSPNfeatureextractorrequiresless leadingeigenvectorstocoverthe98%ofthetotalenergy.Similarly, flat-field referenceimages(with trainingsamples ofhigher qual-ity)alsotends to haveamore compactrepresentationthan their texturedcounterpart,asshownin Table 3 .Itisworth mentioning thatd isinsensitive tothe size oforiginal SPN.According to our experimentalresults,thePCA-SPNderivedfromlargeimageblocks hasasimilarsizetotheonefromsmallimage blocks.This obser-vationindicates that the PCA-SPN iscompressedmore effectively whenitsoriginalSPNisextractedfromlargerimageblocks.

5.3.Distributionsofintra-classandinter-classcorrelations

We evaluate theeffectiveness ofdifferentfeatures interms of thedistributionoftheirinter/intra-classcorrelations.Agreat sepa-rationbetweenintra-classandinter-classdistributionsofafeature suggeststhe feature’shighdiscriminative power.Experimentsare conductedusing3differenttypesofSPNs(i.e.,originalSPN, PCA-SPN,andLDA-SPN)inthe2SCIscenarios (withflat-field/textured referenceaslisted in Table 2 ). Resultsarereportedin Fig. 3 ,from whichwecanseethemeansoftheintra-classcorrelationsare

sig-nificantly increased by using PCA-SPN and LDA-SPN, when com-paredwiththeresultsbasedontheoriginalSPNs.Specifically,for thetwo SCI scenarios, theapplicationofPCA increasesthemean oftheintra-classcorrelationsfrom0.046to0.564fortheflat-field references while from 0.033 to 0.412 when only given the tex-turedimagesasreference.Themeansoftheintra-classcorrelations canbefurtherboostedbyusingLDA-SPNowingtoits supervised-learningnature, to0.883 and0.838, respectively, inthe two sce-narios.

Theincrease inthemeanoftheintra-classcorrelationsresults intherightwardshiftoftheintra-classdistribution,whichwidens the separation between the intra/inter-class similarity distribu-tions. However,the variance oftheinter-class correlationsisalso increasedinthecaseofusingPCA-SPNsandLDA-SPNs.For exam-ple,in the casewithflat-field references, theinter-class variance forPCA-SPN andLDA-SPNare 7.8×10−4 _and6.8×10−3_, respec-tively,whicharehigherthanthatoftheoriginal SPNs,5_.4_×10−6_. However, the increase in variance are trivial when compared to thedisplacementsofthemeansoftheintra-classcorrelations(i.e., 0_.564₋0_.046₌0_.518 and 0_.883₋0_.046₌0_.837) away from the inter-class mean.This suggests the benefits of applyingPCA-SPN andLDA-SPNintheSCItasks.Thisisclearlyreconfirmedin Fig. 3 , wheretheoverlappingareabetweentheintra-classandinter-class distributions ofPCA-SPN andLDA-SPNare much smaller, making thetwodistributionsmoreseparated(especiallywithLDA-SPN).

Inaddition,whenusingtheoriginalSPN(asshowninthefirst columns of Fig. 3 ), the intra-class distribution has small peaks in the overlapping area, which is mainly due to the small neg-ativecorrelation exhibitedamong the matching SPNpairs. These small correlations are probably caused by the strong distortions duetoscenedetailsinsomequeryimages.Nevertheless,when us-ing PCA-SPN andLDA-SPN(as showninthefigures fromthe last two columns), the numbers of smallnegative intra-class correla-tions are significantly reduced. As a result, the overlapping area decreases substantially, again reconfirmingthe merit ofPCA-SPN andLDA-SPN. Moreover,sincethe separationismainly causedby therightwardshiftoftheintra-classdistribution,whichhasa ma-jorinfluenceontheFalseRejectionRate(FRR).Assuch,PCA-SPNs and LDA-SPNs have particular advantage in the situations where lowFRRispreferred.

5.4. Performancecomparison– accuracy

We canuse theafore-mention methods(i.e., trainingset con-struction and PCA/LDA-based SPN feature extraction) as a post-processing for the existing SPN extraction methods. For evalua-tion purpose, here we report the performance (in terms of ROC curves)offourpopularmethods,namely,Basic [3] ,MLE [5] ,BM3D [25] and PCAI8 [26] combined with and without the proposed post-processingmethod.Moreover,sinceourmethodaimsto com-pressthe sizeofSPNs, wealso presentanother SPNcompression method (i.e., SPN Digest [14] ) for comparison.SPN digest is pri-marilyformed by retainingthetop k largestelementsfroma m -dimensional SPN (k<m). Therefore, the size of SPN digest is k, whichislowerthanthatoftheoriginalSPN.WhileareferenceSPN Digestnotonlycontainstopkelementsbutalsothecorresponding locationsofthesekelements.Thislocation informationisusedto extractthedigestsfromthequerySPNssothatitensuresthe ref-erenceandquerydigestareextractedfromthesamelocations.In thisexperiment,wesetk/mequalto10%and20%.

de-Fig.3. Distributionsoftheinter/intra-classcorrelationsw.r.t.differentfeatures(i.e.,originalSPN,PCA-SPNandLDA-SPNfromlefttoright)anddifferentreferencetypes(1st row:flat-fieldreferenceand2ndrow:texturedreference).

Fig.4. OverallROCcurvesofdifferenttypesoffeaturesbasedontheBasicSPNextraction(Left:flat-fieldreference;Right:texturedreference).(Forinterpretationofthe referencestocolorinthisfigure,thereaderisreferredtothewebversionofthisarticle.)

cisionsarecalculatedandthenusedtocalculatethetruepositive rate Pt p andfalse positive rate Pf p,respectively. Since the same

numberofimagesbyeach cameraareuseinourexperiment, we cansimplycalculatethePt p andPf pforathresholdasfollows

Pt p=

c i=1Dt pi

T ,Pf p=

c i=1Dif p

(

c−1

)

T ,i=1,2,...,c, (18)

wherecisthenumberofcameras;T isthenumberofquery im-ages fromall cameras;_Di

t p andDif parethe numberoftrue

posi-tive decisionsandfalsepositive decisionsmadeforcameraC_i.By varying the detectionthreshold fromthe minimumto maximum value,we canobtaintheoverallROCcurve. Inreal-worldforensic applications,itisoftennecessarytoensureasufficientlylowFPR. Therefore,we plotthehorizontalaxisoftheoverall ROCcurvein thelogarithmicscale.

Fig. 4 showstheoverall ROCcurvesofdifferentfeaturesbased on the BasicSPN extractionmethod [3] inthe two SCI scenarios describedin Table 2 ,i.e.,withflat-field/texturedreferenceimages. The black,green,yellow,red andbluecurve indicates the perfor-manceoftheoriginalSPN(i.e.,Basic),SPNDigest-10%,SPN Digest-20%,PCA-SPNandLDA-SPN,respectively.InbothSCIscenarios,we

canseethat SPNDigest performancesverycloselytotheoriginal SPNwhen20%ofthetoplargestelementsareretained,butits per-formancedegradesrapidlywhen theamount oftheretained ele-ments is reduced to 10%. On the other hand, the LDA-SPN(blue line)achievesthebestROCperformanceregardlessofthetype of referenceimages, while thePCA-SPN (red line) takesthe second place.The sameobservation canbe made from Figs. 5 to7 when differentSPN extraction methods (i.e.,MLE, BM3Dand PCIA)are usedrespectively.

5.5.Performancecomparison– compactness

Fig.5. OverallROCcurvesofdifferenttypesoffeaturesbasedontheMLESPNextraction(Left:flat-fieldreference;Right:texturedreference).(Forinterpretationofthe referencestocolorinthisfigure,thereaderisreferredtothewebversionofthisarticle.)

Fig.6. OverallROCcurvesofdifferenttypesoffeaturesbasedontheBM3DSPNextraction(Left:flat-fieldreference;Right:texturedreference).(Forinterpretationofthe referencestocolorinthisfigure,thereaderisreferredtothewebversionofthisarticle.)

Fig.7. OverallROCcurvesofdifferenttypesoffeaturesbasedonthePCAISPNextraction(Left:flat-fieldreference;Right:texturedreference).(Forinterpretationofthe referencestocolorinthisfigure,thereaderisreferredtothewebversionofthisarticle.)

inthisexperiment. Thisobservationshowsthat the dimensional-ityofSPNDigest ismuch higherthan that ofPCA-SPN and LDA-SPN.Consideringthe resultsobtainedin Section 5.4 ,wecan con-cludethat bothPCA-SPN andLDA-SPNare superiortoSPNDigest interms of compactness andidentification accuracy. Experimen-talresults also validate that the proposed SPN feature extraction methodcan beusedasageneralpost-processingmethodapplied aftervariousSPNextractionmethodsintheSCItask.

5.6.Performancecomparison– computationalcomplexity

An efficientSCIsystemplays animportantrole when(i)there isadatabasewithalargenumberofreferencesand(ii)thousands ofquerySPNsarerequiredtobeidentified.Inordertotestify the proposedframeworkonalarge database,weperformthis experi-mentonasyntheticdatabase,whichcontains180camerasderived fromthe36camerasin Table 1 basedonthefactthatSPNsare lo-cationdependent(i.e.,SPNblockscroppedfromdifferentlocations

Table4

Computationalcost(seconds)ofSPNDigest-20%anddifferenttypesoffeatures pro-ducedbyBM3D.

Features I/Ooperations Featureextraction Matching Total

OriginalSPN 0.08 0 12425.48 12425.56 SPNDigest 0.03 1236.25 1770.12 3006.40 PCA-SPN 1.52 4459.35 238.35 4699.22 LDA-SPN 0.08 399.71 154.48 554.27

Table5

Thesize(MB)ofdatarequiredtobeloadedforSPNDigest-20%anddifferenttypes offeaturesproducedbyBM3D.

Features Datasize

Featureextractor Referencesvectors

OriginalSPN 0 45.05

SPNDigest 9.01 9.01

PCA-SPN 621.21 0.43

LDA-SPN 44.80 0.03

Table 4 showstherunningtimeformatchingthe18,000query samples with thesimulated 180 camerasw.r.t. different typesof features. In this case, the size of the original SPN, SPN Digest, PCA-SPN and LDA-SPN are m=262,144, k=52,429, d=2,484 andc₋1₌179_,respectively.Thisexperimentisconductedonthe samePCwithanIntelCorei53.20GHzprocessorand16GRAM.In ordertoreducethestoragerequirement,allthedatainthis exper-imentisstoredintheuint8datatype.Todoso,wefirstprojectall thedataontotherangeof[0,255]andthenconvertthedatatype fromdouble-precisionfloating-pointtouint8beforestoring.

Toquantifytheefficiencyofanidentificationsystem,three fac-tors areconsidered inthisexperiment. Thefirst factoris“I/O op-erations”, which includes the cost of loading the references and theSPNfeatureextractorsintomemoryforprocessing.Thesecond one is“Feature Extraction”,indicating thetime spendon produc-ingSPNDigest,PCA-SPNsorLDA-SPNsfromthe18,000querynoise residuals.Thethirdfactoristhecomputationalcostforcalculating thesimilaritybetweenthe18,000querysamplesandthe180 ref-erences,whichisreferred toas“Matching”.Theoverall computa-tionalcostispresentedas“Total”.

As shown in Table 4 , PCA-SPN incurs the highest computa-tionalcostintermsofI/Ooperations.Itisbecausethedataneeds to be loaded into the memory. It includes not only the 180 m -dimensional referencevectorsbutalso an m_×d-dimensional fea-tureextractor(Md

pca).

Asshownin Table 5 ,PCA-SPNneedsaverysmallspacetostore its 180referencevectors(0.43MB)buta relativelyhuge spacefor thefeatureextractor(621.21MB).Withsuchalargeamountofdata intotal,itisnotsurprisingtoseePCA-SPNincursthehighest com-putationalcost intermsofI/O operations.LDA-SPNalsoneeds to loada feature extractor(M_lda), butits sizeis onlym_×

(

c₋1

)

so that thespace itoccupiesis muchsmallerthan that ofPCA-SPN, which is 44.80MB. Moreover, since the size of LDA-SPN is only c₋1_,its storageoverheadforthe180referencevectors(0.03MB) isthelightestamongallthefeatures.Inthisexperimentalsetting, thetotalstoragerequirementofLDA-SPN(44.83MB)isjustslightly lowerthanthatoftheoriginalSPN(45.05MB),butthismarginwill growinalinearmannerw.r.t.theincreasingnumberofcameras.

SPN Digest requiresthe smalleststorageamong these4types offeatures.Asmentionedearlier,thedigestofanormal-sized ref-erenceSPNconsistsofnotonlythektoplargestelementsbutalso the corresponding location information of thesek elements. This location information will be used to extract query digests from the query SPNs so that the location information of each refer-ence digest can be also treated asa feature extractor.Therefore, when using SPN Digest, the data to be loaded includes not only 180 k-dimensional reference digests but also 180 corresponding k-dimensional SPN feature extractors, which take up a space of 18.02MB intotal. As aresult, SPN Digest incursthe lowest com-putationalcostinI/Ooperations(asshownin Table 4 ).

Asmentioned in [42] ,theprocessofmatchingaqueryfeature withallthereferencesinthedatabasehascomplexityproportional to the product ofthe numberof referencesand thefeature size. Forexample,when usingtheOriginal SPN,thecomplexityofthe matchingphasewouldbeO

(

cm

)

.Sincethenumberofquery

sam-plesandreferencesinthedatabasearefixedinthiscase,therefore LDA-SPN, which is of the lowest dimensionality, incurs the least computationalcostinthematchingphase.PCA-SPNtakesthe sec-ondplace,followedbySPNDigestandOriginalSPN.Although LDA-SPN,PCA-SPNandSPNDigestincuranextracomputationalcostin thefeatureextractionprocess, butwithall aspects takeninto ac-count, we cansee from Table 4 that replacing Original SPNwith LDA-SPN,PCA-SPNorSPNDigestcansignificantlyreducethe over-allcomputationalcost.

Bearinmind,theseabove-mentionedpost-processingmethods wouldalsoincuran extracomputational costinthetraining pro-cessortheprocessofestimatingtheoptimalSPNDigest.However, comparedtotheprocessesthathavetobeconductedon-line(i.e., the processes listed in Table 4 ), PCA/LDA trainingor SPN Digest estimationcanbe performedoff-line, andthereisnoneed to re-runtheseprocessesaslongasthepopulationofdatabasedoesnot change. Moreover, the efficiency of the off-line operations of an SCIsystemisgenerallylessimportantwhenitiscomparedtothe identification accuracy or the on-linematching efficiency. There-fore,thecomputationalcostoftheoff-lineoperations,i.e.,PCA/LDA trainingandSPNDigestestimation,arenotcountedinthis exper-iment.

6. Conclusion

Inthispaper,weintroducedandevaluatedtheconceptofPCA de-noising in the SCI task. Based on this concept, an effective frameworkforde-noisingandcompressingfull-sizedSPNsis pro-posed.We alsoproposed a trainingsetconstruction methodthat minimizesthe impactof interferingartifacts,which plays an im-portantroleinlearningtheSPNfeatureextractorthatisinsensitive tovariousunwantednoise.Boththeoreticalderivationsand exper-imental results suggest that ourmethods can be used as a gen-eral post-processing framework for effective and efficient source camera identification. It is worth mentioning that the proposed frameworkalsoachievesverycompetitiveperformanceinthe chal-lengingtasks when only textured references are available, which isusually thecaseinreal-world applications.However, sofarwe focus on the case that the reference SPNs of all the cameras in questionareincludedinthetrainingset,whileinreal-world foren-sicapplications,referenceSPNsofnewcameraswillcontinuously beaddedtothedatabase.Inthiscase,theproposedsystemneeds tore-performthetrainingprocesswiththenewcamerasor refer-enceSPNs ofthecamerasinvolvedso astomaintainthe identifi-cationaccuracy.Anewlineforfutureresearchistodevelopanew methodologythatcan progressivelyupdatethepreviously trained SPN feature extractor to accommodate the newly received refer-enceSPNswithouthavingtore-traintheentireexpandedset.

Acknowledgments

This work is supported by the EU Horizon 2020 – Marie Sklodowska-Curie Actions through the project entitled Computer Vision Enabled Multimedia Forensics and People Identification (ProjectNo. 690907 ,Acronym:IDENTITY).

References

[1] M.J.Sorrell,DigitalcamerasourceidentificationthroughJPEGquantisation,in: C.-T.Li(Ed.),MultimediaForensicsandSecurity,Hershey,PA:IGIGlobal,2009. [2] P.Blythe,J. Fridrich, Securedigital camera,in:Proceedings ofthe 2004Digital

ForensicResearchWorkshop,2004,pp.11–13.

[3] J.Lukas,J.Fridrich,M.Goljan,Digitalcameraidentificationfromsensorpattern noise,IEEETrans.Inf.ForensicsSecur.1(2)(2006)205–214.

[4] M.Chen,J.Fridrich,M.Goljan,J.Lukas,Determiningimageoriginandintegrity usingsensornoise,IEEETrans.Inf.ForensicsSecur.3(1)(2008)74–90. [5] C.-T.Li,Sourcecameraidentificationusingenhancedsensorpatternnoise,IEEE

[6] X.Kang,Y.Li,Z.Qu,J.Huang,Enhancingsourcecameraidentification perfor-mancewithacamerareferencephasesensorpattern noise,IEEETrans.Inf. ForensicsSecur.7(2)(2012)393–402.

[7] A.J.Cooper, Improved photoresponse non-uniformity (PRNU) basedsource cameraidentification,ForensicSci.Int.226(1)(2013)132–141.

[8] L.-H.Chan,N.-F.Law,W.-C.Siu,Aconfidencemapandpixel-basedweighted correlationforPRNU-basedcameraidentification,Digit.Invest.10(3)(2013) 215–225.

[9] L.T.Van,S. Emmanuel, M. Kankanhall, Identifying sourcecell phone using chromaticaberration,in:Proceedingsofthe2007IEEEInternational Confer-enceonMultimediaandExpo,2007,pp.883–886.

[10] A.Swaminathan, M. Wu, K.R.Liu, Nonintrusivecomponent forensics of vi-sualsensorsusingoutputimages,IEEETrans.Inf.ForensicsSecur.2(1)(2007) 91–106.

[11]O.Celiktutan,B.Sankur,I.Avcibas, Blindidentificationofsourcecell-phone model,IEEETrans.Inf.ForensicsSecur.3(3)(2008)553–566.

[12] S. Bayram, H. Sencar, N. Memon, Sensor fingerprintidentification through compositefingerprintsandgrouptesting,IEEETrans.Inf.ForensicsSecur.10 (3)(Mar.2015)597–612.

[13] Y.Hu,C.-T.Li,Z.Lai,Fastsourcecameraidentificationusingmatchingsigns betweenqueryandreferencefingerprints,Multimed.ToolsAppl.74(18)(2015) 7405–7428.

[14] M.Goljan,J.Fridrich,Sensorfingerprintdigestsforfastcameraidentification fromgeometrically distortedimages,in: Proceedingsofthe 2013IS&T/SPIE ElectronicImaging,201386650B.1-86650B.10.

[15] S. Bayram, H. Sencar, N. Memon, Efficient sensor fingerprint matching throughfingerprintbinarization,IEEETrans.Inf.ForensicsSecur.7(4)(2012) 1404–1413.

[16] D.Valsesia,G.Coluccia,T.Bianchi,E.Magli,Compressedfingerprintmatching andcameraidentificationviarandomprojections,IEEETrans.Inf.Forensics Se-cur.10(1)(2015)1472–1485.

[17] D.Valsesia,G.Coluccia,T.Bianchi,E.Magli,Large-scaleimageretrievalbased on compressed camera identification, IEEE Trans. Multimed. 17 (1)(2015) 1439–1449.

[18] S.Taspinar,H.T.Sencar,S.Bayram,N.Memon,Fastcamerafingerprint match-inginverylargedatabases,in:Proceedingsofthe2017IEEEInternational Con-ferenceonImageProcessing,Beijing,China,17–20,2017.

[19] R.Li,C.-T.Li,Y.Guan,Acompactrepresentationofsensorfingerprintfor cam-eraidentificationandfingerprintmatching,in:Proceedingsofthe2015IEEE InternationalConferenceo.Acoustics,Speech,andSignalProcessing,Brisbane, Australia,19–24,2015,pp.1777–1781.

[20]R.Li,C.-T.Li,Y.Guan, Incrementalupdateoffeatureextractorforcamera iden-tification,in:Proceedingsofthe2015IEEEInternationalConferenceonImage Process.,QuebecCity,Canada,27–30,2015.

[21]D.Zhang,R.Lukac,X.Wu,D.Zhang,PCA-basedspatiallyadaptivedenoisingof CFAimagesforsingle-sensordigitalcameras,IEEETrans.ImageProcess.18(4) (2009)797–812.

[22]L.Zhang,W.Dong,D.Zhang,G.Shi, Two-stageimagedenoisingbyprincipal componentanalysiswithlocalpixelgrouping,PatternRecognit.43(4)(2010) 1531–1549.

[23]M.Mihcak,I.Kozintsev,K.Ramchandran,Spatiallyadaptivestatistical model-ingofwaveletimagecoefficientsanditsapplicationtodenoising,in: Proceed-ingsofthe1999IEEEInternationalConferenceoAcoustics,Speech,andSignal Processing,6,March,1999,pp.3253–3256.

[24]G.Chierchia,S.Parrilli,G.Poggi,C.Sansone,L.Verdoliva,Ontheinfluenceof denoisinginPRNUbasedforgerydetection,in:Proceedingsofthe2010ACM WorkshoponMultimediainForensics,SecurityandIntelligence,7880,2010, pp.117–122.

[25]K. Dabov, A.Foi,V.Katkovnik,K. Egiazarian,Imagedenoisingbysparse3D transform-domain collaborative filtering, IEEE Trans. Image Process. 16 (8) (2007)2080–2095.

[26]X.Kang,J.Chen,K.Lin,A.Peng, Acontext-adaptiveSPNpredictorfor trust-worthysourcecameraidentification,EURASIPJ.ImageVideoProcess.2014(1) (2014)1–11.

[27]W.Liu,W.Zeng,L.Dong,Q.Yao, Efficientcompressionofencryptedgrayscale images,IEEETrans.ImageProcess.19(4)(2010)1097–1102.

[28]C.-T.Li,Y.Li,Color-decoupledphotoresponsenon-uniformityfordigitalimage forensics,IEEETrans.CircuitsSyst.VideoTechnol.22(2)(2012)260–271. [29]M.Chen,J.Fridrich,M.Goljan,J.Lukas,Sourcedigitalcamcorder

identifica-tionusingsensorphotoresponsenon-uniformity,in:Proceedingsofthe2007 SPIEElectronicImaging,Security,Steganography,WatermarkingofMultimedia ContentsIX,6505,20071G-1H.

[30]X. Lin, C.-T. Li, Preprocessing reference sensor pattern noise via spectrum equalization,IEEETrans.Inf.ForensicsSecur.11(1)(2016)126–140. [31]J.Fridrich, M.Goljan,DerivationofROCsforcompositefingerprintsand

se-quential trimming, Dept. Elect. Comput. Eng., Binghamton Univ., Bingham-ton,NY,USA,Tech. Rep.[Online].Available:http://www.ws.binghamton.edu/ fridrich/Research/rocs.pdf.Jan.2010.

[32]M.Goljan,DigitalCameraIdentificationfromImages– estimatingfalse accep-tanceprobability,in:Proceedingsofthe2009InternationalWorkshopDigital– forensicsandWatermarking,2009,pp.454–468.

[33]C.-T.Li,R.Satta,Empiricalinvestigationintothecorrelationbetweenvignetting effectandthequalityofsensorpatternnoise,IETComput.Vis.6(6)(2012) 560–566.

[34]K.Fukunaga,IntroductiontoStatisticalPatternRecognition,seconded., Aca-demic,NewYork,1991.

[35]M.Benito,D.Peña,Afastapproachfordimensionalityreductionwithimage data,PatternRecognit.38(12)(2005)2400–2408.

[36]C.Yang,L.Wang,J.Feng,Onfeatureextractionviakernels,IEEETrans.Syst. ManCybern.BCybern.38(2)(2008)553–557.

[37]Y.Wang,K.N.Plataniotis,Ananalysisofrandomprojectionforchangeableand privacy-preservingbiometricverification,IEEETrans.Syst.ManCybern.B Cy-bern.40(5)(2010)1280–1293.

[38]J.Jauregui,Principalcomponentanalysiswithlinearalgebra(2012).

[39]J.Yang,J.-Y.Yang,WhycanLDAbeperformedinPCAtransformedspace? Pat-ternRecognit.36(2)(2003)563–566.

[40]T.Gloe,R.Böhme,Thedresdenimagedatabaseforbenchmarkingdigitalimage forensics,J.Digit.ForensicPract.3(2–4)(2010)150–159.

[41]A.M.Martínez,A.Kak,PCAversusLDA,IEEETrans.PatternAnal.Mach.Intell. 23(2)(2001)228–233.

RuizheLireceivedtheB.Sc.degreeinsoftwareengineeringfromSichuanUniversity,Chengdu,China,in2010,andthePh.D.degreeincomputersciencefromtheUniversity ofWarwick,Coventry,U.K.,in2016.Hisresearchinterestsincludedigitalforensics,multimediasecurity,imageprocessing,machinelearning,andcomputervision.

Chang-TsunLireceivedtheB.Eng.degreeinelectricalengineeringfromNationalDefenceUniversity(NDU),Taiwan,in1987,theM.Sc.degreeincomputersciencefrom U.S.NavalPostgraduateSchool,USA,in1992,andthePh.D.degreeincomputersciencefromtheUniversityofWarwick,UK,in1998.Hewasanassociateprofessorofthe DepartmentofElectricalEngineeringatNDUduring1998–2002andavisitingprofessoroftheDepartmentofComputerScienceatU.S.NavalPostgraduateSchoolinthe secondhalfof2001.HewasaprofessoroftheDepartmentofComputerScienceattheUniversityofWarwick,UK,untilDec2016.HeiscurrentlyaprofessoroftheSchoolof ComputingandMathematics,CharlesSturtUniversity,Australia.Hisresearchinterestsincludemultimediaforensicsandsecurity,biometrics,datamining,machinelearning, dataanalytics,computervision,imageprocessing,patternrecognition,bioinformatics,andcontent-basedimageretrieval.Theoutcomesofhismultimediaforensicsresearch havebeentranslatedintoaward-winningcommercialproductsprotectedbyaseriesofinternationalpatentsandhavebeenusedbyanumberofpoliceforcesandcourtsof lawaroundtheworld.HeiscurrentlyAssociateEditoroftheEURASIPJournalofImageandVideoProcessing(JIVP)andAssociateofEditorofIETBiometrics.Heinvolvedin theorganisationofmanyinternationalconferencesandworkshopsandalsoservedasmemberoftheinternationalprogramcommitteesforseveralinternationalconferences. Heisalsoactivelycontributingkeynotespeechesandtalksatvariousinternationalevents.