AREUS: A Software Framework for ATLAS Readout Electronics Upgrade Simulation

AREUS

A Software Framework for ATLAS Readout Electronics Upgrade

Simulation

Nico

Madysa

1

,

∗

_{on behalf of the ATLAS Liquid Argon Calorimeter Group}

1

_{Institute of Nuclear and Particle Physics, TU Dresden}

Abstract.

The design of readout electronics for the LAr calorimeters of the

ATLAS detector to be operated at the future High-Luminosity LHC (HL-LHC)

requires a detailed simulation of the full readout chain in order to find optimal

solutions for the analog and digital processing of the detector signals. Due to the

long duration of the LAr calorimeter pulses relative to the LHC bunch crossing

time, out-of-time signal pileup needs to be taken into account. For this purpose,

the simulation framework AREUS has been developed. It models

analog-to-digital conversion, gain selection, and analog-to-digital signal processing at bit precision,

including digitization noise and detailed electronics e

ff

ects. Trigger and object

reconstruction algorithms are taken into account in the optimization process.

The software implementation of AREUS, the concepts of its main functional

blocks, as well as optimization considerations will be presented. Various

ap-proaches to introduce parallelism into AREUS will be compared against each

other.

1 Introduction

In order to reach the ambitious goal of collecting data equivalent to 4 ab

−

1

_{of integrated}

luminosity until 2038, the LHC is scheduled to undergo two major upgrades[1]. These will

be taken during two long shutdown phases: the

LS2

in 20192020 and the

LS3

in 20242026.

The end of LS3 marks the beginning of the

High-Luminosity LHC

phase (HL-LHC). After

this point, the peak luminosity is expected to reach a maximum of 75 nb

−

1

_s

−

1

_{, a value about}

4 times higher than today. At the same time, the number of protonproton interactions per

bunch crossing is expected to increase by a factor of 4 to 5, up to a value of 200.

In order to keep up with the increased luminosity, the ATLAS detector[2] will undergo

upgrades at the same time as the LHC: The

Phase-I

upgrade during LS2[3] and the

Phase-II

upgrade during LS3[4]. During the Phase-II upgrade, the full liquid argon (LAr) calorimeter

readout will be replaced. This is necessary because the old electronics have only been

qual-ified for a radiation exposure equivalent to 1 ab

−

1

_{of integrated luminosity. Furthermore, the}

new electronics are to supply more fine-grained information to the trigger. Finally, they must

accommodate an increase in the trigger rate (from 100 kHz to 1 MHz) and a larger data bu

ff

er

due to an increased over-all latency on the lowest-level trigger (from 2.5

µ

s to at least 10

µ

s).

The greatest challenge for the precise energy measurement of signals in the LAr

calorime-ter subsystem (shown in Figure 1a) is the increased number of incalorime-teractions per bunch

cross-ing. These bunch crossings occur once every 25 ns, whereas the electronic pulses caused by

LAr hadronic

end-cap (HEC)

LAr electromagnetic

end-cap (EMEC)

LAr electromagnetic

barrel (EMB)

LAr forward (FCal)

(a)

)

φ

cos(

×

)|

θ

|tan(

-0.08 -0.06 -0.04 -0.02

0

0.02 0.04 0.06 0.08

)

φ

sin(

×

)|

θ

|tan(

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

Visible Energy [MeV]

-2

10

-1

10

1

10

2

10

A1.00.00 A1.01.00 A1.02.00 A1.03.00 A1.04.00 A1.05.00 A1.06.00 A1.07.00 A1.08.00 A1.09.00 A1.10.00 A1.11.00A1.12.00 A1.13.00 A1.14.00 A1.15.00 A1.00.01 A1.01.01 A1.02.01 A1.03.01 A1.04.01 A1.05.01 A1.06.01 A1.07.01 A1.08.01 A1.09.01 A1.10.01 A1.11.01 A1.12.01 A1.13.01 A1.14.01 A1.15.01 A1.00.02 A1.01.02 A1.02.02 A1.03.02 A1.04.02 A1.05.02 A1.06.02 A1.07.02 A1.08.02 A1.09.02 A1.10.02 A1.11.02 A1.12.02 A1.13.02 A1.14.02 A1.15.02 A1.00.03 A1.01.03 A1.02.03 A1.03.03 A1.04.03 A1.05.03 A1.06.03 A1.07.03 A1.08.03 A1.09.03 A1.10.03 A1.11.03 A1.12.03 A1.13.03 A1.14.03 A1.15.03 A1.00.04 A1.01.04 A1.02.04 A1.03.04 A1.04.04 A1.05.04 A1.06.04 A1.07.04 A1.08.04 A1.09.04 A1.10.04 A1.11.04 A1.12.04 A1.13.04 A1.14.04 A1.15.04 A1.00.05 A1.01.05 A1.02.05 A1.03.05 A1.04.05 A1.05.05 A1.06.05 A1.07.05 A1.08.05 A1.09.05 A1.10.05 A1.11.05 A1.12.05 A1.13.05 A1.14.05 A1.15.05 A1.00.06 A1.01.06 A1.02.06 A1.03.06 A1.04.06 A1.05.06 A1.06.06 A1.07.06 A1.08.06 A1.09.06 A1.10.06 A1.11.06 A1.12.06 A1.13.06 A1.14.06 A1.15.06 A1.00.07 A1.01.07 A1.02.07 A1.03.07 A1.04.07 A1.05.07 A1.06.07 A1.07.07 A1.08.07 A1.09.07 A1.10.07 A1.11.07 A1.12.07 A1.13.07 A1.14.07 A1.15.07 A1.00.08 A1.01.08 A1.02.08 A1.03.08 A1.04.08 A1.05.08 A1.06.08 A1.07.08 A1.08.08 A1.09.08 A1.10.08 A1.11.08 A1.12.08 A1.13.08 A1.14.08 A1.15.08 A1.00.09 A1.01.09 A1.02.09 A1.03.09 A1.04.09 A1.05.09 A1.06.09 A1.07.09 A1.08.09 A1.09.09 A1.10.09 A1.11.09 A1.12.09 A1.13.09 A1.14.09 A1.15.09 A1.00.10 A1.01.10 A1.02.10 A1.03.10 A1.04.10 A1.05.10 A1.06.10 A1.07.10 A1.08.10 A1.09.10 A1.10.10 A1.11.10 A1.12.10 A1.13.10 A1.14.10 A1.15.10 A1.00.11 A1.01.11 A1.02.11 A1.03.11 A1.04.11 A1.05.11 A1.06.11 A1.07.11 A1.08.11 A1.09.11 A1.10.11 A1.11.11 A1.12.11 A1.13.11 A1.14.11 A1.15.11 A1.00.12 A1.01.12 A1.02.12 A1.03.12 A1.04.12 A1.05.12 A1.06.12 A1.07.12 A1.08.12 A1.09.12 A1.10.12 A1.11.12 A1.12.12 A1.13.12 A1.14.12 A1.15.12 A1.00.13 A1.01.13 A1.02.13 A1.03.13 A1.04.13 A1.05.13 A1.06.13 A1.07.13 A1.08.13 A1.09.13 A1.10.13 A1.11.13 A1.12.13 A1.13.13 A1.14.13 A1.15.13 A1.00.14 A1.01.14 A1.02.14 A1.03.14 A1.04.14 A1.05.14 A1.06.14 A1.07.14 A1.08.14 A1.09.14 A1.10.14 A1.11.14A1.12.14 A1.13.14 A1.14.14 A1.15.14 A1.00.15 A1.01.15 A1.02.15 A1.03.15 A1.04.15 A1.05.15 A1.06.15 A1.07.15 A1.08.15 A1.09.15 A1.10.15 A1.11.15 A1.12.15 A1.13.15 A1.14.15 A1.15.15 A1.00.16 A1.01.16 A1.02.16 A1.03.16 A1.04.16 A1.05.16 A1.06.16 A1.07.16 A1.08.16 A1.09.16 A1.10.16 A1.11.16 A1.12.16 A1.13.16 A1.14.16 A1.15.16 A1.00.17 A1.01.17 A1.02.17 A1.03.17 A1.04.17 A1.05.17 A1.06.17 A1.07.17 A1.08.17 A1.09.17 A1.10.17 A1.11.17 A1.12.17 A1.13.17 A1.14.17 A1.15.17 A1.00.18 A1.01.18 A1.02.18 A1.03.18 A1.04.18 A1.05.18 A1.06.18 A1.07.18 A1.08.18 A1.09.18 A1.10.18 A1.11.18 A1.12.18 A1.13.18 A1.14.18 A1.15.18 A1.00.19 A1.01.19 A1.02.19 A1.03.19 A1.04.19 A1.05.19 A1.06.19 A1.07.19 A1.08.19 A1.09.19 A1.10.19 A1.11.19A1.12.19 A1.13.19 A1.14.19 A1.15.19 A1.00.20 A1.01.20 A1.02.20 A1.03.20 A1.04.20 A1.05.20 A1.06.20 A1.07.20 A1.08.20 A1.09.20 A1.10.20 A1.11.20 A1.12.20 A1.13.20 A1.14.20 A1.15.20 A1.00.21 A1.01.21 A1.02.21 A1.03.21 A1.04.21 A1.05.21 A1.06.21 A1.07.21 A1.08.21 A1.09.21 A1.10.21 A1.11.21 A1.12.21 A1.13.21 A1.14.21 A1.15.21 A1.00.22 A1.01.22 A1.02.22 A1.03.22 A1.04.22 A1.05.22 A1.06.22 A1.07.22 A1.08.22 A1.09.22 A1.10.22 A1.11.22 A1.12.22 A1.13.22 A1.14.22 A1.15.22 A1.00.23 A1.01.23 A1.02.23 A1.03.23 A1.04.23 A1.05.23 A1.06.23 A1.07.23 A1.08.23 A1.09.23 A1.10.23 A1.11.23 A1.12.23 A1.13.23 A1.14.23 A1.15.23 A1.00.24 A1.01.24 A1.02.24 A1.03.24 A1.04.24 A1.05.24 A1.06.24 A1.07.24 A1.08.24 A1.09.24 A1.10.24 A1.11.24A1.12.24 A1.13.24 A1.14.24 A1.15.24 A1.00.25 A1.01.25 A1.02.25 A1.03.25 A1.04.25 A1.05.25 A1.06.25 A1.07.25 A1.08.25 A1.09.25

A1.10.25A1.11.25 A1.12.25 A1.13.25 A1.14.25 A1.15.25 A1.00.26 A1.01.26 A1.02.26 A1.03.26 A1.04.26 A1.05.26 A1.06.26 A1.07.26 A1.08.26 A1.09.26 A1.10.26 A1.11.26 A1.12.26 A1.13.26 A1.14.26 A1.15.26 A1.00.27 A1.01.27 A1.02.27 A1.03.27 A1.04.27 A1.05.27 A1.06.27 A1.07.27 A1.08.27 A1.09.27 A1.10.27 A1.11.27 A1.12.27 A1.13.27 A1.14.27 A1.15.27 A1.00.28 A1.01.28 A1.02.28 A1.03.28 A1.04.28 A1.05.28 A1.06.28 A1.07.28 A1.08.28 A1.09.28 A1.10.28 A1.11.28A1.12.28 A1.13.28 A1.14.28 A1.15.28 A1.00.29 A1.01.29 A1.02.29 A1.03.29 A1.04.29 A1.05.29 A1.06.29 A1.07.29 A1.08.29 A1.09.29

A1.10.29A1.11.29 A1.12.29 A1.13.29 A1.14.29 A1.15.29 A1.00.30 A1.01.30 A1.02.30 A1.03.30 A1.04.30 A1.05.30 A1.06.30 A1.07.30 A1.08.30 A1.09.30 A1.10.30 A1.11.30 A1.12.30 A1.13.30 A1.14.30 A1.15.30 A1.00.31 A1.01.31 A1.02.31 A1.03.31 A1.04.31 A1.05.31 A1.06.31 A1.07.31 A1.08.31 A1.09.31 A1.10.31 A1.11.31 A1.12.31 A1.13.31 A1.14.31 A1.15.31 A1.00.32 A1.01.32 A1.02.32 A1.03.32 A1.04.32 A1.05.32 A1.06.32 A1.07.32 A1.08.32 A1.09.32 A1.10.32 A1.11.32A1.12.32 A1.13.32 A1.14.32 A1.15.32 A1.00.33 A1.01.33 A1.02.33 A1.03.33 A1.04.33 A1.05.33 A1.06.33 A1.07.33 A1.08.33 A1.09.33 A1.10.33 A1.11.33 A1.12.33 A1.13.33 A1.14.33 A1.15.33 A1.00.34 A1.01.34 A1.02.34 A1.03.34 A1.04.34 A1.05.34 A1.06.34 A1.07.34 A1.08.34 A1.09.34 A1.10.34 A1.11.34 A1.12.34 A1.13.34 A1.14.34 A1.15.34 A1.00.35 A1.01.35 A1.02.35 A1.03.35 A1.04.35 A1.05.35 A1.06.35 A1.07.35 A1.08.35 A1.09.35 A1.10.35 A1.11.35 A1.12.35 A1.13.35A1.14.35 A1.15.35 A1.00.36 A1.01.36 A1.02.36 A1.03.36 A1.04.36 A1.05.36 A1.06.36 A1.07.36 A1.08.36 A1.09.36 A1.10.36 A1.11.36A1.12.36 A1.13.36 A1.14.36 A1.15.36 A1.00.37 A1.01.37 A1.02.37 A1.03.37 A1.04.37 A1.05.37 A1.06.37 A1.07.37 A1.08.37 A1.09.37 A1.10.37

A1.11.37 A1.12.37A1.13.37 A1.14.37 A1.15.37 A1.00.38 A1.01.38 A1.02.38 A1.03.38 A1.04.38 A1.05.38 A1.06.38 A1.07.38 A1.08.38 A1.09.38

A1.10.38 A1.11.38 A1.12.38 A1.13.38 A1.14.38 A1.15.38 A1.00.39 A1.01.39 A1.02.39 A1.03.39 A1.04.39 A1.05.39 A1.06.39 A1.07.39 A1.08.39 A1.09.39 A1.10.39 A1.11.39A1.12.39 A1.13.39 A1.14.39 A1.15.39 A1.00.40 A1.01.40 A1.02.40 A1.03.40 A1.04.40 A1.05.40 A1.06.40 A1.07.40 A1.08.40 A1.09.40 A1.10.40

A1.11.40 A1.12.40A1.13.40 A1.14.40 A1.15.40 A1.00.41 A1.01.41 A1.02.41 A1.03.41 A1.04.41 A1.05.41 A1.06.41 A1.07.41 A1.08.41 A1.09.41

A1.10.41 A1.11.41 A1.12.41 A1.13.41 A1.14.41 A1.15.41 A1.00.42 A1.01.42 A1.02.42 A1.03.42 A1.04.42 A1.05.42 A1.06.42 A1.07.42 A1.08.42 A1.09.42 A1.10.42 A1.11.42A1.12.42 A1.13.42 A1.14.42 A1.15.42 A1.00.43 A1.01.43 A1.02.43 A1.03.43 A1.04.43 A1.05.43 A1.06.43 A1.07.43 A1.08.43 A1.09.43 A1.10.43

A1.11.43 A1.12.43A1.13.43 A1.14.43 A1.15.43 A1.00.44 A1.01.44 A1.02.44 A1.03.44 A1.04.44 A1.05.44 A1.06.44 A1.07.44 A1.08.44 A1.09.44 A1.10.44 A1.11.44 A1.12.44

A1.13.44 A1.14.44 A1.15.44 A1.00.45 A1.01.45 A1.02.45 A1.03.45 A1.04.45 A1.05.45 A1.06.45 A1.07.45 A1.08.45 A1.09.45 A1.10.45 A1.11.45A1.12.45 A1.13.45 A1.14.45 A1.15.45 A1.00.46 A1.01.46 A1.02.46 A1.03.46 A1.04.46 A1.05.46 A1.06.46 A1.07.46 A1.08.46 A1.09.46 A1.10.46 A1.11.46 A1.12.46

A1.13.46 A1.14.46 A1.15.46 A1.00.47 A1.01.47 A1.02.47 A1.03.47 A1.04.47 A1.05.47 A1.06.47 A1.07.47 A1.08.47 A1.09.47 A1.10.47

A1.11.47A1.12.47 A1.13.47 A1.14.47 A1.15.47 A1.00.48 A1.01.48 A1.02.48 A1.03.48 A1.04.48 A1.05.48 A1.06.48 A1.07.48 A1.08.48 A1.09.48 A1.10.48 A1.11.48 A1.12.48 A1.13.48

A1.14.48 A1.15.48 A1.00.49 A1.01.49 A1.02.49 A1.03.49 A1.04.49 A1.05.49 A1.06.49 A1.07.49 A1.08.49 A1.09.49

A1.10.49 A1.11.49A1.12.49 A1.13.49 A1.14.49 A1.15.49 A1.00.50 A1.01.50 A1.02.50 A1.03.50 A1.04.50 A1.05.50 A1.06.50 A1.07.50 A1.08.50 A1.09.50 A1.10.50 A1.11.50 A1.12.50 A1.13.50 A1.14.50 A1.15.50 A1.00.51 A1.01.51 A1.02.51 A1.03.51 A1.04.51 A1.05.51 A1.06.51 A1.07.51 A1.08.51 A1.09.51 A1.10.51 A1.11.51 A1.12.51 A1.13.51 A1.14.51 A1.15.51 A1.00.52 A1.01.52 A1.02.52 A1.03.52 A1.04.52 A1.05.52 A1.06.52 A1.07.52 A1.08.52 A1.09.52 A1.10.52 A1.11.52 A1.12.52 A1.13.52 A1.14.52 A1.15.52 A1.00.53 A1.01.53 A1.02.53 A1.03.53 A1.04.53 A1.05.53 A1.06.53 A1.07.53 A1.08.53 A1.09.53 A1.10.53 A1.11.53A1.12.53 A1.13.53 A1.14.53 A1.15.53 A1.00.54 A1.01.54 A1.02.54 A1.03.54 A1.04.54 A1.05.54 A1.06.54 A1.07.54 A1.08.54 A1.09.54 A1.10.54 A1.11.54 A1.12.54 A1.13.54 A1.14.54 A1.15.54 A1.00.55 A1.01.55 A1.02.55 A1.03.55 A1.04.55 A1.05.55 A1.06.55 A1.07.55 A1.08.55 A1.09.55 A1.10.55 A1.11.55 A1.12.55 A1.13.55 A1.14.55 A1.15.55 A1.00.56 A1.01.56 A1.02.56 A1.03.56 A1.04.56 A1.05.56 A1.06.56 A1.07.56 A1.08.56 A1.09.56 A1.10.56 A1.11.56 A1.12.56 A1.13.56 A1.14.56 A1.15.56 A1.00.57 A1.01.57 A1.02.57 A1.03.57 A1.04.57 A1.05.57 A1.06.57 A1.07.57 A1.08.57 A1.09.57 A1.10.57 A1.11.57A1.12.57 A1.13.57 A1.14.57 A1.15.57 A1.00.58 A1.01.58 A1.02.58 A1.03.58 A1.04.58 A1.05.58 A1.06.58 A1.07.58 A1.08.58 A1.09.58 A1.10.58 A1.11.58 A1.12.58 A1.13.58 A1.14.58 A1.15.58 A1.00.59 A1.01.59 A1.02.59 A1.03.59 A1.04.59 A1.05.59 A1.06.59 A1.07.59 A1.08.59 A1.09.59 A1.10.59 A1.11.59 A1.12.59 A1.13.59 A1.14.59 A1.15.59 A1.00.60 A1.01.60 A1.02.60 A1.03.60 A1.04.60 A1.05.60 A1.06.60 A1.07.60 A1.08.60 A1.09.60 A1.10.60 A1.11.60A1.12.60 A1.13.60 A1.14.60 A1.15.60 A1.00.61 A1.01.61 A1.02.61 A1.03.61 A1.04.61 A1.05.61 A1.06.61 A1.07.61 A1.08.61 A1.09.61 A1.10.61 A1.11.61 A1.12.61 A1.13.61A1.14.61 A1.15.61 A1.00.62 A1.01.62 A1.02.62 A1.03.62 A1.04.62 A1.05.62 A1.06.62 A1.07.62 A1.08.62 A1.09.62 A1.10.62 A1.11.62A1.12.62 A1.13.62 A1.14.62 A1.15.62

(b)

Figure 1. (a)

Cutaway diagram of the ATLAS liquid argon calorimeters[5].

(b)

Visualization of a

Geant4 event, here in the FCal[6, p. 88]. The color of a cell hit indicates the deposited energy.

each calorimeter hit are roughly 450 ns long. A higher number of interactions means an

in-crease of

pileup

e

ff

ects, where pulses overlap and hinder an accurate energy reconstruction.

Techniques that reduce pileup e

ff

ects often exacerbate electronics noise and vice versa, so an

optimum has to be found.

In order to minimize the combined impact of electronics noise and pileup e

ff

ects, the LAr

upgrade requires a detailed simulation of the new readout electronics. This simulation needs

to be modular so that new filtering algorithms can be implemented and studied in a timely

manner. It also needs to be flexible enough that simulated electronics can be replaced with

data from real, prototype components as they become available over time.

This simulation is provided by the AREUS framework, which is described in Section 2. In

Section 3, implementation details on AREUS’ architecture are given. In Section 4, a custom

smart pointer class as an example for the optimization considered in AREUS is presented. In

Section 5, various approaches to apply parallelization to AREUS are considered and weighed

against each other. Finally, a summary is given in Section 6.

2 The AREUS Framework

AREUS[7–9] is a program written in C

++

[10]. It directly depends on the Boost[11] and

ROOT[12] libraries for common functionality and Git[13] for version control. It can be built

using the GNU build system[14–16] or CMake[17]. Other dependencies are bundled into the

source repository, either literally or as Git submodules:

•

ExprTK[18] for parsing of mathematical expressions,

•

Google Test[19] for unit testing,

•

Splines[20] for interpolation between data points,

•

VDT[21] for vectorized implementations of certain mathematical functions.

In addition, it provides a Python[22] and an ATHENA[23] package to automate certain tasks

such as input file generation.

LAr hadronic

end-cap (HEC)

LAr electromagnetic

end-cap (EMEC)

LAr electromagnetic

barrel (EMB)

LAr forward (FCal)

(a)

)

φ

cos(

×

)|

θ

|tan(

-0.08 -0.06 -0.04 -0.02

0

0.02 0.04 0.06 0.08

)

φ

sin(

×

)|

θ

|tan(

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

Visible Energy [MeV]

-2

10

-1

10

1

10

2

10

A1.00.00 A1.01.00 A1.02.00 A1.03.00 A1.04.00 A1.05.00 A1.06.00 A1.07.00 A1.08.00 A1.09.00 A1.10.00 A1.11.00A1.12.00 A1.13.00 A1.14.00 A1.15.00 A1.00.01 A1.01.01 A1.02.01 A1.03.01 A1.04.01 A1.05.01 A1.06.01 A1.07.01 A1.08.01 A1.09.01 A1.10.01 A1.11.01 A1.12.01 A1.13.01 A1.14.01 A1.15.01 A1.00.02 A1.01.02 A1.02.02 A1.03.02 A1.04.02 A1.05.02 A1.06.02 A1.07.02 A1.08.02 A1.09.02 A1.10.02 A1.11.02 A1.12.02 A1.13.02 A1.14.02 A1.15.02 A1.00.03 A1.01.03 A1.02.03 A1.03.03 A1.04.03 A1.05.03 A1.06.03 A1.07.03 A1.08.03 A1.09.03 A1.10.03 A1.11.03 A1.12.03 A1.13.03 A1.14.03 A1.15.03 A1.00.04 A1.01.04 A1.02.04 A1.03.04 A1.04.04 A1.05.04 A1.06.04 A1.07.04 A1.08.04 A1.09.04 A1.10.04 A1.11.04 A1.12.04 A1.13.04 A1.14.04 A1.15.04 A1.00.05 A1.01.05 A1.02.05 A1.03.05 A1.04.05 A1.05.05 A1.06.05 A1.07.05 A1.08.05 A1.09.05 A1.10.05 A1.11.05 A1.12.05 A1.13.05 A1.14.05 A1.15.05 A1.00.06 A1.01.06 A1.02.06 A1.03.06 A1.04.06 A1.05.06 A1.06.06 A1.07.06 A1.08.06 A1.09.06 A1.10.06 A1.11.06 A1.12.06 A1.13.06 A1.14.06 A1.15.06 A1.00.07 A1.01.07 A1.02.07 A1.03.07 A1.04.07 A1.05.07 A1.06.07 A1.07.07 A1.08.07 A1.09.07 A1.10.07 A1.11.07 A1.12.07 A1.13.07 A1.14.07 A1.15.07 A1.00.08 A1.01.08 A1.02.08 A1.03.08 A1.04.08 A1.05.08 A1.06.08 A1.07.08 A1.08.08 A1.09.08 A1.10.08 A1.11.08 A1.12.08 A1.13.08 A1.14.08 A1.15.08 A1.00.09 A1.01.09 A1.02.09 A1.03.09 A1.04.09 A1.05.09 A1.06.09 A1.07.09 A1.08.09 A1.09.09 A1.10.09 A1.11.09 A1.12.09 A1.13.09 A1.14.09 A1.15.09 A1.00.10 A1.01.10 A1.02.10 A1.03.10 A1.04.10 A1.05.10 A1.06.10 A1.07.10 A1.08.10 A1.09.10 A1.10.10 A1.11.10 A1.12.10 A1.13.10 A1.14.10 A1.15.10 A1.00.11 A1.01.11 A1.02.11 A1.03.11 A1.04.11 A1.05.11 A1.06.11 A1.07.11 A1.08.11 A1.09.11 A1.10.11 A1.11.11 A1.12.11 A1.13.11 A1.14.11 A1.15.11 A1.00.12 A1.01.12 A1.02.12 A1.03.12 A1.04.12 A1.05.12 A1.06.12 A1.07.12 A1.08.12 A1.09.12 A1.10.12 A1.11.12 A1.12.12 A1.13.12 A1.14.12 A1.15.12 A1.00.13 A1.01.13 A1.02.13 A1.03.13 A1.04.13 A1.05.13 A1.06.13 A1.07.13 A1.08.13 A1.09.13 A1.10.13 A1.11.13 A1.12.13 A1.13.13 A1.14.13 A1.15.13 A1.00.14 A1.01.14 A1.02.14 A1.03.14 A1.04.14 A1.05.14 A1.06.14 A1.07.14 A1.08.14 A1.09.14 A1.10.14 A1.11.14A1.12.14 A1.13.14 A1.14.14 A1.15.14 A1.00.15 A1.01.15 A1.02.15 A1.03.15 A1.04.15 A1.05.15 A1.06.15 A1.07.15 A1.08.15 A1.09.15 A1.10.15 A1.11.15 A1.12.15 A1.13.15 A1.14.15 A1.15.15 A1.00.16 A1.01.16 A1.02.16 A1.03.16 A1.04.16 A1.05.16 A1.06.16 A1.07.16 A1.08.16 A1.09.16 A1.10.16 A1.11.16 A1.12.16 A1.13.16 A1.14.16 A1.15.16 A1.00.17 A1.01.17 A1.02.17 A1.03.17 A1.04.17 A1.05.17 A1.06.17 A1.07.17 A1.08.17 A1.09.17 A1.10.17 A1.11.17 A1.12.17 A1.13.17 A1.14.17 A1.15.17 A1.00.18 A1.01.18 A1.02.18 A1.03.18 A1.04.18 A1.05.18 A1.06.18 A1.07.18 A1.08.18 A1.09.18 A1.10.18 A1.11.18 A1.12.18 A1.13.18 A1.14.18 A1.15.18 A1.00.19 A1.01.19 A1.02.19 A1.03.19 A1.04.19 A1.05.19 A1.06.19 A1.07.19 A1.08.19 A1.09.19 A1.10.19 A1.11.19A1.12.19 A1.13.19 A1.14.19 A1.15.19 A1.00.20 A1.01.20 A1.02.20 A1.03.20 A1.04.20 A1.05.20 A1.06.20 A1.07.20 A1.08.20 A1.09.20 A1.10.20 A1.11.20 A1.12.20 A1.13.20 A1.14.20 A1.15.20 A1.00.21 A1.01.21 A1.02.21 A1.03.21 A1.04.21 A1.05.21 A1.06.21 A1.07.21 A1.08.21 A1.09.21 A1.10.21 A1.11.21 A1.12.21 A1.13.21 A1.14.21 A1.15.21 A1.00.22 A1.01.22 A1.02.22 A1.03.22 A1.04.22 A1.05.22 A1.06.22 A1.07.22 A1.08.22 A1.09.22 A1.10.22 A1.11.22 A1.12.22 A1.13.22 A1.14.22 A1.15.22 A1.00.23 A1.01.23 A1.02.23 A1.03.23 A1.04.23 A1.05.23 A1.06.23 A1.07.23 A1.08.23 A1.09.23 A1.10.23 A1.11.23 A1.12.23 A1.13.23 A1.14.23 A1.15.23 A1.00.24 A1.01.24 A1.02.24 A1.03.24 A1.04.24 A1.05.24 A1.06.24 A1.07.24 A1.08.24 A1.09.24 A1.10.24 A1.11.24A1.12.24 A1.13.24 A1.14.24 A1.15.24 A1.00.25 A1.01.25 A1.02.25 A1.03.25 A1.04.25 A1.05.25 A1.06.25 A1.07.25 A1.08.25 A1.09.25

A1.10.25A1.11.25 A1.12.25 A1.13.25 A1.14.25 A1.15.25 A1.00.26 A1.01.26 A1.02.26 A1.03.26 A1.04.26 A1.05.26 A1.06.26 A1.07.26 A1.08.26 A1.09.26 A1.10.26 A1.11.26 A1.12.26 A1.13.26 A1.14.26 A1.15.26 A1.00.27 A1.01.27 A1.02.27 A1.03.27 A1.04.27 A1.05.27 A1.06.27 A1.07.27 A1.08.27 A1.09.27 A1.10.27 A1.11.27 A1.12.27 A1.13.27 A1.14.27 A1.15.27 A1.00.28 A1.01.28 A1.02.28 A1.03.28 A1.04.28 A1.05.28 A1.06.28 A1.07.28 A1.08.28 A1.09.28 A1.10.28 A1.11.28A1.12.28 A1.13.28 A1.14.28 A1.15.28 A1.00.29 A1.01.29 A1.02.29 A1.03.29 A1.04.29 A1.05.29 A1.06.29 A1.07.29 A1.08.29 A1.09.29

A1.10.29A1.11.29 A1.12.29 A1.13.29 A1.14.29 A1.15.29 A1.00.30 A1.01.30 A1.02.30 A1.03.30 A1.04.30 A1.05.30 A1.06.30 A1.07.30 A1.08.30 A1.09.30 A1.10.30 A1.11.30 A1.12.30 A1.13.30 A1.14.30 A1.15.30 A1.00.31 A1.01.31 A1.02.31 A1.03.31 A1.04.31 A1.05.31 A1.06.31 A1.07.31 A1.08.31 A1.09.31 A1.10.31 A1.11.31 A1.12.31 A1.13.31 A1.14.31 A1.15.31 A1.00.32 A1.01.32 A1.02.32 A1.03.32 A1.04.32 A1.05.32 A1.06.32 A1.07.32 A1.08.32 A1.09.32 A1.10.32 A1.11.32A1.12.32 A1.13.32 A1.14.32 A1.15.32 A1.00.33 A1.01.33 A1.02.33 A1.03.33 A1.04.33 A1.05.33 A1.06.33 A1.07.33 A1.08.33 A1.09.33 A1.10.33 A1.11.33 A1.12.33 A1.13.33 A1.14.33 A1.15.33 A1.00.34 A1.01.34 A1.02.34 A1.03.34 A1.04.34 A1.05.34 A1.06.34 A1.07.34 A1.08.34 A1.09.34 A1.10.34 A1.11.34 A1.12.34 A1.13.34 A1.14.34 A1.15.34 A1.00.35 A1.01.35 A1.02.35 A1.03.35 A1.04.35 A1.05.35 A1.06.35 A1.07.35 A1.08.35 A1.09.35 A1.10.35 A1.11.35 A1.12.35 A1.13.35A1.14.35 A1.15.35 A1.00.36 A1.01.36 A1.02.36 A1.03.36 A1.04.36 A1.05.36 A1.06.36 A1.07.36 A1.08.36 A1.09.36 A1.10.36 A1.11.36A1.12.36 A1.13.36 A1.14.36 A1.15.36 A1.00.37 A1.01.37 A1.02.37 A1.03.37 A1.04.37 A1.05.37 A1.06.37 A1.07.37 A1.08.37 A1.09.37 A1.10.37

A1.11.37 A1.12.37A1.13.37 A1.14.37 A1.15.37 A1.00.38 A1.01.38 A1.02.38 A1.03.38 A1.04.38 A1.05.38 A1.06.38 A1.07.38 A1.08.38 A1.09.38

A1.10.38 A1.11.38 A1.12.38 A1.13.38 A1.14.38 A1.15.38 A1.00.39 A1.01.39 A1.02.39 A1.03.39 A1.04.39 A1.05.39 A1.06.39 A1.07.39 A1.08.39 A1.09.39 A1.10.39 A1.11.39A1.12.39 A1.13.39 A1.14.39 A1.15.39 A1.00.40 A1.01.40 A1.02.40 A1.03.40 A1.04.40 A1.05.40 A1.06.40 A1.07.40 A1.08.40 A1.09.40 A1.10.40

A1.11.40 A1.12.40A1.13.40 A1.14.40 A1.15.40 A1.00.41 A1.01.41 A1.02.41 A1.03.41 A1.04.41 A1.05.41 A1.06.41 A1.07.41 A1.08.41 A1.09.41

A1.10.41 A1.11.41 A1.12.41 A1.13.41 A1.14.41 A1.15.41 A1.00.42 A1.01.42 A1.02.42 A1.03.42 A1.04.42 A1.05.42 A1.06.42 A1.07.42 A1.08.42 A1.09.42 A1.10.42 A1.11.42A1.12.42 A1.13.42 A1.14.42 A1.15.42 A1.00.43 A1.01.43 A1.02.43 A1.03.43 A1.04.43 A1.05.43 A1.06.43 A1.07.43 A1.08.43 A1.09.43 A1.10.43

A1.11.43 A1.12.43A1.13.43 A1.14.43 A1.15.43 A1.00.44 A1.01.44 A1.02.44 A1.03.44 A1.04.44 A1.05.44 A1.06.44 A1.07.44 A1.08.44 A1.09.44 A1.10.44 A1.11.44 A1.12.44

A1.13.44 A1.14.44 A1.15.44 A1.00.45 A1.01.45 A1.02.45 A1.03.45 A1.04.45 A1.05.45 A1.06.45 A1.07.45 A1.08.45 A1.09.45 A1.10.45 A1.11.45A1.12.45 A1.13.45 A1.14.45 A1.15.45 A1.00.46 A1.01.46 A1.02.46 A1.03.46 A1.04.46 A1.05.46 A1.06.46 A1.07.46 A1.08.46 A1.09.46 A1.10.46 A1.11.46 A1.12.46

A1.13.46 A1.14.46 A1.15.46 A1.00.47 A1.01.47 A1.02.47 A1.03.47 A1.04.47 A1.05.47 A1.06.47 A1.07.47 A1.08.47 A1.09.47 A1.10.47

A1.11.47A1.12.47 A1.13.47 A1.14.47 A1.15.47 A1.00.48 A1.01.48 A1.02.48 A1.03.48 A1.04.48 A1.05.48 A1.06.48 A1.07.48 A1.08.48 A1.09.48 A1.10.48 A1.11.48 A1.12.48 A1.13.48

A1.14.48 A1.15.48 A1.00.49 A1.01.49 A1.02.49 A1.03.49 A1.04.49 A1.05.49 A1.06.49 A1.07.49 A1.08.49 A1.09.49

A1.10.49 A1.11.49A1.12.49 A1.13.49 A1.14.49 A1.15.49 A1.00.50 A1.01.50 A1.02.50 A1.03.50 A1.04.50 A1.05.50 A1.06.50 A1.07.50 A1.08.50 A1.09.50 A1.10.50 A1.11.50 A1.12.50 A1.13.50 A1.14.50 A1.15.50 A1.00.51 A1.01.51 A1.02.51 A1.03.51 A1.04.51 A1.05.51 A1.06.51 A1.07.51 A1.08.51 A1.09.51 A1.10.51 A1.11.51 A1.12.51 A1.13.51 A1.14.51 A1.15.51 A1.00.52 A1.01.52 A1.02.52 A1.03.52 A1.04.52 A1.05.52 A1.06.52 A1.07.52 A1.08.52 A1.09.52 A1.10.52 A1.11.52 A1.12.52 A1.13.52 A1.14.52 A1.15.52 A1.00.53 A1.01.53 A1.02.53 A1.03.53 A1.04.53 A1.05.53 A1.06.53 A1.07.53 A1.08.53 A1.09.53 A1.10.53 A1.11.53A1.12.53 A1.13.53 A1.14.53 A1.15.53 A1.00.54 A1.01.54 A1.02.54 A1.03.54 A1.04.54 A1.05.54 A1.06.54 A1.07.54 A1.08.54 A1.09.54 A1.10.54 A1.11.54 A1.12.54 A1.13.54 A1.14.54 A1.15.54 A1.00.55 A1.01.55 A1.02.55 A1.03.55 A1.04.55 A1.05.55 A1.06.55 A1.07.55 A1.08.55 A1.09.55 A1.10.55 A1.11.55 A1.12.55 A1.13.55 A1.14.55 A1.15.55 A1.00.56 A1.01.56 A1.02.56 A1.03.56 A1.04.56 A1.05.56 A1.06.56 A1.07.56 A1.08.56 A1.09.56 A1.10.56 A1.11.56 A1.12.56 A1.13.56 A1.14.56 A1.15.56 A1.00.57 A1.01.57 A1.02.57 A1.03.57 A1.04.57 A1.05.57 A1.06.57 A1.07.57 A1.08.57 A1.09.57 A1.10.57 A1.11.57A1.12.57 A1.13.57 A1.14.57 A1.15.57 A1.00.58 A1.01.58 A1.02.58 A1.03.58 A1.04.58 A1.05.58 A1.06.58 A1.07.58 A1.08.58 A1.09.58 A1.10.58 A1.11.58 A1.12.58 A1.13.58 A1.14.58 A1.15.58 A1.00.59 A1.01.59 A1.02.59 A1.03.59 A1.04.59 A1.05.59 A1.06.59 A1.07.59 A1.08.59 A1.09.59 A1.10.59 A1.11.59 A1.12.59 A1.13.59 A1.14.59 A1.15.59 A1.00.60 A1.01.60 A1.02.60 A1.03.60 A1.04.60 A1.05.60 A1.06.60 A1.07.60 A1.08.60 A1.09.60 A1.10.60 A1.11.60A1.12.60 A1.13.60 A1.14.60 A1.15.60 A1.00.61 A1.01.61 A1.02.61 A1.03.61 A1.04.61 A1.05.61 A1.06.61 A1.07.61 A1.08.61 A1.09.61 A1.10.61 A1.11.61 A1.12.61 A1.13.61A1.14.61 A1.15.61 A1.00.62 A1.01.62 A1.02.62 A1.03.62 A1.04.62 A1.05.62 A1.06.62 A1.07.62 A1.08.62 A1.09.62 A1.10.62 A1.11.62A1.12.62 A1.13.62 A1.14.62 A1.15.62

(b)

Figure 1. (a)

Cutaway diagram of the ATLAS liquid argon calorimeters[5].

(b)

Visualization of a

Geant4 event, here in the FCal[6, p. 88]. The color of a cell hit indicates the deposited energy.

each calorimeter hit are roughly 450 ns long. A higher number of interactions means an

in-crease of

pileup

e

ff

ects, where pulses overlap and hinder an accurate energy reconstruction.

Techniques that reduce pileup e

ff

ects often exacerbate electronics noise and vice versa, so an

optimum has to be found.

In order to minimize the combined impact of electronics noise and pileup e

ff

ects, the LAr

upgrade requires a detailed simulation of the new readout electronics. This simulation needs

to be modular so that new filtering algorithms can be implemented and studied in a timely

manner. It also needs to be flexible enough that simulated electronics can be replaced with

data from real, prototype components as they become available over time.

This simulation is provided by the AREUS framework, which is described in Section 2. In

Section 3, implementation details on AREUS’ architecture are given. In Section 4, a custom

smart pointer class as an example for the optimization considered in AREUS is presented. In

Section 5, various approaches to apply parallelization to AREUS are considered and weighed

against each other. Finally, a summary is given in Section 6.

2 The AREUS Framework

AREUS[7–9] is a program written in C

++

[10]. It directly depends on the Boost[11] and

ROOT[12] libraries for common functionality and Git[13] for version control. It can be built

using the GNU build system[14–16] or CMake[17]. Other dependencies are bundled into the

source repository, either literally or as Git submodules:

•

ExprTK[18] for parsing of mathematical expressions,

•

Google Test[19] for unit testing,

•

Splines[20] for interpolation between data points,

•

VDT[21] for vectorized implementations of certain mathematical functions.

In addition, it provides a Python[22] and an ATHENA[23] package to automate certain tasks

such as input file generation.

AREUS’ input are files derived from detailed Geant4[24, 25] simulations of the full

AT-LAS detector. These files provide a series of

events

, where each event contains a list of

0

100

200

300

400

500

600

700

Time [ns]

Amplitude

[a.u.]

Cell hit

Analog pulse

Digitized pulse

Reconstructed hit

Figure 2.

Simulation of the readout chain in AREUS. See the text for a description.

0

2

4

6

8

10

12

14

16

18

20

[GeV]

in

E

0

2

4

6

8

10

12

14

16

18

20

[GeV]

out

E

0

50

100

150

200

250

300

350

400

Figure 3.

Exemplary output of AREUS, a

correlation plot between

E

in

, the energy of

the cell hits, and

E

out

, the corresponding

reconstructed energy. This takes the

filter-induced delay into account. The

color represents the number of entries in

each bin. The bins right at the X-axis from

0 to 3 GeV show undetected events.

detector cells that have been hit with a certain amount of energy (cf. Figure 1b). AREUS

simulates each of the calorimeter cells separately and forwards each hit to the corresponding

cell.

Figure 2 illustrates how AREUS processes each cell hit. It calculates the pulse induced

by the hit, accounting for e

ff

ects such as nonlinearity of the amplification stage or electronics

noise that follows an arbitrary spectral density. This pulse is then overlaid with any pulses

from previous hits in order to simulate pileup e

ff

ects

1

. After this, the pulse is digitized both in

time and amplitude, considering quantization noise and the ADC’s finite range and resolution.

The samples are subsequently sent to the digital filters, which can be chained by the user in

an arbitrary manner. The result of this filter chain should be close to the original series of hits,

except for a small, filter-induced delay. AREUS compares these two sequences on-line and

produces a collection of diverse histograms detailing the performance of the filter chain. Two

examples of these histograms are shown in Figure 3 and Figure 4. Figure 5 shows an analysis

that aggregates the results of multiple AREUS runs and has been used in the technical design

report for the LAr Phase-II upgrade[4, p. 32].

3 Implementation of AREUS

The modularity of AREUS is achieved through the Observer pattern[26, pp. 293–303] as well

as a strict separation of concerns[27, p. 61]. The core library, called

Common

, contains only

utility classes used by all other libraries as well as message classes used to interface the other

1

_{To do so, cells need to retain their state between events for a duration at least as long as an electronics pulse:}

10

1

₁₀

2

₁₀

3

₁₀

4

10

−

4

10

−

3

10

−

2

Energy [GeV]

σ

E

/

E

[1/GeV]

Figure 4.

Exemplary output of AREUS. It

shows the energy reconstruction’s relative

resolution over the energy. This graph was

part of a qualitative study of dynamic gain

selection, where pulses from di

ff

erent gain

levels have slightly di

ff

erent shapes. The

circled section shows a transition e

ff

ect

caused by sample mixing of pulses from

di

ff

erent gain levels. This e

ff

ect does not

occur at 2 TeV because the gain switching

there employs a di

ff

erent strategy.

= 0

µ

µ

= 20

µ

= 80

µ

= 140

µ

= 200

0

20

40

60

80

100

120

140

Total Noise [MeV]

= 8ns

τ

= 13ns

τ

= 18ns

τ

ATLAS Simulation

= 0.5

η

EMB Middle Layer at

Figure 5.

Exemplary analysis of a single

calorimeter cell, done using AREUS[4, p. 32].

This shows the total noise due to both

electronics noise and pileup e

ff

ects. Each of

the points represents an individual AREUS

run. Two parameters have been varied:

µ

, the

average number of proton–proton interactions

as a measure of pileup e

ff

ects; and

τ

, the RC

time constant of the bandpass filter that is

employed in the LAr analog electronics.

libraries (cf. Figure 6). The other libraries communicate with each other using the message

classes from

Common

(cf. Figure 7). The advantage of this is twofold:

1. the libraries do not need to know about each other, only about

Common

;

2. the user can combine objects from these libraries to arbitrary chains (and even graphs)

as long as the message types sent between any two connected objects match.

Common

•

utility classes

•

NextTick

class

•

CellHit

class

•

. . .

Event Loop

•

helper classes

•

sends

NextTick

Hit Sample

•

receives

NextTick

•

sends

CellHit

uses

Figure 6.

Visualization of AREUS’ modular architecture. The core library

Common

contains utility

classes and the message classes that other libraries use for communication. The other libraries only

know about the message classes, but not about each other.

10

1

₁₀

2

₁₀

3

₁₀

4

10

−

4

10

−

3

10

−

2

Energy [GeV]

σ

E

/

E

[1/GeV]

Figure 4.

Exemplary output of AREUS. It

shows the energy reconstruction’s relative

resolution over the energy. This graph was

part of a qualitative study of dynamic gain

selection, where pulses from di

ff

erent gain

levels have slightly di

ff

erent shapes. The

circled section shows a transition e

ff

ect

caused by sample mixing of pulses from

di

ff

erent gain levels. This e

ff

ect does not

occur at 2 TeV because the gain switching

there employs a di

ff

erent strategy.

= 0

µ

µ

= 20

µ

= 80

µ

= 140

µ

= 200

0

20

40

60

80

100

120

140

Total Noise [MeV]

= 8ns

τ

= 13ns

τ

= 18ns

τ

ATLAS Simulation

= 0.5

η

EMB Middle Layer at

Figure 5.

Exemplary analysis of a single

calorimeter cell, done using AREUS[4, p. 32].

This shows the total noise due to both

electronics noise and pileup e

ff

ects. Each of

the points represents an individual AREUS

run. Two parameters have been varied:

µ

, the

average number of proton–proton interactions

as a measure of pileup e

ff

ects; and

τ

, the RC

time constant of the bandpass filter that is

employed in the LAr analog electronics.

libraries (cf. Figure 6). The other libraries communicate with each other using the message

classes from

Common

(cf. Figure 7). The advantage of this is twofold:

1. the libraries do not need to know about each other, only about

Common

;

2. the user can combine objects from these libraries to arbitrary chains (and even graphs)

as long as the message types sent between any two connected objects match.

Common

•

utility classes

•

NextTick

class

•

CellHit

class

•

. . .

Event Loop

•

helper classes

•

sends

NextTick

Hit Sample

•

receives

NextTick

•

sends

CellHit

uses

Figure 6.

Visualization of AREUS’ modular architecture. The core library

Common

contains utility

classes and the message classes that other libraries use for communication. The other libraries only

know about the message classes, but not about each other.

The second point is most important for the digital-filter chain, which may consist of many

small and independent algorithms using the same message class. Figure 8 shows the exact

procedure through which AREUS connects objects from di

ff

erent libraries.

Event Loop

Signal Sample

Pile-up Sample

Cell Map

Filter Analysis

EventListener

<

EndOfTick

>

Figure 7.

The Observer pattern as implemented in AREUS. Execution starts in a root object that loops

over events. On each event, it notifies its observers, which recursively notify their observers. Pre- and

post-processing of events is done via inheritance from the special class

EventListener

, which is backed

by a global registry object.

Factory A

•

property 1

•

property 2

•

. . .

Factory B

•

property 1

•

property 2

•

. . .

configs

(a)

Event Loop

Object A

Object B

(b)

notifies

(c)

notifies

notifies

Library A

Library B

Figure 8.

The three phases in which AREUS runs.

(a)

The configuration file is parsed and each

module’s factory reads its properties from it. A subject–observer chain between factories is built.

(b)

The event loop tells the factories to recursively produce objects. A subject–observer chain between the

produced objects is built.

(c)

For each event, the event loop recursively notifies all observers.

4 AREUS Smart Pointers

Due to the inter-library communication in AREUS, many of its runtime objects do not have

a clear hierarchy of ownership and reference counting has to be used throughout the code

base. To this end, AREUS uses a custom smart-pointer system based on two pointer classes:

CSharedPtr

and

CPersistentPtr

. Though unconventional, this design decision has

sev-eral advantages over using smart pointers from Boost or the standard library:

•

AREUS’ smart pointers are

intrusive

, i.e. the reference count is a part of the object that is

being reference-counted. This choice avoids allocation of many small memory blocks and

so decreases the chance of heap fragmentation.

•

The reference counts are regular integers if AREUS is compiled for single-threaded usage

(cf. Section 5 for details) and atomic integers otherwise. In contrast, the standard-library

class

std::shared_ptr

always uses atomic integers

2

_{. Accessing atomic integers usually}

is slightly slower, though on some modern architectures, this additional cost may avoided

via

hardware lock elision

[28].

5 Parallelization Opportunities

By default, AREUS runs in single-threaded mode, taking a set of input files and producing

a set of output files. On modern machines with multiple CPUs, this is often wasteful and

makes run time longer than necessary. Single-CPU clock speeds have stopped increasing

more than a decade ago and most gains in computing nowadays result from parallelization

and vectorization[29, p. 9].

There are several methods to make use of more than one core per CPU with AREUS,

each with its advantages and disadvantages:

Multiprocessing:

This is the simplest method of parallelization and requires no changes to

the program at all. It simply means to split the user’s data into batches and run one instance

of AREUS on each of them in parallel. When comparing di

ff

erent setups, one may also run

multiple instances of AREUS on the same data, each using a di

ff

erent configuration file. One

of the disadvantages of this method is that the user has to do additional data pre- and

post-processing. Another one is that computing time is wasted if the run time of parallel AREUS

configuration varies a lot – most cores will be idle while the user waits for the longest run to

finish.

Vectorization:

Another method is data-level parallelism, i.e. using a CPU’s SIMD

instruc-tions (single instruction on multiple data), which are available on virtually all modern

pro-cessors. This technique may speed up array-based algorithms by a factor between two and

four. It can be used on a small scale and replace unvectorized algorithms without

modify-ing the overarchmodify-ing architecture of the program. In certain situations, optimizmodify-ing compilers

can even apply it automatically (

auto-vectorization

). AREUS makes use of it where possible

e.g. by using VDT[21] and by writing algorithms in a way that facilitates autovectorization.

However, this method is inherently limited in scope and cannot, for example, parallelize

in-dependent portions of the simulation.

Task-level multithreading:

AREUS’

Thread

library, which can be enabled with a

compile-time flag, allows the user to parallelize the calorimeter cell simulation. This is possible

be-cause each of these simulations is independent of the others. However, no appreciable

perfor-mance gain could be observed when introducing this library – possibly because the programs

was not profiled su

ffi

ciently and the actual bottlenecks lie elsewhere. But even if there was

a gain, the considerable number of users that simulate only a single cell per run would not

profit from this optimization.

Stage-level multithreading:

Another approach, that has yet to be investigated for AREUS,

would go a step further and also parallelize the di

ff

erent stages of AREUS’ processing chain.

AREUS’ architecture already is well-suited to such an approach due to its extensive use of

shared ownership and message passing.

6 Conclusions

The High-Luminosity LHC is expected to start operation in 2026 and the electronics of

AT-LAS’ liquid-argon calorimeters will be upgraded for this purpose in two phases. AREUS

is a valuable tool that has been applied successfully in technical design studies for both the

Phase-I upgrade[3, pp. 97–103] and the Phase-II upgrade[4, pp. 31–36]. It faithfully

simu-lates pileup e

ff

ects and allows developers to quickly implement and investigate digital-filter

2

_Technically,

_{std::shared_ptr}

_{only uses atomic reference counters when compiled with the}

_-pthread

_flag.

5 Parallelization Opportunities

By default, AREUS runs in single-threaded mode, taking a set of input files and producing

a set of output files. On modern machines with multiple CPUs, this is often wasteful and

makes run time longer than necessary. Single-CPU clock speeds have stopped increasing

more than a decade ago and most gains in computing nowadays result from parallelization

and vectorization[29, p. 9].

There are several methods to make use of more than one core per CPU with AREUS,

each with its advantages and disadvantages:

Multiprocessing:

This is the simplest method of parallelization and requires no changes to

the program at all. It simply means to split the user’s data into batches and run one instance

of AREUS on each of them in parallel. When comparing di

ff

erent setups, one may also run

multiple instances of AREUS on the same data, each using a di

ff

erent configuration file. One

of the disadvantages of this method is that the user has to do additional data pre- and

post-processing. Another one is that computing time is wasted if the run time of parallel AREUS

configuration varies a lot – most cores will be idle while the user waits for the longest run to

finish.

Vectorization:

Another method is data-level parallelism, i.e. using a CPU’s SIMD

instruc-tions (single instruction on multiple data), which are available on virtually all modern

pro-cessors. This technique may speed up array-based algorithms by a factor between two and

four. It can be used on a small scale and replace unvectorized algorithms without

modify-ing the overarchmodify-ing architecture of the program. In certain situations, optimizmodify-ing compilers

can even apply it automatically (

auto-vectorization

). AREUS makes use of it where possible

e.g. by using VDT[21] and by writing algorithms in a way that facilitates autovectorization.

However, this method is inherently limited in scope and cannot, for example, parallelize

in-dependent portions of the simulation.

Task-level multithreading:

AREUS’

Thread

library, which can be enabled with a

compile-time flag, allows the user to parallelize the calorimeter cell simulation. This is possible

be-cause each of these simulations is independent of the others. However, no appreciable

perfor-mance gain could be observed when introducing this library – possibly because the programs

was not profiled su

ffi

ciently and the actual bottlenecks lie elsewhere. But even if there was

a gain, the considerable number of users that simulate only a single cell per run would not

profit from this optimization.

Stage-level multithreading:

Another approach, that has yet to be investigated for AREUS,

would go a step further and also parallelize the di

ff

erent stages of AREUS’ processing chain.

AREUS’ architecture already is well-suited to such an approach due to its extensive use of

shared ownership and message passing.

6 Conclusions

The High-Luminosity LHC is expected to start operation in 2026 and the electronics of

AT-LAS’ liquid-argon calorimeters will be upgraded for this purpose in two phases. AREUS

is a valuable tool that has been applied successfully in technical design studies for both the

Phase-I upgrade[3, pp. 97–103] and the Phase-II upgrade[4, pp. 31–36]. It faithfully

simu-lates pileup e

ff

ects and allows developers to quickly implement and investigate digital-filter

2

_Technically,

_{std::shared_ptr}

_{only uses atomic reference counters when compiled with the}

_-pthread

_flag.

However, this flag is mandatory when linking with ROOT, even though AREUS itself is fully single-threaded.

algorithms through its flexible and modular design. Its biggest challenge with respect to

com-puting is handling parallelization in the face of strong data dependencies and a deep stack of

function callbacks.

The most promising approach to improve simulation speed is data-level parallelism

through algorithms that facilitate automatic vectorization by the compiler. Another, more

radical approach is parallelizing the distinct simulation stages through multithreading.

This work was supported in part by the German Bundesministerium für Bildung und Forschung

(BMBF) within the FIS research grant

05H15ODCA9

.

Copyright 2018 CERN for the benefit of the ATLAS Collaboration. Reproduction of this article or

parts of it is allowed as specified in the CC-BY-4.0 license.

References

[1] G. Apollinari, I. Béjar Alonso, O. Brüning, P. Fessia, M. Lamont, L. Rossi, L. Tavian,

High-Luminosity Large Hadron Collider (HL-LHC): Technical Design Report V. 0.1

,

CERN Yellow Reports: Monographs (CERN, Geneva, 2017),

https://cds.cern.

ch/record/2284929

[2] ATLAS Collaboration, JINST

3

, S08003 (2008)

[3] ATLAS Collaboration, Tech. Rep. CERN-LHCC-2013-017, ATLAS-TDR-022, CERN,

Geneva (2013),

https://cds.cern.ch/record/1602230

[4] ATLAS Collaboration, Tech. Rep. CERN-LHCC-2017-018, ATLAS-TDR-027, CERN,

Geneva (2017),

https://cds.cern.ch/record/2285582

[5] J. Pequenao,

Computer generated image of the atlas liquid argon

(2008),

https://

cds.cern.ch/record/1095928

[6] ATLAS Collaboration, Tech. Rep. CERN-LHCC-2015-020, LHCC-G-166, CERN,

Geneva (2015),

https://cds.cern.ch/record/2055248

[7] A. Ambler, J.P. Grohs, M. Hils, P. Horn, A. Jansen, T. Kwan, N. Madysa, S. Stärz,

Areus: Atlas readout electronics upgrade simulation

, version 2.6 (2013–2018),

soft-ware,

https://gitlab.cern.ch/AREUS/AREUS

[8] S. Stärz,

Energy Reconstruction and high-speed Data Transmission with FPGAs for the

Upgrade of the ATLAS Liquid Argon Calorimeter at LHC

(TU Dresden, 2015), thesis

presented on 19 May 2015,

https://cds.cern.ch/record/2030122

[9] J.P. Grohs,

Simulation of the upgraded Phase-1 Trigger Readout Electronics of the

Liquid-Argon Calorimeter of the ATLAS Detector at the LHC

(TU Dresden, 2015),

thesis presented on 29 Feb 2016,

https://cds.cern.ch/record/2135931

[10] ISO

/

IEC 14882:2014,

Programming language C

++

(2014),

https://isocpp.org/

[11] D. Abrahams, B. Dawes et al.,

The Boost C++

libraries

, version 1.62 (1999–2018),

software,

https://www.boost.org/

[12] F. Rademakers et al.,

Root

, version 6.12

/

02 (2018), software,

https://doi.org/10.

5281/zenodo.1292566

[13] J. Hamano, L. Torvalds et al.,

Git – fast, scalable, distributed revision control system

(2005–2018), software,

https://git-scm.com/

[14] D.J. MacKenzie, A. Demaille,

autoconf

, version 2.69 (2012), software,

https://www.

gnu.org/software/autoconf/

[15] T. Tromey, A. Duret-Lutz,

automake

, version 1.13.4 (2013), software,

https://www.

gnu.org/software/automake/

[17] Kitware, Inc. et al.,

CMake

, version 3.1 (2000–2018), software,

https://cmake.org/

[18] A. Partow,

The C

++

mathematical expression toolkit library

(2012–2018), software,

http://www.partow.net/programming/exprtk/index.html

[19] The Google Test Project,

Google Test

, version 1.8 (2016), software,

https://github.

com/google/googletest

[20] E. Bertolazzi,

The Splines C++

library

, version 1.0 (2013), software,

https://

github.com/ebertolazzi/Splines

[21] D. Piparo, V. Innocente, T. Hauth, Journal of Physics: Conference Series

513

, 052027

(2014)

[22] Python Software Foundation,

Python language reference

, version 2.7 (1990–2018),

software,

https://www.python.org/

[23] P. Calafiura, W. Lavrijsen, C. Leggett, M. Marino, D. Quarrie,

The Athena Control

Framework in Production, New Developments and Lessons Learned

, in

Computing in

High Energy Physics and Nuclear Physics

(2005), p. 456,

https://cds.cern.ch/

record/865624

[24] S. Agostinelli, J. Allison, K. Amako, J. Apostolakis, H. Araujo, P. Arce, M. Asai,

D. Axen, S. Banerjee, G. Barrand et al., Nuclear Instruments and Methods in Physics

Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment

506

, 250 (2003)

[25] J. Allison, K. Amako, J. Apostolakis, P. Arce, M. Asai, T. Aso, E. Bagli, A. Bagulya,

S. Banerjee, G. Barrand et al., Nuclear Instruments and Methods in Physics Research

Section A: Accelerators, Spectrometers, Detectors and Associated Equipment

835

, 186

(2016)

[26] J. Vlissides, R. Helm, R. Johnson, E. Gamma,

Design patterns: Elements of reusable

object-oriented software

(Addison-Wesley, 1994), ISBN 0-201-63361-2

[27] E.W. Dijkstra, in

Selected writings on Computing: A Personal Perspective

(Springer-Verlag, New York, 1972), pp. 60–66,

https://www.cs.utexas.edu/users/EWD/

transcriptions/EWD04xx/EWD447.html

[28] R.M. Yoo, C.J. Hughes, K. Lai, R. Rajwar, in

Proceedings of the International

Confer-ence on High Performance Computing, Networking, Storage and Analysis

(ACM, New

York, 2013), pp. 19:1–19:11,

https://doi.acm.org/10.1145/2503210.2503232