Data Quality Monitoring. workshop

(1)

Data Quality

Monitoring

(2)

•

What this presentation is not

•

What it is and how it is organized

 _{Definition of DQM}

 _{Overview of systems and frameworks}

 _{Specific chosen aspects}

oData collection, Storage, Visualization, analyses

 _Operations

 _{Qualitative assessment & discussion}

 _Future

(3)

•

Online feedback on the quality of data

•

Make sure to take and record high quality

data

•

Use the data taking time and the precious

bandwidth in an optimal way

•

Identify and solve problem(s) early

Data Quality Monitoring (1)

(4)

•

Data Quality Monitoring (DQM) involves

 _{Online gathering of data}

 _{Analysis by user-defined algorithm(s)}

 _{Storage of monitoring results}  _{Visualization}

Data Quality Monitoring (2)

Dat afl ow Results (histos, values…) Storage Analysis Data samples Vizualization

(5)

•

Requirements

 _{Range from formal documents to oral}

tradition

 _Changing_difficulties

 _{No interference with data taking}

 _{Low latency update during runs}

 _{Centralized results}

 _{Modular, fast update of users code}

Data Quality Monitoring (3)

(6)

• DQParameter  _{An histo} • DQRegion  A subset of detector • DQAlgorithm  A specific analysis • DQResult  _{A color assessment}

Overview: ATLAS

Dat afl ow Web app Configura tion DB DQResults DQMF DQParam DQRegion

• « Generic, yet flexible and

nicely scalable »

• Shared online/offline and

by ~10 independent communities • C++ – Qt – ROOT – XML – JS – CGI - Python MT MT MT 150MTs 50MTs Infor-mation Services Histos 70000 Events Histos EMON 1Hz/MT 10-15%

(7)

• DQM framework part of reconstruction framework

• SMPS allows for trigger path selection

• 1 DQM app per subsystem

 Re-run reco according to needs

 _{Run certain analysis}

• C++, Python, Java, Oracle DB

Overview: CMS

12/03/2013 Barthélémy von Haller, ALICE, CERN

Web Server

Collector Memory _&

Storage WEB GUI • Monitoring element  _Metadata  Reference histo  Quality • Web based

 _{HTTP for data exchange}

 Web UI only WS WBM Condi-tion DB Qua nt iti e s V ia DI P Dat afl ow Events Histos SMPS Summary Shifter Experts DQM DQM DQM 1 app/subsys 6/26 TCP/IP Mon itoring El e ment s 70000 10%

(8)

•

DIM

 _Network  _RPC

•

3 distinct

histo-gramming places

Overview: LHCb

Da ta flows Histogram Presenter Event data From HLT Event data Reco Monitoring

Farm _SavesetsHisto

Automatic analysis Control System Alarms A larms • C++ - ROOT • Monitoring on 70-100% of data

• Auto. analysis every 15

minutes • Automatic actions ! Aggregated histos ~100 ~10 0 Hi st os 20 x500 5kHz 3-5kHz (>70%) 50Hz Alarms

(9)

• AMORE framework

 C++ - ROOT

 _{Plugin architecture}  _{Notifications via DIM}  _{MySQL database}

• Plugins developed in 20

institutes worldwide

Overview: ALICE

Dat

afl

ow

• Histograms or values also

retrieved in other systems

• MonitorObject

 Metadata (run, quality, expert-shifter flag)

 _{ROOT base type encapsulated}

(TH1, TObject, scalar, …)

Events

Monito

-ring _{Expert GUIs}

Generic GUI eLogbook Alarms Orthos Agent Agent Agent ~40 agents Data Pool MonitorObjects ~11500 /min 8/26 HLT DCS Trigger Histos Values …

(10)

•

Main input to DQM

 _{Events (all) and sub-events (ALICE, ATLAS)}

 _{Histograms or values coming from other}

systems, esp. HLT.

•

Challenges

 _{Bandwidth issues: monitoring processes do}

events selection themselves instead of using a proxy (ATLAS, ALICE)

 _{Merge of all histograms from HLT (LHCb,}

CMS)

(11)

•

Most typical analysis include

 _{Check for dead channels (holes in}

distribution)

 _{Check for noisy channels (peaks in}

distribution)

 _{Check for differences with a reference}

 _{Check for non-empty error histograms}

•

Also

 _{Check for infrastructure issues}  _{Check for reconstruction issues}

 _{Check for trigger/dead time issues}

Data analysis (1)

(12)

•

Offline

-

Online

 _{Same framework ?}  _{Same analysis ?}

•

ATLAS

•

CMS

•

LHCb

•

ALICE

Data analysis (2)

(13)

•

Online Storage

 _{In-memory (ATLAS, CMS, LHCb)}

 _{Database (ALICE)}

•

Online recent history

 _{Short-term, detailed (all)}

•

Archiving

 _{Long-term per run/per time interval (all)}

•

Does long-term archiving make sense ?

Is it still online DQM ? Rather offline QA ?

Storage

(14)

•

Desktop GUI (all but CMS)

 _{C++, Qt or ROOT}

•

Very similar display showing online

•

Also past objects (LHCb)

•

Summary panel (ATLAS)

•

Easy customization of

the layout

•

Quality and alarms

Visualization (Desktop)

Object(s) display Objects Tree Extra information on the object(s) Menu bar Tool bar

(15)

Visualization (Desktop)

(16)

• How can experts access DQM results ?

• Remote ssh, interactive access to control room

 LHCb: open to collaboration

 ATLAS: provides tokens for a few hours access + replica

 CMS: closed but has a collaboration replica  ALICE: closed

• Web GUI

 Images and ROOT objects download (ALICE)  Generated static set of pages (ATLAS)

 Full-fledged web application (CMS)

(17)

Visualization (Web, ALICE)

(18)

(19)

Visualization (Web, CMS)

• Remote monitoring

 Layouts and render plugins

 _{Dynamic zoom and}

drawing options

 Caching

 Home-made DB for quick retrieval histograms

(20)

•

A typical DQM shifter

 _{Check alarms}

 _{Go through plots layouts (LHCb, ALICE,}

ATLAS)

 _{Find information in a Wiki or directly the GUI}  _{Contact experts / shifters (if present)}

 _{Inform the shift leader}

(21)

• Is it worth it ? Yes !

• ATLAS

• Transition Radiation Tracker timing problem spotted

• CMS

• Wrong DAQ configuration of SST detector used during

beam "ramp" _{audio alarm triggered by DQM}_{fix before}

stable beam was declared.

• LHCb

• Bad mass plot -> anomaly -> inverse value of the magnetic field used by the HLT

• ALICE

• HLT started compressing the TPC laser events 

recons-truction problems  DQM shifter spotted anomalies fix

Operations (2)

(22)

•

Personal

hit-list

 _{ATLAS: scalability, service-oriented}

 _{CMS: interactive and dynamic web interface}

 _{LHCb: automatic actions}

 _{ALICE: versatility}

 _General:

oFlexibility and reusability

(23)

•

Biggest challenges

 _{ATLAS: initial display was in Java and}

couldn’t cope with the number of histograms (70k)

 _{CMS: handling huge quantities of histograms}

in the web gui and yet be responsive

 _{LHCb: nothing dramatic, summing up HLT}

histograms with data rates of the order of a LEP experiment

 _{ALICE: ensure stability despite of the}

multiple contributions

Qualitative review (2)

(24)

•

Could we have used the same system ?

 _{4 efficient and scalable DQM}

 _{Similar architecture and technologies}  _{I believe the answer is: Yes…}

 _{… up to a certain extent}

•

If not the framework, maybe have the

same (Web) UI ?

•

Could we at least share more ?

 _{(Regular) common meetings ?}

(25)

•

LS1

 _{Algorithm and thresholds adjusting on run}

conditions (ALICE, ATLAS)

 _{Adapt SW to dependencies changes (}_{LHCb, CMS}₎  _{Adapt framework for multi-core multi-thread}

environment (CMS, ALICE)

 _{Web GUI and tools (}_{ALICE, [LHCb]}₎  _{Full archiving (}_ATLAS₎

 _{Proxy monitoring (}_ALICE₎

•

LS2

 _{Complete rewrite along with online-offline}

framework (ALICE)

Future

(26)

•

Much similarities

 _{General LHC DQM framework ?}

 _{Trans-experiments DQM meetings ?}

•

DQM in all 4 experiments

 _{Work well and up to the demand}

 _{Proved to be of great importance}

•

Future

 _{DQM is crucial, still often considered late}  _{Today is an opportunity to attack it early !}

(27)

A big thanks to:

Markus Frank

Clara Gaspar

Serguei Kolos

Marco Rovere

Questions and discussion

(28)

(29)

•

Remote monitoring

 _{Layouts and render plugins}

 _{Dynamic zoom and drawing options}

 _Caching

Visualization (Web, CMS)

Web Server Homemade DB (ROOT files) Web Browser Javascript 6:parsing 7:HTML+CSS generation Python & C++

4:Image & JSON generation ?

1:Async

Request 2:Ask data

3:Monit oring elemen t 5:JSON Histo renderer Shared memory Collector Li ve A rchi v e 28/26

(30)

Visualization (Web, ATLAS)

Web Server Information Service Web Browser JS parsing CGI Request Ask XML XML ATLAS

(31)

•

Services approach

•

C++ – Qt – ROOT – XML – JS

•

Framework DQMF based on DQMCore

Overview ATLAS

• DQParameter  An histo • DQRegion  A subset of detector • DQAlgorithm  A specific analysis • DQResult  A color assessment  [New histo] 30/26

(32)

Overview: ATLAS

Dat afl ow Online Histo- gramming Infor-mation Services MT Data samples Display GUI DQMF EMON _MT MT Histos Histos Histos Web app DQResults Configura tion DB DQParam DQAlgo