Reliability Decisions

As already mentioned, the devices of the CBM DAQ system are exposed to heavy ionizing radiation (see 2.3.4). The wish for total error-free hardware, partially located in an environment very close to the collision zone seems practically utopian. But reliability has to be ensured to a certain point, which leads to the discussion which faults are tolerable and which faults are critical. Not in every case it is necessary to deliver a radiation hardened system, but rather a radiation tolerant

4.2 CBMnet Version 3.0

system like in [26]. In chapter 2 a particular declaration of fault types has been made already and the impact of radiation has been discussed. In chapter 3 a detailed overview about the disadvantages of the former CBMnet protocol implementation has been given and an explanation, that total reliability is not required at all, as the detectors have finite efficiency anyway. This led to the following design decisions with respect to the new CBMnet implementation. Regarding the data path, reliability is not necessary until the data loss stays reasonably small. The highest SER is expected for the STS detector, with an SEU every 428 s for a four-lane CBMnet LVDS core. Within this period, the raw bit throughput is 107 GByte for the whole interface. A rough calculation of data loss is given in table 4.1. The real payload is only a fraction of a CBMnet packet, and the relative value depends on the packet length and the line coding. It is assumed, that one SEU does not damage more than one packet. Thus, the data loss has to be considered in relation to the absolute payload transferred within the mean time between failures. Calculation shows, even in case the link is fully utilized, the resulting data loss is very negligible. Obviously, the read-out chain consists of more devices than one CBMnet core. Also other components and plug-ins are operated. But even multiplied by a factor of hundred or thousand, data loss does still not matter at all, compared to the efficiency of the detectors. Hence, there is no need for redundancy to be implemented in the protocol. Only a checksum is required to detect corrupted messages. Regarding slow control messages, their reliability is ensured with timing redundancy on a higher layer. If inequality between the payload and the checksum is detected, the slow control operation is not being executed and a NACK is sent back to the origin. In the same manner, a unsuccessful CRC indicates, that the value read back from a device may be wrong. Hence, the PUT/GET command has to be performed again. As slow control messages appear very infrequently, chances are small, that a message is corrupted at all.

The reliability of the control path in a module is far more important than an error-free data path. SEUs in the control path lead to wrong behavior of control logic, like FSMs, FIFOs and thereby higher network layer functions, like routing, addressing and traffic control. Therefore, soft errors in the control path have to be prevented or corrected within a very short time frame, so that only very little data loss occurs. Furthermore, errors in the control path need to be detected immediately, as otherwise the whole network is locked and at worst, data have to be dropped or can not be recorded anymore, due to congestion. The reasonable

Chapter 4 The CBMnet Protocol Upgrade

Byte Content Payload normalized 8b/10b efficiency Payload effective Payload GByte Data loss

Short Packet 8 Payload 57.1 % 80 % 45.7 % 48.9 0.0000000163 % 6 Framing

Long Packet 64 Payload 91.4 % 80% 73.1 % 78.3 0.0000000817 %

6 Framing

Buffer purge 512 Payload 100 % - 100 % 48.9 0.000001047 %

Link lock - - - 0.00117 %

Table 4.1: Expected calculated data loss of detector data due to an SEU every

428 seconds in the data path or control patch of a four-lane LVDS CBMnet core running at 2GBit/s. Additionally, the data drop during a link loss is estimated.

buffer size of the implemented SRAMs in one link port lane is at least 256 x 16 bit, because all FPGAs provide enough resources of these BRAM sizes. Also in ASICs this is a reasonable SRAM size and ensures flexible storage of CBMnet packets in case of stopped data flow. Assuming a long message effective payload like in table 4.1 and a link speed of 2.5 Gbit/s, buffers will be occupied after 1.12 µs. Thus, the control path needs to be corrected latest by the end of this period to avoid dropping of messages. Obviously, a buffer with 256 entries depth can consume the payload of eight long CBMnet packets. Thus, in case of an error in the control path, which leads to a purge of the whole buffer, the data loss compared to an error in the data path is actually increased by a factor of up to around ten, but still does not come into effect. The factor slightly differs, as the buffer logic stores framing characters parallel to the data. Thus, no memory storage is completely utilized by payload.

Finally, most severe errors are configuration faults, as they can lead to wrong synchronization settings, alteration of electrical calibration or shutdown of a device. Unfortunately the detection of wrong timing settings is very complicated and if reliability is not ensured, a corrupted time-stamp counter would not be noticed at all, resulting in wrong event interpretation. The configuration settings of the high-speed serial transceivers are actually the most sensitive part within the CBMnet protocol. Wrong tuning or calibration, as well as alignment, can disturb the transceiver’s operation in a way, that a whole link reinitialization is

4.2 CBMnet Version 3.0

necessary. While the initialization of the serial transceiver is done within 1 µs, the initialization of the physical coding sublayer, word-alignment and deterministic latency adjustments can last up to 5 ms. Within this time, recorded data have to be dropped, as soon as buffers are occupied. Moreover, in case of a link breakdown, the whole sub-tree needs to be resynchronized. Therefore nearly all recorded data have to be dropped, since they can not correlated correctly. Depending on the location of the device within the read-out chain, data loss can happen to a very large extend. Therefore, configuration settings need to be protected accurately against corruption. When running the CBMnet cores in an FPGA design, scrubbing has to be used to secure the FPGA configuration.