Hardware platform

Besides the algorithm, specific hardware for the CMS track trigger is being developed by the groups belonging to the AM approach. The here described hardware components are developed for the review in December 2016. The goal of this system is not primarily to achieve the full performance needed by the CMS track trigger but to demonstrate that a CMS track trigger based on AM Chips and FPGAs is feasible for 2025.

5.3.1. Crate system

A system based on the ATCA standard is proposed for the hardware of the AM approach CMS track trigger. The ATCA standard originates from the telecom- munication industry [110, 111]. The standard has been defined with high data throughput in mind. Therefore, ATCA also suits well the high requirements for communication bandwidth of the CMS track trigger. Two factors enable this high data throughput: on the one hand the large size of the system components and on the other hand the support of full-mesh backplanes.

5.3. Hardware platform

The crates, as shown in Figure 5.9, of the ATCA standard are relatively large. They can host large Printed Circuit Boards (PCBs)—called blades in the ATCA standard—which are 280 mm deep and 322 mm high. A crate that fits into a 19- inch rack can host up to 14 blades. The blades are often used as carrier boards, and smaller boards are plugged onto the blades. The long front of the blades makes it possible to place many optical or electrical connectors. Additionally, a so-called Rear Transition Module (RTM) may be plugged to each blade in the back of the crate. The RTM is foreseen for the communication towards the back side of the crate. This allows the cables to stay connected to the crate when a blade is exchanged within the crate. The RTM may also be used to increase the bandwidth as connectors can be placed on both sides of the crate.

A backplane connects the blades with each other within the crate. Different network topologies exist for the backplanes [111]. The backplanes used for the AM approach CMS track trigger has a full-mesh topology [112], which is illustrated in Figure 5.10. On a full-mesh backplane, each blade is connected to each other blade by designated communication lines. The advantages of the full-mesh backplane are maximum throughput and low latency. The maximum throughput is achieved by the fact that blades do not share the lines, and each blade can send data constantly with maximum speed to any other blade constantly No switch is necessary as every blade can directly send data to all the other blades. Thus, it is also possible to communicate by a protocol with very little overhead, which increases the throughput further. Currently, a backplane capable of communications with up to 10 Gbit/s is installed in the test crates of the AM approach. In the future, backplanes with 40 Gbit/s per link may be installed, and backplanes with 100 Gbit/s have already been announced [113].

5.3.2. Carrier blade: Pulsar IIb

The Pulsar IIb is the proposed ATCA blade for the CMS track trigger. It serves as the carrier for the PRMs and is responsible for the data distribution both between the carrier blades and to the PRMs. It is a further develop- ment of the Pulsar IIa [115, 116] and was originally developed for the ATLAS FTK [117]. The target of the Pulsar II developments was a blade with very high communication capabilities. Figure 5.11 shows a photo of the Pulsar IIb blade.

The central element of the Pulsar IIb is a large Xilinx Virtex-7 FPGA. Different pin-compatible FPGAs with 410 to 690 thousand logic cells can be mounted on the Pulsar IIb. As the schematic overview of the Pulsar IIb in Figure 5.12

Figure 5.9.: 12-slot ATCA crate. Courtesy: Pentair/Schroff. Node 1 Node 2 Node 3 Node 4 Node 5 _{Node 6} Node 7 Node 8 Node 9

Figure 5.10.: A full-mesh network topology with nine nodes. Every node is connected to all the other nodes by designated links. Source: [114]

5.3. Hardware platform

Figure 5.11.: The Pulsar IIb ATCA carrier blade with an old version of the RTM attached on the right side. Courtesy of Fermilab.

shows, all the data communication is connected to the FPGA. Additionally, a 256 MB DDR3 RAM is directly connected to the FPGA.

The Pulsar IIb can host up to four mezzanines that follow the FMC stan- dard [118] and defines the connector type, the pin assignment, etc. It is also possible to plug a double-wide FMC to two adjacent FMC connectors. Each FMC connector provides three bidirectional High-Speed Serial Links (HSSLs) with up to 10 Gbit/s each that are connected to GTH transceivers of the FPGA. Additionally, 34 unidirectional Low-Voltage Differential Signaling (LVDS) sig- nals at 1 Gbit/s are available at each FMC connector. Thus, the total band- width is up to 64 Gbit/s for a single FMC and up to 128 Gbit/s for a double-wide FMC.

Within the crate, the Pulsar IIb blades communicate over a full-mesh back- plane with each other. The Pulsar IIb provides 28 GTH transceivers for the communication over the backplane. With 14 blades in one crate, two HSSLs connect each pair of blades. For the communication off the crate, an RTM has

FMC Mezzanine FMC Mezzanine FMC Mezzanine FMC Mezzanine FPGA Virtex-7 Config Flash RAM Clocks IPMC Power Zone 3 Zone 1 Zone 2 Bac kplane QSFP+ QSFP+ QSFP+ QSFP+ QSFP+ QSFP+ QSFP+ QSFP+ QSFP+ QSFP+ JTAG 40 GTH 28 GTH 3 GTH LVDS

Figure 5.12.: Schematic of Pulsar IIb with the RTM (version 1.0). Source: [117]. been developed together with the Pulsar IIb, see Figure 5.13. This RTM is connected to 40 of the GTH receivers of the FPGA, providing a total band- width of 400 Gbit/s. To provide this bandwidth to the outside, 10 Quad Small Form-factor Pluggable (QSFP+) cages are mounted on the RTM. A QSFP+ cage is a mechanical device to which a QSFP+ transceiver can be plugged. The QSFP+ transceiver itself is joint with a fiber-optical cable or a copper cable and provides a bandwidth of up to 4 × 10 Gbit/s.

Besides these core functionalities, the Pulsar IIb also contains other compo- nents that provide different services. The Intelligent Platform Management Controller (IPMC) module implements the board control, monitors the tem- peratures, voltages and currents, and provides an Ethernet connection for the configuration of the FPGA. The FPGA configuration can also be stored in a flash memory on the Pulsar IIb. Other components provide power and clocks for the Pulsar IIb.