A.2 PPT Processor design
The processor for the DSSC camera use case is designed to include an MMU to access the external memory, no FPU, and cache sizes of 4096 bytes for instructions and data. Controller logic for the external DDR3 memory chip is included in the processor design running at 400 MHz with a 16 bit wide interface.
The Mixed-Mode Clock Manager module (MMCM) on the FPGA fabric is used to generate all necessary clocks for the processor and its periphery modules. In total, 6 different clocks are needed: 100 MHz for the processor and the AXI bus structure, 25, 200 and 400 MHz for the DDR3 controller, and a 125 MHz for the ethernet interface. These are all derived in an FPGA-internal PLL from an oscillator delivering 200 MHz to the FPGA input pins.
Further modules attached to the processor include
- a debug module for accessing the processor registers through the JTAG programming inter- face,
- UART interfaces to the linux kernel brought out via USB interface and a serial interface to be used with external programmable power supplies through the Patch Panel,
- an ethernet core with DMA access to the DRAM,
- access to FPGA fabric pins realized by a general purpose I/O module (GPIO), - two SPI modules for read and write operations to flash memory chips on the PCB
- and a controller for external devices with a 32 bit wide address and data bus (External Pe- ripheral Controller - EPC).
The main task of the GPIO module is to control and read the JTAG lines towards the four ladder IO Boards and the JTAG chain comprising the 16 ASICs of each ladder.
Attached to the EPC, user devices and registers are attached in the PPT firmware. Using the EPC, other logic in the FPGA fabric can be configured. Response to commands from XFEL is enabled through the EPC, for example.
The total number of registers and LUTs occupied by the processor and its interface to peripheral components is 20075 (registers) / 21554 (LUTs) or 5% (registers) / 3.5% (LUTs) of the available slices. The processor runs at 100 MHz. The ethernet connection speed has been measured between 2000 and 3000 kByte/s for receiving, and about 1000 kByte/s for sending.
While being sufficient for the DSSC application, the low speed of the MicroBlaze architecture has been addressed by Xilinx by integrating an ARM-based processor core on the FPGA die [83]. Suitable devices based on the Kintex-7 FPGA with sufficient numbers of gigabit transceivers are now available in the Zynq family during the writing of this thesis.
A.3 Software
The MicroBlaze processor runs a standard linux kernel, cross-compiled for the MicroBlaze archi- tecture. While this is a standard procedure, some aspects must be considered. First, during the build of the linux kernel, a description of the hardware architecture must be provided. Mainly the memory addresses of the peripheral blocks must be known for the processor to properly address control and data registers. Second, a root file system can be prepared and included in the kernel binary. The root file system will be mounted during the boot procedure and may contain a basic set of user software. In the case of the PPT, the BusyBox software package is used to provide common
linux utilities in a small footprint [84]. More DSSC-specific software for access to the board serial number, board and detector configuration, and even a lightweight webserver to view the system status are provided.
The kernel sources itself had to be changed due to flaws in the implementation of the SPI device driver. Only a default bus number is assigned to each SPI devices, even if several devices are available in the processor design. More serious, the use of the STARTUPE2 primitive in the SPI device for the FPGA bitstream flash memory was necessary. The STARTUPE2 primitive allows to control the configuration pins from user logic after the FPGA has been programmed. This is used in the PPT implementation to allow reprogramming of the flash memory with files received over the ethernet connection. However, the STARTUPE2 primitive ignores the first three clock cycles from user logic. During the boot process, the kernel tries to identify the flash memories connected via SPI, but fails to correctly to do so due to the missing clock cycles. The kernel is therefore patched to simply run the identification procedure twice.
Patch Panel Transceiver
I/O Board - Spartan-6 FPGA
MicroBlaze Soft-core JTAG Quadrant Ladders Slow-Control Ethernet JTAG 4 4 4 4 4
Xilinx Virtual Cable Server
ChipScope
Kintex-7 FPGA
Vacuum feed-through
I/O Board - Spartan-6 FPGA JTAG
I/O Board - Spartan-6 FPGA JTAG
I/O Board - Spartan-6 FPGA JTAG
Figure A.2: Connection scheme of the FPGA JTAG signals in the DSSC system. The I/O board FPGAs are programmed by the MicroBlaze. The Xilinx Virtual Cable Server allows access to the ChipScope debugging cores in each FPGA in the DSSC system through the PPT ethernet connection.
The I/O Boards of four ladders are connected to the PPT (figure A.2). The FPGAs on the IOBs are programmed via JTAG pins accessed by the MicroBlaze. A software running on the processor decodes the JTAG commands in the programming files and sets the programming pins accordingly, monitoring the proper output from the I/O Board FPGA.
An important debugging tool for Xilinx FPGAs is provided in the ChipScope tool. The Chip- Scope cores for FPGAs allow analysis of logic signals in the FPGA fabric without putting them on output pins. Instead, the signals to be analysed are sampled and stored in FPGA-internal memory by extra logic inserted in the user design. Such cores have been implemented in both the I/O Board and the PPT design. The ChipScope cores are accessed through the FPGA JTAG pins. A dedicated software on the MicroBlaze accesses the JTAG pins of the I/O Boards and the PPT FPGA itself to make the ChipScope cores available via ethernet. This feature allows debugging of the digital repro- grammable parts of the detector without any extra cables like the dedicated Xilinx programming cable connection usually needed.