FPGA Manager
PCIe, USB 3.0 and Ethernet
Streaming, made simple.
Embedded Computing Conference 2014 Marc Oberholzer
Enclustra – We speak FPGA!
Content
Enclustra Company Profile
Overview
FPGA Design Center FPGA Solution Center
FPGA-to-Host Communication Challenges
Multiplicity
Data streaming and memory mapped accesses Multiplexing Performance Error Handling Device enumeration Design security Compatibility
FPGA specific challenges
PCIe, USB and Ethernet specific challenges
- 2 -
FPGA Manager
Overview Paradigms Design goals
Host – Master API class diagram Host – C# streaming example FPGA – Ethernet block diagram FPGA – USB block diagram FPGA – PCIe block diagram
Integration into Xilinx Vivado IP Integrator Licensing
Use case – 3D scanner for forensics applications Future plans
Evaluation hardware and further information Questions
Content
Enclustra Company Profile
Overview
FPGA Design Center FPGA Solution Center
FPGA-to-Host Communication Challenges
Multiplicity
Data streaming and memory mapped accesses Multiplexing Performance Error Handling Device enumeration Design security Compatibility
FPGA specific challenges
PCIe, USB and Ethernet specific challenges
FPGA Manager
Overview Paradigms Design goals
Host – Master API class diagram Host – C# streaming example FPGA – Ethernet block diagram FPGA – USB block diagram FPGA – PCIe block diagram
Integration into Xilinx Vivado IP Integrator Licensing
Use case – 3D scanner for forensics applications Future plans
Evaluation hardware and further information Questions
Enclustra – We speak FPGA!
Enclustra Company Profile
Overview
Quick Facts
Founded in 2004 – 10 years!
Located in Technopark Zurich, Switzerland
17 FPGA Engineers, 2 technicians, 3 sales, 1 executive staff Vendor independent
Business Units
FPGA Design Center (services) FPGA Solution Center (products)
FPGA Partnerships
Xilinx Alliance Program – Certified member
Altera Design Services Network – Certified member Lattice Semiconductor Leader – Member
Enclustra Company Profile
FPGA Design Center
FPGA System Design
Hardware (High-Speed, Analog, RF) HDL Firmware (VHDL, Verilog)
Embedded Software (for FPGA processors)
Focus Areas
Embedded processing Test & measurement Software-defined radio Smart vision systems Motion & drive control
Experience
~100 person-years of FPGA hardware, firmware & software design >50 customers, >200 projects
Enclustra – We speak FPGA!
Enclustra Company Profile
FPGA Solution Center
FPGA and SoC Modules
Mars family (MX1, MX2, AX3, ZX3)
SO-DIMM 67.6 x 30 mm, 96-108 user I/Os, 0-2 MGTs
1-2 Ethernet ports, 0-1 USB port, 3.3V single supply voltage Mercury family (KX1, CA1, ZX5, ZX1, SA1)
56 x 54 to 72 x 54 mm, 146-188 user I/Os, 0-8 MGTs
1-2 Ethernet ports, 1 USB port, 5-15V single supply voltage Designed & priced for integration into your product
Planned availability: 10 years >500 customers in >50 countries
Module Selection Guide & Roadmap available online
IP Cores and Solutions
Universal Drive Controller IP Core UDP/IP Ethernet IP Core
FPGA Manager Solution
Content
Enclustra Company Profile
Overview
FPGA Design Center FPGA Solution Center
FPGA-to-Host Communication Challenges
Multiplicity
Data streaming and memory mapped accesses Multiplexing Performance Error Handling Device enumeration Design security Compatibility
FPGA specific challenges
PCIe, USB and Ethernet specific challenges
FPGA Manager
Overview Paradigms Design goals
Host – Master API class diagram Host – C# streaming example FPGA – Ethernet block diagram FPGA – USB block diagram FPGA – PCIe block diagram
Integration into Xilinx Vivado IP Integrator Licensing
Use case – 3D scanner for forensics applications Future plans
Evaluation hardware and further information Questions
Enclustra – We speak FPGA!
FPGA-to-Host Communication Challenges
Multiplicity
Multiple interfaces, speeds and generations (PCIe Gen1/2/3/4, USB2/3/4, 100Mbps/1Gbps/10Gbps ETH) Multiple FPGA vendors and FPGA boards (Altera, Xilinx, etc. / eval boards, custom boards, etc.)
Multiple operating systems and programming languages (Linux, Windows, etc. / C++, .NET, Matlab, etc.)
Data streaming
Stream framing (frame streams vs. byte streams, meta-data association (e.g. time stamp, etc.)) Frame fragmentation and re-assembly (frame vs. packet, frame vs. buffer)
Frame size can be larger than buffer size (cut-through forwarding, frame poisoning)
Memory-mapped access
Different access types (single word, burst, same address burst, atomic read-modify-write, etc.)
Read from non-prefetchable targets (e.g. FIFOs) and execution ordering under packet loss scenarios
Multiplexing
Multiple channels over the same physical interface (e.g. stereo vision, 4-channel oscilloscope, etc.) Multiple simultaneous connections to the same device (e.g. monitoring deamon, user GUI, etc.)
FPGA-to-Host Communication Challenges
Performance
Bandwidth (MBytes/sec) vs. Latency (sec)
High bandwidth asks for large frame sizes while low latency asks for small frame sizes High bandwidth also asks for the least possible number of data copies in the host Operation throughput (MOps/sec)
Pipelining of operations in the host and in the FPGA, multiple operations per frame CPU utilization
Event-driven instead of polling Scheduling
Minimize the impact of one stream‘s transfer on other streams
Error Handling
Non-fatal error handling (buffer overflows, checksum errors, packet losses, timeouts, etc.)
Fatal error handling (illegal packet format, network protocol errors, exceptions, stack overflow, etc.)
Device Enumeration
Enclustra – We speak FPGA!
FPGA-to-Host Communication Challenges
FPGA configuration
FPGA firmware download and/or fail-safe FPGA firmware update
Design security
FPGA bitstream copy protection, encrypted data flows, firmware/software authentication, licensing, etc.
Compatibility
FPGA firmware to host software version / network protocol backward & forward compatibility
FPGA interfaces
Multiple bus protocols (AXI4, Avalon, etc.)
Multiple data widths and stream formats (byte, frame, block) Multiple clock domains
FPGA data buffering
Data/packet rate throttling (e.g. if host or link is not as fast as the data source could be) Retransmission capability in packet loss scenarios
PCIe, USB and Ethernet specific challenges
Content
Enclustra Company Profile
Overview
FPGA Design Center FPGA Solution Center
FPGA-to-Host Communication Challenges
Multiplicity
Data streaming and memory mapped accesses Multiplexing Performance Error Handling Device enumeration Design security Compatibility
FPGA specific challenges
PCIe, USB and Ethernet specific challenges
FPGA Manager
Overview Paradigms Design goals
Host – Master API class diagram Host – C# streaming example FPGA – Ethernet block diagram FPGA – USB block diagram FPGA – PCIe block diagram
Integration into Xilinx Vivado IP Integrator Licensing
Use case – 3D scanner for forensics applications Future plans
Evaluation hardware and further information Questions
Enclustra – We speak FPGA!
FPGA Manager
Overview
- 12 -
One common host SW
API for all link types
One common FPGA IP
core user interface for
all link types
Up to 16 streaming ports per direction
Memory mapped option (consumes one
upstream and one downstream port)
FPGA Manager
Paradigms
System
Up to 16 independent, equivalent streams per direction
Stream to memory-mapped converter (e.g. AXI4-Stream to AXI4-MM) The user never receives wrong or wrongly ordered data.
Frames
Data is organized into a sequence of frames of variable size.
A frame number and a frame byte offset is used in each packet header to associate packets to a specific stream, frame and position.
Frames received on the host may be shorter or longer than the pending receive buffers. Frames may span multiple receive buffers to support large dynamics in frame sizes without wasting large amounts of memory.
Error handling
Data loss is tolerated (except PCIe).
The user is notified of all error conditions using callbacks or exceptions. Liveness is assured with (optional) timeouts on all operations.
Enclustra – We speak FPGA!
FPGA Manager
Paradigms
FPGA IP core
The IP core contains asynchronous FIFOs to allow the user-side AXI interfaces to run at a user-selectable clock frequency.
The user can specify the upstream & downstream buffer sizes of each stream at compile time.
Host software API
Multi-threaded API access (e.g. one user thread per data stream)
Any number of transmit and receive operations can be pending on each stream. The user is notified when a transfer completes using callbacks or exceptions.
FPGA Manager
Design Goals
The solution shall be easy to learn and work out-of-the-box.
The user shall see the least possible differences when using FPGA Manager with different link
types, FPGA vendors, programming languages, and operating systems.
The IP components shall integrate well with FPGA vendor tools (drag & drop).
The licensing shall be as flexible as possible in order to keep license costs low for applications
not requiring the full set of features.
The highest possible performance shall be achieved.
One stream shall not stall the other streams.
Memory-mapped accesses shall support retransmission and respect execution order.
User application design is simplified substantially if this goal is achieved.
All data transfers shall be specified at the byte (and not the word) level.
It shall be possible to read and write any number of bytes on each transfer. There shall be no alignment restrictions on the source and destination addresses.
Enclustra – We speak FPGA!
Content
Enclustra Company Profile
Overview
FPGA Design Center FPGA Solution Center
FPGA-to-Host Communication Challenges
Multiplicity
Data streaming and memory mapped accesses Multiplexing Performance Error Handling Device enumeration Design security Compatibility
FPGA specific challenges
PCIe, USB and Ethernet specific challenges
- 16 -
FPGA Manager
Overview Paradigms Design goals
Host – Master API class diagram Host – C# streaming example
FPGA – Ethernet block diagram FPGA – USB block diagram FPGA – PCIe block diagram
Integration into Xilinx Vivado IP Integrator Licensing
Use case – 3D scanner for forensics applications Future plans
Evaluation hardware and further information Questions
FPGA Manager
Host – Master API Class Diagram
Send() Open() Close() Receive() GetStatus() SetStatusEvtCallback() SetProperty() GetProperty() CreateStream() Device Open() Close() CreateMmAccess() Stream Write() Open() Close() Read() GetStatus() SetStatusEvtCallback() SetProperty() GetProperty() MmAccess SetCallback() Transfer GetStatus() GetTransferredBytes() GetFrameNumber()
Can be used to track status (optional) System
GetSwVersion() GetErrorTrace() GetErrorMessage()
Enclustra – We speak FPGA! - 18 -
FPGA Manager
Host – C# Streaming Example
void main() {
//Create send and receive buffers
byte[] receiveArray = new byte[4];
byte[] sendArray = new byte[4] { 1, 2, 3, 4 }; //Open a device with one stream
CDevice myDevice = new CDevice("udp://192.168.33.12", 1);
//Create Stream 0, Frame based, Upstream Enabled, Downstream Enabled
CStream myStream = myDevice.CreateStream(0, true, true, true); //Open
myDevice.Open(); myStream.Open(); //Blocking send
myStream.Send(ref sendArray, null); //Blocking receive
myStream.Receive(ref receiveArray, null); //Close (non-forcing)
myStream.Close(false); myDevice.Close(false); }
Content
Enclustra Company Profile
Overview
FPGA Design Center FPGA Solution Center
FPGA-to-Host Communication Challenges
Multiplicity
Data streaming and memory mapped accesses Multiplexing Performance Error Handling Device enumeration Design security Compatibility
FPGA specific challenges
PCIe, USB and Ethernet specific challenges
FPGA Manager
Overview Paradigms Design goals
Host – Master API class diagram Host – C# streaming example
FPGA – Ethernet block diagram FPGA – USB block diagram FPGA – PCIe block diagram
Integration into Xilinx Vivado IP Integrator Licensing
Use case – 3D scanner for forensics applications Future plans
Evaluation hardware and further information Questions
Enclustra – We speak FPGA! Down-stream Buffer Upstream Buffer Down-stream Buffer Upstream Buffer UDP/IP/ETH Transmit Engine UDP/IP/ETH Receive Engine Ethernet PHY Interface UDP Transmit Buffer UDP Receive Buffer UNISCP Transmit Controller UNISCP Receive Controller Upstream Controller Down-stream Controller
Command Processor Context RAM
Upstream Buffer Down-stream Buffer MM Converter UDP/IP/Ethernet IP Core FPGA Manager IP Core
AXI4-S: Stream 1 AXI4-S: Stream 15 AXI4-S: Stream 1 AXI4-S: Stream 15 AXI4-MM RGMII MII RMII GMII - 20 -
FPGA Manager
FPGA – Ethernet Block Diagram
FPGA Manager
FPGA – USB Block Diagram
Down-stream Buffer Upstream Buffer Down-stream Buffer Upstream Buffer USB Packet Transmit Engine
FTDI USB 2.0 FIFO Interface or FX3 USB 3.0 Slave FIFO Interface UNISCP Transmit Controller UNISCP Receive Controller Upstream Controller Down-stream Controller
Command Processor Context RAM
Upstream Buffer Down-stream Buffer MM Converter FTDI USB 2.0 FIFO Interface FPGA Manager IP Core
AXI4-S: Stream 1 AXI4-S: Stream 15 AXI4-S: Stream 1 AXI4-S: Stream 15 AXI4-MM USB 3.0 USB 2.0
USB Packet Receive Engine FX3 USB 3.0 Slave FIFO Interface
Enclustra – We speak FPGA! - 22 -
FPGA Manager
FPGA – PCIe Block Diagram
Downstream Buffer Upstream Buffer TLP
Engine
Target Controller AXI4-MM Master
Register Bank & Interrupt Controller
Address Lookup Engine Page Table
DMA Read Data Unit
Command Processor Command
Table
Upstream Buffer
Downstream Buffer DMA Write Data Unit
AXI4-MM AXI4-S: Stream 0 AXI4-S: Stream 0 AXI4-S: Stream 15 AXI4-S: Rx TLPs AXI4-S: Stream 15 AXI4-S: Tx TLPs
FPGA Manager PCIe IP Core
Content
Enclustra Company Profile
Overview
FPGA Design Center FPGA Solution Center
FPGA-to-Host Communication Challenges
Multiplicity
Data streaming and memory mapped accesses Multiplexing Performance Error Handling Device enumeration Design security Compatibility
FPGA specific challenges
PCIe, USB and Ethernet specific challenges
FPGA Manager
Overview Paradigms Design goals
Host – Master API class diagram Host – C# streaming example FPGA – Ethernet block diagram FPGA – USB block diagram FPGA – PCIe block diagram
Integration into Xilinx Vivado IP Integrator
Licensing
Use case – 3D scanner for forensics applications Future plans
Evaluation hardware and further information Questions
Enclustra – We speak FPGA! - 24 -
FPGA Manager
Integration into Xilinx Vivado IP Integrator
FPGA Manager
Enclustra – We speak FPGA! - 26 -
FPGA Manager
Integration into Xilinx Vivado IP Integrator
Content
Enclustra Company Profile
Overview
FPGA Design Center FPGA Solution Center
FPGA-to-Host Communication Challenges
Multiplicity
Data streaming and memory mapped accesses Multiplexing Performance Error Handling Device enumeration Design security Compatibility
FPGA specific challenges
PCIe, USB and Ethernet specific challenges
FPGA Manager
Overview Paradigms Design goals
Host – Master API class diagram Host – C# streaming example FPGA – Ethernet block diagram FPGA – USB block diagram FPGA – PCIe block diagram
Integration into Xilinx Vivado IP Integrator
Licensing
Use case – 3D scanner for forensics applications Future plans
Evaluation hardware and further information Questions
Enclustra – We speak FPGA! - 28 -
FPGA Manager
Licensing
Type Ordering Code Description
Base EN-MGR FPGA Manager
2 data streams, 1 memory-mapped interface C/C++/C# language support
EN-MGR-OPT-USB2FT USB 2.0 FTDI support EN-MGR-OPT-USB3CY USB 3.0 Cypress support
EN-MGR-OPT-ETH1G Fast & Gigabit Ethernet support EN-MGR-OPT-PCIE64 PCIe 64-Bit support
EN-MGR-OPT-XIL Xilinx FPGA support EN-MGR-OPT-ALT Altera FPGA support
EN-MGR-OPT-WIN Windows operating system support EN-MGR-OPT-LIN Linux operating system support EN-MGR-OPT-ADV Advanced feature pack
16 total streams, multiple memory-mapped interfaces, multiple stream widths
EN-MGR-OPT-ML MATLAB language support Options
Targets Operating Systems Link types
Modular feature sets for flexible licensing and cost efficiency
Industry standard SignOnce license agreement
Source code or binary/encrypted license types Project or site license types
Content
Enclustra Company Profile
Overview
FPGA Design Center FPGA Solution Center
FPGA-to-Host Communication Challenges
Multiplicity
Data streaming and memory mapped accesses Multiplexing Performance Error Handling Device enumeration Design security Compatibility
FPGA specific challenges
PCIe, USB and Ethernet specific challenges
FPGA Manager
Overview Paradigms Design goals
Host – Master API class diagram Host – C# streaming example FPGA – Ethernet block diagram FPGA – USB block diagram FPGA – PCIe block diagram
Integration into Xilinx Vivado IP Integrator Licensing
Use case – 3D scanner for forensics applications
Future plans
Evaluation hardware and further information Questions
Enclustra – We speak FPGA! - 30 -
FPGA Manager
Use Case – 3D Scanner for Forensics Applications
Stream Buffer Controller
UART HDMI Display Controller UART
FPGA Manager Ethernet DDR3 Memory Controller MicroBlaze Soft Processor Camera Link Receiver Camera Link Receiver Stream Buffer Controller AXI4 AXI4-Stream AXI4-Stream
AXI4-Stream AXI4 Lite
Register Bank Trigger Unit Trigger Unit Camera 2048 x 2048 px 60 fps Camera 2048 x 2048 px 60 fps Projector 1280 x 800 px 60 Hz Host Computer Industrial PC DDR3 SDRAM 256 MB
Scan procedure
Project a fringe pattern image to the object,
simultaneously trigger both cameras and store both images in the DDR3 SDRAM (repeat with more patterns) Transfer the acquired images
to the host computer (offline)
FPGA Manager Ethernet
1 AXI4-Light master port for memory mapped
configuration register accesses
1 AXI4 master port for memory mapped access to the DDR3 SDRAM
1 AXI4-Stream upstream port for each camera
Content
Enclustra Company Profile
Overview
FPGA Design Center FPGA Solution Center
FPGA-to-Host Communication Challenges
Multiplicity
Data streaming and memory mapped accesses Multiplexing Performance Error Handling Device enumeration Design security Compatibility
FPGA specific challenges
PCIe, USB and Ethernet specific challenges
FPGA Manager
Overview Paradigms Design goals
Host – Master API class diagram Host – C# streaming example FPGA – Ethernet block diagram FPGA – USB block diagram FPGA – PCIe block diagram
Integration into Xilinx Vivado IP Integrator Licensing
Use case – 3D scanner for forensics applications
Future plans
Evaluation hardware and further information
Enclustra – We speak FPGA!
FPGA Manager
Future Plans
SoC edition
Communication between ARM subsystems and programmable logic via FPGA Manager API Introduces up to 16 independent streams and scatter/gather DMA over one AXI4 bus port
Android edition
Provide a FPGA Manager API for the Android JAVA layer for communication with programmable logic inside a SoC device.
FPGA Manager Slave API
Replace the FPGA with a software implementation, communicate via a reliable TCP/IP connection Use case 1: FPGA modeling (golden model in software, implement user software and FPGA in parallel)
Use case 2: FPGA in the loop (communicate between host PC and SoC ARM subsystem via TCP/IP)
Additional link types (10 Gbps Ethernet, PCIe Gen3 x8, USB 4.0, Thunderbolt, etc.)
Additional programming languages and environments
Additional operating systems
Your own ideas...
FPGA Manager
Evaluation Hardware and Further Information
Evaluation Hardware
Enclustra Mercury KX1 PE1 Kit: Xilinx Kintex-7 FPGA with USB 3.0, PCIe Gen2 x4 and Gigabit Ethernet Enclustra Mercury CA1 Starter Kit: Altera Cyclone IV FPGA with Gigabit Ethernet
Enclustra Mars ZX3 PM3 Kit: Xilinx Zynq-7020 SoC with USB 3.0 and Gigabit Ethernet Enclustra Mars EB1 AX3 Kit: Xilinx Artix-7 FPGA with Gigabit Ethernet
Altera evaluation boards Xilinx evaluation boards
3rd party boards (e.g. Digilent Zedboard)
Your own custom board (make sure to follow our FPGA Manager hardware design guide)
Further Information
FPGA Manager Product Page
www.enclustra.com/en/products/fpga-manager/
UDP/IP Ethernet IP Core
www.enclustra.com/en/products/ip-cores/udp-ip-ethernet/
Mars PM3 USB 3.0 Quick-Start Kit
Enclustra – We speak FPGA!
Content
Enclustra Company Profile
Overview
FPGA Design Center FPGA Solution Center
FPGA-to-Host Communication Challenges
Multiplicity
Data streaming and memory mapped accesses Multiplexing Performance Error Handling Device enumeration Design security Compatibility
FPGA specific challenges
PCIe, USB and Ethernet specific challenges
- 34 -
FPGA Manager
Overview Paradigms Design goals
Host – Master API class diagram Host – C# streaming example FPGA – Ethernet block diagram FPGA – USB block diagram FPGA – PCIe block diagram
Integration into Xilinx Vivado IP Integrator Licensing
Use case – 3D scanner for forensics applications Future plans
Evaluation hardware and further information
Questions
