P4080DS U-boot & Linux

(1)

FTF-NET-F0683

P4080DS U-boot & Linux

®

June 22, 2010

Srikanth Srinivasan & Ted Peters

(2)

Board-bring up is a key activity for new silicon. Firmware and OS

are the first software components run on the reference platforms.

Initial bring-up involves debug of silicon, board and the software.

This presentation details the important parts of firmware (u-boot)

and OS (Linux) porting/bring-up for P4080DS reference platform.

Freescale’s Board Support Package (BSP) provides software that

can be modified to run on the customer development systems.

The talk will drive home the strategies, steps, concepts, and

highlight relevant parts of the code needed most commonly

during this effort.

(3)

►

Learn more about the Freescale BSP for P4080/P4080DS

►

Details of u-boot and Linux porting, and hypervisor

concepts.

• Obtain and build u-boot and Linux from source

• Peruse source trees for making changes specific to your

platforms

• Identify separation of tasks between SW components

• Identify key components of the reference platform as well as

the software

• Learn multi-core boot sequence

• Hypervisor capabilities and use

(4)

►

Introduction – 5 min

►

U-boot – 30 min

►

Hypervisor – 25 min

►

Linux – 40 min

►

LWE – 5 min

Agenda

(5)

SRIO

Message

Unit

DMA

PCIe

18-Lane 5GHz SERDES

PCIe

SRIO

CoreNet™

1024-Kbyte

Frontside

L3 Cache

64-bit

DDR-2 / 3

Memory Controller

P4080

Power Architecture™

e500-mc Core

D-Cache

I-Cache

128-Kbyte

Backside

L2 Cache

SRIO

Watchpoint

Cross

Trigger

Perf

Monitor

CoreNet

Trace

Aurora

Real Time Debug

Security

4.0 Pattern

Match

Engine

2.0 Queue

Mgr.

Buffer

Mgr.

eLBIU

M2SB

Test

Port/

SAP

Frame Manager

1GE 1GE

10GE

Parse, Classify,

Distribute

Buffer

32-Kbyte

1024-Kbyte

Frontside

L3 Cache

P4080 Block Diagram

64-bit

DDR-2 / 3

Memory Controller

PAMU

Coherency Fabric

PAMU

1GE 1GE

10GE

Parse, Classify,

Distribute

Buffer

Frame Manager

Peripheral Access Mgmt Unit

eOpenPIC

Power Mgmt

2x USB 2.0/ULPI

SD/MMC

Clocks/Reset

DUART

2x I 2C

SPI

GPIO

PreBoot Loader

Security Monitor

Internal BootROM

CCSR

(6)

EC1 – USB1

EC2 – FM1[dTSEC2] 10/100/1G RGMII

SerDes Lanes [0-17]:

• Slot 1: Bank1 A-B (PEX x2)

• Slot 2: Bank1 C-D (PEX x2)

• Slot 3: Bank1 E-H (4x SGMII x1, PEX x4, SRIO

x4)

• ---: Bank1 I-J (Aurora x2 only)

• Slot 4: Bank 2 A-D (4x SGMII x1 or XAUI x4)

• Slot 5: Bank 3 A-D (4x SGMII x1 or XAUI x4)

u-boot RCW[srds_prtcl] = 0x10 (RSX configuration)

PCIe1 (5/2.5Gb)

PCIe3 (5/2.5Gb)

4x SGMII

(1)

FM2 dTSEC[1:4]

Debug (5/2.5Gb)

XAUI

FM2 10GEC

4x SGMII

FM1 dTSEC[1:4]

(7)

Getting Started

► Master u-boot repository

• http://git.denx.de

► Master linux repository

• http://git.kernel.org

► BSP release

• http://www.freescale.com/powerbsp

• Follow instructions in the BSP documentation to extract software

packages

(8)

P4080 SDK Beta 2.1.0 Components

► Board Support Package (BSP)

• GNU Toolchain

gcc-4.3.2, eglibc2.8, binutils 2.18, CodeSourcery release 4.3.74-6

• U-Boot Boot Loader

• Linux 2.6 kernel

Linux device drivers with P4080 Datapath Acceleration Architecture (DPAA)

support

• Embedded Hypervisor Software

• Light Weight Executive (LWE)

LWE Libraries (LWE Libs)

(9)

Extracting Sources And Building Images

►

System requirements

• x86 linux with cross-tools

►

LTIB

$ mount –o loop <BSP.iso> /mnt/p4080ds-ltib

$ cd my-ltib-dir

$ /mnt/p4080ds-ltib/install

From the installed directory :

$ ./ltib –m prep –p u-boot

(Sources available in rpm/BUILD/u-boot)

$ ./ltib –m scbuild –p u-boot

To configure packages :

$./ltib --configure

(10)

git

► Version control done through git

• SCM of choice for many open-source projects including Linux

► Clone the tree locally

• git clone git://git.denx.de/u-boot.git

• Replicates the remote tree with change logs and tags

► When ready to contribute

1. Commit changes locally

2. Create a patch and mail it to the u-boot list

3. Peer review

4. Patch applied to master tree for everyone to use

5. Clean up patch for external world and repeat steps 2-4

► A few keepers for the master tree

• Ensures safety of code base while still allowing group contribution and higher

(11)

Summary Of Boot Flow From System-Wide Reset

Load hypervisor image and at least one guest

OS image. Load device trees for guests.

Hypervisor initializes and then starts one or more guests. It passes them a device tree

from which they get information on resources allocated to them (and more).

One guest is designated to do global initialization of BMan, QMan, and FMan since

they are used by many guests. Other guests await this initialization.

Guests can bring up their network interfaces.

►uboot

►Guest OS

►Hypervisor

A guest can be authorized to act as a partition manager. It can load and run more

guests or reboot them.

(12)

(13)

Introduction

► What is u-boot?

• Open source firmware for hardware platforms

• A boot loader widely used in embedded space

► What does it do?

• Brings the board to a sane state out of reset

• Load OS image onto board and start OS

► Origins

• DENX Software Engineering

(14)

U-boot Directories

Relevant directories for modifications

u-boot/.

u-boot/include

u-boot/board

u-boot/cpu

u-boot/lib_ppc

u-boot/common

fs

./u-boot

board

common

cpu

disk

drivers

include

lib_*

net

rtc

tools

(15)

Boot Sequence

► Reset vector points to FLASH (0xfffffffc)

► Begins execution in FLASH

► Initializes DUARTs

► Checks for information on version numbers, clocks and displays on screen

► Initializes DDR

► Relocates itself to RAM

► Continues execution in RAM

► Releases other cores

► Initializes higher level devices – ethernet, PCIe, SATA, USB, etc.

(16)

U-boot Memory Map for P4080DS

/*

* Memory map

*

* 0x0000_0000 0x7fff_ffff

DDR

2G Cacheable

*

* 0x8000_0000 0x9fff_ffff

PEX 1 Mem

512M non-cacheable

* 0xa000_0000 0xbfff_ffff

PEX 2/SRIO Mem 512M non-cacheable

* 0xc000_0000 0xdfff_ffff

PEX 3 Mem

512M non-cacheable

*

* 0xf800_0000 0xf802_ffff

PCI IO range

256K non-cacheable

* 0xf400_0000 0xf41f_ffff

Bman

2M

* 0xf420_0000 0xf43f_ffff

Qman

2M

*

* Localbus non-cacheable

*

* 0xe000_0000 0xe80f_ffff

Promjet/free

128M non-cacheable

* 0xe800_0000 0xefff_ffff

FLASH

128M non-cacheable

* 0xffdf_0000 0xffdf_7fff

PIXIS

32K non-cacheable TLB0

*

* 0xffd0_0000 0xffd0_3fff

L1 for stack

16K Cacheable TLB0

* 0xfe00_0000 0xfeff_ffff

CCSR

16M non-cacheable

*/

(17)

Initialization

► MMU, LAW setup

► Clocks

► Serial

► DDR

► Misc CPU init

► Relocate to DDR space

► Flash

► PCIe

► Misc init/MP init

► FM init

(18)

Stack In Cache

► Drivers written in C code

Need stack space to run

Lock the cache

Unlock before linux boot

* 0xe800_0000 0xefff_ffff

FLASH

128M non-cacheable

* 0xffdf_0000 0xffdf_7fff

PIXIS

32K non-cacheable TLB0

*

* 0xffd0_0000 0xffd0_3fff

L1 for stack

16K Cacheable TLB0

* 0xffe0_0000 0xffef_ffff

CCSR

1M non-cacheable

Files:

cpu/mpc85xx/start.S

Code Snippets:

lis r6,FSL_BOOKE_MAS0(1, 14, 0)@h

ori r6,r6,FSL_BOOKE_MAS0(1, 14, 0)@l

lis r7,FSL_BOOKE_MAS1(1, 1, 0, 1, BOOKE_PAGESZ_16K)@h

ori r7,r7,FSL_BOOKE_MAS1(1, 1, 0, 1, BOOKE_PAGESZ_16K)@l

lis r8,FSL_BOOKE_MAS2(CONFIG_SYS_INIT_RAM_ADDR, 0)@h

ori r8,r8,FSL_BOOKE_MAS2(CONFIG_SYS_INIT_RAM_ADDR, 0)@l

(19)

MMU, LAW Setup

► Configuring memory map

►

Files:

include/configs/P4080DS.h, corenet_ds.h

board/freescale/corenet_ds/tlb.c, law.c

cpu/mpc85xx/tlb.c

drivers/misc/fsl_law.c

include/asm-ppc/immap_85xx.h

►

Code Snippets:

/* IG* - Flash, localbus */

/* This will be changed to IG* after relocation to RAM. */

SET_TLB_ENTRY(1, CONFIG_SYS_FLASH_BASE, CONFIG_SYS_FLASH_BASE_PHYS,

MAS3_SX|MAS3_SR, MAS2_W|MAS2_G, 0, 2, BOOKE_PAGESZ_256M, 1),

(20)

SYSCLK

► SYSCLK value needed to get started

• Platform frequency

► FPGA registers SCLK0, 1, 2

• Input to ICS307 clock generator code

► Obtain the board and ddr clocks

► Files:

include/configs/corenet_ds.h

board/freescale/corenet_ds/corenet_ds.c

► Code Snippets:

/* Calculate frequency being generated by ICS307-02 clock chip based upon

* the control bytes being programmed into it. */

static unsigned long ics307_clk_freq(unsigned char cw0, unsigned char cw1,

unsigned char cw2)

{

const unsigned long InputFrequency = CONFIG_ICS307_REFCLK_HZ;

unsigned long VDW = ((cw1 << 1) & 0x1FE) + ((cw2 >> 7) & 1);

:

}

unsigned long calculate_board_sys_clk(ulong dummy)

{

ulong val;

(21)

Serial

► 2 DUARTs ( 4 UARTs) on P4080

• 2 serial outputs on DS

► u-boot provides driver for NS16550 compatible DUARTs

• Needs platform frequency

► Serial port very useful for early debug

► Files:

include/configs/corenet_ds.h

drivers/serial/ns16550.c, serial.c

► Code Snippets:

#define CONFIG_CONS_INDEX 2

#undef CONFIG_SERIAL_SOFTWARE_FIFO

#define CONFIG_SYS_NS16550

#define CONFIG_SYS_NS16550_SERIAL

#define CONFIG_SYS_NS16550_REG_SIZE 1

#define CONFIG_SYS_NS16550_CLK (get_bus_freq(0)/2)

#define CONFIG_SYS_NS16550_COM1 (CONFIG_SYS_CCSRBAR+0x11C500)

#define CONFIG_SYS_NS16550_COM2 (CONFIG_SYS_CCSRBAR+0x11C600)

#define CONFIG_SYS_NS16550_COM3 (CONFIG_SYS_CCSRBAR+0x11D500)

#define CONFIG_SYS_NS16550_COM4 (CONFIG_SYS_CCSRBAR+0x11D600)

(22)

DDR Setup

► Relocating code

► Supports multiple controllers and SPD-based (I2C eeprom) initialization

► Supports Interleaving

► Files:

cpu/mpc8xxx/ddr

include/configs/corenet_ds.h

board/freescale/corenet_ds/ddr.c Å board specific parameters

► Code Snippets:

#define CONFIG_NUM_DDR_CONTROLLERS 2

#define CONFIG_DIMM_SLOTS_PER_CTLR 1

#define CONFIG_CHIP_SELECTS_PER_CTRL (2 * CONFIG_DIMM_SLOTS_PER_CTLR)

#define CONFIG_DDR_SPD

#define CONFIG_FSL_DDR3

#define CONFIG_SYS_SPD_BUS_NUM 1

#define SPD_EEPROM_ADDRESS1 0x51

#define SPD_EEPROM_ADDRESS2 0x52

Memory

Controller

1 Memory

Controller

2 CS0

CS0

CS1

(23)

Misc CPU Init

► Set up local bus memory map

• BRn, ORn

► Enable timebase register

► Files:

cpu/mpc85xx/cpu_init.c

► Code Snippets:

void cpu_init_f (void)

{

volatile ccsr_lbc_t memctl = (void )(CONFIG_SYS_MPC85xx_LBC_ADDR);

:

#ifdef CONFIG_FSL_CORENET

corenet_tb_init();

#endif

init_used_tlb_cams();

/* Invalidate the CPC before DDR gets enabled */

invalidate_cpc();

(24)

Initialization

► MMU, LAW setup

► Clocks

► Serial

► DDR

► Misc CPU init

► Relocate to DDR space

► Flash

► PCIe

► Misc init, MP init

► FM init

(25)

Relocate To DDR Space

► Relocates u-boot code to top of memory

• For 2GB on P4080DS, in the last 1MB

• Set up of stack space, etc.

► Continues to run from DDR after relocation

► Files:

cpu/mpc85xx/start.S

lib_ppc/board.c

► Code Snippets:

/*

* void relocate_code (addr_sp, gd, addr_moni)

*

* This "function" does not return, instead it continues in RAM

* after relocating the monitor code.

*

* r3 = dest

* r4 = src

* r5 = length in bytes

* r6 = cachelinesize

*/

.globl relocate_code

relocate_code:

mr r1,r3 /* Set new stack pointer */

mr r9,r4 /* Save copy of Init Data pointer */

mr r10,r5 /* Save copy of Destination Address */

(26)

Initialization

► MMU, LAW setup

► Clocks

► Serial

► DDR

► Misc CPU init

► Relocate to DDR space

► Flash

► PCIe

► Misc init, MP init

► FM init

(27)

NOR Flash

► CFI flash support

► Reads on POR

• Sets up structures much later in the driver

* Localbus non-cacheable

*

* 0xe000_0000 0xe80f_ffff Promjet/free

128M non-cacheable

* 0xe800_0000 0xefff_ffff FLASH

128M non-cacheable

* 0xffdf_0000 0xffdf_7fff PIXIS

32K non-cacheable TLB0

*

► Files:

include/configs/corenet_ds.h

drivers/mtd/cfi_flash.c

► Code Snippets:

#define CONFIG_FLASH_CFI_DRIVER

#define CONFIG_SYS_FLASH_CFI

#define CONFIG_SYS_FLASH_AMD_CHECK_DQ7

#define CONFIG_SYS_FLASH_BANKS_LIST {CONFIG_SYS_FLASH_BASE_PHYS + 0x8000000,

CONFIG_SYS_FLASH_BASE_PHYS}

/* Protect RCW and FMAN microcode sectors for the NOR flash boot bank */

#define CONFIG_SYS_FLASH_AUTOPROTECT_LIST { {CONFIG_SYS_FLASH_BASE + 0x8000000, 1}, \

{CONFIG_SYS_FMAN_FW_ADDR, 1} }

This information is used by the

driver to detect the flash using CFI query

at both possible addresses and populate

its info table

(28)

PCIe

► Check for PCIe interfaces on the SOC

► Basic PCIe enumeration

• Useful when pcie devices are needed at boot time.

e.g. RTC/NVRAM on nVIDIA 1575

/*

* 0x8000_0000 0x9fff_ffff

PEX 1 Mem

512M non-cacheable

* 0xa000_0000 0xbfff_ffff

PEX 2 Mem

512M non-cacheable

* 0xc000_0000 0xdfff_ffff

PEX 3 Mem

512M non-cacheable

► Files:

include/configs/corenet_ds.h

drivers/pci/fsl_pci_init.c

board/freescale/corenet_ds/corenet_ds.c

► Code Snippets:

void fsl_pci_init(struct pci_controller *hose)

{

:

}

#define CONFIG_SYS_PCIE1_ADDR (CONFIG_SYS_CCSRBAR+0x200000)

#define CONFIG_SYS_PCIE2_ADDR (CONFIG_SYS_CCSRBAR+0x201000)

#define CONFIG_SYS_PCIE3_ADDR (CONFIG_SYS_CCSRBAR+0x202000)

:

#define CONFIG_SYS_PCIE3_MEM_PHYS 0xc000000

#define CONFIG_SYS_PCIE3_IO_PHYS 0xf8020000

(29)

Initialization

► MMU, LAW setup

► Clocks

► Serial

► DDR

► Misc CPU init

► Relocate to DDR space

► Flash

► PCIe

► Misc init, MP init

► FM init

(30)

Misc Init/MP Init

► Power-on Reset

• Core0 comes out of reset at the reset vector 0xFFFF_FFFC and all

other cores are in boot hold off mode

Core0 runs u-boot

–

After setting BSTRL/BSTAR to secondary core’s start page, core0 kicks off each

additional secondary core

–

Each secondary core comes out of reset at __secondary_start_page(boot page),

initialize resources specific that that core (caches, MMU, etc.) and enters a spin

loop

(31)

Multicore Boot Architecture

► ePAPR describes specifics on how secondary CPUs are booted for a system with

multiple CPUs

► Default boot architecture

• The boot program releases all CPUs from hardware reset

• One CPU is designated to be the client program’s boot CPU

• All other CPUs are secondary and are placed into loop where the CPUs spin,

waiting for a spin table field to change that directs them where to go

• Control is transferred to the client program on the boot CPU

• When the client program is ready for secondary cores to start, it releases them

by writing the spin table field with the desired address

► The architecture allows for other custom-defined secondary CPU release

mechanisms as well

► Reference: ePAPR

(

Power.org Standard for Embedded Power Architecture™ Platform Requirements (ePAPR). power.org,

(32)

SMP Boot Process: U-boot Boot Process

► In the u-boot source, the entry point is the reset vector 0x0_FFFF_FFFC

containing a simple branch to _start_e500, which is at the base of default

boot page at 0x0_FFFF_F000; both are found in cpu/mpc85xx/start.S

_start_e500() will:

Reset vector

call _start_e500()

_start_e500()

1. Enable L1 caches

2. Config interrupt vectors

3. Config MMU and LAWs

4. Config L1D RAM

5. Config local bus

6. Config DDR

7. Relocate to DDR

Prepare device

trees

1. Copy HW device tree to

DDR

2. Copy HV config tree to

DDR

3. Copy hv.uImage to DDR

4. Launch hypervisor

Boot other cores

Start command

_interpreter

1. Setup BSTRL and

BSTAR

2. Release all other cores

from boot holdoff using

BRR and wait for them

to boot.

(33)

Initialization

► MMU, LAW setup

► Clocks

► Serial

► DDR

► Misc CPU init

► Relocate to DDR space

► Flash

► PCIe

► Misc init, MP init

► FM init

(34)

FM Init

► FMan Independent mode to get basic

networking for tftp downloads

► Needs configuration data

► Configuration data resides at 0xEF000000

MAC

(RX data)

FM BMI

(put MAC data in

internal buffers)

FM BMI

(allocate internal

context buffers)

Enable

Rx Data

RISC

(generate DMA to read

data from BDs)

RISC

(generate DMA to write

data to BDs)

DMA

(read data)

DMA

(write data)

BMI

(read data)

MAC

(transmit TX data)

FM BMI

(free internal buffers)

(35)

FM Init – Independent mode

► Files:

include/configs/corenet_ds.h

drivers/net/fm/*

board/freescale/corenet_ds/corenet_ds.c

► Code Snippets

:

**int board_eth_init(bd_t *bis)**

{

:

p4080_setup_serdes();

**/* FM 1 DTSEC */**

if (is_pcie_configured(prtcl, SGMII_FM1)) {

printf("FM1 SGMII\n");

#ifdef CONFIG_FM1_ETH0

SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 0)

fm1_idx++;

#endif

#ifdef CONFIG_FM1_ETH1

SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 1)

fm1_idx++;

#endif

#ifdef CONFIG_FM1_ETH2

SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 2)

fm1_idx++;

#endif

#ifdef CONFIG_FM1_ETH3

SET_STD_FM_ETH_INFO(fm1_info[fm1_idx], 1, 3)

fm1_idx++;

#endif

:

(36)

Initialization

► MMU, LAW setup

► Clocks

► Serial

► DDR

► Misc CPU init

► Relocate to DDR space

► Flash

► PCIe

► Misc init, MP init

► FM init

(37)

Other New Pieces

► BMAN, QMAN, FMAN

• LIODNs

• Device tree fixups for HV/linux

cpu/mpc85xx/portals.c, liodn.c

► CoreNet Platform Cache

• Enabled by a flag in

(38)

Building u-boot

►

Using LTIB

$ ./ltib –m scbuild –p u-boot

►

From command line

• Set the compiler toolchain to compile u-boot

Set PATH variable on the build machine

Use ARCH=ppc and CROSS_COMPILE flags

make CROSS_COMPILE=powerpc-linux-gnu- ARCH=ppc distclean

make CROSS_COMPILE=powerpc-linux-gnu- ARCH=ppc P4080DS_config

make CROSS_COMPILE=powerpc-linux-gnu- ARCH=ppc

(39)

P4080DS NOR Flash Map Cheat Sheet

(from Bank 0 perspective after U-Boot DDR relocation)

SW7[1:4] = 0b0111

U-Boot PBL @ 0xE0000000

SW7[1:4] =0b0110

SW7[1:4] = 0b0101

SW7[1:4] = 0b0100

SW7[1:4] = 0b0011

SW7[1:4] = 0b0010

SW7[1:4] = 0b0001

SW7[1:4] = -0b0000

U-boot @ 0xEFF80000

0xE8000000 – 0xE8FFFFFF

0xE9000000 – 0xE9FFFFFF

0xEA000000 – 0xEAFFFFFF

0xEB000000 – 0xEBFFFFFF

0xEC000000 – 0xECFFFFFF

0xED000000 – 0xEDFFFFFF

0xEE000000 – 0xEEFFFFFF

0xEF000000 – 0xEFFFFFFF

Bank 2

Bank 5 PBL Image

Bank 0

Bank 1

Bank 3

Bank 4

Bank 5

Bank 6

Bank 7

Bank 0 PBL Image

Bank 1 PBL Image

BANK 2 PBL Image

Bank 3 PBL Image

Bank 4 PBL Image

Bank 6 PBL Image

Bank 7 PBL Image

(40)

Common (Helpful) Commands

⇒

help

⇒

Environment variables

⇒

printenv

⇒

setenv

⇒

saveenv

⇒

NOR Flash

⇒

flinfo

⇒

erase <start addr> <end addr>

⇒

cp.b <from addr > <to addr in flash> <#bytes>

⇒

Network

⇒

setenv [ipaddr][gatewayip][netmask][serverip] <ip address>

⇒

tftp <dest address in RAM> <file on server>

⇒

Other

⇒

imls - Lists all u-boot recognizable images in flash

(41)

U-boot Hooks To Launch Hypervisor

► 36-bit addressing

• U-boot uses 36-bit addressing but is limited to 4GB

HV uses 36-bit addressing to support multiple partitions and resources

• Physical addresses of the IO are moved to the end of the 64GB address

space

DDR address space starts at 0; now has room to expand

► New commands

• Device trees for HW and configuration

Pass HW device tree address to “bootm” command

Location of configuration device tree is passed “config-addr” env variable

=> setenv bootcmd bootm e8700000 - e8800000

=> setenv bootargs config-addr=0xfe8900000

(42)

(43)

Terminology

► SMP – Symmetric Multiprocessing

► ASMP – Asymmetric Multiprocessing (aka. AMP)

► CAMP – Cooperative AMP

► Guest OS – OS running in guest supervisor state under a hypervisor

► Hypervisor - Manages globally shared resources, more privileged

than operating systems, enforces system security, virtualizes some

resources

► Light Weight Executive (LWE) - Provides a communications

mechanism between an application running in a partition and the

hypervisor software

► Device Tree - A data structure used for representing a partition’s

(44)

Partitioning With A Hypervisor

CPU

partition

App

Memory

partition

App

partition

App

Memory

I/O

Hypervisor

Shared

Cache

I/O

Interrupt

Controller

Linux

®

_RTOS

_{Legacy OS}

Multicore

System

Hardware

► Hypervisor Software

• Analogous to role of an operating

system kernel in managing user

processes

• More privileged than operating systems

• Enforces system security

• Manages globally shared resources

• Virtualizes some resources– e.g.

(45)

e500mc Privilege Levels – Guest State (GS) MSR bit

OS

App

partition

User

MSR[PR=1][GS=1]

Kernel/Supervisor

MSR[PR=0][GS=1]

Hypervisor

MSR[PR=0][GS=0]

Under Hypervisor

OS

App

partition

User

MSR[PR=1][GS=0]

Kernel/Supervisor

MSR[PR=0][GS=0]

Bare Metal

CPU

Memory

I/O

CPU

Memory

I/O

Hypervisor

(46)

Hypervisor Features

virtual CPU

(e500vcpu)

services

boot services

_(ePAPR)

Emulation

(privileged instructions)

guest operating system

debug stub

Hypervisor

Debug console

hypercalls

device tree

system hardware

UART

device

tree

direct

I/O

mux

UART

Doorbells

GPIO

PIC

IOMMU

_Channels

Byte

Partition

Mgmt

► Operating System sees a virtual core plus hypervisor services

• Virtual CPU (like e500mc

minus hypervisor features)

• Services via hypercall

• Debug stub interface for

debugging guest operating

systems

(47)

Virtual CPU

► Like e500mc minus the hypervisor extensions

► Full emulation is used

► MMU

• tlb access instructions are emulated

• Guest receive and manage TLB miss interrupts directly

► Timer

• DEC, FIT, watchdog, timebase synchronization

► Cache control

• L1,L2

► Intra-partition signaling (msgsnd/msgclr)

► Debug services (DBCRx, IACx, DACx, etc)

► Core performance monitor

(48)

Virtual CPU – Emulation Example

► Operating system manages it’s own page tables. Hypervisor has no

knowledge of OS page table format.

► Hypervisor traps and authenticates all TLB updates

► lwz

r3,(r4)

► Applicati

on

► OS

► (tlb miss handler)

tlbwe

► tlb miss

_►

_privilege

trap

► Hypervisor

► validate

► tlb writes

(49)

Hypervisor Services – hcalls

► Interrupt controller (MPIC)

► Byte-channels – character I/O stream

► Inter-partition signaling – doorbell

► Partition management

• Start/stop/image-loading

In future: Create/destroy

• Partition management interrupts

► Power management– change clock frequency, power states

► GPIO– supports partitioning of GPIO pins

(50)

Device Trees

u-boot

hypervisor

HW

Dev tree

Guest #2

dynamically

created and

loaded into

guest

memory

Loaded into

hypervisor

memory by

u-boot.

Hardware dev

tree /chosen

node points to

HV config tree

Guest #1

Guest Dev tree

(51)

Boot Process

► Hypervisor initialization

• PAMU initialization

• Coherence domain setup– set up LAWS, CSDIDs

• Error Configuration– e.g. single bit ECC error thresholds

• Driver initialization

DDR

CPC

CCM

UART

• Release secondary CPUs

• Partition instantiation/creation

The boot CPU for each partition takes care of partition creation

(52)

HV Commands

► Commands list and uses

HV> ?

Commands:

help (?) - Print command usage information

version - Print the hypervisor version

list-partitions (lp) - List partitions

guest-device-tree (gdt) - Guest device tree operation

master-device-tree (mdt) - Display master device tree

paact - Dump PAMU's PAACT table entries

start - Start a partition

restart - Start a partition

stop - Stop a partition

pause - Stop a partition

resume - Stop a partition

HV>

(53)

(54)

Overview

► P4080 Linux Overview

• P4080 BSP Linux – Bare Metal and Guest OS

• Linux Kernel Configuration

• Device Tree

• SMP Boot Process

► DPAA and the Linux Ethernet Driver

• DPAA Components

• Life of a Packet

Ingress

Egress

(55)

P4080 Linux

► Bare metal Linux

• Linux runs directly on HW

• User state MSR[PR]=1 MSR[GS]=0

• Supervisor state MSR[PR]=0 MSR[GS]=0

► Guest OS Linux

• Hypervisor SW layer between Linux and HW

• Guest User state MSR[PR]=1 MSR[GS]=1

• Guest Supervisor state MSR[PR]=0 MSR[GS]=1

• Hypervisor state MSR[PR]=0 MSR[GS]=0

(56)

Bare Metal SMP Linux

Multicore

System

Hardware

SMP Linux

®

Shared

Cache

I/O

Interrupt

Controller

I/O

CPU

App

(57)

Guest OS Linux

Multicore

System

Hardware

SMP Linux

®

_RTOS

_{Legacy OS}

App

Memory

partition

Hypervisor

Shared

Cache

I/O

Interrupt

Controller

I/O

Memory

CPU

App

(58)

Porting Guest OS (Linux) to Hypervisor

► Resources controlled by Hypervisor:

• Physical memory map

• Core: caches, MMU, SPRs (Guest OS sees a virtual CPU via emulation)

• Platform: CPC, MPIC, Partion management, Doorbell, Byte-channels,

IOMMU, GUTs, DDR (Guest OS access via HCALLs)

► Direct resources (Direct I/O access by Guest OS)

► Device Tree aware (per ePAPR, libfdt library available)

► State of HW when Guest OS Runs (per ePAPR)

► Hypercall API (HCALLs for shared services, ie VMPIC)

► Traps (emulation - vCPU)

(59)

Building Linux

► LTIB BSP

• Configure default LTIB settings

$ ./ltib

• Configure menu for P4080DS, then automatically build

$ ./ltib --configure

• In configure menu, select:

Select “Configure kernel” to alter the default kernel build configuration

Select “Leave the sources after building” to leave the linux source in rpm/BUILD/linux

► From GIT tree

• make CROSS_COMPILE=powerpc-linux-gnu- ARCH=powerpc menuconfig

• make CROSS_COMPILE=powerpc-linux-gnu- ARCH=powerpc uImage

(60)

P4080 Linux Kernel Configuration

► The easiest way to create a .config file is to simply copy the appropriate

config file in ./arch/powerpc/configs to .config.

• cp arch/powerpc/configs/p4080_ds_defconfig .config

► Typing “make menuconfig” results in a console-based, menu-driven utility

(61)

Linux Directory Structure

net

./linux-2.6.xx

Doc.

arch

block

crypto

drivers

fs

include

init

scripts

security

kernel

lib

mm

sound

ipc

usr

► arch/powerpc/boot/dts/p4080ds.dts

► arch/powerpc/configs/p4080_ds_defconfig

► arch/powerpc/boot/

(62)

P4080 Linux Device Tree

► Baremetal Linux

► Use the device tree compiler to build the appropriate dts file in the

./arch/powerpc/boot/dts directory.

dtc -I dts -O dtb -R 8 -b 0 -S 0x3000 p4080ds.dts > p4080ds.dtb

OR

make p4080ds.dtb (from linux-2.6/)

► Guest OS Linux

• The device tree for each partition is generated by Hypervisor from a system

(63)

Kernel Configuration Options - Hypervisor

► CONFIG_P4080_HV - Freescale P4080 Hypervisor platform

► CONFIG_FSL_HYPERVISOR - Freescale P4080 Hypervisor

► CONFIG_VMPIC - Freescale P4080 Hypervisor - Virtualized

(64)

SMP Boot Process: Linux

®

_{SMP Boot Process}

smp_prepare_cpu()

smp_init()

smp_ops->probe()

cpu_up()

smp_ops->kick_cpu()

smp_ops->setup_cpu()

...

in main.c

► In the Linux® kernel, the entry point is _start found in

arch/powerpc/kernel/head_fsl_booke.S

start _kernel() is in init/main.c, it will:

_start

call _start_kernel()

_start_kernel()

Kernel_init()

setup_arch()

smp_cpus_done()

…

mpc85xx_smp_init() {

smp_ops =

&smp_85xx_ops;

} ……

(65)

SMP Boot Process: Core1 Start Up (continued)

► Let’s look at smp_mpc85xx_kick_cpu() found in:

arch/powerpc/platforms/85xx/mpc85xx_smp.c

smp_mpc85xx_kick_cpu() {

…….

**/* Get the BPTR */**

bptr_vaddr = ioremap(get_immrbase() + MPC85xx_BPTR_OFFSET, 4);

**/* Set the BPTR to the secondary boot page */**

oldbptr = in_be32(bptr_vaddr);

bptr = (BPTR_EN | (pa((unsigned)secondary_start_page) >> 12));

out_be32(bptr_vaddr, bptr);

**/* Kick that CPU */**

smp_85xx_release_core(nr);

**/* Wait a bit for the CPU to take the exception. */**

while ((__secondary_hold_acknowledge != nr) && (++n < 1000))

mdelay(1);

**/* Restore the BPTR */**

(66)

SMP Boot Process: Core1 “Kick-Off” Process

► Core 0 kicks off core 1 by setting EEBPCR[CPU1_EN]

► Let’s look at smp_85xx_release_core() in

arch/powerpc/mpc85xx/mpc85xx_smp.c

smp_85xx_release_core(int nr)

{

……

/*

* Startup Core #nr.

*/

ecm_vaddr = ioremap(get_immrbase() + MPC85xx_ECM_OFFSET,

MPC85xx_ECM_SIZE);

pcr = in_be32(ecm_vaddr + (ECM_PORT_CONFIG_OFFSET >> 2));

pcr |= EEBPCR_CPU1_EN;

out_be32(ecm_vaddr + (ECM_PORT_CONFIG_OFFSET >> 2), pcr);

(67)

SMP Boot Process: Core1 Kernel Code Flow

►

Core1 starts up from __secondary_start_page defined in

arch/powerpc/kernel/head_fsl_booke.S

• __secondary_start_page:

4K

The code in this page must not exceed 1023 instruction. The1024th is a branch instruction.

It will initialize I-cache and D-cache

Use TLB1[1] to map 16M memory

Jump to __early_start

• __early_start

If it is core1, it will not call start_kernel(), but jump to __secondary_start

• __secondary_start

jumps to start_secondary()

►

start_secondary() defined in arch/powerpc/kernel/smp.c

• Call smp_ops->setup_cpu()

• ……

(68)

(69)

Multicore Datapath Issues and Requirements

► Multicore SoCs, like the P4080, have a

number of new requirements related to

packet processing when compared to

single core SoCs:

• Load spreading of arriving packets across

pools of cores for parallel processing

• Packet ordering issues after processing

• Pipelined processing of packets using

cores

• Share network I/O between cores

• “Virtualizes” hardware accelerators

• Inter-core communication

Core

D$

I$

Network

I/O

Hardware

Accelerator

Network

Core

D$

Core

I$

D$

Core

I$

D$

I$

Core

D$

I$

(70)

More Multicore Datapath Requirements

► Addressing these requirements can lead to new requirements:

• Hardware managed queues lead to the need for

hardware-supported active queue management

• Network interfaces must able to parse, classify, and distribute (PCD)

frames.

► High-bandwidth network I/O as found on P4080 also drive datapath

requirements:

• Queue congestion driven flow control

• Resource depletion driven flow control

• Hardware buffer management

(71)

Frame Manager

1GE 1GE

10GE

QorIQ

™

_{DPAA Components}

► “Infrastructure” components

• Queue Manager (QMan)

• Buffer Manager (BMan)

► Network I/O

• Frame Manager (FMan)

► Hardware accelerators

• SEC – cryptographic accelerator

• PME – Pattern matching engine

► Cores

► CoreNet is not part of the DPAA

but it provides the interconnect

between the cores and the DPAA

infrastructure as well as access

to memory (DRAM)

D$

I$

D$

I$

L2$

e500mc

Core

D$

I$

Multi-Lanes SERDES

CoreNet™

Coherency Fabric

Sec 4.0

PME 2

Buffer

Mgr

Frame Manager

1GE 1GE

10GE

D$

I$

D$

I$

L2$

e500mc

Core

D$

I$

Queue

Manager

…

(72)

FMan/QMan/BMan Ingress Packet Processing

10G

_1G

Buffer Acquisition Request

Buffer Reference

Packets

in

process

FMan

Buffer

QMan

Packet Data written to main

memory subsystem

BMan

Frontside Cache

_{DDR SDRAM}

…

References to Packet

(Frame Descriptors)

Packet Data Stored in

H/W managed buffers

1

2

4

3 References to Buffers

Parse, classify/hash, select queue

₅

(73)

Priority based packet

scheduling

FMan Egress Packet Processing

10G

1G

Packets Transmitted

FMan Releases

Buffer to BMan

Packets

in

process

FMan

Buffer

QMan

Bman

Frontside Cache

DDR SDRAM

…

4

5 Class scheduler

8 Priority Work Queues

Packet Data read from

main memory subsystem

3

6 Optional packet

response

2 Think ARP response

(74)

Aggregate Driver Structure

DPA net_device Linux Driver

(fsl-dpa)

drivers/net/dpa

BMan Linux Driver

QMan Linux Driver

FMan/Port Linux

Driver

FMan/MAC Linux

Driver

drivers/hwalloc

drivers/hwqueue

drivers/net/dpa/NetComSw

_{drivers/net/dpa}

Low Level Driver

FMan/PCD/Port

(FMD)

(75)

Kernel Configuration Options – Bare Metal

► CONFIG_FSL_BMAN - Freescale BMan support

► CONFIG_FSL_BMAN_PORTAL - BMan portal support - compiles support to detect and support BMan software

CoreNet portals (as provided by the DT)

► CONFIG_FSL_BMAN_CONFIG - BMan device management - if this Linux image is running natively, you need this

option. If this Linux image is running as a guest OS under the hypervisor, only one guest OS ("the control plane")

needs this option and the DT node

► CONFIG_FSL_QMAN - Freescale QMan support

► CONFIG_FSL_QMAN_PORTAL - QMan portal support - compiles support to detect and support QMan software

CoreNet portals (as provided by the DT)

► CONFIG_FSL_QMAN_CONFIG - QMan device management - if this Linux image is running natively, you need this

option. If this Linux image is running as a guest OS under the hypervisor, only one guest OS ("the control plane")

needs this option and the DT node

► CONFIG_FSL_QMAN_FQALLOCATOR - QMan FQ allocator - implemented using BMan BP 0 as a system wide

simple FQ allocator/manager

(76)

(77)

Light Weight Executive (LWE)

► LWE is a set of lower level platform libraries used for development

of data plane networking applications

► LWE libraries enable application software to run in OS-less

environment.

► Applications based on LWE run in a partitioned environment under

(78)

LWE Application Programming Model

► Single application per partition

► Run to completion

► Thread model

► Global variables

► Per CPU variables

(79)

LWE Applications

► LWE Apps

• Provide an example of how to use LWE Libraries

• Out-of-box benchmark code

• Starting point for users to develop their own high performance data path

code

► LWE Apps Examples

• Ethernet Bridging

• Hello World

• Simple Queueing (Pktwire)

• Queueing Torture Test

• Simple Crypto

• IPSecFwd

• FMan Test

• Simple PME

(80)

Partition Scenarios

► hv-4p

• 2-core Linux partition

• Single core Linux partition

• 4-core LWE partition

• Single core LWE partition

• Requires use of Hypervisor

CPU

White Board Shared Memory

Linux

LWE

Data Path Shared Memory

Private

Memory

Private

Memory

Linux

LWE

Private

Memory

Private

Memory

Ethernet

Shared

Memory

(81)

Session Summary

► U-boot, Hypervisor, Linux for P4080DS BSP

• Concepts

• Porting

(82)

(83)

What is DPAA and Why do I Need It?

► A multicore SoC must be able to:

• Load spread RX’ed packets across multiple cores to facilitate “parallel processing”

• Ensure packets are TX’ed in the same order that they were RX’ed especially after

parallel processing

• Support “pipelining” of cores

• Support inter-core communication

• Share network interfaces between cores

On ingress when the different cores providing different services

On egress

• Share hardware offload accelerators (e.g. SEC, PME) between multiple cores

• Address performance issues that arise from high bandwidth network interfaces (e.g.

10GE) prevalent on such an SoC

• Continue to support key functionality that exists in network interfaces today e.g.

IEEE1588

QoS

(84)

Addressing Multi-Core Datapath Requirements

► Many of these requirements (“share …”, “pipeline …”, “… inter-core communication

…”) result in the need for the ability to define communications paths between all the

many blocks in the “datapath”

• Thus … QMan

► Load spreading packets and sharing network interfaces require hardware to “parse,

classify, and distribute”

• FMan’s PCD functionality

► Effective load spreading requires that multiple “consumers” can share queued data