Using Embedded Linux for Infrastructure Systems

(1)

Using Embedded Linux for

Infrastructure Systems

Embedded Linux Conference Europe 2014 14 Oct 2014

(2)

Scope of this presentation



Make a Place for collaboration



NOTE

 This project currently belongs to CE workgroup, Linux

Foundation

(3)

Outline



Definition of Social Infrastructure Systems



Problem statement



Expected goal



Project scope



Activities

(4)

Definition of Social Infrastructure Systems (SI systems)



Embedded systems which uses for the following

infrastructure systems

 Transport infrastructure  Energy infrastructure

 Water management infrastructure  Communications infrastructure

 Earth monitoring and measurement systems



The system includes

 Automation systems  Controllers  PLC  Microcontrollers  Signal controller  Multi-purpose controller

(5)

An Example of Infrastructure System

sensor Controller

(6)

IoT?

sensor Controller (e.g. PLC) THE INTERNET OF THINGS

(7)

Infrastructure System becomes a part of IoT

sensor

THE INTERNET OF THINGS THE INTERNET

OF THINGS

Moving Forward

Controller (e.g. PLC)

(8)

Problem statement



The SI systems will be work with other systems

such as:

 Cloud computing

(9)

Problem statement



To connect with other systems, SI systems

 Need to have same quality with new requirement

 Need to have new technology to have new functionality



New requirements

 Ensure a complicated performance requirement

 Nondeterministic and deterministic programs runs on a

same board

 Collaborate with cloud platform



New functionality

(10)

Goal



Main goal

 Develop reference implementations for SI system



Reference implementation may includes:

 Linux kenel and filesystem for some target board

 Target board: TBD

 Open spec hardware is better for the implementation

 Linux kernel related implementations should post to LKML

for mainlining

(11)

Project scope

 Security

 Protect from irregular activities (e.g. viruses)  Integrity check

 Dependability  Robustness

 Run-Time Maintenance / Update

 e.g. Kpatch / kGraft

 Real-time

 Hard real-time

 Multi-purpose real-time system

 Tests

 Reliability test

 Long Term Support

 VLTS = Very Long Term Support

 Legal/Compliance Issue

 (Possibly) Functional Safety

(12)

Activities



This project will be separated into several

different sub-project by its key technology



Initial activity

 Survey to determine current common key issues for SI

systems

 Create a mail list

 F2F meeting at Linux Foudation’s conference

 Embedded Linux Conference / LinuxCon

 Workgroup meeting will be arranged

 Reviewed by members of the CE Workgroup Architecture

(13)

Estimated output



Publicly available pages on the elinux wiki which

includes the follows:

 A list of issues

 Specification for reference implementations

 A list of link to reference implementations

 All kernel related implementation should post to LKML



Reference implementations for key issues will be

(14)

Current status



Under survey to determine actual issues by

discussion

(15)

Project scope

 Security

 e.g. kpatch / kGraft

 Real-time

 Hard real-time

 Tests

 (Possibly) Functional Safety

(16)

Project scope

 Security

 e.g. kpatch / kGraft

 Real-time

 Hard real-time

 Tests

(17)



Real-time

 Improve scheduring granularity for SCHED_DEADLINE

 Hardware resource isolation to run RT and GP process



Test

 Deta reliability evaluation for Linux filesystems



Other

(18)

Issue on SCHED_DEADLINE

dl_timer 1.5ms task_tick_dl 1ms dl_timer 1.5ms 1ms

Step1: Refill budget

Step2: Use budget

2ms



Difficult to keep task’s budget if the budget has

micro seconds granularity

(19)

Support for micro seconds granularity

Period: 1ms, Budget: 0.5ms

Wakeup

Period Task execution

Set HRTICK here Step1: Refill budget

Step2: Use budget

Time 3.2ms

1ms 2ms 3ms

Wakeup

 When a task’s budget is less than 1ms, set HRTICK for the rest of budget

(20)

Let’s play a movie

Modified

Original

“Big Buck Bunny” http://www.bigbuckbunny.org/

(21)



Realtime



Reliability



Functionality

(22)

RT process and GP process runs on a board

CPU Device 1 Device 2 Hardw are Appli ca ti on s OS Linux RT App Core 1 Core 2 GP App 

Goal

 Run RT processes and GP processes on a hardware platform

CPU Device 1 Device 2 RT App RT core GP core GP App Linux

(23)

An example for isolation demo

Eth0 SD UART Core1 Core0 Web server RT control

Pandaboard

Workload

(Web clients)

Puppy

Motor Sensors (Gyroscope, etc.) Motor torque Sensor values CAN RS232C Serial-CAN converter

Point

RT Control and UART interrupts are isolated from the Web server and other interrupts.

IRQ IRQ

(24)

(25)

An issue while control the robot

Latency [μs]

Latency for RT app while running with GP appliation

Evaluation Environment

Hardware

・ Pandaboard

- ARM Dual core 1.2GHz - Memory 1GB - SDcard OS ・ Linux 3.6.11.6-RT38 Application ・ RT App - Periodic task

Period = 300 micro sec ・ GP App

- Random I/O on SDcard 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 600 700 800 900 Num ber of sampl es Before (CPUset)

This latency cause a failure

(26)

Realtime performance improvement with Timer-Shielding

Timers Timers Timer management are Appl ications Linux _Kernel RT core GP core RT App GP App Timers Timers Timer management are App lica tio ns Linux _Kernel RT core GP core RT App GP App Migrator

Hardware Partition Management （Create / Modify / Delete）

Migrator

Prohibit to resister non RT related

timers

Migrate

Migrate non RT relate timers API to manage hardware partitions Re giste r Issue

Some kernel task does not able to allocate to target core

(27)

An issue while control the robot

0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 Number of sampl es Latency [μs] After Latency is less than 60us Latency [μs]

Latency for RT app while running with GP appliation

Evaluation Environment

Hardware

・ Pandaboard

- ARM Dual core 1.2GHz - Memory 1GB - SDcard OS ・ Linux 3.6.11.6-RT38 Application ・ RT App - Periodic task

Period = 300 micro sec ・ GP App

- Random I/O on SDcard 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 600 700 800 900 Num ber of sampl es Before (CPUset)

(28)



Realtime



Test



Other

(29)

Data reliability test

Checker Target file LOG file AAAAA BBBBB CCCCC DDDDD EEEEE OK FFFFF AAAAA BBBBB CCCCC DDDDD EEEEE OK AAAAA BBBBB CCCCC DDDDD AAAAA NG AAAAA BBBBB CCCCC DDDDD NG ? size mismatch data mismatch

Estimated file contents

Estimated file size



Power off while running writer processes



Verify all data

 File size

(30)

Results of data reliability tests on Linux filesystems

0.00 0.50 1.00 1.50 2.00 EXT3-ORDERED EXT3-JOURNAL * * * JFS XFS Error rate [%] 45.9% Error rate [%] 1.00 1.50 2.00 82.4% 84.7% 43.4% 41.4% 43.2% rate [%] 0.00 0.50 1.00 1.50 2.00 EXT3-ORDERED EXT3-JOURNAL EXT4-JOURNAL EXT4-ORDERED * JFS XFS 13.3% kernel 2.6.18 kernel 2.6.31 kernel 2.6.33

File size mismatch rate Data mismatch rate

Point 1

：

An filesystem has

different characteristics on different kernel

Point 2:

2.6.33 has high error rate on ordered and writeback mode

Point 3:

kernel

vers

(31)



Realtime



Test



Other

(32)

Long-term running test by accelerated kernel

START GOAL

Accelerate

Accelerate clock counter jiffies = jiffies + (1 * ratio)

Issue

• Long-term testing takes really long time

→ _{We want results as fast as possible}

Things that NOT able to be accelerated.

• CPU frequency

(33)

Evaluation with accelerated kernel

Xdaliclock works as a stopwatch

About 40 times faster

to get 100% CPU usage

returned incorrect value

after about 450 days

(34)

Enough?



Real-time latency is short enough



Data is safe enough

(35)

Summary

 There are many issues need to be solved for SI system

 CE workgroup is considering to have sub project for it

 Current status

 Under survey for actual issues  Start call for participation

 Initial activity will be the follows:

 Clarifying issues to be solved  Create a mail list

 Have F2F meetings at Linux Foundaton’s conference

 Estimated output

 Publicly available pages on the elinux wiki or Linux Foundation’s web site

which includes the follows:

 A list of issues

 Specification for reference implementations  A list of link to reference implementations

 All kernel related implementation should post to LKML  All source code for key issues will be opened

(36)

Discussion



Any comments and suggestions are welcome



Contact information

 To get the latest information, please send an email to the

following adderss:

 Yoshitake Kobayashi [email protected]

(37)

(38)

(39)

https://github.com/ystk/sched-deadline

https://github.com/ystk/fs-test