Yocto Project Experience: Continuous Integration

(1)

Yocto Project Experience:

Continuous Integration

Mark Hatle

Senior Member of Technical Staff Wind River

(2)

Agenda

• Our experiences as an OSV, productizing the Yocto

Project

• Software Lifecycle

•  Big-Bang Example

•  Continuous Integration Example

(3)

Productization

What does it take to turn the Yocto Project into

a commercial product?

(4)

Yocto Project Productization

• What does an OSVs customer’s require?

•  Up-to-date kernel

•  Up-to-date toolchain

•  Up-to-date userspace

•  One or more specific BSP (hardware support)

•  Quality improvements

(5)

Yocto Project Productization

• Up-to-date

•  When the Yocto Project release is complete, it is generally

considered to be very Up-to-date (nothing older than 6 months)

•  Up-to-date in the customer world is roughly nothing older

than one generation, or 12-18 months

•  Toolchain is at the current community supported version

•  Kernel is at the generally accepted stable version (LTSI or

otherwise)

(6)

Yocto Project Productization

• Hardware Support

•  Customers require the hardware of their choice to be

supported.

•  Generally new hardware requires newer versions of the

Linux kernel

•  Semiconductor specific optimizations for toolchains, drivers

(7)

Yocto Project Productization

• Quality

•  Anything that is released from an OSV needs to be at or

better than Open Source quality

(8)

Yocto Project Productization

• Timely support

•  When something doesn’t work, the OSV is expected to be

the expert on the problem!

•  The OSV must understand the system as a whole

•  The OSV must work with the community to find existing fixes

(9)

Software Lifecycle

(10)

Software Lifecycle

• Yocto Project Lifecycle

• Commercial Product Lifecycle

(11)

Yocto Project Lifecycle

• 6 month development cycle

•  4 – 4 week development milestones

•  5th milestone is stabilization

(12)

Yocto Project Lifecycle

4 - 4 week development milestones

(13)

Commercial Product Lifecycle Examples

Big-Bang Continuous Integration

Start with community release Work in parallel with community Add ‘missing’ requirements Influence community work

Add new value-add features Add new value-add features QA/Verify OSS

QA/Verify new components QA/Verify value-add features

QA/Verify OSS

QA/Verify value-add features Work to resolve bugs internally Work with community to fix bugs Release to customers Release to customers

Maintain Release (5-10 years) Maintain Release (5-10 years)

(14)

Big-Bang Lifecycle Example

• Big-Bang refers to the work

•  Starts with a large amount of community software

•  Need to learn how it works

•  Learn what required product features need to be

implemented

•  More of the traditional approach

(15)

Big-Bang Lifecycle Example

Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun

Yocto Project

Commercial

Update Update EOL

U1 U2 U3 U4 U5 U7

• No opportunity to influence design, must follow

• Enhancements should be contributed to the next

version, and backported

• Bugs found may or may not have been found by

community, may require additional resources to

resolve

(16)

Big-Bang Lifecycle Example

Yocto Project

Commercial

U1 U2 U3 U4 U5 U6

• At start of commercialization, product components

are up to 6 months old

• By the time of release, it’s nearly 12+ months old

• Has a “shelf life” of only 6-12 months from release

• Decision extend shelf-life or uprev?

(17)

Big-Bang Lifecycle Example

Apr-Jun

Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun

Yocto Project

Commercial

U1 U2 U3 U4 U5

Customer Adoption Commercial

Jul-Sep Oct-Dec Jan-Mar

U1 U2 U3 U4 …

• Extend shelf life?

•  Update kernel, toolchain, BSPs and other critical elements

•  Backport additional select features

•  6 months work, only gains 6-12 months Customer Adoption

•  No community help

(18)

Big-Bang Lifecycle Example

Apr-Jun

Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun

Yocto Project

Commercial

U1 U2 U3 U4 U5

Customer Adoption

Commercial

Jul-Sep Oct-Dec Jan-Mar

U1 U2 U3 U4 …

• Uprev?

•  Rebase changes (15-45 days)

•  Add commercialization time

•  Finish date has now slipped back a bit more

•  Still following, not leading…

(19)

Commercial Product Lifecycle Examples

Big-Bang Continuous Integration

Start with community release Work in parallel with community Add ‘missing’ requirements Influence community work

Add new value-add features Add new value-add features QA/Verify OSS

QA/Verify new components QA/Verify value-add features

QA/Verify OSS

QA/Verify value-add features Work to resolve bugs internally Work with community to fix bugs Release to customers Release to customers

Maintain Release (5-10 years) Maintain Release (5-10 years)

(20)

Continuous Integration Lifecycle Example

• Continuous Integration refers to tracking and

contributing to the community development

•  Work with the community on development

•  Learn capabilities and feature deficits as development

continues

•  Ability to influence the community by discussing

requirements and/or providing patches

•  Ability to monitor OSS quality over a longer period of time

•  Requires resources to follow the community

(21)

Continuous Integration Lifecycle Example

Yocto Project Commercial

U1 U2 U3 U4 U5 U6

• Ability to identify issues and work with the community

to resolve them

• Enhancements can be contributed during

development

• Bugs can be filed with the community and worked on

as a group

… Customer Adoption

(22)

Continuous Integration Lifecycle Example

U1 U2 U3 U4 U5 U6

• During commercialization components are current

• By the time of release, it’s only 6-7 months old

• Has a “shelf life” of 12-24 months from release

• No reason to extend the shelf-life!

(23)

Continuous Integration Lifecycle Example

Apr-Jun

Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar

U1 U2 U3 U4 U5 U6

…

Yocto Project Update Update EOL

Commercial U1 U2 U3 U4 U5 U6

Yocto Project Update Update EOL

• Uprev!

•  No rebase required…

(24)

Real World Examples

• Big-Bang

•  Took a large product team 6 months to commercialize

•  Required 6 one month cycles to add enhancements and

backport upstream features

•  First cycle was devoted to investigation

•  Required significant developer resources

(25)

Real World Examples

• Big-Bang Extended life

•  Required 6 months development to update kernel, BSPs,

toolchain and other customer essential systems

•  Required roughly the same development effort as a new

product

•  Release occurred at appox time of EOL of the base Yocto

(26)

Real World Examples

• Big-Bang Uprev

•  Requires 45 days (of one engineer) to update the tree 2

Yocto Project releases. This 45 days simply enabled the main development team.

•  Projected to required 6 months of development to

commercialize and add new features, QA, etc.

(27)

Real World Examples

• Continuous Integration

•  Work done in parallel with the community.

•  Able to ramp up a small team to full team over the course of

development.

•  As bugs were found, many filed with the community and

fixed in a timely manner. (Many critical bug fixes were submitted to the community.)

•  As missing features were identified, worked with the

(28)

Real World Examples

• Continuous Integration

•  Required 1 full time resource to manage integration and

tracking of the community.

•  Resource was the go-to person for questions about

community quality, bug triage, etc.

•  Continuous uprev averaged every 1-2 weeks for the first 4

milestones. Bug fixes and QA took most of the two week time. (Longer than we expected.)

•  Unexpected change in the 5th milestone caused a single 4

week integration cycle.

(29)

Real World Examples

• Continuous Integration

•  Estimated to be the same amount of “improvement” and

commercialization required

•  Smaller teams required, as less unexpected ‘reactionary’

(30)

Recommendation

(31)

Recommendations

• Semiconductor Mfg – Kernel/BSP support

•  ‘big-bang’ – address customers wanting a stable approach

•  ‘continuous integration’ – keep changes timely, and ready to

release for the next stable release

•  May depend on chip market, release schedules and

(32)

Recommendations

• OSV

•  Use continuous integration. Quicker time to market, more

up-to-date software, and longer shelf-life.

•  Should allow time to sync to semiconductor and customer

(33)

Recommendations

• ISV / Application Developers

•  Follow the YP versions your customers need. Most likely to

‘follow’ the stable release, but for large applications the continuous-integration model may make sense.

(34)

Recommendations

• Device Developers

•  Look at what your needs are.

•  If you don’t need ‘work-in-progress’ features, it’s better to

start with a stable release!

•  If you expect to be updating the OS over the life of the

(35)