Using Toaster in a Production Environment

(1)

Yocto Project | The Linux Foundation

Using Toaster in a

Production Environment

Alexandru Damian , David Reyna, Belén Barros Pena

Yocto Project Developer Day  ELCE  17 Oct 2014

(2)

Introduction

Agenda:

• What is Toaster

• Toaster out of the box

• Toaster in a production environment

•Comparison of database servers

•Comparison of web servers

•Toaster: interaction with auto build and auto test tools

• How-to guide of a best practice configuration

(3)

What is Toaster

• The new UI interface

for the Yocto Project

• Project and

management isolation

and scalability

• System scalability,

remote management

(4)

Toaster out of the box

• Toaster provides a simple out-of-box configuration,

where a few commands will automatically set up an

sqlite database and start a Django web server.

• This default setup is sufficient to immediately support

a small number of users and projects, before you move

on to the more robust and scalable solutions.

Host

Django Server Bitbake

Toaster

Project

sqlite

(5)

How-to: Starting Toaster out of the box

• Set up the project

• Start Toaster data collection and web server

• Build the project

• Open Toaster in browser and see results

$ source poky/oe-init-build-env $ source toaster start

$ bitbake core-image-minimal

$ xdg-open http://localhost:8000/ $

(6)

Yocto Project | The Linux Foundation Yocto Project | The Linux Foundation

(7)

Advantages of Production Environment for Toaster

• Multiple clients

• Multiple build machines

• Scalability and reliability

• Ability for long-term data trending analysis

DB Host sql

Web Host

_Client

s

Web Server

Build Host

Build Blade(s) Bitbake Toaster Projects Browser

(8)

Review of Database Servers

• Sqlite3 default server

• Advantages:

• No configuration needed, easy installation

• Emphasizes economy, efficiency, independence, and simplicity for data storage local applications and devices

• Disadvantages:

• No remote host access, slow, no transactional support

• MySQL or PostgreSQL server

• Advantages:

• Advanced SQL features, fast, reliable

• Emphasizes scalability, concurrency, centralization, and

control

• Disadvantages:

• Some more complex setup,

(9)

Review of Web Servers

• Django development server

• Advantages:

• Available with out-of-box Django installations

• No configuration needed

• Disadvantages:

• Very poor performance, no customization, security issues

• Apache or nginx server

• Advantages:

• Very fast, very configurable

• Disadvantages:

• Needs complex configurations, and a system

administrator to look after it

(10)

Toaster and autobuilders/auto-testers

• Toaster is not a replacement for your continuous

integration software

• Toaster interacts with your continuous integration

software

• Toaster provides an interface that is customized and

fine-tuned to the capabilities (and quirks) of the Yocto

Project build system

• Which means an easier way to configure and debug

your Yocto Project builds

(11)

How-to Guide

Setting up a Toaster production environment

with MySQL and Apache

(12)

Checkout toaster for server use

• Select location for production toaster, and checkout

bitbake

• Install all dependencies

• Make it reachable by the Apache user

$ mkdir /opt/ && cd /opt/

$ git clone git://git.openembedded.org/bitbake #production

...Or for the test-to-merge branch

$ git clone -b toaster/install_test --single-branch \ http://git.yoctoproject.org/git/poky-contrib bitbake

$ pip install -r /opt/bitbake/toaster-requirements.txt

(13)

Configuring MySQL

• Install mysql-server

• Create a sql user account, and database with minimal

rights (not like in the example)

• Update bitbake/lib/toaster/toastermain/settings.py

root$ apt-get install mysql-server root$ mysql -u root

mysql> CREATE USER 'newuser'@'localhost' IDENTIFIED BY 'password'; mysql> GRANT ALL PRIVILEGES ON * . * TO 'newuser'@'localhost'; mysql> GRANT ALL PRIVILEGES ON * . * TO 'newuser'@'localhost'; mysql> CREATE DATABASE 'toaster';

root$ cat /opt/bitbake/lib/toaster/toastermain/settings.py …

DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'toaster', 'USER': 'newuser', 'PASSWORD': 'password', 'HOST': '127.0.0.1', 'PORT': '3306', }

(14)

Configuring Apache

• Install Apache and mod_wsgi

• Configure Apache to load Toaster

root$ apt-get install apache2 libapache2-mod-wsgi root$ a2enmod wsgi

root$ cat /etc/apache2/sites-available/000-default.conf

… # the WSGIPythonPath is global

WSGIPythonPath /opt/bitbake/lib/toaster/

… #snip – in VirtualHost

WSGIScriptAlias / /opt/bitbake/lib/toaster/toastermain/wsgi.py <Directory //opt/bitbake/lib/toaster/toastermain/>

<Files wsgi.py>

Require all granted </Files>

</Directory> …

(15)

Configuring Apache (continued)

• Collect static media from Toaster

• Configure Apache to serve static media

$ mkdir /var/www/html/static && cd /var/www/html/static $ /opt/bitbake/lib/toaster/manage.py collectstatic

$ cat /etc/apache2/sites-available/000-default.conf

… # in VirtualHost, AHEAD of the WSGIScriptAlias definition

Alias /static/ /var/www/html/static/ <Directory /var/www/html/static/> Require all granted

</Directory> …

WSGIScriptAlias / /opt/bitbake/lib/toaster/toastermain/wsgi.py …

(16)

Starting up Toaster

• Sync databases for toaster

• Start apache

root$ /opt/bitbake/lib/toaster/manage.py syncdb

root$ /opt/bitbake/lib/toaster/manage.py migrate orm

root$ service apache2 restart

# when happy with the toaster settings.py, # save it in your local git:

root$ cd /opt/bitbake

(17)

Use Toaster to log your builds

• Configure Database settings in your source tree

settings.py, build and see results

# set database in the poky source tree to point to the same database instance

user$ cat ~/poky/bitbake/lib/toaster/toastermain/settings.py …

DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'toaster', 'USER': 'newuser', 'PASSWORD': 'password', 'HOST': '127.0.0.1', 'PORT': '3306', }

# build as normal with toaster

user$ source oe-init-build-env user$ source toaster start

user$ bitbake core-image-minimal

# open up the web interface to see your builds

(18)

Some Big Gotchas

• Under wsgi, all Python environment (like libs, interpreters, etc) are shared on the same instance of Apache. This may lead to some very un-intuitive results if you want to run two different django apps on the same apache server. The recommended way to multiplex django apps under a single server is run each Django app in its own apache

instance, and provide a single routing point using a reverse-proxy nginx server.

• Apache/wsgi aggressively caches the .py files in the application, in other words if you make changes to your application they won't be picked up on the fly like with the Django development server, so you need to restart apache.

• Neither the database nor the queries have been tuned for the high workloads such as on a build farm. It may work out of the box, it may not, so further testing is needed.

(19)

(20)

Real world uses of Toaster

• Wind River has been using Toaster for the last 6 months

in production environments. Lessons learned include:

•

The toaster captured build event data was being clobbered if an eventhandler was defined in any bitbake class that didn't have an eventmask

• We found that the 'best practice' to provide an event mask for each event handler (which filtered the 'right' events being sent to an event handler) was not always implemented in Wind River bbclasses

• A race condition was discovered when starting bitbake server, observer, and a build of a specific target when the launches were split out of the original upstream script