Continuous Integration/Testing and why you should assume every change breaks your code

(1)

Continuous Integration/Testing

(2)

kno

wl

edg

e

Mün

s

ter

Continuous

I completely automatic triggering (nightly, on branch updates, ...)

I automatic reporting

I simple visualization

I wide accessibility

(3)

kno

wl

edg

e

Mün

s

ter

No quadrature rules contained in this talk either

Integration

I does your software stack still build after changes?

I compile, run, link, work as a dependency for other software?

(4)

kno

wl

edg

e

Mün

s

ter

Testing

I automated, repeatable, deterministic, reliable

I Unit Testing

I System Testing

(5)

kno

wl

edg

e

Mün

s

ter

assume your every change breaks everybody’s code

I saves time, no need to argue about possible effects of changes

I It’s better to err on the side of caution.You will break code from time to time.

I software is modular, codebases are huge, dependency graphs are complex

I distributed development: multiple people, branches, features/fixes, ...

(6)

kno

wl

edg

e

Mün

s

ter

I free for FOSS

I integrates with GitHub repositories

I explicitly supports C++, python, ruby, but if your test/software runs on a standard-ish Ubuntu 12.04 you’re good

I triggers on branch push, visual feedback in pull requests and badges

(7)

kno

wl

edg

e

Mün

s

ter

Tools for CI: Travis

language: python python:

-"2.7"

script:

- DISPLAY=:99.0 py.test -k "${PYTEST_MARKER}" install:

- python setup.py build_ext -i notifications: email: on_success: change on_failure: change after_success: - coveralls branches: except: - gh-pages

(8)

kno

wl

edg

e

Mün

s

ter

sudo: false language: cpp matrix: include: - os: linux compiler: gcc addons: &gcc49 apt: sources: [’ubuntu-toolchain-r-test’] packages: [’g++-4.9’,’gcc-4.9’] env: CXX_COMPILER=g++-4.9 COMPILER=gcc-4.9 - os: linux

compiler: gcc addons: *gcc49

(9)

kno

wl

edg

e

Mün

s

ter

(10)

kno

wl

edg

e

Mün

s

ter

(11)

kno

wl

edg

e

Mün

s

ter

Tools for CI: Buildbot

I Python framework to build CI systems, not a ready made solution

I Master/Slave setup

I good for testing complex stacks on diverse architectures

(12)

kno

wl

edg

e

Mün

s

ter

I setup to deploy new masters with only 5 LOC

I buildbot code for that to work is 2300 LOC

I git repository tracks individual DUNE-modules´repositories -> dynamic build steps -> make test, result log

I tied into redmine

(13)

kno

wl

edg

e

Mün

s

ter

Tools for CI: Buildbot - Example (master config)

from common import config, passwords, genconfig

config.CURRENT_DB = passwords.dune_db_url

import redmine.db.connection

redmine.db.connection.setup(config.CURRENT_DB)

BuildmasterConfig = genconfig.genconfig(

"http://users.../dune-stuff-demos.git",

(14)

kno

wl

edg

e

Mün

s

ter

[clang_3.5] file = config.opts/clang-3.5 cc = clang-3.5 [gcc_5.0] file = config.opts/gcc-snapshot cc = gcc-snapshot [variants]

# name = semicolon seperated list of directory names # to delete prior to buildbot config

no_fem = dune-fem

minimal = dune-grid;dune-localfunctions;dune-fem; dune-typetree;dune-pdelab;dune-istl

(15)

kno

wl

edg

e

Mün

s

ter

Buildbot

(16)

kno

wl

edg

e

Mün

s

ter

utopic

Every line of code is unit tested, with function level granularity. System tests cover all possible software configurations and inputs. Performance testing is run on a wide variety of benchmark problems, in a restricted, possibly virtualized, environment. All of that on a very diverse set of computer architectures, operating systems, support library versions.

Not viable in the real world. Configuration space is just too big. Compute time, and available hardware is limited. Test cycles need to be short to not impede development.

(17)

liv

in

g

kno

wl

edg

e

WWU

Mün

s

ter

Continuous Integration/Testing 17/16

Testing: What?

realistic

Every non-trivial piece of code is unit tested, with algorithm level granularity. System tests cover the most common configuration options. Benchmark runs are recorded.

(18)

liv

in

g

kno

wl

edg

e

WWU

Mün

s

ter

Testing: What?

minimal

(19)

liv

in

g

kno

wl

edg

e

WWU

Mün

s

ter

Testing: How?

I Use a test harness: google-test, Boost.Test, py.test

Simplifies selective test running, result gathering, behavior expectation

I Writing a DUNE module? Use dune-testtools for simple test parameterization

I record algorithm results, error norms and compare in system tests

I Make sure exceptions get raised on invalid data input

(20)

liv

in

g

kno

wl

edg

e

WWU

Mün

s

ter

Testing: How? - Gtest Example

typedeftesting::Types<YaspGrid<1>, YaspGrid<4>Grids;

template<classT>

structCornerRangeTest:public::testing::Test { CornerRangeTest():grid_prv(0.,1., level) {}

voidcheck() {

for(autoit: grid_prv.grid().leafGridView()) check_range(it, cornerRange(it->geometry()));