Jolt Documentation. Robert Andersson. May 18, 2021

(1)

Robert Andersson

(2)

(3)

1 Installing 3

1.1 Prerequisites . . . 3

1.2 Installing with pip . . . 3

2 Architecture 5 2.1 Overview . . . 5 2.2 Artifact Cache . . . 6 2.3 Execution . . . 7 3 Tutorial 9 3.1 First example . . . 9 3.2 Publishing Files . . . 10 3.3 Parameters . . . 11 3.4 Dependencies. . . 11 3.5 Tools . . . 12 3.6 Resources. . . 12 3.7 Tests . . . 13 3.8 Influence . . . 14 3.9 Ninja . . . 15 4 Distributed Builds 21 4.1 Deploying with Docker Swarm . . . 21

5 Configuration 25 5.1 Jolt . . . 25 5.2 Network . . . 26 6 Plugins 27 6.1 Alias . . . 27 6.2 Allure . . . 27 6.3 AMQP . . . 28 6.4 HTTP . . . 28 6.5 Autoweight . . . 29 6.6 Dashboard . . . 29 6.7 Email . . . 29 6.8 FTP . . . 30 6.9 Logstash (HTTP) . . . 31

(4)

6.12 Selfdeploy . . . 31 6.13 Symlinks . . . 32 6.14 Telemetry . . . 32 7 Reference 33 7.1 Alias . . . 33 7.2 Artifact . . . 33 7.3 Context . . . 36 7.4 Decorators . . . 36 7.5 Download. . . 37 7.6 Influence . . . 38 7.7 Parameter . . . 40 7.8 BooleanParameter . . . 41 7.9 ListParameter. . . 42 7.10 Resource . . . 43 7.11 Task. . . 44 7.12 Test . . . 46 7.13 Tools . . . 46 7.14 Conan. . . 55 7.15 Docker . . . 57 7.16 Ninja . . . 58 8 Changelog 67 8.1 Version 0.9.10 . . . 67

Python Module Index 69

(5)

Jolt is an easy to use, yet powerful task execution tool. Tasks are defined in Python scripts using a class based API. A task typically produces output which can be published as a binary artifact into a content addressable cache for later consumption by other tasks.

(6)

(7)

Installing

There are several methods to install Jolt, but the recommendation is to use pip.

1.1 Prerequisites

You need a working installation of Python 3.2+ (or deprecated 2.7) to run Jolt. Plugins can have additional require-ments. Consult plugin documentation for details.

1.2 Installing with pip

Run:

(8)

(9)

Architecture

2.1 Overview

Jolt is a tool executing user-defined tasks. Execution is carried out either locally on a user’s computer or remotely on clustered workers. Different types of cluster infrastructures are supported through plugins.

When a task is executed an artifact containing files and metadata is created. The artifact is shared among users and workers in the network by the means of storage providers (file servers). Again, different types are supported through plugins.

In a typical configuration, automation software such as Jenkins may trigger tasks to be executed on a cluster of workers. Resulting artifacts are stored on the file server, ready to be downloaded by users attempting to execute the same tasks.

(10)

2.2 Artifact Cache

All task artifacts are stored in a local cache directory. They are content addressable, meaning they all have a unique and reproducible identity. This identity allows the artifacts to be consistently exchanged between local and remote caches.

(11)

The identity is a SHA1 sum of different attributes that may influence the output of the task, as illustrated below. Such influence includes the source code of the task, task parameters and their assigned values, the content of files and scm repositories, the identity of dependency tasks, etc.

The SHA1 digest is used as a key when looking up artifacts both locally and remotely.

2.3 Execution

(12)

(13)

Tutorial

Let’s kick off with a few examples, getting more into details as we move along.

3.1 First example

Our first example is a classic. Copy/paste the code below into a file called first_example.jolt. from jolt import _*

class HelloWorld(Task):

def run(self, deps, tools):

print("Hello world!")

This task will simply print “Hello world!” on the console when executed. The task name is automatically derived from the class name, but can be overridden by setting the name class attribute.

Now try to execute the task: $ jolt build helloworld

Once the task has been executed, executing it again won’t have any effect. This is because the task produced an empty artifact which is now stored in the local artifact cache. When Jolt determines if a task should be executed or not, it first calculates a task identity by hashing different attributes that would influence the output of the task, such as:

• Source code • Name • Parameters • Dependencies

When the task identity is known, Jolt searches its artifact cache for an artifact with the same identity. If one is found, no action is taken. You can try this out by changing the Hello world! string to something else and executing the

(14)

task again. If the string is reverted back to Hello world!, the task will regain its first identity and no action will be taken because that artifact is still present in the cache.

To clean the local cache and remove all artifacts, run: $ jolt clean

To selectively remove a specific task’s artifacts, run: $ jolt clean helloworld

3.2 Publishing Files

Tasks that don’t produce output are not very useful. Let’s rework our task to instead produce a file with the Hello world!message. We also shorten its name to hello.

from jolt import _*

class HelloWorld(Task):

""" Creates a text file with cheerful message """

name = "hello"

def run(self, deps, tools): b = tools.builddir() with tools.cwd(b):

tools.write_file("message.txt", "Hello world!")

def publish(self, artifact, tools): b = tools.builddir()

with tools.cwd(b):

artifact.collect(_"*.txt")

The implementation of the task is now split into two methods, run and publish.

The run method performs the main work of the task. It creates a file called message.txt containing our greeting from the first example. The file is written into a temporary build directory that will persist for the duration of the task’s execution. The directory is removed afterwards.

The publish method collects the output from the work performed by run. It does so by instructing the artifact to collect all textfiles from the build directory.

$ jolt build hello

After executing the task an artifact will be present in the local cache. Let’s investigate its contents, but first we need to know the identity of the task in order to know what artifact to look for. Run:

$ jolt info -a hello

The info command shows information about the task, including the documentation written in its Python implemen-tation. We’re looking for the identity:

Identity 50a215905eb28a0911ff83828ac56b542525bce4

With this identity digest at hand, we can dive into the artifact cache. By default, the cache is located in $HOME/. cache/jolt. To list the content of the current hello artifact, run:

(15)

$ ls $HOME/.cache/jolt/hello/50a215905eb28a0911ff83828ac56b542525bce4

You will see the message.txt file just created.

3.3 Parameters

Next, we’re going to use a task parameter to alter the Hello world! message. Instead of greeting the world, we’ll allow the executor to specify an alternative recipient. We rename the class to reflect this change and we also add a parameter class attribute. The run method is changed to use the new parameter’s value when writing the message. txtfile.

class Hello(Task):

""" Creates a text file with a cheerful message """

recipient = Parameter(default="world", help="Name of greeting recipient.")

def run(self, deps, tools): b = tools.builddir() with tools.cwd(b):

tools.write_file("message.txt", "Hello {recipient}!")

def publish(self, artifact, tools): b = tools.builddir()

with tools.cwd(b):

artifact.collect(_"*.txt")

By default, the produced message will still read Hello world! because the default value of the recipient parameter is world. To produce a different message, try this:

$ jolt build hello:recipient=John

3.4 Dependencies

To better illustrate the flexibility of the new parameterized task, let’s add another task class, Print, which prints the contents of the message.txt file to the console. Print will declare a dependency on Hello.

class Print(Task):

""" Prints a cheerful message """

recipient = Parameter(default="world", help="Name of greeting recipient.") requires = "hello:recipient={recipient}"

cacheable = False

hello = deps["hello:recipient={recipient}"]

with open(os.path.join(hello.path, "message.txt")) as f:

print(f.read())

The output from this task is not cacheable, forcing the task to be executed every time. It’s dependency hello however, will only be re-executed if its influence changes, for example by passing new values to the recipient parameter. Try it out:

(16)

$ jolt build print:recipient=John $ jolt build print:recipient=Lisa $ jolt build print:recipient=Kelly

3.5 Tools

The run and publish methods take a tools argument as their last parameter. This toolbox provides a large set of tools useful for many different types of tasks. See the reference documentation for more information.

However, Jolt was originally created with compilation tasks in mind. Below is a real world example of a task compiling the e2fsprogs package containing EXT2/3/4 filesystem utility programs. It uses AutoTools to configure and build its sources into different binary applications. Luckily, the tools object provides utilities for building autotools projects as seen below. In addition to AutoTools, there is also support for CMake and Meson as well as generic support for running any third-party build tool.

from jolt import *

from jolt.plugins import git

class E2fsprogs(Task):

""" Ext 2/3/4 filesystem utilities """

requires = "git:url=git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git"

def run(self, deps, tools): ac = tools.autotools() ac.configure("e2fsprogs") ac.build()

ac.install()

def publish(self, artifact, tools): ac = tools.autotools()

ac.publish(artifact)

artifact.environ.PATH.append("bin")

The autotools ac object automatically creates temporary build and install (–prefix) directories which are used when configuring, building and installing the project. All files installed in the installation directory will be published. Both directories are removed when execution has finished, i.e. the project will be completely rebuilt if the task’s influence changes.

The task also extends the environment of consumer tasks by adding the artifact’s bin directory to the PATH. That way, any task that depends on e2fsprogs will be able to run its published programs directly without explicitly referencing the artifact where they reside. Use this method to package tools required by other tasks.

Also, note that the task requires a git repository hosted at kernel.org. This git task, implemented by a builtin plugin, is actually not a task but a resource. You can read more about resources next.

3.6 Resources

Resources are a special kind of task only executed in the context of other tasks. They are invoked to acquire and release a resource before and after the execution of a task. No artifact is produced by a resource.

A common use-case for resources is to allocate and reserve equipment required during the execution of a task. Such equipment could be a build server or a mobile device on which to run tests.

(17)

Below is a skeleton example providing mutual exclusion: from jolt import *

class Exclusivity(Resource):

""" Resource providing mutual exclusion to an object """

to = Parameter(help="Name of shared object")

def acquire(self, artifact, deps, tools): # TODO: Implement locking

self.info("{to} is now locked")

def release(self, artifact, deps, tools): # TODO: Implement unlocking

self.info("{to} is now unlocked")

class RebootDevice(Task):

""" Reboots the specified test device """

device = Parameter(help="Name of device to reboot") requires = "exclusivity:to={device}"

cacheable = False

tools.run("ssh {device} reboot")

3.7 Tests

After implementing the e2fsprogs task above, the next logical step is to write a few test-cases for the utility programs it builds. Luckily, Jolt has integrated test support.

Test tasks are derived from the Test base class instead of Task and they are implemented like a regular Python unittest.TestCase. You can use all assertions and decorators like you normally would. In all other respects, a Testtask behaves just like a regular Task.

Below is an example: from jolt import * class E2fsTest(Test):

requires = "e2fsprogs"

def setup(self, deps, tools):

self.tools = tools

def test_mke2fs(self):

self.assertTrue(self.tools.run("mke2fs"))

def test_badblocks(self):

self.assertTrue(self.tools.run("badblocks"))

def test_tune2fs(self):

(18)

3.8 Influence

It is important that all attributes that define the output of a task are known and registered to avoid false cache hits. For example, in a compilation task all compiled source files should influence the task’s identity and trigger re-execution of the task if changed, otherwise binary compatibility will be lost quickly.

When using an external third-party build tool such as make, Jolt has no way of knowing what source files to monitor. This information must be explicitly provided by the task’s implementor. Luckily, Jolt provides a few builtin class decorators to make it easier.

Let’s revisit the e2fsprogs task from earlier, but this time we assume that the repository is already cloned and managed by external tools and not through the builtin Jolt git resource. We can no longer rely on the resource to automatically influence the hash of the task. We instead use the git.influence decorator:

from jolt import _*

from jolt.plugins import git

@git.influence(path="path/to/e2fsprogs") class E2fsprogs(Task):

def run(self, deps, tools): ac = tools.autotools()

ac.configure("path/to/e2fsprogs") ac.build()

ac.install()

The decorator adds the git repository’s tree hash as hash influence. It will also add the git diff output as influence to simplify iterative local development.

There are a number of other useful influence decorators as well: from jolt import *

from jolt import influence from jolt.plugins import git

@influence.files(_{"path/to/e2fsprogs/*.c"}) @influence.environ("CFLAGS")

@influence.weekly

@influence.attribute("webstatus") class E2fsprogs(Task):

@property

def webstatus(self):

r = requests.get("http://statusindicator/") return r.text

def run(self, deps, tools): ac = tools.autotools()

ac.configure("path/to/e2fsprogs") ac.build()

ac.install()

self.report()

Above, the git.influence decorator has been replaced by influence.files. The result is virtually the same, the content of all files matched by the provided pattern will influence the hash of the task. However, the Git tree hash implementation is more effecient and faster, but it obviously doesn’t work if sources reside in a different type of repository.

The influence.environ decorator is used to influence the hash of the task based on the value of the CFLAGS environment variable. If the value of the variable changes the task will be re-executed.

(19)

The influence.weekly decorator adds the week number as hash influence. If nothing else changes, the task will be re-executed once every week. This can be useful to verify that external resources, such as files downloaded from the Internet, are still available. Other time-based decorators include:

• influence.yearly • influence.montly • influence.daily • influence.hourly

The influence.attribute decorator adds the value of an attribute or property as hash influence. Above, the webstatusproperty is registered to influence the task with data obtained from a web service. The source code of the property itself is monitored automatically.

3.9 Ninja

Ninja is a fast third-party build system. Where other build systems, such as Jolt, are high-level languages, Ninja aims to be an assembler. Together they form a powerful couple. Jolt has builtin Ninja tasks which automatically generate Ninja build files and build your projects for you. All you have to do is to tell Jolt which source files to compile. You can also define custom build rules for file types not recognized by Jolt, see theRuleclass.

3.9.1 Basics

Below is an example of a library and a program. The library contains a function returning a message. The program calls this function and prints the message.

// lib/message.cpp

#include "message.h"

const char _*message() {

return "Hello " RECIPIENT "!"; } // program/main.cpp #include <cstdlib> #include <iostream> #include "lib/message.h" int main() {

std::cout << message() << std::endl; return EXIT_SUCCESS;

}

To build the library and the program we use this Jolt recipe: from jolt import _*

from jolt.plugins.ninja import *

class Message(CXXLibrary):

recipient = Parameter(default="world", help="Name of greeting recipient.")

(20)

(continued from previous page)

headers = ["lib/message.h"] sources = ["lib/message.cpp"]

macros = ['RECIPIENT="{recipient}"']

class HelloWorld(CXXExecutable): requires = "message"

sources = "program/main.cpp"

3.9.2 Metadata

Jolt automatically configures include paths, link libraries, and other build attributes for the HelloWorld program based on metadata found in the artifact of the Message library task. In the example, the Message library task relies upon CXXLibrary.publish to collect public headers and to export the required metadata such as include paths and linking information. Customization is possible by overriding the publish method as illustrated below. These two implementations of Message are equivalent.

recipient = Parameter(default="world", help="Name of greeting recipient.") sources = [_{"lib/message.*"}]

macros = ['RECIPIENT="{recipient}"']

def publish(self, artifact, tools): with tools.cwd("{outdir}"):

artifact.collect(_"*.a", "lib/")

artifact.cxxinfo.libpaths.append("lib") artifact.collect(_"lib/*.h", "include/") artifact.cxxinfo.incpaths.append("include")

The cxxinfo artifact metadata can be used with other build systems too, such as CMake, Meson and Autotools. It enables your Ninja tasks to stay oblivious to whatever build system their dependencies use as long as binary compati-bility is guaranteed.

3.9.3 Parameterization

To support build customization based on parameters, several class decorators can be used to extend a task with condi-tional build attributes.

The first example uses a boolean debug parameter to disable optimizations and set a preprocessor macro. The dec-orators enable Ninja to consider alternative attributes, in addition to the default cxxflags and macros. The names of alternatives are expanded with the values of parameters. When the debug parameter is assigned the value true, the cxxflags_debug_true and macros_debug_true attributes will be matched and included in the build. If the debug parameter value is false, no extra flags or macros will be included because there are no cxxflags_debug_falseand macros_debug_false attributes in the class.

@ninja.attributes.cxxflags("cxxflags_debug_{debug}") @ninja.attributes.macros("macros_debug_{debug}") class Message(ninja.CXXLibrary):

debug = BooleanParameter()

cxxflags_debug_true = ["-g", "-Og"] macros_debug_true = ["DEBUG"] sources = [_{"lib/message.*"}]

(21)

@ninja.attributes.sources("sources_{os}") class Message(ninja.CXXLibrary):

os = Parameter(values=["linux", "windows"]) sources = [_"lib/*.cpp"]

sources_linux = [_{"lib/posix/*.cpp"}] sources_windows = [_{"lib/win32/*.cpp"}]

3.9.4 Influence

The Ninja tasks automatically let the content of the listed header and source files influence the task identity. However, sometimes source files may #include headers which are not listed. This is an error which may result in objects not being correctly recompiled when the header changes. To protect against such errors, Jolt uses output from the compiler to ensure that files included during a compilation are properly influencing the task.

In the example below, the message.h header is no longer listed in headers, nor in sources. from jolt import _*

from jolt.plugins.ninja import *

sources = ["lib/message.cpp"]

Assuming message.cpp includes message.h, this would be an error because Jolt no longer tracks the content of the message.h header and message.cpp would not be properly recompiled. However, thanks to the builtin sanity checks, trying to build this library would fail:

$ jolt build message

[ ERROR] Execution started (message b9961000) [ STDOUT] [1/2] [CXX] message.cpp

[ STDOUT] [1/2] [AR] libmessage.a [WARNING] Missing influence: message.h

[ ERROR] Execution failed after 00s (message b9961000) [ ERROR] task is missing source influence (message)

The solution is to ensure that the header is covered by influence, either by listing it in headers or sources, or by using an influence decorator such as @influence.files.

sources = ["lib/message.h", "lib/message.cpp"]

Headers from artifacts of dependencies are exempt from the sanity checks. They already influence the consuming task implicitly. This is also true for files in build directories.

3.9.5 Custom Rules

Rules are used to transform files from one type to another. An example is the rule that compiles a C/C++ file to an object file. Ninja tasks can be extended with additional rules beyond those already builtin and the builtin rules may also be overridden.

To define a new rule for a type of file, assign a Rule object to an arbitrary attribute of the compilation task being defined. Below is an example where a rule has been added to generate Qt moc source files from headers.

class MyQtProject(CXXExecutable):

sources = ["myqtproject.h", "myqtproject.cpp"]

(22)

(continued from previous page)

moc_rule = Rule(

command="moc -o $out $in", infiles=[".h"],

outfiles=["{outdir}/{in_path}/{in_base}_moc.cpp"])

The moc rule applies to all .h header files listed as sources, i.e. myqtproject.h. It takes the input header file and generates a corresponding moc source file, myqtproject_moc.cpp. The moc source file will then automatically be fed to the builtin compiler rule from which the output is an object file, myqtproject_moc.o.

Below, another example illustrates how to override one of the builtin compilation rules. The example also defines an environment variable that will be accessible to the rule.

class MyQtProject(CXXExecutable):

sources = ["myqtproject.h", "myqtproject.cpp"]

custom_cxxflags = EnvironmentVariable()

cxx_rule = Rule(

command="g++ $custom_cxxflags -o $out -c $in", infiles=[".cpp"],

outfiles=["{outdir}/{in_path}/{in_base}{in_ext}.o"])

$ CUSTOM_CXXFLAGS=-DDEBUG jolt build myqtproject

3.9.6 Toolchains

Maintaining binary compatibility between libraries can be a pain. To ensure that a chain of dependencies stay compat-ible you could inject a synthetic toolchain task at the bottom of your dependency tree and use it to control all compiler options. This methods also enables easy cross-compilation.

First, define a toolchain task: class Toolchain(Task):

arch = Parameter("i386", values=["i386", "arm"]) host = Parameter(platform.system())

debug = BooleanParameter(False)

def publish(self, artifact, tools): if self.arch.get_value() == "arm":

artifact.environ.CC = "arm-linux-gnueabi-gcc" if self.arch.get_value() == "i386":

artifact.environ.CC = "x86_64-linux-gnu-gcc -m32" if self.debug.is_true:

artifact.cxxinfo.cflags.append("-g") artifact.cxxinfo.cflags.append("-Og") else:

artifact.cxxinfo.cflags.append("-O2")

Flags can also be exported as environment variables, CFLAGS, CXXFLAGS, etc. Secondly, declare the toolchain as a dependency of all your compilation tasks: class HelloWorld(CXXExecutable):

requires = "toolchain" sources = "src/main.cpp"

(23)

Default toolchain parameter values can be overridden from the command line when you need to. For example, to build the HelloWorld task for the ARM architecture, run:

$ jolt build helloworld -d toolchain:arch=arm

The -d toolchain:arch=arm command line argument instructs Jolt to overide the default value of the arch parameter of the toolchain task. The new value changes the identity of the toolchain artifact which triggers a rebuild of all depending tasks.

To build the HelloWorld task without optimizations and with debug information: $ jolt build helloworld -d toolchain:debug=true

This approach with default valued parameters can also be used to enable other use-cases where you temporarily may want:

• code coverage builds • builds with custom cflags • etc

3.9.7 Conan Package Manager

The Conan package manager is an excellent way to quickly obtain prebuilt binaries of third-party libraries. It has been integrated into Jolt allowing you to seemlessly use Conan packages with your Jolt Ninja tasks.

In the example below, Conan is used to collect the Boost C++ libraries. Boost is then used in our example application. All build metadata is automatically configured.

from jolt.plugins.conan import Conan

class Boost(Conan):

requires = "toolchain" packages = ["boost/1.74.0"]

class HelloWorld(CXXExecutable): requires = ["toolchain", "boost"] sources = "src/main.cpp"

With the toolchain as a dependency also for Boost, Conan will be able to fetch the appropriate binaries that match your toolchain. If no such binaries are available, Conan will build them for you.

(24)

(25)

Distributed Builds

To support distributed and incremental task execution in a build cluster, two things are typically needed: • a centralized file server where resulting task artifacts can be stored

• workers executing tasks and uploading the artifacts to the file server

Task recipies may also have to be adapted to work in a network build environment. In particular, all task dependencies such as tools and source code repositories should be explicitly declared to allow the task to run anywhere. The more explicit the better. Your workers should be capabable of handling any workload. With proper dependency management they can provision the correct environment for any task on demand.

4.1 Deploying with Docker Swarm

Docker’s Swarm mode is an easy to use container orchestration tool which can be used to deploy and manage a cluster of Jolt workers. In this example, the Jolt AMQP plugin is used to facilitate communication between Jolt clients and workers. The workers are deployed using the standard Jolt Docker container available in Docker Hub. RabbitMQ has been chosen as the AMQP message broker and Nginx serves as the centralized HTTP artifact cache. The Docker compose file for this setup looks like this:

version: "3.5" services: amqp: image: rabbitmq:latest ports: - "5672:5672" cache: image: nginx:latest deploy: replicas: 1 configs: - source: nginx.conf target: /etc/nginx/nginx.conf ports:

(26)

(continued from previous page) - "8080:80" volumes: - cache-http:/usr/share/nginx/html dashboard: image: robrt/jolt-dashboard:latest ports: - "80:80" worker: image: robrt/jolt:latest deploy: replicas: 2

command: ["-c", "/opt/jolt/worker.conf", "amqp-worker"] configs: - source: worker.conf target: /opt/jolt/worker.conf volumes: - cache-node:/root/.cache/jolt configs: worker.conf: file: ./worker.conf nginx.conf: file: ./nginx.conf volumes: cache-node: cache-http:

The Jolt workers are configured through the worker.conf file. It enables the AMQP and HTTP plugins. The hidden amqp-workercommand provided by the plugin will connect to the configured AMQP message broker and start processing execution requests. The HTTP plugin will store the resulting artifacts on the configured HTTP server. Jolt clients can then download these artifacts to the local host. In the example, a simple local Docker volume is used as server storage for the artifacts. In a real deployment, you probably want to use something else.

[amqp] host = amqp config = /opt/jolt/worker.conf [http] uri = http://cache/ [dashboard]

To deploy the system into a swarm, run:

$ docker stack deploy -c jolt.yaml jolt

You can then scale up the the number of workers to a number suitable for your swarm: $ docker service scale jolt_worker=10

Scaling is possible even with tasks in progress as long as they don’t cause any side effects. If a task is interrupted because the worker is terminated, RabbitMQ will redeliver the execution request to another worker.

The newly deployed swarm of Jolt workers is utilized by configuring the Jolt client as follows: [jolt]

upload = false

(27)

(continued from previous page) [http] uri = http://localhost:8080/ [amqp] host = localhost port = 5672 [dashboard] uri = http://localhost/

These configuration keys can also be set from command line: $ jolt config amqp.host localhost

$ jolt config http.uri http://localhost/

If your local machine is not part of the swarm you will need to replace localhost with the IP-address of one of the swarm nodes.

To schedule a task in the swarm, pass the –network flag to the build command: $ jolt build --network <task>

Alternatively, if you are using a separate configuration file: $ jolt -c client.conf build --network <task>

(28)

(29)

Configuration

By default, Jolt loads its configuration from $HOME/.config/jolt/config. It uses this format: [section]

key = value

All available sections and their respective keys are detailed below.

5.1 Jolt

The [jolt] config section contains global configuration. • cachesize = <size>

Maximum size of the local artifact cache. When this size is reached, Jolt will start to evict artifacts from the cache until it no longer exceeds the configured limit. Artifacts which are required to execute a task currently present in the dependency tree are never evicted. Therefore, it may not be possible for Jolt to evict enough artifacts to reach the limit. Consider this size advisory. The size is specified in bytes and SI suffixes such as K, M and G are supported. Example: cachesize = 5G. The default size is 1G.

• colors = <boolean>

Colorize output. When enabled, Jolt uses colors to make it easier to read the console log and to spot errors. The default is true.

• default = <task>

When invoked without any arguments, Jolt by default tries to build a task called default. The name of the default task can be overridden by setting this configuration key.

• download = <boolean>

Configures if Jolt is allowed to download artifacts from remote storage providers or not when building locally. The option has no effect on distributed network builds. The default value is true.

(30)

• log = <filepath>

Location of Jolt’s logfile. By default, the logfile is written in $HOME/.jolt/jolt.log. • upload = <boolean>

Configures if Jolt is allowed to upload artifacts to remote storage providers or not when building locally. The option has no effect on distributed network builds. The default value is true.

• pager = <str>

The pager to use, e.g. when viewing the logfile. Defaults to the PAGER environment variable followed by less, more and cat, in that order.

• pluginpath = <str>

A list of one or more directory names, separated by colon, specifying additional search paths for plugins. • shell = <str>

The shell to use when entering the interactive task debug shell.

• threads = <integer> Used to limit the number of threads used by third party tools such as Ninja. The environment variable JOLT_THREADS can also be used. The default value is the number of CPUs available. • parallel_tasks = <integer> Used to control the number of tasks allowed to execute in parallel on

the local machine. The environment variable JOLT_PARALLEL_TASKS can also be used as well as the -j/ --jobsbuild command option. The default value is 1.

5.2 Network

The [network] section contains keys applicable when Jolt is started in network execution mode. • config = <text>

The config key contains config file content for Jolt to be used when Jolt is executed on a different machine during distributed execution. The configuration is automatically passed to the remote worker and may contain all subsections and keys detailed in this document. Lines must be properly indented for the key to be considered multiline. Example:

[network] config = [jolt]

upload = true download = true

(31)

Plugins

Jolt can be extended through plugins. Those already built-in and their configuration options are described below.

6.1 Alias

This plugin can be used to create user-defined task aliases through configuration keys. An alias points to one or many other tasks. For example, to create an alias called deploy which deploys a fictitious smartphone app to all supported devices, run:

$ jolt config alias.deploy "deploy/android deploy/iphone" $ jolt build deploy

Alternatively, edit the configuration manually: [alias]

deploy = deploy/android deploy/iphone

Aliases cannot be used to override the names of tasks loaded from recipes.

6.2 Allure

This plugin generates an Allure test report on the outcome of executed tasks. The report includes: • status of tasks, i.e. successful, failed or skipped

• duration of tasks • hostname of executor • logs

The plugin is enabled by adding a [allure] section in the Jolt configuration. Available configuration keys:

(32)

• loglevel - The level of detail to include in task logs: INFO, VERBOSE or DEBUG

• path - Path to directory where result files are written. Default: <workspace>/allure-results.

6.3 AMQP

The AMQP plugin implements distributed task execution with the help of an AMQP message queue broker such as RabbitMQ. When the plugin is used, tasks execution requests are submitted to an AMQP queue. Execution requests are then consumed by workers that run tasks and build artifacts.

The plugin enables a special subcommand amqp-worker that is used to run a worker. It is recommended to deploy multiple workers as well as a message queue broker using an orchestrator such as Kubernetes.

As long as a connection to the message queue broker can be maintained, the number of workers may be scaled up and down transparently without affecting tasks already in progress. Workers may even be redeployed completely. If a worker is stopped before completing a task, the task is restarted as soon as another worker becomes available. Note, however, that tasks can’t be safely interrupted if they have side-effects outside of the worker.

To use this plugin, a networked artifact storage provider must also be configured to enable the workers to share artifacts between each other.

The plugin is enabled by adding a [amqp] section in the Jolt configuration. These configuration keys exist:

• host - Hostname or address of the AMQP service. Default: amqp-service • port - Port number of the AMQP service. Default: 5672

• max-priority - Optional worker configuration. Enables task priority queues. Tasks that are waiting in queue for a worker will be dequeued in order of priority. See priority. This value configures the number of priority levels that will be available and should be a positive integer between 1 and 255. Values between 1 and 10 are recommended. Default: 1.

• priority - Optional client configuration. Configures the default priority of all tasks submitted to the queue. Default: 0.

• routing_key - Optional. By using routing keys, tasks can be directed to different types of workers. When starting a worker by using the amqp-worker command, the worker will only consume tasks tagged with the configured key. To tag a task, set the routing_key task attribute. Default: default

• workers - Optional client configuration. The maximum number of tasks Jolt is allowed to run in parallel. Default: 16.

• keyring.username - Username to use when authenticating with the AMQP service.

• keyring.password - Password to use when authenticating with AMQP service. Should normally never need to be set in the configuration file. By default, Jolt asks for the password when needed and stores it in a keyring for future use.

• keyring.service - Keyring service identifier. Defaults to amqp.

6.4 HTTP

The HTTP plugin implements an artifact storage provider. When used, artifacts can be automatically uploaded to and downloaded from a configured HTTP server when tasks are executed.

(33)

• To support distributed task execution. Task executors must be able to share artifacts between each other. Using a networked storage provider is an easy way to meet that requirement.

• To reduce execution time by letting multiple users share the same artifact cache. If one user has already executed a task, its artifact is simply downloaded to others who attempt execution.

• To reduce the amount of disk space required locally. Jolt can be configured to evict artifacts more aggressively from the local cache. Artifacts will still be available on the server if needed.

The HTTP plugin is enabled by adding an [http] section in the Jolt configuration. These configuration keys exist:

• download - Boolean. Allow/disallow artifacts to be downloaded from the HTTP server. Defaults to true. • upload - Boolean. Allow/disallow artifacts to be uploaded to the HTTP server. Defaults to true.

• uri - URL to the HTTP server.

• keyring.service - Keyring service identifier. Currently, only basic authentication is supported. Authenti-cation is disabled if left unset.

• keyring.username - Username to use when authenticating with the HTTP server.

• keyring.password - Password to use when authenticating with the HTTP server. Should normally never need to be set in the configuration file. By default, Jolt asks for the password when needed and stores it in a keyring for future use.

6.5 Autoweight

The autoweight plugin automatically collects statistics about task execution times. The data is used to assign weights to task, allowing the Jolt scheduler to favor tasks along the critical path. This improves overall execution time in a distributed execution configuration where many tasks are executed in parallel.

The plugin is enabled by adding an [autoweight] section in the Jolt configuration. These configuration keys exist:

• samples - Integer. The number of execution time samples to store per task in the database. Once the number is exceeded, samples are evicted in FIFO order.

6.6 Dashboard

The dashboard plugin automatically submits required telemetry to the Jolt Dashboard. It should be enabled on both clients and workers.

The plugin is enabled by adding a [dashboard] section in the Jolt configuration. These configuration keys exist:

• uri - Base URI of the Jolt Dashboard. Default:http://dashboard

6.7 Email

The email plugin sends an HTML email report to configured recipients when builds have completed. The email includes a list of interpreted errors in case of failure.

(34)

The plugin is enabled by adding a [email] section in the Jolt configuration. These configuration keys exist:

• server - SMTP server used to send emails. • from - Sender email address.

• to - Receiver email address. May also be read from environment, e.g. {ENV|GERRIT_PATCHSET_UPLOADER_EMAIL}. Multiple addresses should be separated by a sin-gle space.

• stylesheet - An optional custom XSLT stylesheet used to transform the Jolt result manifest into an HTML email.

• on_success - Send emails when builds are successful. Default: true • on_failure - Send emails when builds failed. Default: true

6.8 FTP

The FTP plugin implements an artifact storage provider. When used, artifacts can be automatically uploaded to and downloaded from a configured FTP server when tasks are executed.

The plugin is enabled by adding an [ftp] section in the Jolt configuration. These configuration keys exist:

• download - Boolean. Allow/disallow artifacts to be downloaded from the FTP server. Defaults to true. • host - Hostname/IP address of the FTP server.

• path - Path to directory where artifacts should be stored on the FTP server. Defaults to jolt. The directory is created if it doesn’t exist.

• tls - Use a TLS connection to the FTP server.

• upload - Boolean. Allow/disallow artifacts to be uploaded to the FTP server. Defaults to true. • keyring.username - Username to use when authenticating with the FTP server.

• keyring.password - Password to use when authenticating with the FTP server. Should normally never need to be set in the configuration file. By default, Jolt asks for the password when needed and stores it in a keyring for future use.

(35)

• keyring.service - Keyring service identifier. Defaults to ftp.

6.9 Logstash (HTTP)

The logstash plugin is used to collect task logs into a common place. This is useful in distributed execution environ-ments where detailed logs may not always be immediately accessible to ordinary users. Unlike the terminal log output, stashed logs are always unfiltered and include statements from all log levels as well as exception callstacks.

The plugin is enabled by adding a [logstash] section in the Jolt configuration. These configuration keys exist:

• http.uri - An HTTP URL where logs will be stashed. The HTTP PUT method is used. • failed - Boolean. Stash logs when tasks fail.

• finished - Boolean. Stash logs when tasks finish successfully.

6.10 Ninja Compilation Database

This plugin enables compilation database generation for Ninja C++ tasks. The database is automatically published in task artifacts. Note that commands are recorded exactly as invoked by Ninja and they are therefore not immediately usable because of how Jolt sandboxes dependencies. A special command, compdb is made available to post-process published databases into a database that is usable with IDEs. The command takes an already built task as argument: $ jolt compdb <task>

Upon completion, a path to the resulting database is printed. The database aggregates the databases of the task and all its dependencies.

The plugin is enabled by adding a [ninja-compdb] section in the Jolt configuration. Ninja version >= 1.10.0 is required.

6.11 Ninja Compilation Cache

This plugin enables a compilation cache allowing C/C++ object files to be reused from libraries residing in the artifact cache. Reuse is permitted if the source file, included headers and the compiler command line are unmodified since the object was built.

The plugin is enabled by adding a [ninja-cache] section in the Jolt configuration.

6.12 Selfdeploy

The Selfdeploy plugin automatically deploys the running version of Jolt into all configured artifact storage providers. This is useful when using distributed task execution to ensure that the same version of Jolt is used everywhere. Before starting execution of a task, a network executor can download and install Jolt from a storage provider.

The plugin is enabled by adding a [selfdeploy] section in the Jolt configuration. These configuration keys exist:

• extra - Comma separated list of paths to additional python modules to be deployed. The paths should be relative to the workspace root.

(36)

Once enabled, the plugin automatically passes two parameters to distributed network builds: • jolt_url - A URL to a compressed tarball with the sources of the running Jolt version. • jolt_identity - The identity of the Jolt artifact.

• jolt_requires - A list of additional Python modules to install on the executor.

6.13 Symlinks

The symlink plugin automatically creates symlinks to task artifacts in the jolt workspace (relative to the topmost .joltfile). The symlinks are kept updated and always points to the latest built artifact.

The plugin is enabled by adding a [symlinks] section in the Jolt configuration. These configuration keys exist:

• path - Path, relative to the workspace root, where symlinks will be created. Defaults to artifacts.

6.14 Telemetry

The telemtry plugin posts task telemetry to a configured HTTP endpoint. The payload is a JSON object with these fields:

• name - The name of the task.

• identity - The identity of the task artifact.

• instance - A UUID representing the lifecycle of the task. Tasks can be executed multiple times with the same identity, for example if the first execution attempt failed and a subsequent attempt succeeded. The instance ID may be used to distingush between such attempts.

• hostname - hostname of the machine from which the telemetry record originated. • role - client or worker depending on where the record originated.

• event - queued, started, failed or finished.

The plugin is enabled by adding a [telemetry] section in the Jolt configuration. These configuration keys exist:

• uri - Where telemetry records should be posted.

• local - Submit telemetry for locally executed tasks. Default: true.

• network - Submit telemetry for tasks executed by a network worker. Default: true. • queued - Enable queued event. Default: true.

• started - Enable started event. Default: true. • failed - Enable failed event. Default: true. • finished - Enable finished event. Default: true.

(37)

Reference

7.1 Alias

class jolt.Alias(*args, **kwargs) An alias task.

Aliases are a special kind of task which can be used to introduce new names for other tasks or groups of tasks. They are useful as milestones when building continuous integration pipelines since they won’t be executed, thus saving time compared to a regular task.

name

Name of the alias. Derived from class name if not set.

requires = []

List of dependencies to other tasks.

7.2 Artifact

class jolt.Artifact(cache, node)

An artifact is a collection of files and metadata produced by a task.

Task implementors call artifact methods to collect files to be published. In addition to files, other metadata can be provided as well, such as variables that should be set in the environment of consumer tasks.

collect(files, dest=None, flatten=False, symlinks=False, cwd=None) Collect files to be included in the artifact.

Parameters

• files (str) – A filename pattern matching the files to be include in the artifact. The pattern may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

• dest (str, optional) – Destination path within the artifact. If the string ends with a path separator a new directory will be created and all matched source files will be copied

(38)

into the new directory. A destination without trailing path separator can be used to rename single files, one at a time.

• flatten (boolean, optional) – If True, the directory tree structure of matched source files will flattened, i.e. all files will be copied into the root of the destination path. The default is False, which retains the directory tree structure relative to the current work-ing directory.

• symlinks (boolean, optional) – If True, symlinks are copied. The default is False, i.e. the symlink target is copied.

• cwd (str, optional) – Change current working directory before starting collection.

copy(pathname, dest, flatten=False, symlinks=False) Copy files from the artifact.

Parameters

• pathname (str) – A pathname pattern, relative to the root, matching the files to be copied from the artifact. The pattern may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

• dest (str, optional) – Destination path. If the string ends with a path separator a new directory will be created and all matched source files will be copied into the new directory. A destination without trailing path separator can be used to rename single files, one at a time.

• flatten (boolean, optional) – If True, the directory tree structure of matched source files will flattened, i.e. all files will be copied into the root of the destination path. The default is False, which retains the directory tree structure.

• symlinks (boolean, optional) – If True, symlinks are copied. The default is False, i.e. the symlink target is copied.

cxxinfo = {}

Artifact C/C++ build metadata.

A task can add compilation metadata to an artifact. Such metadata will be automatically applied when consumer compilation tasks are executed. A common use-case is to add preprocessor definitions, link libraries, etc. These string fields are supported:

• asflags - assembler flags (string) • cflags - compiler flags (string) • cxxflags - compiler flags (string) • ldflags - linker flags (string)

• libraries - libraries to link with (list, use append()) • macros - preprocessor macros to set (list, use append())

Values appended to PATH-type metadata fields are relative to the artifact root. They will be automatically expanded to absolute paths. These PATH-type fields are supported:

• incpaths - preprocessor include paths (list, use append()) • libpaths - linker library search paths (list, use append())

(39)

Example

def publish(self, artifact, tools): artifact.collect(_"*.h", "include/")

artifact.cxxinfo.incpaths.append("include")

artifact.cxxinfo.macros.append("PACKAGE_VERSION=1.0")

environ = {}

Artifact environment variables.

A task can add environment variables to an artifact. Such a variable will automatically be set in the environment when consumer tasks are executed. A common use-case is to add programs to the PATH. Values appended to PATH-type variables are relative to the artifact root. They will be automatically ex-panded to absolute paths. These PATH-type variables are supported:

• PATH

• LD_LIBRARY_PATH • PKG_CONFIG_PATH

Example

def publish(self, artifact, tools): artifact.environ.PATH.append("bin")

artifact.environ.JAVA_HOME = artifact.final_path

final_path

The final location of the artifact in the local cache. Type str

path

The current location of the artifact in the local cache. Type str

python = {}

Artifact Python configuration.

A task can add Python configuration to an artifact. Such configuration will automatically be set in the environment when consumer tasks are executed. A common use-case is to add Python modules to the PATH so that they can be easily imported by a consumer.

Values appended to PATH-type variables are relative to the artifact root. They will be automatically ex-panded to absolute paths.

Example

def publish(self, artifact, tools):

artifact.python.PATH.append("my_module")

strings = {}

Artifact strings.

A task can add arbitrary string values to an artifact. Such a string will be available for consumer tasks to read.

(40)

Example

def publish(self, artifact, tools): artifact.strings.version = "1.2"

7.3 Context

class jolt.Context(cache, node)

Execution context and dependency wrapper.

A Context gathers dependencies and initializes the environment for an executing task. It is passed as an argument to the Task’s run() method.

A task implementor can use the context as a dictionary of dependencies where the key is the name of a depen-dency and the value is the dependepen-dency’sArtifact.

__getitem__(key)

Get artifact for a task listed as a requirement.

Parameters key (str) – Name of the task listed as a requirement. Returns TheArtifactassociated with the task.

Example

requires = "dependency"

dependency_artifact = deps["dependency"]

items()

List all requirements and their artifacts.

Returns Requirement dictionary items. Each item is a tuple with the requirement name and the artifact.

7.4 Decorators

class jolt.attributes

static requires(attrib)

Decorates a task with an alternative requires attribute.

The new attribute will be concatenated with the regular requires attribute.

Parameters attrib (str) – Name of alternative attribute. Keywords are expanded.

static system(cls)

Decorates a task with a property returning the operating system name. Examples: “linux”, “windows”

(41)

7.5 Download

class jolt.tasks.Download(parameters=None, **kwargs) Downloads a file over HTTP(S).

Once downloaded, archives are extracted and all of their files are published. If the file is not an archive it is published as is. Recognized archive extensions are:

• .tar • .tar.bz2 • .tar.gz • .tar.xz • .zip Example class NodeJS(Download):

""" Downloads and publishes Node.js. Adds binaries to PATH. """

version = Parameter("14.16.1")

url = "https://nodejs.org/dist/v{version}/node-v{version}-win-x64.zip"

def publish(self, artifact, tools):

super(publish).publish(artifact, tools)

artifact.environ.PATH.append("node-v{version}-win-x64")

collect = ['*']

A list of file publication instructions.

Items in the list are passed directly toArtifact.collect()and can be either strings, tuples or dic-tionaries.

Example

collect = [

"*", # Collect all files

(_"*", "src/"), # Collect all files into the artifact's ˓→src/ directory

{"files": _"*", cwd="subdir"}, # Collect all files from the archive's ˓→subdir/ directory

]

extract = True

Automatically extract archives.

url = None

(42)

7.6 Influence

jolt.influence.always(cls) Always execute the task.

Example

from jolt import influence

@influence.always class Example(Task):

jolt.influence.attribute(name, sort=False) Add task attribute value as hash influence.

Parameters name (str) – Name of task class attribute/property. Example:

@influence.attribute("attribute") class Example(Task):

attribute = False

jolt.influence.daily(cls) Add daily hash influence.

If nothing else changes, the task is re-executed once every day.

Example

@influence.daily class Example(Task):

jolt.influence.environ(variable) Add environment variable hash influence.

Parameters variable (str) – Name of an environment variable that will influence the hash of the task.

Example:

@influence.environ("CFLAGS") class Example(Task):

jolt.influence.files(pathname) Add file content hash influence.

Parameters pathname (str) – A pathname pattern used to find files that will influence the hash of the task The pattern may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

(43)

Example:

@influence.files(_"*.cpp") class Example(Task):

jolt.influence.hourly(cls) Add hourly hash influence.

If nothing else changes, the task is re-executed once every hour.

Example

@influence.hourly class Example(Task):

jolt.influence.monthly(cls) Add monthly hash influence.

If nothing else changes, the task is re-executed once every month.

Example

@influence.monthly class Example(Task):

jolt.influence.weekly(cls) Add weekly hash influence.

If nothing else changes, the task is re-executed once every week.

Example

@influence.weekly class Example(Task):

jolt.influence.yearly(cls) Add yearly hash influence.

If nothing else changes, the task is re-executed once every year.

Example

@influence.yearly class Example(Task):

(44)

7.7 Parameter

class jolt.Parameter(default=None, values=None, required=True, const=False, influence=True, help=None)

Generic task parameter type.

__init__(default=None, values=None, required=True, const=False, influence=True, help=None) Creates a new parameter.

Parameters

• default (str, optional) – An optional default value.

• values (list, optional) – A list of accepted values. An assertion is raised if an unlisted value is assigned to the parameter.

• required (boolean, optional) – If required, the parameter must be assigned a value before the task can be executed. The default is True.

• const (boolean, optional) – If const is True, the parameter is immutable and cannot be assigned a non-default value. This is useful in a class hierarchy where a subclass may want to impose restrictions on a parent class parameter. The default is False. • influence (boolean, optional) – If influence is False, the parameter value will

not influence the identity of the task artifact. The default is True.

• help (str, optional) – Documentation for the parameter. This text is displayed when running the info command on the associated task.

Raises ValueError – If the parameter is assigned an illegal value.

__str__()

Returns the parameter value as a string

get_default()

Get the default value of the parameter.

Returns The default value or None if no default was given.

get_value()

Get the parameter value.

is_default()

Check if parameter is set to its default value.

Returns True if the assigned value is the default value.

is_set()

Check if the parameter is set to a non-default value.

Returns True if the assigned value is not the default value.

is_unset()

Check if the parameter is unset.

Returns True if the assigned value is None.

set_value(value) Set the parameter value.

Parameters value (str) – The new parameter value.

(45)

7.8 BooleanParameter

class jolt.BooleanParameter(default=None, required=True, const=False, influence=True, help=None)

Boolean task parameter type.

__bool__()

Returns True if the parameter value is a non-empty string.

__getitem__(key)

Returns a substitution string depending on the parameter value. Parameters

• key (str) – A special key syntax, enabled,disabled, where • either enabled or disabled would be returned (and) –

• on the paramter value. See the example below. (depending) – Returns Substitution string.

Example

class Example(Task):

debug = BooleanParameter()

self.info("debug is {debug[enabled,disabled]}")

$ jolt build example:debug=true

[INFO] debug is enabled (example)

$ jolt build example:debug=false

[INFO] debug is disabled (example)

__init__(default=None, required=True, const=False, influence=True, help=None) Creates a new parameter.

Parameters

• default (boolean, optional) – An optional default boolean value.

not influence the identity of the task artifact. The default is True.

__str__()

(46)

get_default()

get_value()

is_default()

is_false

The parameter value is False.

is_set()

is_true

The parameter value is True.

is_unset()

Parameters value (boolean) – The new parameter value. Accepted values are: False, True, “false, and “true”, 0 and 1, “no” and “yes”.

7.9 ListParameter

class jolt.Parameter(default=None, values=None, required=True, const=False, influence=True, help=None)

Generic task parameter type.

__init__(default=None, values=None, required=True, const=False, influence=True, help=None) Creates a new parameter.

Parameters

• default (str, optional) – An optional default value.

• values (list, optional) – A list of accepted values. An assertion is raised if an unlisted value is assigned to the parameter.

(47)

__str__()

Returns the parameter value as a string

get_default()

get_value()

is_default()

is_set()

is_unset()

Parameters value (str) – The new parameter value.

7.10 Resource

class jolt.tasks.Resource(*args, **kwargs) Bases:jolt.tasks.Task

A resource task.

Resources are special tasks executed in theContextof other tasks. They are invoked to acquire and release a resource, such as hardware equipment, before and after the execution of a task. No artifact is produced by a resource.

Implementors should overrideacquire()andrelease().

acquire(artifact, deps, tools) Called to acquire the resource.

An implementor overrides this method in a subclass. The acquired resource must be released manually if an exception occurs before the method has returned.

Parameters

• artifact (Artifact) – The artifact associated with the resource. It is not possible to publish files from a resource, but the implementor can still use the resource to pass information to consuming tasks.

(48)

• deps (Context) – Task execution context used to access the artifacts of dependencies. • tools (Tools) – A task specific toolbox.

release(artifact, deps, tools) Called to release the resource.

An implementor overrides this method in a subclass. Parameters

• artifact (Artifact) – The artifact associated with the resource. It is not possible to publish files from a resource, but the implementor can still use the resource to pass information to consuming tasks.

• deps (Context) – Task execution context used to access the artifacts of dependencies. • tools (Tools) – A task specific toolbox.

7.11 Task

class jolt.tasks.Task(parameters=None, **kwargs)

cacheable = True

Whether the task produces an artifact or not.

expires = Immediately()

An expiration strategy, defining when the artifact may be evicted from the cache. When the size of the artifact cache exceeds the configured limit an attempt will be made to evict artifacts from the cache. The eviction algorithm processes artifacts in least recently used (LRU) order until an expired artifact is found.

By default, an artifact expires immediately and may be evicted at any time (in LRU order). An exception to this rule is if the artifact is required by a task in the active task set. For example, if a task A requires the output of task B, B will never be evicted by A while A is being executed.

There are several expiration strategies to choose from: • jolt.expires.WhenUnusedFor

• jolt.expires.After • jolt.expires.Never Examples:

# May be evicted if it hasn't been used for 15 days expires = WhenUnusedFor(days=15)

# May be evicted 1h after creation expires = After(hours=1)

# Never evicted expires = Never()

extends = ""

Name of extended task.

A task with this attribute set is called an extension. An extension is executed in the context of the extended task, immediately after the extended task has executed.