Matthias Erll
Jan 16, 2018
1 Installation 3
1.1 Salt master . . . 3
1.2 Salt minions . . . 3
2 Configuration 5 3 State module 7 4 Execution module 13 5 Renderer module 15 6 Change History 17 6.1 0.2.2 . . . 17
6.2 0.2.1 . . . 17
6.3 0.2.0 . . . 17
6.4 0.2.0rc1 . . . 17
6.5 0.2.0b2 . . . 18
6.6 0.2.0b1 . . . 18
6.7 0.1.12 . . . 18
6.8 0.1.11 . . . 18
6.9 0.1.10 . . . 18
6.10 0.1.9 . . . 18
6.11 0.1.9b1 . . . 18
6.12 0.1.8 . . . 19
6.13 0.1.7 . . . 19
6.14 0.1.6 . . . 19
6.15 0.1.5 . . . 19
6.16 0.1.4 . . . 19
6.17 0.1.3 . . . 19
6.18 0.1.2 . . . 19
6.19 0.1.1 . . . 19
6.20 0.1.0 . . . 20
7 Indices and tables 21
Python Module Index 23
created in state files.
• Dependencies between services are implicitly defined through links and shared volumes. Therefore, they are followed automatically during creation, start, stop, and removal of containers and do not have to be modelled explicitly via require or require_in.
• Additional volumes can be created automatically to isolate shared resources.
• When images are updated on the Docker host, it is usually for the purpose of updating the containers depending on that image. This action can be triggered via watch in the dockerio module. With the container_map.
updatestate, the re-creation and restart takes places automatically, also considering dependencies.
• Changes in volumes, network parameters, and the command or entrypoint parameter also trigger a container re-creation when using container_map.update.
• In case paths in the virtual filesystem of shared volumes between containers become inconsistent, e.g. due to manual actions or broken containers during a restart of the host system, containers are also re-created as far as necessary.
Along with the state and execution modules comes a custom renderer lazy_yaml. It builds on Salt’s built-in YAML renderer, but adds two custom tags: !pillar and !grain. The !pillar tag is to help the fact that usually pillar files cannot include other pillar items. Instead of being processed by the template renderer, variables are resolved to their actual values just before container_map uses them.
Contents:
2 Contents
Installation
The package consists of a custom renderer module, and two minion modules. The former is used by the Salt master, the latter needs to be distributed to minions.
1.1 Salt master
Install the package by running
pip install salt-container-map
Then place the installed modules in the appropriate Salt master directories, preferably as a symlink. The easiest way of installing the modules is through the provided script install_container_map, which attempts to look up the target directories in the master config file. Since the script uses the Salt API for that purpose, the following locations are checked for the configuration:
• The path described by the environment variable SALT_MASTER_CONFIG;
• the directory from the environment variable SALT_CONFIG_DIR, which may contain a file master;
• and if none of these is available, the default /etc/salt/master.
1.2 Salt minions
Install the module dependency docker-map using pip. For example, the following state can accomplish this:
python_pip:
pkg.installed:
- name: curl cmd.run:
- name: curl -s https://bootstrap.pypa.io/get-pip.py|python - unless:
- which pip
- reload_modules: True - require:
- pkg: curl docker-map:
pip.installed:
- upgrade: True - require:
- cmd: python_pip - reload_modules: True
If pip is already installed, you can skip the python_pip state.
Finally, distribute the modules from the master through saltutil.sync_all or state.highstate, e.g.
salt '*' saltutil.sync_all
4 Chapter 1. Installation
Configuration
There are a few optional configuration settings, that adjust the behavior of the provided modules. For all of them, the order of precedence is as follows, where available:
• Pillar values have the highest priority.
• In absence of the pillar value, a grain value is read.
• If there is no matching grain, the configuration option applies.
• Where applicable, a default value is used in case no other setting is available.
Pillar / Grain Config Description Default Comment docker:url docker.url Docker API URL. http+unix://var/run/docker.socksame as for dock-
erio docker:timeout docker.timeout Docker API request timeout. 60
docker:version docker.version Docker API version. depends on docker- pyversion docker-
registries
Authentication data for Docker registries.
docker:wait_timeoutdocker.wait_timeoutMore specific timeout for waiting for a container to finish.
60
docker:stop_timeoutdocker.stop_timeoutTimeout for stopping a container. More precisely, time span between sending SIG- INT and SIGKILL to the main process.
10
con-
tainer_map:domainname (grain: do-
main)
Domain name for creating containers.
con-
tainer_map:interface_aliases con-
tainer_map.interface_aliases
Mapping network interface alias names to actual interface names.
Example:
private:
eth1 public:
eth0 con-
tainer_map:use_attached_parent_name con-
tainer_map.use_attached_parent_name
Include the name of the parent container in attached containers’ names.
False public:
eth0 con-
tainer_map:skip_checks con-
tainer_map.skip_checks
Skip integrity checks when loading con- tainer maps.
False skip_check
can be set to Trueon map con-
tainer_map:raise_map_errors con-
tainer_map.raise_map_errors
For maps loaded from pillars, raise errors immediately.
True If False, only appears in minion log
con-
tainer_map.pillar_name
Pillar to load container maps from on mod- ule init.
con- tainer_maps con-
tainer_map:base_image con-
tainer_map.base_image
Base image to use for attached volumes. tianon/
true:latest con-
tainer_map:core_image con-
tainer_map.core_image
Core image for initializing attached vol- umes’ permissions.
busybox:latest
6 Chapter 2. Configuration
State module
Module for managing containers, configured throughDocker-Map, along with dependencies in Salt states. It comple- ments Salt’s dockerng module with the following functionality:
• Implicit dependency check of containers based on volumes and links.
• Automated provision of containers for the purpose of selectively sharing data between containers (attached containers).
• Automated re-creation of containers when their running state diverts from the configuration in – image id,
– volume paths, – ports,
– entrypoint, command, or environment.
Maps can be stored in a pillar container_maps (or any other name set in the configuration variable container_map.pillar_name). When using the lazy_yaml renderer, pillar values can be used as far as custom tags are valid in YAML. The lazy_yaml module has to be accessible by the master.
#!lazy_yaml container_maps:
webapp:
repository: registry.example.com host_root: /var/lib/site
containers:
web_server:
image: nginx binds:
!pillar web:host_paths:config:
- !pillar web:host_paths:config - ro
uses: app_server_socket attaches: web_log
exposes:
80: !pillar web:port_http 443: !pillar web:port_https app_server:
image: app instances:
- instance1 - instance2 binds:
- app_config: ro - app_data:
attaches:
- app_log
- app_server_socket user: !pillar app:user permissions: u=rwX,g=rX,o=
volumes:
web_log: !pillar web:container_paths:log
app_server_socket: !pillar app:container_paths:socket app_config: !pillar app:container_paths:config
app_log: !pillar app:container_paths:log app_data: !pillar app:container_paths:data host:
app_config:
instance1: !pillar app1:host_paths:config instance2: !pillar app2:host_paths:config app_data:
instance1: !pillar app1:host_paths:data instance2: !pillar app2:host_paths:data
For skipping the dependency check of incomplete maps, add skip_check: True on the same level as containers.
For merging a pillar into an existing one, use merge_into, e.g.
container_map:
extra_app:
merge_into: webapp containers:
...
During the pillar set up, maps can also copied and extended from others using extend_copy, so that per-application adaptions are possible. For example
will create a map webapp_custom that includes all containers from webapp. It also contains all contents that have been merged into webapp, as merge_into on extended maps is processed beforehand.
As an alternative to the pillar setup, set_up / merged can be used to add container configurations, but these need to be set as a prerequisite to any action.
Aforementioned example will ensure that the following state starts containers in the required order and maps host volumes to the web_server container, and each of the instances app_config.instance1 and app_config.
instance2.
When the image app changes, on re-execution of the state containers are stopped, removed, re-created and restarted as necessary.
webapp.web_server:
container_map.updated: []
8 Chapter 3. State module
- reload_signal: SIGHUP - watch:
- file: webapp_config
For a full description ofDocker-Map, please refer toits documentation.
saltcontainermap.states.container_map.all_removed(name, **kwargs)
Removes all containers from the host. Note this also applies to containers that are not on any map.
name State name - has no effect.
kwargs Keyword arguments forwarded to container_map.remove_all_containers.
saltcontainermap.states.container_map.built(name, **kwargs) name Image tag to apply.
ignore_existing [False] Rebuild the image if it exists. Note this does not imply nocache, so might not actually generate a new image.
show_log [True] Return the build output.
source Dockerfile source (e.g. salt://... for a file loaded from the master).
saltenv [‘base’] Salt environment to use for loading source files.
template Template engine to use for the source file.
context: Additional template context.
contents The script can be passed in here directly as a multiline string or list. Ignored if source is set.
content_pillar Pillar to load the script contents from. Ignored if contents or source is set.
baseimage: Image to base the build on. Ignored if source is used. Can also be included directly using the FROMDockerfile command.
maintainer: Maintainer to state in the image. Ignored if source is used. Can also be included using the MAINTAINERDockerfile command.
kwargs Additional keyword arguments for building the Docker image.
saltcontainermap.states.container_map.containers_clean(name, in-
clude_initial=False, ex- clude=None)
Removes all containers from the host which are not running, not attached volume containers, and not marked as persistent. Note this also applies to containers that are not on any map.
name State name - has no effect.
include_initial [False] If set to True, also removes containers that have never been running.
exclude List of container names or ids to exclude from the removal.
saltcontainermap.states.container_map.created(name, instances=None,
map_name=None, extra_kwargs=None) Ensures that a container exists, along with its dependencies.
name Container configuration name.
instances Optional list of instance names.
map_name Container map name.
extra_kwargs Extra keyword arguments for the container creation.
saltcontainermap.states.container_map.images_clean(name, remove_old=False, keep_tags=None)
Removes all images from the host which are not in use by any container and have no tag. Optionally can also remove images with a repository tag that is not latest, or all tags which are not in the specified whitelist.
name State name - has no effect.
remove_old [False] Remove images that have a tag, but not latest. Does not affect additional (e.g. version) tags of latest images.
keep_tags Remove images that have none of the specified tags.
saltcontainermap.states.container_map.images_updated(name, map_name=None, utility_images=True, inse- cure_registry=False)
Ensures that all images on a map are updated to their latest version or the specified tag.
name State name - has no effect.
map_name Container map name.
map_names Multiple container map names. Can be used instead of, or in conjunction with map_name.
utility_images [True] Unless set to False, also updates utility images such as busybox and tianon/true.
insecure_registry [False] Allow insecure registries for retrieving images.
saltcontainermap.states.container_map.logged_in(name, username=None, pass-
word=None, email=None,
reauth=False)
Ensures authentication to a registry. All parameters are optional, and in case they are no provided information will be read from the docker-registries pillar, if available.
name Registry name.
username Login user name.
password Login password.
email Login email (optional in most cases).
reauth [False] Force re-authentication, even if authentication data has been cached for this registry.
saltcontainermap.states.container_map.merged(name, target_map, containers=None, vol- umes=None, host=None, host_root=None, repository=None, default_domain=None, lists_only=False)
Merges the given map into an existing container map. This means that list-like properties are extended and dictionaries on the target map are updated.
name State name - has no effect.
target_map Name of the container map to merge the following contents into.
containers Container configurations to load.
volumes Volume aliases, each with the path of their container mount point.
host Volume aliases, each with the path of their host mount point.
host_root Host root path to prefix all host paths with, unless they are absolute paths.
10 Chapter 3. State module
saltcontainermap.states.container_map.mod_watch(name, sfun=None, **kwargs)
saltcontainermap.states.container_map.removed(name, instances=None,
map_name=None, extra_kwargs=None) Ensures that a container is removed, along with its dependent containers.
name Container configuration name.
instances Optional list of instance names.
map_name Container map name.
extra_kwargs Extra keyword arguments for the container removal.
saltcontainermap.states.container_map.restarted(name, instances=None,
map_name=None, ex-
tra_kwargs=None) Restarts a container.
name Container configuration name.
instances Optional list of instance names.
map_name Container map name.
extra_kwargs Extra keyword arguments for the container restart.
saltcontainermap.states.container_map.set_up(name, containers=None, volumes=None, host=None, host_root=None, repos- itory=None, default_domain=None,
check_integrity=True, ig-
nore_existing=False) Sets up a container map.
name Name of the container map.
containers Container configurations to load.
volumes Volume aliases, each with the path of their container mount point.
host Volume aliases, each with the path of their host mount point.
host_root Host root path to prefix all host paths with, unless they are absolute paths.
repository Default repository name to prefix image names with, unless they are specified with a different prefix.
default_domain Default domain to set for new containers.
check_integrity [True] Whether to check the map for missing containers and volume aliases. Set to False if the map is to be complemented with merged.
ignore_existing [False] In case a map with the given name exists, this state will fail. Setting this to True overwrites existing maps instead.
saltcontainermap.states.container_map.shut_down(name, instances=None, map_name=None)
Ensures that a container is stopped and removed, along with its dependent containers.
name Container configuration name.
instances Optional list of instance names.
map_name Container map name.
saltcontainermap.states.container_map.signaled(name, instances=None, map_name=None, signal=None)
Sends a signal to a container. By default this is SIGKILL, but can be set to other signals, e.g. SIGHUP for reloading configurations.
name Container configuration name.
instances Optional list of instance names.
map_name Container map name.
signal Signal name or number.
saltcontainermap.states.container_map.started(name, instances=None,
map_name=None, extra_kwargs=None) Ensures that a container is started, along with its dependencies.
name Container configuration name.
instances Optional list of instance names.
map_name Container map name.
extra_kwargs Extra keyword arguments for the container start.
saltcontainermap.states.container_map.started_up(name, instances=None, map_name=None)
Ensures that a container exists and that it is started, along with its dependencies.
name Container configuration name.
instances Optional list of instance names.
map_name Container map name.
saltcontainermap.states.container_map.stopped(name, instances=None,
map_name=None, extra_kwargs=None) Ensures that a container is stopped, along with its dependent containers.
name Container configuration name.
instances Optional list of instance names.
map_name Container map name.
extra_kwargs Extra keyword arguments for the container stop.
saltcontainermap.states.container_map.updated(name, instances=None,
map_name=None, reload_signal=None, send_signal=False, **kwargs)
Ensures that a container is up-to-date, i.e. * the image id corresponds with the image tag from the configuration
* the existing container still has access to all dependent volumes * linked containers are available * command, entrypoint, or environment have not been changed.
Non-existing containers are created and started. Outdated containers are removed and re-created and restarted along the dependency path.
name Container configuration name.
instances Optional list of instance names.
map_name Container map name.
reload_signal Optional signal to send to the main process for reloading.
send_signal [False] Whether to send the reload_signal. Set to True by the watch directive.
12 Chapter 3. State module
Execution module
14 Chapter 4. Execution module
Renderer module
16 Chapter 5. Renderer module
Change History
6.1 0.2.2
• Added state distinction for host-config updates, as implemented in Docker-Map 1.0.0.
• Added show command for displaying imported maps as a dictionary.
6.2 0.2.1
• Fixed bug in processing output from pull_images.
• Added alias pull for pull_images.
6.3 0.2.0
• Fixed bug in template tag expansion.
• Fixed that command output could be overwritten by another command on the same configuration.
• Fixed behavior of remove_all_containers, with and without arguments.
6.4 0.2.0rc1
• Fixed script runner input argument handling.
• Added utility method get_volumes to show volumes of a running container instance.
• Improved change reporting for runtime container updates - network changes and exec commands.
6.5 0.2.0b2
• Fixed input of config references including an instance name.
• Improved detection of container restarts.
• Fixed state for signal action.
6.6 0.2.0b1
• Adapted to upcoming release of Docker-Map, which brings a lot of simplifications and more reliability in detecting changes. Furthermore it should be noted that methods can now be called on multiple configurations at once, and groups make it a lot easier to manage sets of containers.
6.7 0.1.12
• Fixed sending wrong signal in kill function.
• Fixed container removal in remove_all_containers.
6.8 0.1.11
• Fixed access to a method removed from Docker-Map.
6.9 0.1.10
• Fixed import for compatibility with Docker-Map.
6.10 0.1.9
• Added build function and built state.
• Added controlled shutdown to container removal (remove_all_containers).
• De-duplication of image names before pull.
• Configurable base_image and core_image.
• Key out is only returned by module functions for non-empty output.
• Various bugfixes from 0.1.9b1.
6.11 0.1.9b1
• Added tag whitelist to cleanup_images module method and images_clean state.
• Adapted to changes in Docker-Map 0.6.6b1.
18 Chapter 6. Change History
6.13 0.1.7
• Added registry authentication before pulling images in execution module.
• Separated IPv6 from IPv4 addresses in the client configuration.
6.14 0.1.6
• Added extend_copy on map level, which allows for re-using extending maps under a different name.
• Added stop_timeout to remove_all_containers module function, using configuration.
6.15 0.1.5
• Added wrapper for exec_create and exec_start for tracking post-container-start commands.
• Removed extra keyword arguments attached by Salt.
6.16 0.1.4
• Improved behavior of watch directive on updated state.
6.17 0.1.3
• Added signaled state.
• Fixed image update for extended configurations.
6.18 0.1.2
• More sensitive error handling.
6.19 0.1.1
• Added configuration for new DockerMap features.
• Minor cleanups.
6.20 0.1.0
Initial release.
20 Chapter 6. Change History
Indices and tables
• genindex
• search
22 Chapter 7. Indices and tables
s
saltcontainermap.states.container_map, 7
24 Python Module Index
A
all_removed() (in module saltcontain- ermap.states.container_map),9
B
built() (in module saltcontain-
ermap.states.container_map),9
C
containers_clean() (in module saltcontain- ermap.states.container_map),9
created() (in module saltcontain-
ermap.states.container_map),9
I
images_clean() (in module saltcontain- ermap.states.container_map),10
images_updated() (in module saltcontain- ermap.states.container_map),10
L
logged_in() (in module saltcontain-
ermap.states.container_map),10
M
merged() (in module saltcontain-
ermap.states.container_map),10
mod_watch() (in module saltcontain-
ermap.states.container_map),11
R
removed() (in module saltcontain-
ermap.states.container_map),11
restarted() (in module saltcontain-
ermap.states.container_map),11
S
saltcontainermap.states.container_map (module),7
set_up() (in module saltcontain-
ermap.states.container_map),11
shut_down() (in module saltcontain-
ermap.states.container_map),11
signaled() (in module saltcontain-
ermap.states.container_map),12
started() (in module saltcontain-
ermap.states.container_map),12
started_up() (in module saltcontain- ermap.states.container_map),12
stopped() (in module saltcontain-
ermap.states.container_map),12
U
updated() (in module saltcontain-
ermap.states.container_map),12
