It is likely that services running in a swarm will need to be upgraded. Traditionally, upgrades involved stopping a service, performing the upgrade, then restarting the service.
If everything goes well, the service starts and works as expected and the downtime is minimized. If not, there can be an extended outage as the administrator and developers debug what went wrong and restore the service.
Docker makes it easy to test new images before they are deployed and one can be confident that the service will work in production just as it did during testing. The question is, how does one deploy the upgraded service without a noticeable downtime for the users? For busy services, even a few seconds of downtime can be problematic. Docker Swarm provides a way to update services in the background with zero downtime.
There are three options which are passed to docker service create that control how rolling updates are applied. These options can also be changed after a service has been created with docker service update. The options are as follows:
--update-delay: This option sets the delay between each container upgrade.
The delay is defined by a number of hours, minutes, and seconds indicated by a number followed by h, m, or s, respectively. For example, a 30 second delay will be written as 30s. A delay of 1 hour, 30 minutes, and 12 seconds will be written as 1h30m12s.
--update-failure-action: This tells swarm how to handle an upgrade failure. By default, Docker Swarm will pause the upgrade if a container fails to upgrade. You can configure swarm to continue even if a task fails to upgrade.
The allowed values are pause and continue.
--update-parallelism: This tells swarm how many tasks to upgrade at one time. By default, Docker Swarm will only upgrade one task at a time. If this is set to 0 (zero), all running containers will be upgraded at once.
For this example, a service named web is started with six replicas using the nginx:1.10 image. The service is configured to update two tasks at a time and wait 30 seconds between updates. The list of tasks from docker service ps shows that all six tasks are running and that they are all running the nginx:1.10 image:
$ docker service create --name web --update-delay 30s --update-parallelism 2 --replicas 6 nginx:1.10
$ docker service ps web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
83p4vi4ryw9x6kbevmplgrjp4 web.1 nginx:1.10 worker2 Running Running 20 seconds ago
2yb1tchas244tmnrpfyzik5jw web.2 nginx:1.10 worker1 Running Running 20 seconds ago
f4g3nayyx5y6k65x8n31x8klk web.3 nginx:1.10 worker2 Running Running 20 seconds ago
6axpogx5rqlg96bqt9qn822rx web.4 nginx:1.10 worker1 Running Running 20 seconds ago
2d7n5nhja0efka7qy2boke8l3 web.5 nginx:1.10 manager Running Running 16 seconds ago
5sprz723zv3z779o3zcyj28p1 web.6 nginx:1.10 manager Running Running 16 seconds ago
Updates have started using the docker service update command and specifying a new image. In this case, the service will be upgraded from nginx:1.10 to nginx:1.11. Rolling updates work by stopping a number of tasks defined by --update-parallelism and starting new tasks based on the new image. Swarm will then wait until the delay set by --update-delay elapses before upgrading the next tasks:
$ docker service update --image nginx:1.11 web
When the updates begin, two tasks will be updated at a time. If the image is not found on the node, it will be pulled from the registry, slowing down the update. You can speed up the process by writing a script to pull the new image on each node before you run the update.
The update process can be monitored by running docker service inspect or docker service ps:
$ docker service inspect --pretty web ID: 4a60v04ux70qdf0fzyf3s93er
83p4vi4ryw9x6kbevmplgrjp4 web.1 nginx:1.10 worker2 Running Running 35 seconds ago
29qqk95xdrb0whcdy7abvji2p web.2 nginx:1.11 manager Running Preparing 2 seconds ago
2yb1tchas244tmnrpfyzik5jw \_ web.2 nginx:1.10 worker1 Shutdown Shutdown 2 seconds ago
f4g3nayyx5y6k65x8n31x8klk web.3 nginx:1.10 worker2 Running Running 35 seconds ago
6axpogx5rqlg96bqt9qn822rx web.4 nginx:1.10 worker1 Running Running 35 seconds ago
3z6ees2748tqsoacy114ol183 web.5 nginx:1.11 worker1 Running Running less than a second ago
2d7n5nhja0efka7qy2boke8l3 \_ web.5 nginx:1.10 manager Shutdown Shutdown 1 seconds ago
5sprz723zv3z779o3zcyj28p1 web.6 nginx:1.10 manager Running Running 30 seconds ago
As the upgrade starts, the web.2 and web.3 tasks have been updated and are now running nginx:1.11. The others are still running the old version. Every 30 seconds, two more tasks will be upgraded until the entire service is running nginx:1.11.
Docker Swarm does not care about the version that is used on the image tag. This means that you can just as easily downgrade the service. In this case, if the docker service update --image nginx:1.10 web command were to be run after the upgrade, swarm will go through the same update process until all tasks are running nginx:1.10. This can be very helpful if an upgrade does work as it was supposed to.
It is also important to note that there is nothing that says that the new image has to be the same base as the old one. You can decide to run Apache instead of Nginx by running docker service update --image httpd web. Swarm will happily replace all the web tasks which were running an nginx image with one running the httpd image.
Rolling updates require that your service is able to run multiple containers in parallel. This may not work for some services, such as SQL databases. Some updates may require a schema change that is incompatible with the older version. In either case, rolling updates may not work for the service. In those cases, you can set --update-parallelism 0 to force all tasks to update at once or manually recreate the service. If you are running a lot of replicas, you should pre-pull your image to ease the load on your registry.