We already discussed service discovery tools and the nginx playbook we run earlier made sure that Consul, Registrator, and Consul Template are properly configured on the proxy node. That means that Registrator detected the service container we ran and stored that information to the Consul registry. All that is left is to make a template, feed it to Consul Template that will output the configuration file and reload nginx.
Let’s make the situation a bit more complicated and scale our service by running two instances.
Scaling with Docker Compose is relatively easy.
1 docker-compose scale app=2 2
3 docker-compose ps
The output of the latter command is as follows.
1 Name Command State Ports
2 ---3 vagrant_app_1 /run.sh Up 0.0.0.0:32768->8080/tcp 4 vagrant_app_2 /run.sh Up 0.0.0.0:32769->8080/tcp 5 vagrant_db_1 /entrypoint.sh mongod Up 27017/tcp
We can observe that there are two instances of our service, both using different random ports.
Concerning nginx, this means several things, most important being that we cannot proxy in the same way as before. It would be pointless to run two instances of the service and redirect all requests only to one of them. We need to combine proxy with load balancing.
We won’t go into all possible load balancing techniques. Instead, we’ll use the simplest one called round robin that is used by nginx by default. Round robin means that the proxy will distribute requests equally among all services. As before, things closely related to a project should be stored in the repository together with the code and nginx configuration files and templates should not be an exception.
Let us first take a look at thenginx-includes.conf¹²¹configuration file.
1 location /api/v1/books {
2 proxy_pass http://books-ms/api/v1/books;
3 proxy_next_upstream error timeout invalid_header http_500;
4 }
This time, instead of specifying IP and port, we’re using books_ms. Obviously, that domain does not exist. It is a way for us to tell nginx to proxy all requests from the location to an upstream.
Additionally, we also added proxy_next_upstream instruction. If an error, timeout, invalid header or an error 500 is received as a service response, nginx will pass to the next upstream connection.
That is the moment when we can start using the second include statement from the main configuration file. However, since we do not know the IPs and ports the service will use, the upstream is the Consul Template filenginx-upstreams.ctmpl¹²².
1 upstream books-ms {
2 {{range service "books-ms" "any"}}
3 server {{.Address}}:{{.Port}};
4 {{end}}
5 }
What this means is that the upstream request books-ms we set as the proxy upstream will be load balanced between all instances of the service and that data will be obtained from Consul. We’ll see the result once we run Consul Template.
First things first. Let’s download the two files we just discussed.
¹²¹https://github.com/vfarcic/books-ms/blob/master/nginx-includes.conf
¹²²https://github.com/vfarcic/books-ms/blob/master/nginx-upstreams.ctmpl
1 wget http://raw.githubusercontent.com/vfarcic\
2 /books-ms/master/nginx-includes.conf 3
4 wget http://raw.githubusercontent.com/vfarcic\
5 /books-ms/master/nginx-upstreams.ctmpl
Now that the proxy configuration and the upstream template are on the cd server, we should run Consul Template.
1 consul-template \
2 -consul proxy:8500 \
3 -template "nginx-upstreams.ctmpl:nginx-upstreams.conf" \
4 -once
5
6 cat nginx-upstreams.conf
Consul Template took the downloaded template as the input and created the books-ms.conf upstream configuration. The second command output the result that should look similar to the following.
1 upstream books-ms {
2
3 server 10.100.193.200:32768;
4
5 server 10.100.193.200:32769;
6 7 }
Since we are running two instances of the same service, Consul template retrieved their IPs and ports and put them in the format we specified in the books-ms.ctmpl template.
Please note that we could have passed the third argument to Consul Template, and it would run any command we specify. We’ll use it later on throughout the book.
Now that all the configuration files are created, we should copy them to the proxy node and reload nginx.
1 scp nginx-includes.conf \
2 proxy:/data/nginx/includes/books-ms.conf # Pass: vagrant 3
4 scp nginx-upstreams.conf \
5 proxy:/data/nginx/upstreams/books-ms.conf # Pass: vagrant 6
7 docker kill -s HUP nginx
All that’s left is to double check that proxy works and is balancing requests among those two instances.
1 curl http://proxy/api/v1/books | jq '.' 2
3 curl http://proxy/api/v1/books | jq '.' 4
5 curl http://proxy/api/v1/books | jq '.' 6
7 curl http://proxy/api/v1/books | jq '.' 8
9 docker logs nginx
After making four requests we output nginx logs that should look like following (timestamps are removed for brevity).
1 "GET /api/v1/books HTTP/1.1" 200 268 "-" "curl/7.35.0" "-" 10.100.193.200:32768 2 "GET /api/v1/books HTTP/1.1" 200 268 "-" "curl/7.35.0" "-" 10.100.193.200:32769 3 "GET /api/v1/books HTTP/1.1" 200 268 "-" "curl/7.35.0" "-" 10.100.193.200:32768 4 "GET /api/v1/books HTTP/1.1" 200 268 "-" "curl/7.35.0" "-" 10.100.193.200:32769
While ports might be different in your case, it is obvious that the first request was sent to the port 32768, the next one to the 32769, then to the 32768 again, and, finally, to the 32769. It is a success, with nginx not only acting as a proxy but also load balancing requests among all instances of the service we deployed.
Figure 9-3: Services with automatic proxy with Consul Template
We still haven’t tested the error handling we set up with the proxy_next_upstream instruction. Let’s remove one of the service instances and confirm that nginx handles failures correctly.
1 docker stop vagrant_app_2 2
3 curl http://proxy/api/v1/books | jq '.' 4
5 curl http://proxy/api/v1/books | jq '.' 6
7 curl http://proxy/api/v1/books | jq '.' 8
9 curl http://proxy/api/v1/books | jq '.'
We stopped one service instance and made several requests. Without the proxy_next_upstream instruction, nginx would fail on every second request since one of the two services set as upstreams are not working anymore. However, all four requests worked correctly. We can observe what nginx did by taking a look at its logs.
1 docker logs nginx
The output should be similar to the following (timestamps are removed for brevity).
1 "GET /api/v1/books HTTP/1.1" 200 268 "-" "curl/7.35.0" "-" 10.100.193.200:32768 2 [error] 12#12: *98 connect() failed (111: Connection refused) while connecting t\
3 o upstream, client: 172.17.42.1, server: _, request: "GET /api/v1/books HTTP/1.1\
4 ", upstream: "http://10.100.193.200:32769/api/v1/books", host: "localhost"
5 [warn] 12#12: *98 upstream server temporarily disabled while connecting to upstr\
6 eam, client: 172.17.42.1, server: _, request: "GET /api/v1/books HTTP/1.1", upst\
7 ream: "http://10.100.193.200:32768/api/v1/books", host: "localhost"
8 "GET /api/v1/books HTTP/1.1" 200 268 "-" "curl/7.35.0" "-" 10.100.193.200:32768,\
9 10.100.193.200:32768
10 "GET /api/v1/books HTTP/1.1" 200 268 "-" "curl/7.35.0" "-" 10.100.193.200:32768 11 "GET /api/v1/books HTTP/1.1" 200 268 "-" "curl/7.35.0" "-" 10.100.193.200:32768
The first request went to the port 32768 served by the instance that is still running. As expected, nginx sent the second request to the port 32768. Since the response was 111 (Connection refused), it decided to temporarily disable this upstream and try with the next one in line. From there on, all the rest of requests were proxied to the port 32768.
With only a few lines in configuration files, we managed to set up the proxy and combine it with load balancing and failover strategy. Later on, when we get to the chapter that will explore self-healing systems, we’ll go even further and make sure not only that proxy works only with running services, but also how to restore the whole system to a healthy state.
When nginx is combined with service discovery tools, we have an excellent solution. However, we should not use the first tool that comes along, so we’ll evaluate a few more options. Let us stop the nginx container and see how HAProxy behaves.
1 docker stop nginx
HAProxy
Just like nginx,HAProxy¹²³is a free, very fast and reliable solution offering high availability, load balancing, and proxying. It is particularly suited for very high traffic websites and powers quite many of the world’s most visited ones.
We’ll speak about the differences later on when we compare all proxy solutions we’re exploring. For now, suffice to say that HAProxy is an excellent solution and probably the best alternative to nginx.
We’ll start with practical exercises and try to accomplish with HAProxy the same behavior as the one with have with nginx. Before we provision the proxy node with HAProxy, let us take a quick look at the tasks in the Ansible rolehaproxy¹²⁴.
¹²³http://www.haproxy.org/
¹²⁴https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/roles/haproxy/tasks/main.yml
1 - name: Directories are present
8 - name: Files are present 9 copy:
16 - name: Container is running 17 docker:
The haproxy role is very similar to the one we used for nginx. We created some directories and copied some files (we’ll see them later on). The major thing to note is that, unlike most other containers not built by us, we’re not using the official haproxy container. The main reason is that the official image has no way to reload HAProxy configuration. We’d need to restart the container every time we update HAProxy configuration, and that would produce some downtime. Since one of the goals is to accomplish zero-downtime, restarting the container is not an option. Therefore, we had to look at alternatives, and the user million12 has just what we need. Themillion12/haproxy¹²⁵ container comes with inotify (inode notify). It is a Linux kernel subsystem that acts by extending filesystems to notice changes, and report them to applications. In our case, inotify will reload HAProxy whenever we change its configuration.
Let us proceed and provision HAProxy on the proxy node.
1 ansible-playbook /vagrant/ansible/haproxy.yml \ 2 -i /vagrant/ansible/hosts/proxy