As before, we’ll start by exploring manual installation commands and, later on, automate them with Ansible. We’ll configure it on the cd node as an exercise.
1 sudo apt-get install -y unzip 2
We started by installing unzip since it is not included in default Ubuntu distribution. Then we downloaded the Consul ZIP, unpacked it, moved it to the /usr/local/bin directory, removed the ZIP file since we won’t need it anymore and, finally, created few directories. Consul will place its information to the data directory and configuration files into config.
We can also install Consul WEB UI that will give us an excellent visual presentation of Consul features.
1 wget https://releases.hashicorp.com/consul/0.6.0/consul_0.6.0_web_ui.zip 2
3 unzip consul_0.6.0_web_ui.zip 4
5 sudo mv index.html static /data/consul/ui/
Since the Web UI consists of only static files, all we had to do is download, unpack and move the files. We’ll take a look at the UI later on when we generate some data.
Next we can run Consul.
1 sudo consul agent \ 2 -server \
3 -bootstrap-expect 1 \ 4 -ui-dir /data/consul/ui \ 5 -data-dir /data/consul/data \ 6 -config-dir /data/consul/config \ 7 -node=cd \
8 -bind=10.100.198.200 \ 9 -client=0.0.0.0 \ 10 >/tmp/consul.log &
Running Consul was very straight forward. We specified that it should run the agent as a server and that there will be only one server instance (-bootstrap-expect 1). That is followed by locations of key directories; ui, data and config. Then we specified the name of the node, address it will bind to and which client can connect to it (0.0.0.0 refers to all). Finally, we redirected the output and made sure that it’s running in the background (&).
Let’s verify that Consul started correctly.
1 cat /tmp/consul.log
The output of the log file should be similar to the following (timestamps are removed for brevity).
1 ==> Starting Consul agent...
2 ==> Starting Consul agent RPC...
3 ==> Consul agent running!
4 Node name: 'cd'
5 Datacenter: 'dc1'
6 Server: true (bootstrap: true)
7 Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400) 8 Cluster Addr: 10.100.198.200 (LAN: 8301, WAN: 8302)
9 Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false 10 Atlas: <disabled>
11
12 ==> Log data will now stream in as it occurs:
13
14 [INFO] serf: EventMemberJoin: cd 10.100.198.200 15 [INFO] serf: EventMemberJoin: cd.dc1 10.100.198.200
16 [INFO] raft: Node at 10.100.198.200:8300 [Follower] entering Follower state 17 [WARN] serf: Failed to re-join any previously known node
18 [INFO] consul: adding LAN server cd (Addr: 10.100.198.200:8300) (DC: dc1) 19 [WARN] serf: Failed to re-join any previously known node
20 [INFO] consul: adding WAN server cd.dc1 (Addr: 10.100.198.200:8300) (DC: dc1) 21 [ERR] agent: failed to sync remote state: No cluster leader
22 [WARN] raft: Heartbeat timeout reached, starting election
23 [INFO] raft: Node at 10.100.198.200:8300 [Candidate] entering Candidate state 24 [INFO] raft: Election won. Tally: 1
25 [INFO] raft: Node at 10.100.198.200:8300 [Leader] entering Leader state 26 [INFO] consul: cluster leadership acquired
27 [INFO] consul: New leader elected: cd
28 [INFO] raft: Disabling EnableSingleNode (bootstrap)
We can see that the Consul agent we run in server mode elected itself as the leader (which is to be expected since it’s the only one).
With Consul up and running, let’s see how we can put some data into it.
1 curl -X PUT -d 'this is a test' \
The first command created the msg1 key with the value this is a test. The second had nested the key msg2 into a parent key messages. Finally, the last command added the flag with the value 1234.
Flags can be used to store version number or any other information that can be expressed as an integer.
Let’s take a look how to retrieve the information we just stored.
1 curl http://localhost:8500/v1/kv/?recurse \ 2 | jq '.'
The output of the command is as follows (order is not guaranteed).
1 [
Since we used the recurse query, keys were returned from the root recursively.
Here we can see all the keys we inserted. However, the value is base64 encoded. Consul can store more than “text” and, in fact, it stores everything as binary under the hood. Since not everything can be represented as text, you can store anything in Consul’s K/V, but there are size limitations.
We can also retrieve a single key.
1 curl http://localhost:8500/v1/kv/msg1 \ 2 | jq '.'
The output is the same as before but limited to the key msg1.
1 [
2 {
3 "CreateIndex": 140, 4 "Flags": 0,
5 "Key": "msg1", 6 "LockIndex": 0, 7 "ModifyIndex": 140,
8 "Value": "dGhpcyBpcyBhIHRlc3Q="
9 }
10 ]
Finally, we can request only the value.
1 curl http://localhost:8500/v1/kv/msg1?raw
This time, we put the raw query parameter and the result is only the value of the requested key.
1 this is a test
As you might have guessed, Consul keys can easily be deleted. The command to, for example, delete the messages/msg2 key is as follows.
1 curl -X DELETE http://localhost:8500/v1/kv/messages/msg2
We can also delete recursively.
1 curl -X DELETE http://localhost:8500/v1/kv/?recurse
The Consul agent we deployed was set up to be the server. However, most agents do not need to run in the server mode. Depending on the number of nodes, we might opt for three Consul agents running in the server mode and many non-server agents joining it. If, on the other hand, the number of nodes is indeed big, we might increase the number of agents running in the server mode to five. If only one server is running, there will be data loss in case of its failure. In our case, since the cluster consists of only three nodes and this is a demo environment, one Consul agent running in the server mode is more than enough.
The command to run an agent on the serv-disc-02 node and make it join the cluster is as follows (please don’t run it yet).
1 sudo consul agent \
The only difference we did when compared with the previous execution is the removal of arguments -server and -bootstrap-expect 1. However, running Consul in one of the cluster servers is not enough.
We need to join it with the Consul agent running on the other server. The command to accomplish that is as follows (please don’t run it yet).
1 consul join 10.100.198.200
The effect of running this command is that agents of both servers would be clustered and data synchronized between them. If we continued adding Consul agents to other servers and joining them, the effect would be an increased number of cluster nodes registered in Consul. There is no need to join more than one agent since Consul uses a gossip protocol to manage membership and broadcast messages to the cluster. That is one of the useful improvements when compared to etcd that requires us to specify the list of all servers in the cluster. Managing such a list tends to be more complicated when the number of servers increases. With the gossip protocol, Consul is capable of discovering nodes in the cluster without us telling it where they are.
With Consul basics covered, let’s see how we can automate its configuration across all servers in the cluster. Since we are already committed to Ansible, we’ll create a new role for Consul. While the configuration we’re about to explore is very similar to those we did by now, there are few new details we have not yet seen.
The first two tasks from the Ansible roleroles/consul/tasks/main.yml⁹⁶are as follows.
1 - name: Directories are created 2 file:
8 - name: Files are copied 9 copy:
⁹⁶https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/roles/consul/tasks/main.yml
10 src: "{{ item.src }}"
11 dest: "{{ item.dest }}"
12 mode: "{{ item.mode }}"
13 with_items: files 14 tags: [consul]
We started by creating directories and copying files. Both tasks use variables array specified in the with_items tag.
Let’s take a look at those variables. They are defined in theroles/consul/defaults/main.yml⁹⁷.
1 logs_dir: /data/consul/logs
9 { src: 'consul', dest: '/usr/local/bin/consul', mode: '0755' }, 10 { src: 'ui', dest: '/data/consul', mode: '0644' }
11 ]
Even though we could specify all those variables inside theroles/consul/tasks/main.yml⁹⁸file, having them separated allows us to change their values more easily. In this case, have a simple list of directories and the list of files in JSON format with source, destination and mode.
Let’s continue with the tasks in theroles/consul/tasks/main.yml⁹⁹. The third one is as follows.
1 - name: Is running
2 shell: "nohup consul agent {{ consul_extra }} \ 3 -ui-dir /data/consul/ui \
9 >{{ logs_dir }}/consul.log 2>&1 &"
10 tags: [consul]
⁹⁷https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/roles/consul/defaults/main.yml
⁹⁸https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/roles/consul/tasks/main.yml
⁹⁹https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/roles/consul/tasks/main.yml
Since Consul makes sure that there is only one process running at the time, there is no danger running this task multiple times. It is equivalent to the command we run manually with an addition of a few variables.
If you remember the manual execution of Consul, one node should run Consul in the server node and the rest should join at least one node so that Consul can gossip that information to the whole cluster. We defined those differences as the (consul_extra) variable. Unlike those we used before that are defined in roles/consul/defaults/main.yml¹⁰⁰file inside the role, consul_extra is defined in the hosts/serv-disc¹⁰¹inventory file. Let’s take a look at it.
1 [consul]
2 10.100.194.201 consul_extra="-server -bootstrap -ui-dir /ui"
3 10.100.194.20[2:3] consul_server_ip="10.100.194.201"
We defined variables to the right of the server IPs. In this case, the .201 is acting as a server. The rest is defining the consul_server_ip variables that we’ll discuss very soon.
Let’s jump into the fourth (and last) task defined in theroles/consul/tasks/main.yml¹⁰²file.
1 - name: Has joined
2 shell: consul join {{ consul_server_ip }}
3 when: consul_server_ip is defined 4 tags: [consul]
This task makes sure that every Consul agent, except the one running in the server mode, joins the cluster. The task runs the same command like the one we executed manually, with the addition of the consul_server_ip variable that has a double usage. The first usage is to provide value for the shell command. The second usage is to decide whether this task is run at all. We accomplished that using thewhen: consul_server_ip is defineddefinition.
Finally, we have theconsul.yml¹⁰³playbook, that is as follows.
¹⁰⁰https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/roles/consul/defaults/main.yml
¹⁰¹https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/hosts/serv-disc
¹⁰²https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/roles/consul/tasks/main.yml
¹⁰³https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/consul.yml
1 - hosts: consul
There’s not much to say about it since it follows the same structure as the playbooks we used before.
Now that we have the playbook, let us execute it and take a look at Consul nodes.
1 ansible-playbook \
2 /vagrant/ansible/consul.yml \ 3 -i /vagrant/ansible/hosts/serv-disc
We can confirm whether Consul is indeed running on all nodes by sending the nodes request to one of its agents.
1 curl serv-disc-01:8500/v1/catalog/nodes \ 2 | jq '.'
The output of the command is as follows.
1 [
All three nodes in the cluster are now running Consul. With that out of the way, we can move back to Registrator and see how it behaves when combined with Consul.
Figure 8-9: Multiple nodes with Docker containers and Consul
Setting Up Registrator
Registrator¹⁰⁴has two Consul protocols. We’ll take a look at consulkv first since its results should be very similar to those obtained with the etcd protocol.
1 export DOCKER_HOST=tcp://serv-disc-01:2375 2
3 docker run -d --name registrator-consul-kv \ 4 -v /var/run/docker.sock:/tmp/docker.sock \ 5 -h serv-disc-01 \
6 gliderlabs/registrator \
7 -ip 10.100.194.201 consulkv://10.100.194.201:8500/services
Let’s take a look at the Registrator log and check whether everything seems to be working correctly.
1 docker logs registrator-consul-kv
The output should be similar to the following (timestamps were removed for brevity).
¹⁰⁴https://github.com/gliderlabs/registrator
1 Starting registrator v6 ...
2 Forcing host IP to 10.100.194.201
3 consulkv: current leader 10.100.194.201:8300
4 Using consulkv adapter: consulkv://10.100.194.201:8500/services 5 Listening for Docker events ...
6 Syncing services on 1 containers
7 ignored: 19c952849ac2 no published ports
8 ignored: 46267b399098 port 443 not published on host 9 added: 46267b399098 nginx
The result is the same as when we run Registrator with the etcd protocol. It found the nginx container running (the one that we started previously while practicing etcd) and published the exposed port 4321 to Consul. We can confirm that by querying Consul.
1 curl http://serv-disc-01:8500/v1/kv/services/nginx/nginx?raw
As expected, the output is the IP and the port exposed through the nginx container.
1 10.100.194.201:4321
However, Registrator has another protocol called consul (the one we just used is consulkv) that utilizes Consul’s format for storing service information.
1 docker run -d --name registrator-consul \ 2 -v /var/run/docker.sock:/tmp/docker.sock \ 3 -h serv-disc-01 \
4 gliderlabs/registrator \
5 -ip 10.100.194.201 consul://10.100.194.201:8500
Let’s see what information Registrator sent to Consul this time.
1 curl http://serv-disc-01:8500/v1/catalog/service/nginx | jq '.' This time, the data is a bit more complete yet still in a very simple format.
1 [
Besides the IP and the port that is normally stored with etcd or consulkv protocols, this time, we got more information. We know the node the service is running on, service ID and the name. We can do even better than that with few additional environment variables. Let’s bring up another nginx container and see the data stored in Consul.
1 docker run -d --name nginx2 \ 2 --env "SERVICE_ID=nginx2" \
8 curl http://serv-disc-01:8500/v1/catalog/service/nginx | jq '.'
The output of the last command is as follows.
1 [
12 "ServiceEnableTagOverride": false
The second container (nginx2) was registered and, this time, Consul got tags that we might find useful later on. Since both containers are listed under the same name Consul considers them to be two instances of the same service.
Now that we know how Registrator works in conjunction with Consul, let’s configure it in all nodes of the cluster. The good news is that the role is already created, and we set the protocol to be defined with the variable protocol. We also put the name of the container as the registrator_name variable so that we can bring the Registrator container with the consul protocol without getting in conflict with the etcd one we configured earlier.
The playbookregistrator.yml¹⁰⁵is as follows.
1 - hosts: registrator
Theregistrator-etcd.yml¹⁰⁶has the registrator_protocol variable set to etcd and registrator_port to 2379. We didn’t need it in this case since we already had default values set to consul and 8500 in the roles/registrator/defaults/main.yml¹⁰⁷file. On the other hand, we did overwrite the default value of the registrator_name.
With everything ready, we can run the playbook.
1 ansible-playbook \
2 /vagrant/ansible/registrator.yml \ 3 -i /vagrant/ansible/hosts/serv-disc
Once the execution of this playbook is finished, Registrator with the consul protocol will be configured on all nodes in the cluster.
Figure 8-10: Multiple nodes with Docker containers, Consul and Registrator
How about templating? Should we use confd or something else?