Let us set up the etcd. First, we should create the first node in the cluster (serv-disc-01) together with the, already familiar, cd VM.
1 vagrant up cd serv-disc-01 --provision 2
3 vagrant ssh serv-disc-01
With the cluster node serv-disc-01 up and running, we can install etcd and etcdctl (etcd command line client).
⁷⁷https://github.com/coreos/etcd
1 curl -L https://github.com/coreos/etcd/releases/\
We downloaded, uncompressed and moved the executables to /usr/local/bin so that they are easily accessible. Then, we removed unneeded files and, finally, run the etcd with output redirected to /tmp/etcd.log.
Let’s see what we can do with etcd.
Basic operations are set and get. Please note that we can set a key/value inside a directory.
1 etcdctl set myService/port "1234"
2
3 etcdctl set myService/ip "1.2.3.4"
4
5 etcdctl get myService/port # Outputs: 1234 6
7 etcdctl get myService/ip # Outputs: 1.2.3.4
The first command put the key port with the value 1234 into the directory myService. The second did the same with the key ip, and the last two commands were used to output values of those two keys.
We can also list all the keys in the specified directory or delete a key with its value.
1 etcdctl ls myService 2
3 etcdctl rm myService/port 4
5 etcdctl ls myService
The last command output only the /myService/ip value since previous command removed the port.
Besides etcdctl, we can also run all commands through HTTP API. Before we try it out, let’s install jq so that we can see the formatted output.
1 sudo apt-get install -y jq
We can, for example, put a value into etcd through its HTTP API and retrieve it through a GET request.
Thejq '.'is not required, but I tend to use it often to format JSON. The output should be similar to the following.
HTTP API is especially useful when we need to query etcd remotely. In most, I prefer the etcdctl, when running ad-hoc commands while HTTP is a preferred way to interact with etcd through some code.
Now that we’ve seen (briefly) how etcd works on a single server, let us try it inside a cluster.
The cluster setup requires a few additional arguments to be passed to etcd. Let’s say that we’ll have a cluster of three nodes with IPs 10.100.197.201 (serv-disc-01), 10.100.197.202 (serv-disc-02) and 10.100.197.203 (serv-disc-03). The etcd command that should be run on the first server would be the following (please don’t run it yet).
1 NODE_NAME=serv-disc-0$NODE_NUMBER
I extracted parts that would change from one server (or a cluster) to another into variables so that you can see them clearly. We won’t go into details of what each argument means. You can find more information in theetcd clustering guide⁷⁸. Suffice to say that we specified the IP and the name of the server where this command should run as well as the list of all the servers in the cluster.
Before we start working on the etcd deployment to the cluster, let us kill the currently running instance and create the rest of servers (there should be three in total).
1 pkill etcd 2
3 exit 4
5 vagrant up serv-disc-02 serv-disc-03
Doing the same set of tasks manually across multiple servers is tedious and error prone. Since we already worked with Ansible, we can use it to set up etcd across the cluster. This should be a fairly easy task since we already have all the commands, and all we have to do is translate those we already
⁷⁸https://github.com/coreos/etcd/blob/master/Documentation/clustering.md
run into the Ansible format. We can create the etcd role and add it to the playbook with the same name. The role is fairly simple. It copies the executables to the /usr/local/bin directory and runs etcd with the cluster arguments (the very long command we examined above). Let us take a look at it before running the playbook.
The first task in theroles/etcd/tasks/main.yml⁷⁹is as follows.
1 - name: Files are copied 2 copy:
The name is purely descriptive and followed with thecopy module⁸⁰. Then, we are specifying few of the module options. The copy option src indicates the name of the local file we want to copy and is relative to the files directory inside the role. The second copy option (dest) is the destination path on the remote server. Finally, we are setting the mode to be 755. The user that runs with roles will have read/write/execute permissions, and those belonging to the same group and everyone else will be assigned read/execute permissions. Next is the with_items declaration that allows us to use a list of values. In this case, the values are specified in theroles/etcd/defaults/main.yml⁸¹file and are as follows.
1 files: [
2 {src: 'etcd', dest: '/usr/local/bin/etcd'}, 3 {src: 'etcdctl', dest: '/usr/local/bin/etcdctl'}
4 ]
Externalizing variables is a good way to keep things that might change in the future separated from the tasks. If, for example, we are to copy another file through this role, we’d add it here and avoid even opening the tasks file. The task that uses the files variable will iterate for each value in the list and, in this case, run twice; once for etcd and the second time for etcdctl. Values from variables are represented with the variable key surrounded with {{ and }} and use theJinja2 format⁸². Finally, we set etcd to be the tag associated with this task. Tags can be used to filter tasks when running playbooks and are very handy when we want to run only a subset of them or when we want to exclude something.
The second task is as follows.
⁷⁹https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/roles/etcd/tasks/main.yml
⁸⁰http://docs.ansible.com/ansible/copy_module.html
⁸¹https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/roles/etcd/defaults/main.yml
⁸²http://docs.ansible.com/ansible/playbooks_variables.html#using-variables-about-jinja2
1 - name: Is running
2 shell: "nohup etcd -name {{ ansible_hostname }} \ 3 -initial-advertise-peer-urls \
11 -initial-cluster-token {{ cl_token }} \ 12 -initial-cluster \
13 {{ cl_node_01 }},{{ cl_node_02 }},{{ cl_node_03 }} \ 14 -initial-cluster-state new \
15 >/var/log/etcd.log 2>&1 &"
16 tags: [etcd]
Shell module⁸³is often the last resort since it does not work with states. In most cases, commands run as through shell will not check whether something is in the correct state or not and run every time we execute Ansible playbook. However, etcd always runs only a single instance and there is no risk that multiple executions of this command will produce multiple instances. we have a lot of arguments and all those that might change are put as variables. Some of them, like ansible_hostname, are discovered by Ansible. Others were defined by us and placed in theroles/etcd/defaults/main.yml⁸⁴.
With all the tasks defined, we can take a look at the playbooketcd.yml⁸⁵.
1 - hosts: etcd
When this playbook is run, Ansible will configure all the servers defined in an inventory, use vagrant as the remote user, run commands as sudo and execute the common and etcd roles.
Let us take a look at the hosts/serv-disc⁸⁶file. It is our inventory that contains the list of all hosts we’re using.
⁸³http://docs.ansible.com/ansible/shell_module.html
⁸⁴https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/roles/etcd/defaults/main.yml
⁸⁵https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/etcd.yml
⁸⁶https://github.com/vfarcic/ms-lifecycle/blob/master/ansible/hosts/serv-disc
1 [etcd]
2 10.100.194.20[1:3]
In this example, you can a different way to define hosts. The second line is Ansible’s way of saying that all addresses between 10.100.194.201 and 10.100.194.203 should be used. In total, we have three IPs specified for this purpose.
Let’s run the etcd playbook and see it in action.
1 vagrant ssh cd 2
3 ansible-playbook \
4 /vagrant/ansible/etcd.yml \
5 -i /vagrant/ansible/hosts/serv-disc
We can check whether etcd cluster was correctly set by putting a value through one server and getting it from the another.
1 curl http://serv-disc-01:2379/v2/keys/test \ 2 -X PUT \
3 -d value="works" | jq '.' 4
5 curl http://serv-disc-03:2379/v2/keys/test \ 6 | jq '.'
The output of those commands should be similar to the following.
1 {
16 "modifiedIndex": 8, 17 "value": "works"
18 }
19 }
We sent the HTTP PUT request to the serv-disc-01 server (10.100.197.201) and retrieved the stored value through the HTTP GET request from the serv-disc-03 (10.100.197.203) node. In other words, data set through any of the servers in the cluster is available in all of them. Isn’t that neat?
Our cluster (after we deploy few containers), would look as presented in the figure 8-6.
Figure 8-6: Multiple nodes with Docker containers and etcd
Now that we have a place to store the information related to our services, we need a tool that will send that information to etcd automatically. After all, why would we put data to etcd manually if that can be done automatically? Even if we would want to put the information manually to etcd, we often don’t know what that information is. Remember, services might be deployed to a server with least containers running and it might have a random port assigned. Ideally, that tool should monitor Docker on all nodes and update etcd whenever a new container is run, or an existing one is stopped. One of the tools that can help us with this goal is Registrator.