4.5 Nested LXC Container Kubernetes Cluster
4.5.1 Creating LXC Containers
In order to deploy the cluster, we need to create the scenario and to do so we need some configurations. First of all we need to install LXD in our host and create a LXC container, in our case is defined as Devops. To do so we are creating it with the next commands:
$ snap install lxd
$ lxd init
$ lxc launch ubuntu :20.04 devops
Figure 24:Creating a LXC Container
Also we need to create the profile for microk8s that are going to use the nodes containers, which is going to be inherited from the host.
$ lxc profile create microk8s
$ wget https :// raw . githubusercontent . com / ubuntu / microk8s / master / tests / lxc / microk8s - zfs . profile
\ -O microk8s . profile
$ cat microk8s . profile | lxc profile edit microk8s
$ lxc profile assign devops default, microk8s
Figure 25: Creating the microk8s profile
Then inside the LXC container, Devops, we are going to create the nodes of the cluster, and we are going to do the same as before, create the profile and then create the containers.
To create the multi-node Cluster, we are going to add node-02 to the node-01 and to do so, we are going to use the next microk8s commands
Node -01# microk8s add - node
microk8s join 10.234.156.79:25000/72 b949125b30d2801955db837ef26020 ...
Node -02# microk8s join 10.234.156.79:25000/72 b949125b30d2801955db837ef26020
Figure 26:Adding a node to the cluster
Once the first container is initialized, node-01, we are going to follow the same procedure to develop the Kubernetes Cluster using microk8s, and so on for the rest of the nodes.
Note that we only need to enable the addons for the rest of the nodes, and in one node we can deploy all, that is going to automatically choose in which node will the pods be deployed. So now we are able to deploy the same cluster as in the Kubernetes Cluster section.
5 Results
The final result of this TFG project is the creation of three different clusters using different technologies, Docker Compose and Microk8s.
Also at the end of the document there are attached the Docker and Kubernetes slides for an academic purpose and a guide to develop the recreated clusters. This guide consists of configuration files and instructions to make it easy for any people that want to develop the same applications or for the collaboration of new students.
In this section, there is a brief summary about the three clusters and the the basic config-uration of each one. All the detailed information of the procedure or the specific settings can be found in the documentation attached to this project.
In addition, there is also a technical explanation of how each functionality has been performed.
5.1 Docker Compose Cluster
The Docker Cluster has been deployed with almost all the applications that we desired.
The only application we could not develop was OpenVPN. That does not mean that the application is not working, the issue was that we were no capable of configuring the VPN server to establish a TAP interface connection in order to access all the services of the cluster.
As we can see, with a simple command we can know what is the status of the services running inside our Docker cluster and which ports of the Host are exposed to the exterior and which are the internal ports for each Docker Application.
Figure 27: Docker Compose Cluster Status
As we can observe the applications hosted are not not exposing any ports of the host, only Nginx and Jitsi applications, because they need it to work. We can access the applications via the created network for the cluster and its going to gain access through the FQDN, that are the names showed above, like redmine_redmine_1 and the port of the container.
To show in a more visual way, the next figure shows all the Proxy Hosts configured with Nginx Proxy Manager, that host the web applications of the other services deployed in the cluster.
Figure 28:Proxy Hosts of the Docker Cluster
5.2 Kubernetes Cluster
To deploy the Kubernetes Cluster was more challenging than deploying it in Docker. The main reason is because K8s is evolving every day, and you must adapt to the changes.
We have to take in mind that not all the desired applications are developed in K8s so we could not deploy all of them. GitLab has nowadays a PoC in K8s and Jitsi is not offering any support for it, also we have to remark that a GitHub repository[4] with Helm charts is deprecated and it has no longer development. This is due to the Helm 3’s public release and it ended the support on Nov 13, 2020.
However we were able to deploy enough applications to cover a company’s needs, we could cover the communication, despite the conferences, the repositories of data, knowledge or
As K8s has integrated ingress, an API object that manages external access to the services in a cluster, also it may provide load balancing, SSL termination and name-based virtual hosting, we do not need to deploy Nginx Proxy Manager.
So with the help of Lens, a K8s IDE, it is easy to show the cluster management. In the next figure are represented all the deployments of the cluster.
Figure 29: Deployments of the Kubernetes Cluster
We know that with the deployment of an application in k8s it will generate the pods, that are the containers running the applications and the services, that expose the application to an externally accessible port. So the next figures represent our pods and services.
Figure 30: Pods of the Kubernetes Cluster
Figure 31: Services of the Kubernetes Cluster
The final figure that represents the accessibility of the applications is the one with the Ingresses of the applications, that provides external access to the cluster. So editing our /etc/host and with a DNS provider we can access to the cluster.
Figure 32: Ingresses of the Kubernetes Cluster
5.3 Nested LXC Container Kubernetes Cluster
This cluster needed a lot of configurations, as mentioned in the Incidences section, we had a lot of issues to make possible the creation of the cluster. Being the main problem the creation of a zpool in the LXC host. Finally we solve it creating the pool manually using the next command.
$ zfs create zpool /lxd - devops
Figure 33: Creating a Zpool
The results are very similar to the previous section, but this time we have deployed three nodes in the cluster, that are hosting the deployments of each application.
Again, with the help of Lens, we can show the cluster specifications. The first figure represents the cluster nodes.
Figure 34:Nodes of the Nested LXC Kubernetes Cluster
To be able to access the services from outside the cluster, from the host, devops, we needed to edit the /etc/hosts to point to the services inside the cluster, that are exposed, they have an external IP. Inside our master node, in this case node-01, we also need to configure the /etc/hosts to point to the desired service.
Figure 35:Ingresses of the Nested LXC Kubernetes Cluster
root@devops :~# cat / etc / hosts 127.0.0.1 localhost
10.234.156.21 lxc . adminer . com 10.234.156.22 lxc . mattermost . com 10.234.156.23 lxc . nextcloud . com 10.234.156.24 lxc . ldap . com 10.234.156.25 lxc . redmine . com 10.234.156.26 lxc . xwiki . com
Figure 36: Hosts of Devops
Now we are able to access to an application by the URL defined in the hosts file. For example, we are going to access lxc.mattermost.com:8065
Figure 37:Mattermost application of the Nested LXC Kubernetes Cluster
As we can see, after all the creation of the cluster and deploying the same applications as before, we can observe that the size of the zpool of the container has increased.
$ zfs list zpool /lxd - devops
NAME USED AVAIL REFER MOUNTPOINT
zpool /lxd - devops 73.4 G 463 G 24K none
Figure 38: Zpool status
6 Budget
If the object of the thesis is not a prototype at least you should include in this section an estimation of the number of hours you have dedicated to the thesis, evaluated at cost of junior engineer. If you have used a specific software you should also include the license and amortization costs.
In terms of the budget involved in this project we are talking of a total amount of almost 50.000€. This budget is determined taking into account different terms.
First of all, we have the salary of the team, this is distributed into two different wages, one for the project leader that is 20€/h and 15€/h for the two junior engineers. In order to finish the project in time, the team is working for 6 months, though the project leader and one of the junior engineers are spending 20h per week and the other junior engineer is spending 40h per week, with a total amount of 520 hours and 960 hours respectively.
The second important thing is the equipment needed, this consists of one laptop for each team member and one special laptop to save all the data. As we are maintaining an Open Source methodology we do not have any cost for the software.
The last thing we have to take into account is that the team needs an office which should include some facilities like internet connection and dispose of water and energy.
So the breakdown of the budget is in the next table that shows the different parts of it.
Table 8:Total Budget for the Project
7 Costs
Once we had done a budget for the project, we can determine the real costs of it, so the costs of the project are divided into three different concepts, the team, the material and the utilities.
First of all we have the team, that is composed by three members. The project leader as it has more responsibilities has a wage of 15€/h and the rest of the team, the junior enginees have a wage of 12€/h. In order to complete the project the team is going to work during six months with a total amount of 320 hours, for the project leader and one of the engineers, they are working 20h per week, and the other engineer, that is working 40h per week, has a total amount of 960 hours. So the salary of the team is 24.480 € and 8.568 € for the Social Security.
Table 9:Cost of the Project Members
To be able to do the project the team needs some equipment such as tables, chairs and technical laptops. This laptops should run linux and have at least 8GB of ram, with a SSD hard drive of 128GB and an Intel Core I7 processor. So the next table shows the material needed to complete the project that has a total investment of 11.200 €.
Table 10: Cost of the Project Material
One important aspect that we need to take into account is the amortization, that will determine the cost of our investment. In order to work we buyed the equipment men-tioned earlier and it has an amortization cost. As the table below shows, we can see the devaluation of the material buyed. The team only needs this material for 6 months, but the amortization is calculated annually, so the total depreciation of the material used for the project is 1.896 €.
Table 11:Cost of the Project Amortization
The last thing that suppose a cost for the team is all the utilities needed, these utilities are the water, energy, telephony and internet and the rent of the office. As shown below it does not support a very expensive cost as it cost 3.320 € for the whole duration of the project.
Table 12: Cost of the Project Utilities
With all the tables represented above we can determine that is not an expensive project as one of the goals we have for it. So we are talking about 45.000 € for a project that is going to improve the deployment of the services of a company and can reduce millions due to the simplicity of it.
Table 13:Total Cost of the Project
8 Environmental Impact
We know that almost every thing we use uses energy, and more when it comes to the IT sector. In 2019 the global data center energy use was 250 TWh, about 1% of global electricity consumption and based on current efficiency improvement trends, electricity consumption is projected to rise to around 270 TWh in 2022. [5]
Also we need to figure out if we are using the machines well. There are a lot of data centers and machines that we are not using them efficiently and we could extract more of them. So in terms of energy reduction, is where the containers come in. Containers, that are packaged software for faster build, ship and deploy, affects in a very positive way to the environment. This is because if they increases the efficiency of the data centers, it will reduce the energy they consume.
So if containers can reduce the equipment needed to deploy applications, it also will reduce the energy we spend on maintaining this equipments. Also we need to note that the objective of containerized applications was not to reduce the energy consumption, but is a factor that benefits of it.
On the other hand, there is and article[6] that states that Docker containers use more energy than a traditional deployment.
Nonethless we can determine that is the begining of an important change to the society and if a lot of developers come together to obtain solutions to the enviromental impact it can be very profitable.
9 Conclusions
First of all, personally, I am very happy of the work done during this project, since it has taught me to work in a didactic way and I have learned a lot. I have faced new challenges and unknown issues, that had led me to acquire new knowledge in order to be able to carry out the project, that is highly complex.
This project allowed us to see the large amount of options there are to host services between our systems. We have gone from the use of methods to improve the performance of the services of a company, like docker, using YAML files, to systems that permits us to integrate a huge amount of separated services and databases, such as the utilization of Helm in Kubernetes.
Each of them has its own benefits in their specific scenarios, and it is worth checking which ones to choose before starting to develop a project or trying to integrate two separate ones.
In one hand, we have Docker, that nowadays is much more developed and offers a lot of different applications to work with, though it only works on a single node.
On the other hand Kubernetes, is simple, powerful and scaleable, though it is still a lot to learn and does not have a lot of deployed applications.
As a final statement, both scenarios are much better than the traditional way of deploying applications. They provide a faster way of deploying applications, more secure, and they have less storage capacity. Also is has proven to be pretty easy to manage and deploy. In addition they can be monitored through Prometheus and managed through Lens.
Finally, it could be good to only use Kubernetes, though its learning curve makes it not as attractive yet, compared to Docker.
10 Future Work
This project opens a wide range of windows for future development. First of all there is a lot of research to do, for both Kubernetes and Docker. Also there is a lot of work to do in Dev, because the services we know nowadays will be deployed using this technology. For example, GitLab, has a PoC for Kubernetes, that it is not deployed on this project.
As regards of nested LXC Containers were developed in a educational sphere and the reliability of them is still being checked.
During the development of this project, Kubernetes has announced that is going to drop support to Docker, note that all the applications use Docker images, but it is not a big deal. In reality it drops the CRI, Container Runtime Interface, support for Docker. The debate now is which runtime is going to be used, Containerd or CRI-O.
Finally, we have to note that this project is very upgradable and since the sector keeps evolving every year, it would be easier to develop newer and safer applications.
References
[1] Docker Inc. Docker Compose Requirements. [Online] Available: https://docs.
docker.com/compose/install/, 2020. [Accessed: 15 Nov 2020].
[2] Canonical Ltd. Microk8s Requirements. [Online] Available: https://microk8s.io/
docs, 2020. [Accessed: 15 Nov 2020].
[3] The Artifact Hub Authors. XWiki Helm Chart. [Online] Available: https://
artifacthub.io/packages/helm/keyporttech/xwiki, 2020. [Accessed: Dec 2020].
[4] Helm Authors. Helm Charts. [Online] Available:https://github.com/helm/charts, 2020. [Accessed: Dec 2020].
[5] IEA 2020. Data Centres and Data Transmission Networks. [Online] Available:https:
//www.iea.org/reports/data-centres-and-data-transmission-networks, 2020.
[Accessed: 27 July 2020].
[6] Christopher Solinas Eddie Antonio Santos, Carson McLean and Abram Hindle. How does docker affect energy consumption? evaluating workloads in and out of docker containers. May 2017. [Online] Available:https://arxiv.org/pdf/1705.01176.pdf [Accessed: Dec 2020].
[7] The Kubernetes Authors. Kubernetes Documentation. [Online] Available: https:
//kubernetes.io/docs/home/, 2020. [Accessed: 27 July 2020].
[8] Kubernetes By Example. Kubernetes Examples. [Online] Available: https://docs.
docker.com/get-started/overview/, 2020. [Accessed: 27 July 2020].
[9] Docker Inc. Quickstart: Compose and WordPress. [Online] Available:https://docs.
docker.com/compose/wordpress/, 2020. [Accessed: Nov 2020].
[10] Jakub Vrána. Adminer. [Online] Available: https://www.adminer.org, 2020. [Ac-cessed: Nov 2020].
[11] GitLab. GitLab. [Online] Available: https://about.gitlab.com, 2020. [Accessed:
Nov 2020].
[12] Jitsi. Jitsi. [Online] Available: https://jitsi.org, 2020. [Accessed: Nov 2020].
[13] Mattermost Inc. Mattermos. [Online] Available: https://mattermost.com, 2020.
[Accessed: Nov 2020].
[14] Jamie Curnow. OpenLDAP. [Online] Available: https://nginxproxymanager.com, 2020. [Accessed: Nov 2020].
[15] OpenLDAP Fundation. OpenLDAP. [Online] Available:https://www.openldap.org, 2020. [Accessed: Nov 2020].
[16] Nextcloud GmbH. Nextcloud. [Online] Available: https://nextcloud.com, 2020.
[Accessed: Nov 2020].
[17] OpenVPN Inc. OpenVPN. [Online] Available: https://openvpn.net, 2020.
[Ac-[18] Redmine. Redmine. [Online] Available:https://www.redmine.org, 2020. [Accessed:
Nov 2020].
[19] XWiki SAS. XWiki. [Online] Available: https://www.xwiki.org, 2020. [Accessed:
Nov 2020].
[20] Canonical Ltd. Microk8s Addons. [Online] Available: https://microk8s.io/docs/
addons, 2020. [Accessed: Dec 2020].
[21] Canonical Ltd. Linux Containers. [Online] Available: https://linuxcontainers.
org/, 2020. [Accessed: Dec 2020].
[22] Docker Inc. Why Docker? [Online] Available: https://www.docker.com/
what-docker, 2020. [Accessed: Dec 2020].
[23] Docker Inc. Docker Compose Documentation. [Online] Available: https://docs.
docker.com/compose/, 2020. [Accessed: Dec 2020].
[24] Cloud Native Computing Foundation. CRI-O. [Online] Available: https://cri-o.
io/, 2020. [Accessed: Dec 2020].
[25] Containerd Authors. Containerd. [Online] Available:https://containerd.io/, 2020.
[Accessed: Dec 2020].
[26] VMware inc. Kubernetes Cluster. [Online] Available: https://www.vmware.com/
topics/glossary/content/kubernetes-cluster, 2020. [Accessed: Dec 2020].
[27] The Kubernetes Authors. Kubernetes Commands. [Online] Available: https:
//kubernetes.io/docs/reference/generated/kubectl/kubectl-commands, 2020.
[Accessed: Dec 2020].
[28] Helm Authors. Using Helm. [Online] Available: https://helm.sh/docs/intro/
using_helm/, 2020. [Accessed: Dec 2020].