It’s great that we have the ELK stack running; Elasticsearch will store and make available log data with one search index per day, Logstash will listening for log entries, Logstash Forwarder will send entries in /var/log/auth.log to Logstash, and Kibana will organize the logged data with useful visualizations.
Configuring additional servers to direct their logs to our new Logstash server is fairly simple using Logstash Forwarder. The basic steps we’ll follow are:
1. Set up another server in the Vagrantfile.
2. Set up an Ansible playbook to install and configure Logstash Forwarder alongside the application running on the server.
⁹⁰http://www.elasticsearch.org/guide/en/kibana/current/index.html
Chapter 8 - Ansible Cookbooks 166 3. Boot the server and watch as the logs are forwarded to the main ELK server.
Let’s begin by creating a new Nginx web server. It’s useful to monitor webserver access logs for a variety of reasons, not the least of which is to watch for traffic spikes and increases in non-200 responses for certain resources. Add the following server definition inside the Vagrantfile, just after theendof the ELK server definition:
28 # Web server.
We’ll next set up the simple playbook to install and configure both Nginx and Logstash Forwarder, atprovisioning/web/playbook.yml:
8 - name: Update apt cache if needed.
9 apt: update_cache=yes cache_valid_time=86400
16 - name: Set up virtual host for testing.
17 copy:
18 src: files/example.conf
19 dest: /etc/nginx/conf.d/example.conf
20 owner: root
21 group: root
22 mode: 0644
23 notify: restart nginx
This playbook installs the geerlingguy.nginx and geerlingguy.logstash-forwarder roles, and in the tasks, there is an additional task to configure one virtualhost in a Nginx configuration directory, via the fileexample.conf. Create that file now (at the path provisioning/web/files/ex-ample.conf), and define one Nginx virtualhost for our testing:
1 server {
Since this is the onlyserverdefinition, and it’s set as thedefault_serveron port 80, all requests will be directed to it. We routed theaccess_log to /var/log/nginx/access.log, and told Nginx to write log entries using thecombinedformat, which is what our Logstash server will expect for nginx access logs.
Next, set up the required variables to tell thenginxandlogstash-forwarderroles how to configure their respective services. Insideprovisioning/web/vars/main.yml:
1
Chapter 8 - Ansible Cookbooks 168 Thenginxvariables ensure Nginx will run optimally on our Ubuntu server, and remove the default virtualhost entry. Thelogstash_forwardervariables tellgeerlingguy.logstash-forwarderwhat logs to forward to our central log server:
• logstash_forwarder_logstash_serverand_port: Defines the server IP or domain and port to which logs should be transported.
• logstash_forwarder_files: Defines a list of pathsand fields, which identify a file or list of files to be transported to the log server, along with a type for the files. In this case, the authentication log (/var/log/secure) is asyslog-formatted log file, and /var/log/nginx/ac-cess.logis of typenginx(which will be parsed correctly on the Logstash server since it’s in the combined log format popularized by Apache).
Note that this demonstration configuration is not using a custom certificate to authenticate logging connections. You should normally configure your own secure certificate and give the logstash-forwarder role the path to the certificate using the logstash_forwarder_ssl_-certificate_filevariable. If you use the example provided with the project, you could expose your logging infrastructure to the outside, plus you’ll get a***SECURITY RISK***
warning in the logs every time the Logstash role is run.
To allow Vagrant to pass the proper connection details to Ansible, create provisioning/web/inven-torywith thewebshost details:
1 webs ansible_ssh_host=192.168.9.91 ansible_ssh_port=22
Runvagrant upagain. Vagrant should verify that the first server (logs) is running, then create and run the Ansible provisioner on the newly-definedwebsNginx server.
You can load http://192.168.9.91/ or http://webs/ in your browser, and you should see a Welcome to nginx!message on the page. You can refresh the page a few times, then switch back over tohttp://logs/to view some new log entries on the ELK server:
Entries populating the Logstash Search Kibana dashboard.
If you refresh the page a few times, and no entries show up in the Kibana Logstash dashboard, it could be that Nginx is buffering the log entries. In this case, keep refreshing a while (so you generate a few dozen or hundred entries), and Nginx will eventually write the entries to disk (thus allowing Logstash Forwarder to convey the logs to the Logstash server). Read more about Nginx log buffering in the Nginx’s ngx_http_log_module documentation⁹¹.
A few requests being logged through logstash forwarder isn’t all that exciting. Let’s use the popular abtool available most anywhere to put some load on the web server. On a modest MacBook Air, running the command below resulted in Nginx serving around 1,200 requests per second.
ab -n 20000 -c 50 http://webs/
During the course of the load test, I set Kibana to show only the past 5 minutes of log data, automatically refreshed every 5 seconds, and I could monitor the requests on the ELK server just a few seconds after they were served by Nginx:
⁹¹http://nginx.org/en/docs/http/ngx_http_log_module.html
Chapter 8 - Ansible Cookbooks 170
Monitoring a deluge of Nginx requests in near-realtime.
Logstash Forwarder uses a highly-efficient TCP-like protocol, Lumberjack, to transmit log entries securely between servers. With the right tuning and scaling, you can efficiently process and display thousands of requests per second across your infrastructure! For most, even the simple example demonstrated above would adequately cover an entire infrastructure’s logging and log analysis needs.
Summary
Log aggregation and analysis are two fields that see constant improvements and innovation. There are many SaaS products and proprietary solutions that can assist with logging, but few match the flexibility, security, and TCO of Elasticsearch, Logstash and Kibana.