As we have been discussing in this chapter, most of the reports aggregate information to present summaries that are easily consumable for the intended audience. The amount of time that it takes to create a report is directly proportional to the amount of data to be processed. Calculating the average value of a specific field or something as simple as counting the number of events implies reading every single event, which in a large data set can take a long time. When you have reports that summarize large data sets on a regular basis this can be inefficient, especially if different users are running similar reports, thus incrementing the load on the system, which can quickly become a big problem.
The core of the problem is that every time you run a report it goes through every one of the required events every single time. One way to solve this repetitive problem is creating summaries of the data for that particular report on a regular basis. That way, when you run a report, it uses the summarized data, significantly reducing the processing time. For that chunk of data that has not been summarized yet, the normal processing applies, but because it is a
Splunk offers two methods to create data summaries for reports, report acceleration and summary indexing. Most of the reports that you have can probably be handled with the report acceleration feature, which is also the easiest of both methods. All you have to do is click on a checkbox and specify a time range when defining the search and Splunk will automagically take care of the rest. Not only that, but any search that is similar to the one you defined for your report will also benefit from the summary.
To show how report acceleration works we will use the Longest Flight by Airline search we defined earlier. When we were building this search we limited the time period to the most recent month of data, September 2012. To make an interesting example, we will state that this search will run on a monthly basis presenting the previous 12 months, that is, a rolling report of the last 12 months. Because TranStats takes a couple of months to upload the most recent month, we also define that the search starts 2 months before the current date and goes back 12 months from that point, that is, 14 months. In this case, earliest is defined as -14mon and latest as -2mon. Remember that when using months as a time modifier you have to use mon, month, or months instead of m, which stands for minutes. This seems to be a common mistake.
Without accelerating this report it takes about four minutes to go over almost five million events. In a world where waiting more than five seconds is unacceptable, four minutes is an eternity, so let us go through the necessary steps to accelerate this report. There are a couple of ways for doing this, when saving a report or by editing an existing report. We will show the latter, for which we go to the “Searches and reports” page of the Manager in the user interface. There we click on the name of our search, Longest Flight by Airline. In Figure 10-30 you can see that the dialog box contains all the information we have provided in the past. When we click on “Accelerate this search,” a pull down menu shows up, presenting various choices for the summary range. We choose one year, as our report covers approximately the last 12 months. After that, all we do is click on the save button.
Figure 10-30. Accelerating a report
From this point on Splunk will take care of updating the summary on a regular basis in the background so that we will not even notice. The first time the summary is created immediately, but it can take a while to complete. To know if
Chapter 10 ■ analyzing airlines, airports, Flights, and delays
the summary is done you can go to the “Report Accelerations Summaries” page of the Manager in the user interface. In Figure 10-19 you will see under the title “Summary Status” that the summary for our report is complete. If it were in the process of updating the summary with the most recent information it would specify the amount of processing it has completed as a percentage. The “Summarization Load” is an important metric as it reflects the effort that Splunk has put into updating the summary. The bigger this number is, the more processing was involved. However, be aware that the calculation is based on the amount of time it takes to update the summary and how often it happens. For a report as ours, which will be updated every month the number will be very low. As a matter of fact, in Figure 10-31 you can see that the number is 0.00004 as it has not been updated for the last few weeks.
The other important metric is the “Access Count,” which also shows the last time the report was run. If you have a report with a high load and it is seldomly used or it has not been used in a long time, you should seriously consider deleting this summary. In addition to Complete, the summarization status can also be Pending, Suspended and Not enough data to summarize. Pending means that it is close to review if an update is needed, whereas suspended means that Splunk has determined that the search is not worth summarizing because the summary is too large. This usually happens with searches that have a high data cardinality, that is, they produce a lot of events, almost as many as there are in the original data set.
If you are interested in more details of the summary, you can click on the strange number under the “Summary ID” column. Figure 10-32 shows the detailed information of a summary.
Within the details, it is worth mentioning the size on disk. This feature is so powerful that it is easy to forget that it has a cost, that of regularly running the search and that the summary uses additional space on disk. Considering that the flight data occupies about 17GBs, 45MBs for this summary is almost negligible. Of course, this will grow over time and we should be mindful of the impact it can have on the longer term.
In Figure 10-32, you can also see that there are various actions available. Because data, fields, and the way that we manipulate them tend to change, sometimes without our knowledge, we can use the verify action to make sure that the summary is still producing correct results and that the data is still consistent. The other actions, update, rebuild, and delete, are standard maintenance issue.
The question now is, how fast is the search that we accelerated? Really fast, clocking at about three seconds. That is a couple of orders of magnitude faster, which in this case is well worth it. Again, remember that you cannot defy the laws of physics; you have to pay the price somewhere. In this case, we increase disk usage and distribute the processing over time in the background. In the summarization load and usage we have two good metrics that let us know if this powerful and very simple feature is worth it for every report.
Of course, not all searches qualify for report acceleration as there are a few limitations. The first one being that the searches must use reporting commands, such as chart, timechart, stats, top, rare, and others. Additionally, the commands feeding the reporting commands must be streaming commands, such as search, eval, fields, regex, lookup,
multikv, and others. The second limitation is that the data to be summarized has to have at least 100,000 recent events. The way the summarization works is that Splunk will regularly run the search in the background and store the summary information so that it is available the next time it is needed. Splunk will figure out the frequency it needs to run the background search, but we can say that it is dependent on the time range you define.
All of the searches we have done in this chapter qualify for report acceleration. You might have thought that the searches for the longest and shortest flights would not qualify because they use the dedup command, which is not a streaming command, but it is used after the reporting commands, not before; thus, it meets the first requirement. Regarding the recent event count, we know that every monthly update brings well over 400,000 events, so that is not an issue.
The second method for creating data summaries is summary indexing. It works basically the same as report acceleration, but internally Splunk handles things differently. Setting it up and using it is a more involved process. As most of the reports can be done with report acceleration, summary indexing is relegated for those cases where the search includes nonstreamable commands before a reporting command. We will not be covering summary indexing in this book. If for any reason you may need to use it, please refer to the Splunk documentation.