Pattern Detection (order)

The pattern detection based approaches can be applied on a single node as well as mul- tiple nodes. Applying the pattern detection approach on a group of nodes in the vicinity of each other significantly increases the accuracy. Various pattern detection based meth- ods shown acceptable results in detecting anomalies using HPC systems monitoring data. Among them, suffix arrays and sequence analysis provides higher accuracy and less false positives. Pattern detection methods in contrast to statistical analysis mainly utilize the order of events for anomaly detection.

11_{The workflow of behavioral analysis method including all major building blocks is shown in Figure 3.1} 12_{This is the only manual step of the entire workflow.}

13_{The default size is 8 characters.}

Suffix trees and arrays

Suffix trees and suffix arrays are powerful data structures to detect recurring patterns in long sequences. Recurring patterns play an important role in describing the normal be- havior of HPC systems. Numerous Cron jobs15_{generate periodic syslog entries that always}

follow a certain order. Furthermore, multi-step procedures, such as authentication, gen- erate blocks of syslog entries with predefined orders. Therefore, the goal is to detect the recurring sequences of events in syslog entries. Sudden changes in the pattern of recurring events might indicate anomalies.

Suffix arrays are constructed via performing a depth-first traversal of a suffix tree. Both suffix trees and suffix arrays solve the same problem with similar time complexity. How- ever, suffix arrays require less space and provide better cache locality. Similar to previous method, various granularities of monitoring data can be used by suffix trees and suffix arrays. Furthermore, an independent suffix tree is built for every computing node.

A sequence of 7 syslog entries is shown in Figure 4.13a. Each entry is encoded into its event pattern (hash key) via PαααRS. For better readability, every event pattern in this ex- ample is further encoded into a single character (symbol). The collection of all symbols forms the alphabet (i.e., alphabet={J,A,M,E}). To extract the patterns of recurring sequences of events, a simpler form of the classic problem of the longest repeated sub-string (LRS) should be solved. Using suffix trees, the LRS can be found in O(n). The suffix tree of sys- log entries from Figure 4.13a is shown in Figure 4.13b. The longest path16 _{from the root to}

non-leaf vertexes identifies the LRS. In this example, among the non-leaf vertexes of 5, 7 and 9 the path from root to vertex 5 is the longest path. Therefore, the longest repeated sequence is J » A » M or 23666bbc » 760c5208 » 85f5c18b.

Any repeated sequence of events (two and more events) represents a pattern and is significant. Thus, instead of extracting only the longest repeated sub-string, all repeated sub-strings (RS) will be identified. In a suffix tree, repeated sub-strings are represented by paths from root to the parent of each leaf. Among the overlapping paths with similar repetition rate, the largest path is chosen. Paths should have a minimum length of two symbols. Therefore, according to the suffix tree shown in Figure 4.13b, although both paths from root to 5 and 7 represent a repeated sub-string, due to the overlaps only the pat h to 5is considered.

The detected RS among system logs identifies the block of syslog entries which are ex- pected to appear in a certain order. Any derivation from these patterns might be a sign of anomalous behavior. An example of applying pattern detection on Taurus syslog entries is shown in Figure 4.14.

Sequence Analysis

The structure of syslog entries has resemblance to Human’s DNA. Both are long se- quences of unites (log entries / base pairs) with numerous repeated sub-sequences [257].

15_{Jobs which are executed using Cron, the time-based job scheduler of Unix-like operating systems.} 16_{The path with the most number of symbols.}

(a) Encoding event patterns into symbols (b) The suffix tree of encoded event patterns. Each path from root to a non-leaf vertex is a repetitive sub-sequence. TIMESTAMP EVENT 1488424393 23666bbc -> J 1488424398 760c5208 -> A 1488424414 85f5c18b -> M 1488424459 bba3d47c -> E 1488424570 23666bbc -> J 1488424596 760c5208 -> A 1488424611 85f5c18b -> M

Figure 4.13: Creation of suffix tree for sample syslog entries

# Message Event

1 (siavash) CMD (/usr/bin/check >/dev/null 2>&1) 66dc2742

2 (parya) CMD (/usr/lib32/lm/lm1 1 1) 66dc2742

3 (siavash) CMD (run-parts /etc/cron.hourly) 66dc2742

4 starting 0anacron dd740712

5 Anacron started on 2018-01-30 e5a59462

6 Jobs will be executed sequentially f1e7eac3

7 Normal exit (0 jobs run) eac7924f

8 finished 0anacron a5803a8a

9 (siavash) CMD (/usr/lib32/lm/lm1 1 1) 66dc2742

10 (root) CMD (/usr/lib32/cl/cl2 1 1) 66dc2742

11 (root) CMD (/usr/lib64/lm/lm1 1 1) 66dc2742

12 (siavash) CMD (/usr/bin/check >/dev/null 2>&1) 66dc2742

13 (parya) CMD (/usr/bin/run >/dev/null 2>&1) 66dc2742

14 (siavash) CMD (/usr/bin/exec >/dev/null 2>&1) 66dc2742

15 (siavash) CMD (run-parts /etc/cron.hourly) 66dc2742

16 starting 0anacron dd740712

17 Anacron started on 2018-01-31 e5a59462

18 Jobs will be executed sequentially f1e7eac3

19 Normal exit (4 jobs run) eac7924f

20 finished 0anacron a5803a8a

1: Starting the daemon 2: Successfully started

3: Successfully finished

1: Starting the daemon 2: Successfully started

3: Successfully finished

Figure 4.14: Recurring blocks of syslog entries. Similar colors indicate similar events. The events are automatically derived from syslog messages using PαααRS.

Throughout the years various techniques and tools have been developed to analyze the sequence of base pairs in DNA. The goal of this section is to employ DNA sequence ana-

lyzing methods for syslog analysis based on the similarities between DNA structures and syslog entries (e.g., long range correlations [258]).

Sequence alignment map (SAM) is a text-based format suitable for storing DNA se-

quences aligned to a reference sequence [259]. Since the SAM format is widely used by various DNA sequence analyzing tools, in order to apply the DNA sequence analyzing meth- ods, Taurus syslog entries were transformed into the SAM format.

Sequence alignmentis used to detect anomalies with a reduced false positive rate. Us-

ing majority voting among the nodes in the vicinity of each other, the reference sequence of events on HPC system is extracted. The system logs are aligned to the reference se- quence in their SAM format. Derivations from the reference sequence can be detected via sequence alignment methods. All existing derivations are not erroneous, in fact a certain degree of change is expected due to the dynamic nature of the HPC systems and the users behavior.

Point mutation is a change in the bases of DNA. This concept is used to mitigate the

noises in system logs and reduce the false positives. Single mismatches during the com- parison of current sequence and the reference sequence is interpreted as noise (point mutation).

Sequence motifs are significant sequence patterns in DNA. For system logs the se-

quence motifs are defined according to their length, frequency, and accuracy of repetition. Longer sequence patterns that are more frequent and have precise recurring time inter- vals are considered as sequence motifs. Derivations of sequence motifs from the reference sequence are interpreted as signs of anomalies.

The main advantage of using tools and methods that are originally built for analyzing DNA sequences is their capability in processing radically large sequences. However, to be able to use those tools and methods, the monitoring data (e.g., system logs) must be transformed into compatible formats. Furthermore, the important features of the moni- toring data must be preserved. In this work the transformation of Taurus system logs into sequence alignment map is performed using PαααRS.

PαααRS encodes syslog entries into anonymized event patterns (anonymized list). Fre- quency of each event pattern in a fixed interval (e.g., 24 hours) is calculated. A reverse ordered (descending) list of all event patterns based on their frequency is generated (fre-

quency list). The frequency list is divided into subsections, each with 4 entries (frequency sub-list). The anonymized list is divided into subsections according to the event patterns in

each frequency sub-list (anonymized sub-list). A unique symbol from a 4-letter predefined alphabet (e.g., {A,T,C,G}) is assigned to each event pattern of a frequency sub-list. The event patterns of each anonymized sub-list are substituted by the relevant symbols listed in the corresponding frequency sub-list (sequence sub-list). Each sequence sub-list is converted into a SAM file and is ready for further analysis. Table 4.2 provides an example of applying the proposed workflow on a collection of syslog entries.

Table 4.2: Encoding syslog entries into DNA-like sequences

Anonymized list Frequency sub-list Sequence sub-list

Timestamp Event pattern Event pattern Frequency Symbol timestamp Symbol

1490997601 34b25731 1c6f9d5e 7 A 1490997601 C 1490997601 1c6f9d5e 1808e388 3 T 1490997601 A 1490997601 1808e388 34b25731 2 C 1490997601 T 1490997602 90a389bc fa144f05 1 G 1490997661 G 1490997661 2e307f37 e92704ab 1 A 1490997901 A 1490997661 fa144f05 ddab3d0a 1 T 1490998201 A 1490997661 6234343a 90a389bc 1 C 1490998201 T 1490997661 ddab3d0a 8c41e908 1 G 1490998501 A 1490997661 8c41e908 8372c3dc 1 A 1490998801 T 1490997661 e92704ab 6234343a 1 T 1490998801 A 1490997661 8372c3dc 35d87b50 1 C 1490999101 A 1490997661 35d87b50 2e307f37 1 G 1490999401 A 1490997901 1c6f9d5e 1490999401 C 1490998201 1c6f9d5e 1490997661 A 1490998201 1808e388 1490997661 T 1490998501 1c6f9d5e 1490997661 C 1490998801 1808e388 1490997661 G 1490998801 1c6f9d5e 1490997661 A 1490999101 1c6f9d5e 1490997661 T 1490999401 1c6f9d5e 1490997602 C 1490999401 34b25731 1490997661 G

The size of alphabet can be increased to any arbitrary number of symbols. However, large alphabets may negatively impact the detection mechanisms. According to Taurus syslog analysis, high frequency entries deliver less significant information in comparison to low frequency entries. Therefore, grouping the anonymized list on syslog entries into smaller sub-lists according to their frequency improves the detectability of low frequency (but significant) entries via comparing events only within groups of events with similar fre- quency. It is important to note that in sequence analysis, similar to other methods pro- posed in this work, the normal behavioral pattern is inferred via majority voting within nodes vicinity.

Figure 4.15 shows a 24-hour sample of system logs (in SAM format) collected from 100 nodes visualized using Tablet (Table B.1). Each row represents a single node (100 rows) and each column represents a one-minute time window (1, 440 columns). Several potential anomalies in form of horizontal red zones are visible in the middle and bottom of the figure.