In this section, we show how three network monitoring problems—reflection attack monitoring, application performance analysis, and port scan detection—can be expressed as streaming analytics problems.
Our approach regarding real –time stream processing of big data for enhanced Security may have numerous applications. Therefore here we can briefly enumerate some basic examples; however this list is not at all exhaustive.
11.2.1 Blocking Blacklisted IPs – Analysis of an attacker
This application resembles basically firewalls. Similar to packet filters’ function, we can deploy a Storm Topology for real –time processing of high rates of incoming requests to a web server, in order to timely and without latencies distinguish malicious IPs from benign IPs requesting access to a website or web application and avoid establishing a TCP connection with potentially malicious users. The feed of malicious IPs can be extracted from various sources, like open APIs, public feeds or subscripted Security Intelligence feeds, which provide in regular intervals, ongoing blacklisted IPs. Moreover security intelligence feeds can even be created from internal processing of incoming packets which may present some potentially malicious activity.
Therefore, a deployed storm topology may discriminate between various packets or requests and swiftly drop any possibly malicious IPs trying to establish a TCP connection to our Web Server, while simultaneously retain a list of malicious IPs striving to gain access to our resources and process further the malicious requests to obtain security intelligence about the attacker or malicious user.
11.2.2 DNS traffic analysis – Confront DNS poisoning
Taking into account that “DNS is widely abused by Internet criminals” and that “DNS is used by cyber criminals in order to provide stealthy, flexible and resilient communication within malicious communication infrastructure” (Stevanovic et al, 2016), we should mention that DNS traffic abused for malicious purposes is commonly referred to as malicious DNS traffic. “DNS can facilitate stealthy and undisturbed communication as DNS traffic is present in all networks and it is not usually filtered by firewalls” (), therefore stream processing could be deployed for real –time processing of high rate DNS queries and responses in large scale operational networks.
Breaking down and maybe oversimplifying the DNS query process, we consider that the client’s stub resolver after consulting local cashed records (like /etc/hosts, ldap, nis+, etc.) sends via the network a recursive DNS query to a (full DNS resolver) recursive DNS server (RDNS) which again after consulting its cache (if it is also a DNS cache server) and then it performs the desired recursion by which it discovers iteratively Authoritative Name Servers (ANS) for each zone and ultimately produces a mapping between the queried domain and the resolving IP address.
During this process we can deploy a Storm Topology for real – time processing of high rates of DNS queries and replies, in order to timely and without latencies distinguish malicious IPs and/or domains from benign IPs and/or domains and avoid originating a connection to a potentially malicious resource or more notably to prevent an unknown and obtrusive locally running malware to establish connection to its Command and Control - C&C Servers and extend its botnet, rendering our assets fully vulnerable to further exploitation. Again the feed of malicious IPs and/or domains can be extracted from various sources, like open APIs, public feeds or subscripted Security Intelligence feeds, which provide in regular intervals ongoing notorious - C&C Servers’ IPs and/or domains. Again security intelligence feeds can even be created from internal processing of DNS queries and corresponding responses which may present some potentially malicious pattern.
Therefore, a deployed storm topology may set aside requested malicious domains or replied malicious IPs swiftly inform the user and/or network administrator about the potential harms of accessing the particular IP and/or domain, while simultaneously creating a log file about the exact procedure to understand the occurring attack i.e. either a benign domain was requested and a DNS Poisoning occurred
with falsified DNS reply or a malicious domain was requested probably by an already entrenched malware on our resources. Further process of the weird activity can take place to obtain significant security intelligence about the malware or the attacker.
11.2.3 Distributed port scanning detection.
Another interesting and ambitious deployment would be (distributed) port scanning detection. Existing intrusion prevention system (IPS) devices often cannot process traffic at high rates, and they typically only operate at a single network location. Instead, our topology could be tweaked to count the number of distinct SYN packets that never have a corresponding ACK packet (SYN scanning type) or FIN packets without previous corresponding SYN, ACK packets (FIN scanning type) as long as we are referring to a stateless firewall. We can convert each packet into a tuple, and Storm Bolts could couple each SYN with a matching ACK, if any, or retain a list of source IPs that sent a lone FIN. Many assumptions are made here, on the grounds that port scanning techniques should be carefully studied to apply the respective rules for each method. However, a Topology could easily play the role of a distributed real-time and fault- tolerant Intrusion Prevention System – IPS (or IDS) confronting with high rates of input data.
When referring to distributed port scanning our mind travels to Nmap uses a variety of active probing techniques and offers its users the ability to randomize destination IPs and change the order of and timing between packets. This functionality can obscure the port scanning activity and thus fool intrusion detection systems - IDS.
Non exhaustive list of examples
We remain confident that the aforementioned examples comprise a non exhaustive compiled list, regarding potential applications of Apache Storm (or other real-time processing paradigm) for enhanced security. To our knowledge practical use cases of Apache Storm can be encompassed into real-time IDS and IPS to optimize anomaly based detection implementations.