A Real Time Unsupervised NIDS for Detecting Unknown and Encrypted Network Attacks in High Speed Network

A Real Time Unsupervised NIDS for Detecting

Unknown and Encrypted Network Attacks in High

Speed Network

Payam Vahdani Amoli

Prof. Timo Hämäläinen, Prof. Gil David

Department of Mathematical Information Technology

Faculty of Information Technology, Jyväskylä University Jyväskylä, Finland

Agenda

•

Review

•

Introduction

•

Related Works

•

Proposed Solution

•

Conclusions and Future Works

•

Acknowledgments

•

Questions

Review

3

•

M. CS Information Security, from University Technology Malaysia (UTM),

Malaysia, 2008-2010

•

B. IT (Hons) Information Systems Engineering, from Multimedia University

(MMU), Malaysia, 2004-2007

Review

•

" New Detection Technique Using Correlation of Network Flows For NIDS",

Proceedings of the International Conf. on Security Management (SAM

2011), Las Vegas, Nevada, USA

•

" Botnet Detection Based on Traffic Monitoring", IEEE International Conf.

on Networking and Information Technology (ICNIT 2010), Manila,

Philippines

•

" A taxonomy of Botnet detection techniques", 3rd IEEE International Conf.

on Computer Science and Information Technology (ICCSIT 2010), Chengdu,

China

•

"All About Malwares (Malicious Codes)", Proceedings of the International

Conf. on Security Management (SAM 2010), Las Vegas Nevada, USA

•

"Automated tools for manipulating files in a distributed environment with

RSYNC", 12th IEEE International Conf. on Advanced Communication

Introduction

•

NIDS (Network Intrusion Detection System) monitors the behavior of

the network to detect intrusions.

•

Generally there are two types of NIDS:

–

Signature-based

–

Anomaly-based

Introduction

–

Signature-based : which detects intrusions by matching the

behavior of the network with the predefined “attack signatures”.

•

Issues: can not detect “Zero day” attacks, creating attack

signature is costly and time consuming

–

Anomaly-based: which detects intrusions by matching the current

behavior of the network to the training samples

•

Issues: Imbalance traffic increase false alarms, training

samples or labeled data are costly and time consuming,

difficult to declare a normal state

Introduction

There are different types of input for each NIDS:

–

Byte

–

Packets (header and content)

–

Network flows: “A flow is defined as a set of IP packets passing an

observation point in the network during a certain time interval. All

packets belonging to a particular flow have a set of common

properties.”

Introduction

Issues of Byte or packets (header and content)

•

Difficulties of monitoring in high-speed network

•

High rate of false alarms for complex attacks

•

No sufficient data

Introduction

Advantages of Network flows

•

Higher rate of detection and Better understanding from networks

behavior (to detect complex attacks)

•

Faster detection (minimize the computation time for detection

engine)

•

0.1% Storage

•

Real time

•

Encrypted data

Introduction

•

Machine learning algorithms improved detection rate of complex and new

attacks, there are three categories of machine learning algorithms:

•

Supervised

•

Semi supervised

Introduction

Supervised and Semi supervised

–

They can be trained by labeled or attack free dataset (need

supervision)

–

Issues: producing labeled or attack-free dataset is difficult

Introduction

Unsupervised

–

They formulate the invisible structure of an unlabeled data set

without any supervision

Introduction

Clustering:

Clustering is the process of assigning a set of objects into group or

groups (which called cluster) while the objects in the same cluster are

more similar (in some way) compare to other objects

Related Works

Unsupervised machine learning have been proposed in:

–

Neural network [26]

•

Real-time NIDS

•

Combination of misuse and anomaly detection

•

Combination of several neural network (SOM, ART1

and ART2)

•

High rate of detection

–

Clustering [27]

•

Combination of Sub-Space Clustering and DBSCAN

Related Works

Unsupervised machine learning have been proposed in:

–

Clustering Encrypted Traffics [9,11,12]

•

Analysing network flows provides sufficient

information to detect intrusion within encrypted

•

Distinguish the behaviour of networks based on

statistics

In order to have a real-time NIDS, the system should receive

packets from different sniffers in the network. At the

beginning, the NIDS should filter the duplicated packets, and

then put all of the packets into correct time synchronized

queue, then the content of packets and packets’ header will

be used to create network flows.

The proposed NIDS has two different window sizes:

–

The first window will be monitored by network change

measurement formula

,based on “time-series analyses”.

If any element passes the threshold the system will mark

that time slot as anomaly and send it to the first engine

for further analyses to detects the intrusion in real-time

manner.

–

If the first engine detects distributed network attacks

such as DDOS then the second engine will find the

similarities of past communication of attackers (multiple

machines) to find the Bot-Master in the Botnet attacks.

Proposed Solution

Proposed Solution

Packets Sniffer Z Packets First Engine Packets Packets Network’s Behaviour (Features in Table 1) Second Engine

Network Flows of Attackers (Distributed Attackers)

Filtered and Time Synchronized

Packets

IP Add & Port # of Attackers

(During Distributed Attacks)

Report of BotMasters

Report of Current Live Attacks Input Preprocessor

(Filtering, Time Synchronization, Network Flow Creation) Packets

Traffic Aggregator

Proposed Solution

19

N

ETWORK

F

LOW SPECIFICATION FOR EACH TYPE OF PACKET

Packet Type Specification

IP Src-IP, Dest-IP, Time of First Packet, Time of last packet, Duration

TCP Src-Port, Dest-Port, #Packets, #SYN, #SYN-ACK, #RST, #RST-ACK, #FIN-ACK, Average Packet Size from Source, Average Packet Size from Destination, Biggest Packet Size, Smallest Packet Size, Time of Last packet from Source, Time of last Packet From Destination, Average latency of packets from Source, Average latency of packets from Destination

UDP Src-Port, Dest-Port, #Packets, Average Packet Size from Source, Average Packet Size from Destination, Biggest Packet Size, Smallest Packet Size

ICMP Average Packet Size from Source, Average Packet Size from Destination, Biggest Packet Size, Smallest Packet Size, #Eco Request, #Eco Reply

Proposed Solution (First Engine )

1-#Flows 6-Ratio of #Flows to # Live Destination 2-#Live Sources 7-XLOGX (5) & XLOGX (6)

3-#Live Destinations 8- STD (65-50,50-35,35-20,20-5) seconds 4-#Suspicious Flows 9-Threshold=(MAX(7)+MAX(8))*α

5-Ratio of #Flows to # Live Sources 10-(XLOGX)/(Threshold)

•

Sub-Space clustering

–

DBSCAN(α=10% of data,β=Mean of Euclidian Distance and

Mahalanobis Distance )

Proposed Solution (First Engine )

One of the way to detect Bot-Master is to aggregate and correlate

network flows of DDOS, Spam senders or other distributed types of

attacks. The reason of storing the entire past hour of network traffic is to

be able to find the relation and similarities of previous connections from

attackers to a suspicious machine (the Bot-Master).

Proposed Solution (First Engine )

Proposed Solution (First Engine )

While NIDS detects distributed network attacks the DBSCAN will cluster

the IP addresses and Port numbers to find similar destination or sources

of network flows. By this way the system will flag the similar destination

machines as suspicious Bot-Master.

Proposed Solution (Second Engine )

Conclusions and Future Works

•

The goal of this Project is to propose a model for NIDS to work in

Real-Time

•

Proposing “Unsupervised NIDS” allow us to over come the issues of

creating labeled or attack free dataset which is costly and time

Conclusions and Future Works

29

•

Found the weaknesses and limitations of current unsupervised NIDS and

apply new features

–

Applying Mahalanobis distance for DBSCAN to apply the factor of

“density”

–

Decrease the computation burden by monitoring the volume of

traffic before processing it by “network change measurement

formula ”

–

Dividing the process of detection by applying multi-stage engines to

decrease the load of the computation processes

Conclusions and Future Works

•

More accurate parameters for BDSCAN

Publication

31

• " Artificial Immune System Based Intrusion Detection: Innate Immunity using an Unsupervised Learning Approach", International Journal of Digital Content

Technology and its Applications(JDCTA), Volume8, Number5, Oct 2014

• " A Real Time Unsupervised NIDS for Detecting Unknown and Encrypted Network Attacks in High Speed Network", 2nd IEEE International Workshop on

Measurements and Networking (M&N 2013), Naples, Italy

• " Distributed Agent Based Model for Intrusion Detection System Based on Artificial Immune System", International Journal of Digital Content Technology and its

Applications(JDCTA), Volume7, Number9, May 2013

• " Real Time Multi Stage Unsupervised Intelligent Engine for NIDS to Enhance Detection Rate of Unknown Attacks", 3rd IEEE International Conf. on Information Science and Technology (ICIST 2013), Jiangsu, China

• " Real-time Botnet command and control characterization at the host level", 6th IEEE International Symposium on Telecommunications (IST 2012), Tehran, Iran

Acknowledgments

We wish to acknowledge the sponsorships of CIMO (Centre

for International Mobility) in Helsinki, Finland and COMAS

(Doctoral Program in Computing and Mathematical Sciences)

by the University of Jyväskylä, Finland which have made it

possible to undertake this research.

Questions

33

References

1. A. Sperotto, G. Schaffrath, R. Sadre, C. Morariu, A. Pras, B. Stiller, “An overview of IP flow-based intrusion detection,” Communications Surveys & Tutorials, IEEE , vol.12, no.3, pp.343-356, Third Quarter 2010

2. V. Engen, “Machine learning for network based intrusion detection: an

investigation into discrepancies in findings with the KDD cup '99 data set and

multi-objective evolution of neural network classifier ensembles from imbalanced data,” PhD Thesis, Bournemouth University, 2010

3. P. Vahdani Amoli, A.R. Ghobadi, G. Taherzadeh, R. Karimi, S. Maham, “New Detection Technique Using Correlation of NetworkM. H. Bhuyan, D. K.

Bhattacharyya, J. K. Kalita. “An effective unsupervised network anomaly detection method,” In Proceedings of the International Conference on Advances in

Computing, Communications and Informatics (ICACCI '12). ACM, pp.533-539, New York, NY, USA, 2012

4. A. Lakhina, M. Crovella, C. Diot, “Characterization of network-wide anomalies in traffic flows,” Proc. of the 4th ACM SIGCOMM conference on Internet

References

35

5. G. Tedesco, U. Aickelin, “An Immune Inspired Network Intrusion Detection System Utilising Correlation Context,” Proceedings of the Workshop on Artificial Immune Systems and Immune System Modelling (AISB '06), Bristol, 2006

6. T. Peng, C. Leckie, K. Ramamohanarao, “Proactively Detecting Distributed Denial of Service Attacks Using Source IP Address Monitoring,” Proceedings of the Third

International IFIP-TC6 Networking Conference (Networking 2004), pp.771-782, 2004

7. A.L. Mark, M. Crovella, C. Diot, “Characterization of Network-Wide Anomalies in Traffic Flows,” IMC '04 Proceedings of the 4th ACM SIGCOMM conference on Internet measurement, pp.201-206, New York, NY, USA, 2004

8. B. Claise, “Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information,” RFC 5101 (Proposed Standard), [Online]. Available: http://www.ietf.org/rfc/rfc5101.txt, Jan. 2012

References

9. R. Koch, G.D. Rodosek, "Security System for Encrypted Environments (S2E2)," RAID 2010, LNCS, vol. 6306, pp.505-507, Springer, Heidelberg, 2010

10. R. Koch, G.D. Rodosek , "Command Evaluation in Encrypted Remote Sessions," Network and System Security (NSS), 2010 4th International Conference on , vol., no., pp.299-305, 1-3 Sept. 2010

11. M. Augustin, A. Balaz, "Intrusion detection with early recognition of encrypted application," Intelligent Engineering Systems (INES), 2011 15th IEEE International Conference on , vol., no., pp.245-247, 23-25 June 2011

12. F. Alserhani, M. Akhlaq, I.U. Awan, A.J. Cullen, P. Mirchandani, "MARS: Multi-stage Attack Recognition System," Advanced Information Networking and Applications (AINA), 2010 24th IEEE International Conference on , vol., no., pp.753-759, 20-23 April 2010

13. P. García-Teodoro, J. Díaz-Verdejo, G. Maciá-Fernández, E. Vázquez, “Anomaly-based network intrusion detection: Techniques, systems and challenges,” Computers & Security, vol. 28, Issues 1–2, pp. 18-28, February–March 2009

References

37

14. M.N.M. Sap, A.H. Abdullah, S. Srinoy, S. Chimphle, W. Chimphle, “Anomaly

Intrusion Detection Using Fuzzy Clustering Methods,” Jurnal Teknologi Maklumat, FSKSM, UTM, Jurnal Teknologi Maklumat, vol.18, pp.25-32, 2006

15. T.P. Fries, “A Fuzzy-Genetic Approach to Network Intrusion Detection,” Proceedings of the 2008 GECCO conference companion on Genetic and evolutionary

computation, Atlanta, GA, USA, pp.2141-2146, 2008

16. T.T.T. Nguyen, G. Armitage, "A survey of techniques for internet traffic classification using machine learning," Communications Surveys & Tutorials, IEEE , vol.10, no.4, pp.56-76, Fourth Quarter 2008

17. M. H. Bhuyan, D. K. Bhattacharyya, J. K. Kalita. “An effective unsupervised network anomaly detection method,” In Proceedings of the International Conference on Advances in Computing, Communications and Informatics (ICACCI '12). ACM, pp.533-539, New York, NY, USA, 2012

References

18. H.R. Zeidanloo, Bt Manaf, P. Vahdani Amoli, F. Tabatabaei, M. Zamani, “Botnet Detection Based on Traffic Monitoring,” International Conference on Networking and Information Technology (ICNIT), vol., no., pp.97 – 101, Manila, Philippines, 2010

19. H.R. Zeidanloo, M.J.Z. Shooshtari, P. Vahdani Amoli, M. Safari, M. Zamani,“A

taxonomy of Botnet detection techniques,”3rd IEEE International Conference on Computer Science and Information Technology (ICCSIT), vol.2, no., pp.158 – 162, Chengdu, China , 2010

20. H.R. Zeidanloo, F. Tabatabaei, P. Vahdani Amoli, A. Tajpour, “All about Malwares (Malicious Codes),” Proceedings of the 2010 International Conference on Security Management, SAM 2010, pp.342-348, Las Vegas Nevada, USA, 2010

21. F.F. Etemad, P.Vahdani Amoli, “Real-Time Botnet Command and Control Characterization at the Host Level,” 6th International Symposium on

Telecommunication with emphasis on Information and Communication Technology (IST’2012), Tehran, Iran, 2012

References

39

22. A. Karasaridis, B. Rexroad, D. Hoeflin, “Wide-scale botnet detection and

characterization,” Proceedings of the first conference on First Workshop on Hot Topics in Understanding Botnets, pp.7-7, Cambridge, MA, USA, 2007

23. H.C. Lin, C.M. Chen, J.Y. Tzeng, "Flow Based Botnet Detection," Innovative Computing, Information and Control (ICICIC), 2009 Fourth International Conference on , vol., no., pp.1538-1541, 2009

24. W. Hong, G. Zhenghu, G. Qing, Wang Baosheng, "Detection Network Anomalies Based on Packet and Flow Analysis," Seventh International Conference on

Networking, 2008. ICN 2008., vol., no., pp.497-502, 2008

25. Y. Waizumi, H. Tsunoda, M. Tsuji, Y. Nemoto, "A Multi-Stage Network Anomaly Detection Method for Improving Efficiency and Accuracy," Journal of Information Security, vol.3 no. 1, pp.18-24, 2012

26. M. Amini, R. Jalili, H.R. Shahriari, “RT-UNNID: A practical solution to real-time network-based intrusion detection using unsupervised neural networks,” Computers and Security, Elsevier Inc, vol.25, Issue 6, pp.459-468, 2006

References

27. P. Casas, J. Mazel, P. Owezarski, “Unsupervised Network Intrusion Detection Systems: Detecting the Unknown without Knowledge,” Computer

Communications, vol.35, Issue 7, pp.772-783, 2012

28. G. Cormode, S. Muthukrishnan, “What's new: finding significant differences in network data streams,” IEEE/ACM Transactions on Networking (TON), vol.13, Issue 6, pp.1219-1232, 2005

29. Cisco.com, “Cisco IOS NetFlow Configuration Guide, Release 12.4,” http://www.cisco.com, Sep. 2012

30. M. Ester, H.P. Kriegel, J. Sander, X. Xu , "A density-based algorithm for discovering clusters in large spatial databases with noise," Proceedings of the Second

International Conference on Knowledge Discovery and Data Mining (KDD-96), AAAI Press. pp.226–23, 1996

31. P.C. Mahalanobis, "On the generalised distance in statistics," Proceedings of the National Institute of Sciences of India 2 (1) : pp.49–55, 1936

References

41

32. M. Tavallaee, E. Bagheri, Lu Wei, A.A. Ghorbani, "A detailed analysis of the KDD CUP 99 data set," Computational Intelligence for Security and Defense

Applications, CISDA 2009. IEEE Symposium on , vol., no., pp.1,6, 8-10 July 2009 33. A. Shiravi, H. Shiravi, M. Tavallaee, A. A. Ghorbani, Toward developing a systematic

approach to generate benchmark datasets for intrusion detection, Computers & Security, vol.31, Issue 3, May 2012, pp.357-374, ISSN 0167-4048, 2012