FEATURE PRUNING METHOD FOR HIDDEN MARKOV MODEL-BASED ANOMALY DETECTION: A COMPARISON OF PERFORMANCE

(Received: 2018-10-10, Revised: , Accepted: 2018-08-10)

Authors Sulaiman Alhaidari, Mohamed Zohdy,

Keywords #Anomaly detection; Feature pruning; Hidden Markov Model; NSL-KDD; DDoS; UNSW_NB15; IoTPOT.

Abstract Selecting effective and significant features for Hidden Markov Model (HMM) is very important for detecting anomalies in databases. The goal of this research is to identify the most salient and important features in building HMM. In order to improve the performance of HMM, an approach of feature pruning is proposed. This approach is effective in detecting and classifying anomalies, very simple and easily implemented. Also, it is able to reduce computational complexity and time without compromising the model accuracy. In this work, the proposed approach is applied to NSL-KDD (the new version of KDD Cup 99), DDoS, IoTPOT and UNSW_NB15 data sets. Those data sets are used to perform a comparative study that involves full feature set and a subset of significant features. The experimental results show better performance in terms of efficiency and providing higher accuracy and lower false positive rate with reduced number of features, as well as eliminating irrelevant redundant or noisy features.

References

[1] Z. Ghahramani, "An Introduction to Hidden Markov Models and Bayesian Networks," International Journal of Pattern Recognition and Artificial Intelligence, vol. 15, no. 1, pp. 9-42, 2001.

[2] Sulaiman Alhaidari, Ali Alharbi and Mohamed Zohdy, "Detecting Distributed Denial of Service Attacks Using Hidden Markov Models," International Journal of Computer Science Issues (IJCSI), vol. 15, no. 5, 2018.

[3] NSL-KDD Dataset,[Online], Available: http://nsl.cs.unb.ca/nsl-kdd/.

[4] Pa, Yin Minn Pa et al., "Iotpot: A Novel Honeypot for Revealing Current IoT Threats," Journal of Information Processing, vol. 24, no. 3, pp. 522-533, 2016.

[5] N. Moustafa and J. Slay, "UNSW-NB15: A Comprehensive Dataset for Network Intrusion Detection Systems," Proc. of the IEEE Military Communications and Information Systems Conference (MilCIS), Australia, 2015.

[6] A. Alharbi, S. Alhaidari and M. Zohdy, "Denial-of-Service, Probing, User to Root (U2R) and Remote to User (R2L) Attack Detection Using Hidden Markov Models," International Journal of Computer and Information Technology, 2018 (Submitted).

[7] X. Zeng, Y.-W. Chen, C. Tao and D. Alphen, "Feature Selection Using Recursive Feature Elimination for Handwritten Digit Recognition," Proc. of the 5th International Conference on Intelligent Information Hiding and Multimedia Signal Processing, Kyoto, pp. 1205-1208, 2009.

[8] A. Alshammari et al., "Security Threats and Challenges in Cloud Computing," Proc. of the IEEE 4th International Conference on Cyber Security and Cloud Computing (CSCloud), NY, USA, pp. 46-51, 2017.

[9] A. Alharbi et al., "Sybil Attacks and Defenses in Internet of Things and Mobile Social Networks," CyberHunt 2018: IEEE International Workshop on Big Data Analytics for Cyber Threat Hunting, Westin Seattle, WA, USA, 2018 (Submitted).