Intrusion Detection System

Part 3
1) Executive Summary As the next step in the process of creating an Intrusion detection system, this part of the project involves not only detection of an attack but also it need to be recognized as a particular type and need to be classified as known attack. Further the project involves detection of an unknown attack, if it happens to be present, as a new attack. This unknown attack will be different than those 5 attacks that have been pre-selected. In this section of the project it was required to design Artificial Neural Network structures which will consist of NNs detecting a particular type of attack. Based on the output of these NNs it need to be decided whether the given input is an attack of known type or unknown type. If it is of known type then which one of those 5 attacks. Same 5 different training sets created for each of the 5 attacks will then be used to train the networks starting from smaller size files to larger size files. After training a NN structure, involving 5 different NNs detecting a particular attack, using a particular size file, it is then tested based on the test file created in part 1 of the project. The testing will determine how well a NN structure is able to detect and classify an attack and the results is calculated in terms of False Positive and False Negative ratios based on whether an attack is wrongly detected or wrongly not detected and wrongly classified and correctly classified respectively. These results are then plotted and analyzed to determine how the size of the file affects the two types of outputs and the efficiency of the ANN structure as a whole.

2) Requirements In this part of the project it was required to create an anomaly detection system which will not only detect an attack but also identify as particular attack and also classify it as known or unknown attack.

In the part it was required to design an ANN structure that will consist of 5 different NN trained to detect particular type of attack as in part 2 of this project and a NN trained to detect normal situation. Each of the NNs in the NN structure is trained to detect particular attack using particular size file. The output of these NNs will then be used to decide whether its an attack or not. If its an attack then whether it is among known attack or its an unknown attack. This part is divided into 4 sections. First section requires creation of a decision making algorithm that will decide whether there is of particular type of attack or not. This algorithm takes input from all the 6 NNs mentioned above and based on these outputs decide whether it is an attack or not, if yes then what type. Second section requires addition of few unknown attack type records in the testing sets. For each of these records, each of the NNs will give an output and these outputs is then given to the decision making algorithm which will then decide whether the given record is an attack of known type or unknown type. Third section involves replacement of decision making algorithm with a special type of NN called Competitive NN. Competitive NN is an unsupervised type of NN which makes the decision based on Kohonen learning rule. This NN will take input from all the NN trained for particular attack and decide the type attack. This section doesnt include the records for unknown attack. Fourth section involves addition of few unknown attack type records in the testing sets. For each of these records, each of the NNs will give an output and these outputs is then given to the Competitive NN which will then decide whether the given record is an attack of known type or unknown type. It was required to train the networks in the NN structure starting from smaller size training files to larger size files (i.e from 10 then 20 then 40 then 60 and then 100). After training the NNs using each of these file it is then tested with the testing file created in part 1. Based on the outputs of testing, four measures are calculated ie false positive ratio, false negative ratio, right classification and wron classification. Once all the training files are been used for training the NNs for a particular attack and corresponding testing results are calculated these ratios are then

plotted against the size of the training files and the resulting graph analyzed to determine how the size of the training set affect the efficiency of the decision making algorithm or the Competitive NN in detecting and classifying a particular attack. These steps are then repeated for all the 5 types of attack chosen earlier and corresponding graphs are obtained. Based on the results conclusions are drawn. 3) Specifications This part of the project can be divided into 3 steps Design Each of the NN used in NN structure have the same design architecture as mentioned in specification of part 2. Training and Testing Training process involve creating ANN structure which will have 6 NNs (5 for particular attack and 1 for normal condition detection). These networks are trained with the created training files of different sizes. Training is begun by using the smallest size training file i.e 10 and the networks are trained based on it. Once trained, the network structure is then tested based on the test file using decision making algorithm or Competitive NN and results are calculated in the form of false positive ratio, false negative ratio, misclassification and correct classification. Then the training is repeated using other training files in increasing order of their size and corresponding testing results are obtained. Then the training is repeated for other 4 types of attack in the same fashion and corresponding results are obtained. Analysis The results obtained after testing during each of the 4 sections, are then plotted against the size of the training files. These plots are then analyzed to determine how the efficiency of the decision making algorithm and competitive NN get affected in detecting and classifying a particular type

of attack as the size of the training set is gradually increased. Based on these analyses, conclusions are drawn. 4) Implementation Structure As mentioned before this project is conducted using MATLAB toolbox. In this experiment, for each training process an ANN structure that will consist of 6 NNs, 5 for each 5 attacks and 1 for detecting normal condition. These NNs are created using MATLAB commands and each NN structure is in a separate training .m files. These files consist of input and output variables which include the input and corresponding output records for a particular training file of particular size for a particular type of attack. These variables are the used in the training process. A variable TestInput is also created which consist of records for the testing part. All the user need to do is to run these .m files in MATLAB. The results are shown in the form of graph plots in separate plot files for each of the experiment conducted. Software Requirement The given project is performed using MATLAB software tool hence user is required to have MATLAB software to run the given project. No other software is required.

5) Results and Analysis The results obtained in this part of the experiment are shown in the following table
False Positive ratio 8/900 =0.008 24/900=0.026 80/900 = 0.08 21/900 =0.023 7/900=0.007 0 0 0 False Negative ratio 0 0 0 0 0 0 0 0 Misclassification .19 .1 .18 0 0 .18 0 0 Right Classification .81 .99 .82 1 1 .82 1 1

NN 10 NN 20 NN 40 NN 60 NN 100 CNN 10 CNN 20 CNN 40

CNN 60 CNN 100 Unknown 10 Unknown 20 Unknown 40 Unknown 60 Unknown 100 CNN Unknown 10 CNN Unknown 20 CNN Unknown 40 CNN Unknown 60 CNN Unknown 100

0 0 39/900 = 0.043 32/900 = 0.035 82/900 = 0.09 53/900 = 0.058 52/900 = 0.057 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0

0 0 .2 0 .18 0 0 .2 0 0 0 0

1 1 .8 1 .82 1 1 .8 1 1 1 1

Following screenshot shows the graph of False Positive, False Negative misclassification and correct classification against file size for part A which involves decision making algorithm without unknown attack records added in test file.

Analysis The given graph shows that false positive ratio in blue line gradually reduces to zero as size of training file increases. False negative in green line remains zero throughout which shows all the attacks got detected. Red line shows misclassification of attack which remains below .2 and eventually drops to zero for higher sizes. Similarly correct classification shown in light blue remains high and finally reaches 100 percent for higher sizes. Overall graph shows good results and thus decision making algorithm seems quite efficient.

Following screenshot shows the graph of False Positive, False Negative misclassification and correct classification against file size for part B which involves decision making algorithm with unknown attack records added in test file.

Analysis The graph show that false positive ratio in blue line remain almost constant and very low. False negative in red line remain zero throughout. Misclassification in light blue, initially show some ups and down finally drops to zero for higher sizes. Correct classification in converse to misclassification remain high, higher that .8 and finally become 100 percent. The errors present in this experiment show some fluctuation but they are always very low hence the decision making algorithm and thus the experiment seems to be good and satisfactory.

Following screenshot shows the graph of False Positive, False Negative misclassification and correct classification against file size for part A which involves Competitive network without unknown attack records added in test file.

Analysis The given graph pretty good results. False positive and False Negative ratios are 0 throughout. Misclassification drop to zero from size 20 similarly Right Classification also reaches 100 percent from size 20. Overall the result is good and the CNN is able to classify attack efficiently.

Following screenshot shows the graph of False Positive, False Negative misclassification and correct classification against file size for part A which involves Competitive network with unknown attack records added in test file.

Analysis The graph also shows good results with false positive and false negative zero throughout while misclassification drops to zero after size 20 and right classification reaches 100 percent after size 20. Overall good results thus the CNN was able to detect and classify correctly to a very good result.

6) Comparison between Decision making algorithm and CNN

Seeing the results, it can be seen that both the methods of classification results into satisfactory outcome. Both are able to detect and classify attacks almost completely without an error. Certain variations exist with files of smaller sizes. When these two methods are compared to each other CNN method shows better results than decision making algorithm. But this cannot be said for larger test files.