Sample Usage (v5)
RELOAD can be downloaded as a ZIP archive from [17]. The ZIP archive includes three items: the JAR file of RELOAD, a preferences file and a folder that contains configuration files. Once files are extracted from the archive, RELOAD can be launched from command line as java -jar RELOAD.jar. Note that Windows OS users may run the tool by double-clicking the JAR file: this does not show the log of the application (which, however, is saved as a txt file for post-mortem analyses). Overall, RELOAD needs just a reload.preferences file to start, along with the input folder that can be found in the ZIP file. The folder contains main configuration files, which can be managed and adjusted through RELOAD GUI and that are required by algorithms to execute.
RELOAD requires i) Java (Oracle) 8+ or OpenJDK 8, ii) around 30MB of storage space plus the space needed to store the targeted dataset(s). In this case, since we want to download our dataset and the tool from their online repositories [36], [17], we also need internet connection. We remark here that, once downloaded, the tool is fully standalone and does not require external connections.
We applied previous and less-stable versions of RELOAD to different case studies, especially regarding data logs of service-oriented systems [33]. The tool turned out to be helpful in identifying the more fitting algorithms, either for error or intrusion detection. To show the steps that a generic user has to follow for using the tool, in this section we refer to an entirely new case study targeting the most relevant dataset obtained by querying “intrusion detection” in the Kaggle datasets portal. Note that results of the application of RELOAD on some of the other datasets showed by Kaggle i.e., KDDCup and UNSW, may be found at [35].
After downloading the ZIP file of the dataset, we observe that the dataset is partitioned in two (CSV) files, one for training and the other one for testing. The structure of the two files is the same, expect for an unlabelled column in the training set – the first one – that we remove due to the lack of information. The resulting data is structured in 42 columns, with respectively 125.973 and 10.000 data points for train and test set. The last column shows the label for each data point, that can be either normal (43.3% of the test set), or representing an attack dos (33.3%), probe (10.5%), r2l (12.0%), or u2r (0.9%). Information on the attacks are not reported in the portal, we assume that they cover the same categories as their KDDCup [12] and NSL-KDD [9] datasets. All but columns 2 and 42 (1 and 41 if zero-based numbering) are numeric, meaning that RELOAD can process them without needing further categorizations. Note that in some cases datasets report on numeric features which report on categories e.g., network ports. These features can be processed by RELOAD, but it may be better to avoid using them since some operations that algorithms usually do e.g., statistics, are not meaningful in these cases.
Once started, RELOAD shows the GUI below.
Starting from the “Setup” box, we first want to define the reference metric e.g., F-Measure in the Figure), strategies to select features and aggregate them to create aggregated features. For this case study we selected VARIANCE(3) and INFORMATION_GAIN(0.05) as feature selection strategies.
The other option of the “Setup” box allow choosing how to aggregate features i.e. n-dimensional feature spaces, which for this case study is executed whenever the Pearson correlation index between two or more selected indicators is more than 0.8 – PEARSON(0.8) -, and to choose which of the phases in the Figure the user wants to execute. To the sake of this case study, we will run all the phases of RELOAD, thus checking all the Feature Selection, Training, Optimization and Evaluation checkboxes. Note that these are not always mandatory and can be executed separately by ticking/unticking checkboxes.
We also proceed with a 10-fold sampling of the training set as widely suggested [34] in the literature.
The “Path” box should not need further adaptations. Only note that the default folder for datasets is specified as a “datasets” subfolder of the current directory. If the datasets the user wants to analyse is located in another folder, the user should either i) change the default path of RELOAD through GUI, or ii) move the files.
Here the user can i) define the data loader(s), and ii) select algorithms.
When a new data loader has to be defined, as it is the case for this kaggle dataset, pressing the “Create Loader” button in the “Data Analysis” box allows to choose a file name, type of the source (CSV or ARFF - CSV in this case) and opens the GUI below. We filled the fields of the id_kaggle.loader loader in the figure as follows.
The CSV files were put in datasets/intrusion_detection_kaggle folder: we specified train and test file in the TRAIN_FILE and VALIDATION_FILE items. Then, we chose to analyse the performances of RELOAD in identifying probe attacks in this dataset. Therefore, we FAULTY_TAGS fields to “probe” and the SKIP_ROWS fields to “dos”, “u2r”, “r2l”, or rather the remaining attacks we are not interested in. Note that labels are used in the training phase to rank anomaly checkers rather than being provided to algorithms, which work in an unsupervised fashion. We select 50 batches both for training and for validation i.e., RUN_IDS fields, considering batches of 200 data points, as specified by EXPERIMENT_ROWS. Lastly, we specified the LABEL_COLUMN containing labels (xAttack), and the columns to be skipped ("protocol_type", see SKIP_COLUMNS field in the figure).
To show the versatility of the tool, for this example we selected different algorithms as HBOS, KMeans and LOF, a sliding window algorithm (SPS), and a combination of two notoriously fast algorithms such as HBOS and KMeans.
When everything is set, the user should press the RELOAD! button on the bottom of the GUI. This will start the process of selecting algorithms and executing anomaly detection. When the process completes, RELOAD will open a window that summarizes results as shown below.
The first tab to be seen is the “Summary” tab: RELOAD shows a row for each couple <algorithm, data set> analysed. Each row reports on i) the data set, ii) the algorithm, iii) the most performing pair of selection policy and voting strategy, according to the target metric, iv) the percentage of labelled anomalies over all data points in the ED, and v) the score of the target metric. More in detail, this view allows to see which algorithm reached higher metric scores, or rather the optimal algorithm for the dataset or the system under investigation. In addition, for each sliding window algorithm the user selected (SPS in the example), RELOAD will show metric scores for each combination of selection policy and window size the user set through GUI.
As a side note, by looking at SPS results it is easy to observe that the usage of wider sliding windows does not help SPS in identifying attacks i.e., the wider the window, the lower the F-Measure.
In addition, the GUI shows detail of all combinations of data sets and algorithms for the different selection policies and voting strategies. The picture shows metric scores obtained in the training set and on the evaluation set. Metric scores obtained on the two sets should not differ a lot. If huge differences are reported, the user should try to raise the K parameter of the K-Fold validation, to limit overfitting.
The detailed view shows other 3 viewpoints of RELOAD activity.
Clicking the "Selected Features" button opens the picture above, which contains the list of available features, along with the scores feature selectors assigned to them, and an indication (true/false) about the final result of the selection process. Features labelled with true were selected by the tool and used to build dataseries for algorithms.
Instead, by clicking "Training Detail" it is possible to observe all the couples <algorithm, selected feature> which RELOAD explored, trying to understand which feature had better sinergy with the algorithm. The figure below reports on the combined execution of two algorithms, HBOS and KMEANS. As it is possible to observe, different combinations of feature and algorithm appear in the figure, showing how - in this case - the feature "service" allowed both algorithms to reach their maximum scores.
Lastly, the "Plot Results" button shows a new window which depicts a bar chart: on the X axis the user will find the algorithm score (a real number), while the Y axis reports on the frequency. Bars will be as high as the number of data points (rows if files) of the evaluation set that were scored by the algorithm with the same number. Red bars represent data points which are labeled as anomalies in the evaluation set, while blue bars represent the amount of normal data points which were assigned by the algorithm to a given score.
