SpikeLog: Log-based anomaly detection via Potential-assisted Spiking Neuron Network
The increasing volume and complexity of log data generated by modern systems have made it challenging to analyze and extract useful insights manually. To address this problem, many machine learning methods have been proposed for log-based anomaly detection. However, most of these methods lack interpretability, and their underlying premises do not always reflect real scenarios. In this paper, we consider a more reasonable premise scenario where a large number of logs are unlabeled, while only a small number of anomalous logs are labeled. Moreover, a small proportion of anomaly contamination may be present. To handle this practical scenario, we propose a novel hybrid potential-assisted framework (SpikeLog) using the membrane potential of spiking neurons. SpikeLog adopts a weakly supervised approach to train an anomaly score model, which effectively utilizes a limited number of labeled anomalies alongside abundant unlabeled logs while ensuring computational efficiency without compromising accuracy. Extensive experiments have demonstrated that SpikeLog outperforms baseline methods in terms of performance, robustness, interpretability, and energy consumption.
We refer to the hub: https://github.com/LogIntelligence/LogADEmpirical/tree/dev#1-data-preparation.
Raw and preprocessed datasets (including parsed logs and their embeddings) are available at https://zenodo.org/record/8115559.
Download the data to ./dataset/ .
Run following script to generate embedding.json (Default BGL)
python generate_template_embedding.py python -u main_run.py --folder=bgl/ --log_file=BGL_2k.log --dataset_name=bgl --model_name=prolog --window_type=sliding \
--semantics --input_size=1 --embedding_dim=300 \
--data_dir=./dataset/ \
--sample=sliding_window --is_logkey --train_size=0.7 --train_ratio=1 --valid_ratio=0.1 --test_ratio=1 --max_epoch=10 \
--n_warm_up_epoch=0 --n_epochs_stop=10 --batch_size=64 --history_size=100 --lr=5e-4 \
--session_level=entry --window_size=100 --step_size=100 --output_dir=experimental_results/bgl/ \
--is_process