logdata-anomaly-miner：一款安全日志解析与异常检测工具

关于logdata-anomaly-miner

logdata-anomaly-miner是一款安全日志解析与异常检测工具，该工具旨在以有限的资源和尽可能低的权限运行分析，以使其适合生产服务器使用。

为了确保 logdata-anomaly-miner的正常运行，推荐安装了python >= 3.6的Linux 系统。更具体地说，支持的系统包括 Debian Buster、Debian Bullseye、Debian Bookworm、Ubuntu 20.04、Ubuntu 22.04、Fedora（docker image fedora:latest）和 RedHat（docker image redhat/ubi9）。

工具架构

工具要求

scipy==1.10.0

pylibacl==0.5.4

kafka_python==2.0.2

pytz==2020.4

urllib3==1.26.19

numpy==1.22.0

Cerberus==1.3.2

psutil==5.7.3

kafka==1.3.5

pyzmq==20.0.0

python_dateutil==2.8.1

PyYAML==5.4

statsmodels==0.12.2

工具安装

由于该工具基于Python 3开发，因此我们首先需要在本地设备上安装并配置好最新版本的Python 3环境。

源码安装

接下来，广大研究人员可以直接使用下列命令将该项目源码克隆至本地：

git clone https://github.com/ait-aecid/logdata-anomaly-miner.git

然后切换到项目目录中，使用pip命令和项目提供的requirements.txt安装该工具所需的其他依赖组件：

cd logdata-anomaly-miner

pip install -r requirements.txt

Debian安装

在官方 Debian/Ubuntu 存储库中，有适用于 logdata-anomaly-miner 的 Debian 软件包：

apt-get update && apt-get install logdata-anomaly-miner

Wget安装

以下命令将安装最新的稳定版本：

cd $HOME

wget https://raw.githubusercontent.com/ait-aecid/logdata-anomaly-miner/main/scripts/aminer_install.sh

chmod +x aminer_install.sh

./aminer_install.sh

工具配置

现在让我们将 Apache 解析器模型添加到 aminer-config：

alice@ubuntu2004:~$ sudo ln -s /etc/aminer/conf-available/generic/ApacheAccessModel.py /etc/aminer/conf-enabled/

alice@ubuntu2004:~$

在以前版本的 aminer 中，我们必须用 python 编写配置文件。在当前版本中，我们可以使用以 yaml 编写的配置。现在创建并编辑文件 /etc/aminer/config.yml：

LearnMode: True

 

LogResourceList:

        - 'file:///var/log/apache2/access.log'

 

Parser:

        - id: 'START'

          start: True

          type: ApacheAccessModel

          name: 'apache'

 

Input:

        timestamp_paths: "/accesslog/time"

 

Analysis:

        - type: "NewMatchPathValueDetector"

          paths: ["/accesslog/status"]

          output_logline: True

 

EventHandlers:

        - id: "stpe"

          type: "StreamPrinterEventHandler"

如果我们现在启动 aminer，它将读取 access.log 并了解所有解析器路径。我们将在启动 aminer 之前使用“-C”参数清除持久性。（请注意，您可以使用 CTRL+c 终止 aminer）

alice@ubuntu2004:~$ sudo cat /var/log/apache2/access.log

127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"

alice@ubuntu2004:~$ sudo aminer -C --config /etc/aminer/config.yml

2021-05-17 12:12:36 New path(es) detected

NewMatchPathDetector: "DefaultNewMatchPathDetector" (1 lines)

/accesslog: 127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"

/accesslog/host: 127.0.0.1

/accesslog/sp0:

/accesslog/ident: -

/accesslog/sp1:

/accesslog/user: -

/accesslog/sp2:

/accesslog/time: 1621250714

/accesslog/sp3: ] "

/accesslog/fm/request: GET / HTTP/1.1

/accesslog/fm/request/method: 0

/accesslog/fm/request/sp5:

/accesslog/fm/request/request: /

/accesslog/fm/request/sp6:

/accesslog/fm/request/version: HTTP/1.1

/accesslog/sp6: "

/accesslog/status: 200

/accesslog/sp7:

/accesslog/size: 11229

/accesslog/combined:  "-" "Wget/1.20.3 (linux-gnu)"

/accesslog/combined/combined:  "-" "Wget/1.20.3 (linux-gnu)"

/accesslog/combined/combined/sp9:  "

/accesslog/combined/combined/referer: -

/accesslog/combined/combined/sp10: " "

/accesslog/combined/combined/user_agent: Wget/1.20.3 (linux-gnu)

/accesslog/combined/combined/sp11: "

['/accesslog', '/accesslog/host', '/accesslog/sp0', '/accesslog/ident', '/accesslog/sp1', '/accesslog/user', '/accesslog/sp2', '/accesslog/time', '/accesslog/sp3', '/accesslog/fm/request', '/accesslog/sp6', '/accesslog/status', '/accesslog/sp7', '/accesslog/size', '/accesslog/combined', '/accesslog/combined/combined', '/accesslog/combined/combined/sp9', '/accesslog/combined/combined/referer', '/accesslog/combined/combined/sp10', '/accesslog/combined/combined/user_agent', '/accesslog/combined/combined/sp11', '/accesslog/fm/request/method', '/accesslog/fm/request/sp5', '/accesslog/fm/request/request', '/accesslog/fm/request/sp6', '/accesslog/fm/request/version']

127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"

2021-05-17 12:12:36 New value(s) detected

NewMatchPathValueDetector: "NewMatchPathValueDetector2" (1 lines)

{'/accesslog/status': 200}

127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"

工具运行演示

演示视频：【点我观看】

许可证协议

本项目的开发与发布遵循GPL-3.0开源许可协议。

项目地址

logdata-anomaly-miner：【GitHub传送门】

参考资料

AMiner: A Modular Log Data Analysis Pipeline for Anomaly-based Intrusion Detection | Digital Threats: Research and Practice

SciTePress - Publication Details

AECID-PG: A Tree-Based Log Parser Generator To Enable Log Analysis | IEEE Conference Publication | IEEE Xplore

https://securitylab.no/cyberhunt2019/