2023年7月17日发(作者：)

Suricata+ELK集群监控办公⽹流量背景需要利⽤Suricata作为IDS来监控办公⽹出⼝流量，同时利⽤ELK（Elasticsearch+Logstash+Kibana）集群进⾏数据存储与展⽰。准备⼯作：在办公⽹出⼝核⼼交换机上做端⼝流量镜像，将流量镜像端⼝连接到⼀台服务器上，以下的内容都是在这台服务器上开展的。⼀、环境准备基础环境更新：yum updateyum upgrade安装基础依赖：yum -y install gcc libpcap-devel pcre-devel libyaml-devel file-devel zlib-devel jansson-devel nss-devel libcap-ng-devel libnet-devel tar make libnetfilter_queue-devel lua-devel其他基础组件，后⾯遇到缺啥再装啥就可以了。需要安装的软件：suricata、Luajit、Hyperscan、elasticsearch（集群）、elasticsearch-head、logstash、filebeat、kibana其中elasticsearch、logstash、filebeat、kibana需要安装同⼀版本。安装的⽬标机器：192.168.1.101（主）、192.168.1.102、192.168.1.103注意点：出于性能考量，监控系统各组件安装⽅式都不建议使⽤Docker安装⼆、suricata部署部署⽬标机器：192.168.1.101安装依赖：yum install wget libpcap-devel libnet-devel pcre-devel gcc-c++ automake autoconf libtool make libyaml-devel zlib-devel file-devel jansson-devel nss-devel epel-release lz4-devel rustc cargo编译并安装suricata：wget /download/ -xvf suricata-6.0.2# 注意，这⾥的默认参数尽量不要改，否则后⾯各种问题排查起来也是要命的./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --enable-geoip --enable-luajit --with-libluajit-includes=/usr/local/include/luajit-2.0/ --with-libluajit-libraries=/usr/local/lib/ --with-libhs-includes=/usr/local/include/hs/ --with-libhs-libraries=/usr/local/lib/ --enable-profiling编译后的参数：Suricata Configuration: AF_PACKET support: yes eBPF support: no XDP support: no PF_RING support: no NFQueue support: no NFLOG support: no IPFW support: no Netmap support: no DAG enabled: no Napatech enabled: no WinDivert enabled: no Unix socket enabled: yes Unix socket enabled: yes Detection enabled: yes Libmagic support: yes libnss support: yes libnspr support: yes libjansson support: yes hiredis support: no hiredis async with libevent: no Prelude support: no PCRE jit: yes LUA support: yes, through luajit libluajit: yes GeoIP2 support: yes Non-bundled htp: no Hyperscan support: yes Libnet support: yes liblz4 support: yes Rust support: yes Rust strict mode: no Rust compiler path: /usr/bin/rustc Rust compiler version: rustc 1.50.0 (Red Hat 7) Cargo path: /usr/bin/cargo Cargo version: cargo 1.50.0 Cargo vendor: yes Python support: yes Python path: /usr/bin/python3 Python distutils yes Python yaml yes Install suricatactl: yes Install suricatasc: yes Install suricata-update: yes Profiling enabled: yes Profiling locks enabled: no Plugin support (experimental): yesDevelopment settings: Coccinelle / spatch: no Unit tests enabled: no Debug output enabled: no Debug validation enabled: noGeneric build parameters: Installation prefix: /usr Configuration directory: /etc/suricata/ Log directory: /var/log/suricata/ --prefix /usr --sysconfdir /etc --localstatedir /var --datarootdir /usr/share Host: x86_64-pc-linux-gnu Compiler: gcc (exec name) / g++ (real) GCC Protect enabled: no GCC march native enabled: yes GCC Profile enabled: no Position Independent Executable enabled: no CFLAGS -g -O2 -std=gnu99 -march=native -I${srcdir}/../rust/gen -I${srcdir}/../rust/dist PCAP_CFLAGS SECCFLAGS执⾏安装：make && make install安装其他组件：# 这⾥执⾏这⼀个就够了，它相当于安装：configuration、rules、providemake install-full修改配置⽂件：vim /etc/suricata/# 修改的第⼀部分# 修改相关参数，并把不⽤的注释掉vars: address-groups: HOME_NET: "[10.10.11.0/24,172.16.10.0/24]" DNS_NET: "[10.10.10.100,10.10.10.101,10.10.10.102]"# HOME_NET: "[10.0.0.0/8]"# HOME_NET: "[172.16.0.0/12]"# HOME_NET: "any" EXTERNAL_NET: "!$HOME_NET"# EXTERNAL_NET: "any" HTTP_SERVERS: "$HOME_NET"# SMTP_SERVERS: "$HOME_NET"# SQL_SERVERS: "$HOME_NET" DNS_SERVERS: "$DNS_NET"# TELNET_SERVERS: "$HOME_NET"# AIM_SERVERS: "$EXTERNAL_NET"# DC_SERVERS: "$HOME_NET"# DNP3_SERVER: "$HOME_NET"# DNP3_CLIENT: "$HOME_NET"# MODBUS_CLIENT: "$HOME_NET"# MODBUS_SERVER: "$HOME_NET"# ENIP_CLIENT: "$HOME_NET"# ENIP_SERVER: "$HOME_NET" port-groups: HTTP_PORTS: "80,443"# SHELLCODE_PORTS: "!80"# ORACLE_PORTS: 1521 SSH_PORTS: 22# DNP3_PORTS: 20000# MODBUS_PORTS: 502# FILE_DATA_PORTS: "[$HTTP_PORTS,110,143]" FTP_PORTS: 21# GENEVE_PORTS: 6081# VXLAN_PORTS: 4789# TEREDO_PORTS: 3544# 修改的第⼆部分# 修改⽇志存储⽂件位置。因为home分区⽐较⼤，⼀般是⼏百GB，⽽root分区⼀般⼏⼗GBdefault-log-dir: /home/suricata/log/suricata/# 修改的第三部分# 主要是修改部分参数配置 - http: extended: yes # enable this for extended logging information custom: [ accept, accept_charset, accept_datetime, accept_encoding, accept_language, accept_range, age, allow, authorization, cache_control, connection, content_encoding, content_language, content_length, content_location, content_md5, content_range, content_type, cookie, date, dnt, etag, expires, from, last_modified, link, location, max_forwards, org_src_ip, origin, pragma, proxy_authenticate, proxy_authorization, range, referrer, refresh, retry_after, server, set_cookie, te, trailer, transfer_encoding, true_client_ip, upgrade, vary, via, warning, www_authenticate, x_authenticated_user, x_bluecoat_via, x_flash_version, x_forwarded_proto, x_requested_with ] dump-all-headers: [both] - dns: enabled: yes version: 1 requests: yes responses: yes - tls: extended: yes session-resumption: yes custom: [ subject, issuer, session_resumed, serial, fingerprint, sni, version, not_before, not_after, certificate, chain, ja3 ]# 修改第四部分# 全局搜索，将⽹卡名称改为⾃⼰的⽹卡名称。我这⾥⽹卡名称是p2p1，因此将所有eth0/eth1/eth2……都改为p2p1- interface: p2p1# 修改第五部分# 全局搜索，将所有完整校验关掉，不然的话会有⾮常多的误报，并且占空间checksum-checks: no# 修改第六部分# 修改我们要使⽤的规则，把不使⽤的注释掉default-rule-path: /var/lib/suricata/rules# 需要把这个⽬录下的规则复制过去：/usr/share/suricata/rulesrule-files:# - # - # - # - # - - - - # - # - # - # - # - # - # - # - - 更新规则集：pip install --upgrade suricata-updatesuricata-update启动测试，若不报错即成功：/usr/bin/suricata -T正常启动：/usr/local/bin/suricata -c /etc/suricata/ -i p2p1 --init-errors-fatal也可以将其使⽤supervisord守护进程启动：vim /etc/supervisord.d/[program:suricata]directory=/usr/bincommand=suricata -c /etc/suricata/ -i p2p1 --init-errors-fatalautostart=trueautorestart=false#stderr_logfile=/tmp/test_#stdout_logfile=/tmp/test_r=root三、Luajit部署部署⽬标机器：192.168.1.101简介：LuaJIT是采⽤C语⾔写的Lua代码的解释器，LuaJIT试图保留Lua的精髓–轻量级,⾼效和可扩展。安装：wget /download/ -zxf LuaJIT-2.0.5/sudo make && make install编辑配置⽂件：vim /etc/# 添加如下路径，保存退出/usr/local/lib执⾏加载命令：sudo ldconfig四、Hyperscan部署部署⽬标机器：192.168.1.101简介：Hyperscan是⼀个⾼性能的多重正则表达式匹配库。在Suricata中它可以⽤来执⾏多模式匹配。Hyperscan适⽤于部署在诸如DPI/IPS/IDS/FW等场景中,⽬前已经在全球多个客户⽹络安全⽅案中得到实际的应⽤。使⽤ Hyperscan 作为 Suricata 的 MPM（多处理模块)）匹配器（mpm-algo 设置）可以⼤⼤提⾼性能，尤其是在快速模式匹配⽅⾯。Hyperscan 还在快速模式匹配时考虑深度和偏移量。安装依赖：yum install cmake ragel libtool python-devel GyeoIP-develyum install boost boost-devel boost-docyum install libquadmath libquadmath-devel bzip2-devel安装：wget /project/boost/boost/1.66.0/boost_1_66_ xvzf boost_1_66_ boost_1_66_0/./ --prefix=/home/suricata/boost-1.66./b2 install// 不要退出⽬录git clone /intel/ hyperscancmake -DBUILD_STATIC_AND_SHARED=1 -DBOOST_ROOT=/home/suricata/boost-1.66makemake install编辑配置⽂件：vim /etc/# 添加如下路径，保存退出/usr/local/lib64执⾏加载命令：sudo ldconfig⽂件结构如下：五、elasticsearch集群部署部署⽬标机器：192.168.1.101、192.168.1.102、192.168.1.103简介：ElasticSearch简称ES，它是⼀个实时的分布式搜索和分析引擎，它可以⽤于全⽂搜索，结构化搜索以及分析。它是⼀个建⽴在全⽂搜索引擎 Apache Lucene 基础上的搜索引擎，使⽤ Java 语⾔编写。基础配置，三台机器都⼀样：vim /etc/security/# 这些数尽量不要省，不然启动失败还得改回来，⿇烦* soft nofile 65536* hard nofile 131072* soft nproc 2048* hard nproc 4096vim /etc/_map_count=655360安装Java，三台机器都⼀样：tar zxf jdk1.8.0_271/ /usr/local/javavim /etc/profileexport JAVA_HOME=/usr/local/javaexport JRE_HOME=/usr/local/java/jreexport PATH=$PATH:/usr/local/java/binexport CLASSPATH=./:/usr/local/java/lib:/usr/local/java/jre/lib# 让环境变量⽣效source !$java -version# 这⾥最好直接放在/bin下⾯，否则后⾯logstash报错⾮常难排查原因which javaln -s /usr/local/java/bin/* /bin安装elasticsearch，三台机器都⼀样：tar zxf elasticsearch-7.5.1 /usr/local/elasticsearchmkdir /usr/local/elasticsearch/datachown -R admin:admin /usr/local/elasticsearch192.168.1.101机器修改配置⽂件：vim /usr/local/elasticsearch/config/: ELK # 集群名，同⼀个集群，集群名必须⼀致: es-1 # 集群节点，可任意取ss: : /usr/local/elasticsearch/data # 数据存放路径: /usr/local/elasticsearch/logs # ⽇志存放路径: 192.168.1.101 # 监听IP地址，: : _hosts: ["192.168.1.101", "192.168.1.102", "192.168.1.103"]l_master_nodes: ["192.168.1.101", "192.168.1.102", "192.168.1.103"]h_host: : true # 允许成为主节点: true # 允许成为数据节点#d: true # 建议关闭或不设置，若设置了有很多⾮常⿇烦的事d: -origin: "*"_clause_count: _buckets: 100000192.168.1.102机器修改配置⽂件：vim /usr/local/elasticsearch/config/: ELK # 集群名，同⼀个集群，集群名必须⼀致: es-2 # 集群节点，可任意取ss: : /usr/local/elasticsearch/data # 数据存放路径: /usr/local/elasticsearch/logs # ⽇志存放路径: 192.168.1.102 # 监听IP地址，: : _hosts: ["192.168.1.101", "192.168.1.102", "192.168.1.103"]l_master_nodes: ["192.168.1.101", "192.168.1.102", "192.168.1.103"]h_host: : true # 允许成为主节点: true # 允许成为数据节点#d: true # 建议关闭或不设置，若设置了有很多⾮常⿇烦的事d: -origin: "*"_clause_count: _buckets: 100000192.168.1.103机器修改配置⽂件：vim /usr/local/elasticsearch/config/: ELK # 集群名，同⼀个集群，集群名必须⼀致: es-3 # 集群节点，可任意取ss: : /usr/local/elasticsearch/data # 数据存放路径: /usr/local/elasticsearch/logs # ⽇志存放路径: 192.168.1.103 # 监听IP地址，: : _hosts: ["192.168.1.101", "192.168.1.102", "192.168.1.103"]l_master_nodes: ["192.168.1.101", "192.168.1.102", "192.168.1.103"]h_host: : true # 允许成为主节点: true # 允许成为数据节点#d: true # 建议关闭或不设置，若设置了有很多⾮常⿇烦的事d: -origin: "*"_clause_count: _buckets: 100000尽量将所有机器设置为允许成为主节点和数据节点，除⾮机器负载很⾼。修改⽇志，三台机器都⼀样：vim //usr/local/elasticsearch/config/_ = _s = 2GB# 只存储7天的⽇志_ = 7D启动elasticsearch集群，三台机器都⼀样：cd /usr/local/elasticsearch/bin# 后台启动./elasticsearch -d# ⾮后台启动，主要⽤于调试./elasticsearch查看集群健康状态：# curl '192.168.1.101:9200/_cluster/health?pretty'# curl '192.168.1.102:9200/_cluster/health?pretty'# curl '192.168.1.103:9200/_cluster/health?pretty'

{ "cluster_name" : "ELK", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0}在root下执⾏以下命令，三台机器都⼀样，否则在运⾏⼀段时间后就会出错：// ⾃⾏换IPcurl -XPUT -H 'Content-Type: application/json' 192.168.1.101:9200/_all/_settings -d '{"_only_allow_delete": null}'六、elasticsearch-head部署部署⽬标机器：192.168.1.101⽤它来看ES状态⾮常直观，除此外感觉这个没啥卵⽤，可以不安装。nodejs安装：因为head插件是⽤开发的，所以需要此环境。tar -Jxf node-v14.15.4-linux-x64/ /usr/local/nodevim /etc/profileexport NODE_HOME=/usr/local/nodeexport PATH=$NODE_HOME/bin:$PATHexport NODE_PATH=$NODE_HOME/lib/node_modules:$PATHsource !$node -vhead插件安装：wget /mobz/elasticsearch-head/archive/ip elasticsearch-head-master/ /usr/local/elasticsearch-headcd /usr/local/elasticsearch-headnpm install -g cnpm --registry=m install -g grunt-clicnpm install -g gruntcnpm install grunt-contrib-cleancnpm install grunt-contrib-concatcnpm install grunt-contrib-watchcnpm install grunt-contrib-connectcnpm install grunt-contrib-copycnpm install grunt-contrib-jasmine #若报错就再执⾏⼀遍vim /usr/local/elasticsearch-head/ect: { server: { options: { hostname: '0.0.0.0', # 新增这⼀⾏即可，不要忘了后⾯的逗号 port: 9100, base: '.', keepalive: true } }}后台启动cd /usr/local/elasticsearch-headnohup grunt server &eval "cd /usr/local/elasticsearch-head/ ; nohup npm run start >/dev/null 2>&1 & "七、logstash部署部署⽬标机器：192.168.1.101简介：Logstash是⼀个具有实时传输能⼒的数据收集引擎，⽤来进⾏数据收集（如：读取⽂本⽂件）、解析、过滤，并将数据发送给ES。由于需要加⼊数据模板，最好使⽤yum部署安装yum安装部署logstash：vim /etc/.d/[logstash-7.x]name=Elastic repository for 7.x packagesbaseurl=/packages/7.x/yumgpgcheck=1gpgkey=/GPG-KEY-elasticsearchenabled=1autorefresh=1type=rpm-md安装密钥，否则⽆法下载rpm --import /GPG-KEY-elasticsearchyum install logstash-7.5.1安装x-pack：./logstash-plugin install x-pack修改配置⽂件：vim /etc/logstash/s-Xms4g-Xmx4gvim /etc/logstash/ = th = ${sys:} = = ${sys:}/logstash-${sys:}_ = _ = 7Dvim /etc/logstash/: "172.16.10.248": : /usr/share/logstash/: /usr/share/logstash/d: : [ "172.16.10.248:9200","10.10.11.33:9200","192.168.150.134:9200" ]vim /etc/logstash/这⼀步由于使⽤了synesis_lite_suricata模板的配置⽂件，因此⽐较消耗性能，会影响到数据的实时性。如果对数据实时性要求较⾼，这⾥的⽬录可以定向到⾃⼰的配置⽂件。- : synlite_suricata : "/etc/logstash/synlite_suricata/conf.d/*.conf"# 如果是⾃⼰的配置⽂件，例如：- : synlite_suricata : "/etc/logstash/"导⼊数据模板synesis_lite_suricata：在实践中，这⼀步千万不要执⾏，否则报错，⽽且⽆法排查原因，博主就因为这个失去了⼀个周末假期cd /usr/share/logstash./logstash-plugin update logstash-filter-dns将synesis_lite_suricata下的synlite_suricata移动到logstash的配置⽬录下mv synesis_lite_suricata/logstash/synlite_suricata /etc/logstash此时的⽬录结构（和没⽤，是我⽤来测试的。当然，如果要使⽤⾃定义配置⽂件的话，它就有⽤了）：vim e.d/synlite_[Service]# Synesis Lite for Suricata global configurationEnvironment="SYNLITE_SURICATA_DICT_PATH=/etc/logstash/synlite_suricata/dictionaries"Environment="SYNLITE_SURICATA_TEMPLATE_PATH=/etc/logstash/synlite_suricata/templates"Environment="SYNLITE_SURICATA_GEOIP_DB_PATH=/etc/logstash/synlite_suricata/geoipdbs"Environment="SYNLITE_SURICATA_GEOIP_CACHE_SIZE=8192"Environment="SYNLITE_SURICATA_GEOIP_LOOKUP=true"Environment="SYNLITE_SURICATA_ASN_LOOKUP=true"Environment="SYNLITE_SURICATA_CLEANUP_SIGS=false"# Name resolution optionEnvironment="SYNLITE_SURICATA_RESOLVE_IP2HOST=false"Environment="SYNLITE_SURICATA_NAMESERVER=127.0.0.1"Environment="SYNLITE_SURICATA_DNS_HIT_CACHE_SIZE=25000"Environment="SYNLITE_SURICATA_DNS_HIT_CACHE_TTL=900"Environment="SYNLITE_SURICATA_DNS_FAILED_CACHE_SIZE=75000"Environment="SYNLITE_SURICATA_DNS_FAILED_CACHE_TTL=3600"# Elasticsearch connection settingsEnvironment="SYNLITE_SURICATA_ES_HOST=[192.168.1.101:9200, 192.168.1.102:9200, 192.168.1.103:9200]"# 如果是开源的ES，这⾥的⽤户名和密码都不⽤管，它会⾃动忽略Environment="SYNLITE_SURICATA_ES_USER=elastic"Environment="SYNLITE_SURICATA_ES_PASSWD=changeme"# Beats inputEnvironment="SYNLITE_SURICATA_BEATS_HOST=172.16.10.248"Environment="SYNLITE_SURICATA_BEATS_PORT=5044"移动synlite_ e.d/synlite_ /etc/systemd/system/e.d/synlite_# 使配置⽣效systemctl daemon-reload# 启动logstashsystemctl start logstash验证配置⽂件，这⾥就是我测试配置⽂件⽤的：./logstash --gs /etc/logstash/ -f /etc/logstash/ --_and_exit⼋、filebeat安装部署⽬标机器：192.168.1.101下载及安装：wget /downloads/beats/filebeat/filebeat-7.5.1-linux-x86_ -zxvf filebeat-7.5.1-linux-x86_ filebeat-7.5.1-linux-x86_64 filebeatcd filebeat修改配置⽂件：vim :- type: log enabled: true # 必须和/etc/suricata/⼒的⽇志配置路径⼀致 paths: - /home/suricata/log/suricata/ fields: : suricata _under_root: true ite_keys: true

s: path: ${}/modules.d/*.yml d: gs: _of_shards: : host: ["192.168.1.101:5601"]sh: hosts: ["192.168.1.101:5044"]processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~运⾏：./filebeat -c ./也可以将其使⽤supervisord守护进程启动：vim /etc/supervisord.d/[program:filebeat]directory=/home/suricata/filebeatcommand=/home/suricata/filebeat/filebeat -e -c /home/suricata/filebeat/ostart=trueautorestart=falsestderr_logfile=/tmp/test_out_logfile=/tmp/test_r=root九、kibana部署部署⽬标机器：192.168.1.101Kibana为 Elasticsearch 提供了分析和可视化的 Web 平台。它可以在 Elasticsearch 的索引中查找，交互数据，并⽣成各种维度表格、图形。yum安装部署kibana：vim /etc/.d/[kibana-7.x]name=Kibana repository for 7.x packagesbaseurl=/packages/7.x/yumgpgcheck=1gpgkey=/GPG-KEY-elasticsearchenabled=1autorefresh=1type=rpm-md安装密钥，否则⽆法下载rpm --import /GPG-KEY-elasticsearchyum install kibana-7.5.1修改配置⽂件：vi /etc/kibana/: : "0.0.0.0": ["192.168.1.101:9200","192.168.1.101:9200","192.168.1.101:9200"]: /usr/share/kibana/logs/: ".kibana": "zh-CN"启动kibana：systemctl start kibana添加模板⽂件：/kibana/synlite_添加位置：创建索引：要创建⾄少两个索引：suricata-*和suricata_stats-*数据展⽰垃圾数据删除suricata在内⽹跑起来后，短短时间就会有⼤量告警。所以我们得对规则进⾏优化，某些我们不关⼼的规则可以禁⽤掉。禁⽤掉相关规则后，不会再⽣成对应的告警。但是ES中已存在的该规则告警该怎么删除呢？我们可以在kibana中直接删除：使⽤kibana⾯板中的DevTools。告警量不⼤的删除⽅式：POST logstash-suricata_log-*/_delete_by_query{ "query": { "match": { "ure": "SURICATA STREAM 3way handshake wrong seq wrong ack" } }}若告警量⼤，则会报超时错误，此时的删除⽅式：POST logstash-suricata_log-*/_delete_by_query?wait_for_completion=false{ "query": { "match": { "ure": "SURICATA STREAM bad window update" } }}上述步骤若成功会返回⼀个task，检查清空操作是否完成：GET _tasks/NQtjLxAaTiig6ZDZ3nK-cw:126846320若删除完成，则会提⽰"completed": true删除ES中的索引数据这个就很简单了，如下⾯这样