日志收集处理服务框架：ELK + Filebeat

日志收集处理服务框架：ELK + Filebeat

一、需求：压测数据分析和反馈

说明：因为在压测产生异常数据时，需要手动登录到压测机上分析日志比较麻烦（压测服务器因业务需求可能到达10+台）；所以搭建平台，为了快速搜索日志并跟开发沟通解决问题。

二、官方框架

1. filebeat 在服务节点上收集日志信息
2. docker 运行 ELK 服务
3. 用户通过 Kibana Web 页面访问日志服务器: http://ip:5601

三、配置环境

1. ELK 服务器：

$ docker-compose up -d
$ vim docker-compose.yml

version: '3'

services:
   elk:
     image: sebp/elk
     ports:
        - "5601:5601"
        - "9200:9200"
        - "5044:5044"
     volumes:
        - ${pwd}/elk-data:/var/lib/elasticsearch

2. Filebeat 服务器：

 # 获取 ELK cert

$ docker exec -it elk /bin/bash

$ cat /etc/pki/tls/certs/logstash-beats.crt

 # 服务节点配置启动 filebeat

$ vim cert.key # 粘贴 logstash-beats.crt (无需进入容器，docker cp dea76a3b59dc:/etc/pki/tls/certs/logstash-beats.crt .)

$ sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

$ vim /etc/yum.repos.d/elk-elasticsearch.repo

[elasticsearch-7.x]

name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

$ sudo yum install filebeat
$ cp /etc/filebeat/filebeat.yml ${pwd}/filebeat.yml

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:

    - /log_path/locustlogs/*.log #(存放日志的路径）

output.logstash:
  # The Logstash hosts
  hosts: ["IP:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/xxx/elk/cert.key"]

$ nohup filebeat -e -c ${pwd}/filebeat.yml

三、使用说明