logstash对nginx日志进行解析

最新推荐文章于 2024-04-17 15:33:59 发布

转载最新推荐文章于 2024-04-17 15:33:59 发布 · 992 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://blog.51cto.com/liuzhengwei521/2141244

文章标签：

#运维 #数据库 #json

本文介绍如何使用Logstash处理Nginx JSON格式的日志文件，包括解析user_agent、提取地理位置信息及数据类型转换等操作，并通过Filebeat收集日志到Redis，最后将处理后的日志发送至Elasticsearch。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

logstash对nginx日志进行解析过滤转换等操作；
此例中nginx日志提前配置为json格式
配置可以用于生产环境，架构为filebeat读取日志放入redis，logstash从redis读取日志后进行操作；
对user_agent和用户ip也进行了解析操作，便于统计；

input {
    redis {
        host => "192.168.1.109"
        port => 6379
        db => "0"
        data_type => "list"
        key => "test"
    }
}
filter{
    json {
        source => "message"
        remove_field => "message"
    }
    useragent {
        source => "agent"
        target => "agent"
        remove_field => ["[agent][build]","[agent][os_name]","[agent][device]","[agent][minor]","[agent][patch]"]
    }
    date {
        match => ["access_time", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
    mutate {
        remove_field => ["beat","host","prospector","@version","offset","input","source","access_time"]
        convert => {"body_bytes_sent" => "integer"}
        convert => {"up_response_time" => "float"}
        convert => {"request_time" => "float"}

    }
    geoip {
        source => "remote_addr"
        target => "geoip"
        remove_field => ["[geoip][country_code3]","[geoip][location]","[geoip][longitude]","[geoip][latitude]","[geoip][region_code]"]
        add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"]
        add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"]
    }
    mutate {
        convert => ["[geoip][coordinates]","float"]
    }
}
output {
    if [tags][0] == "newvp" {
        elasticsearch {
                hosts  => ["192.168.1.110:9200","192.168.1.111:9200","192.168.1.112:9200"]
                index  => "%{type}-%{+YYYY.MM.dd}"
        }
        stdout {
                codec => rubydebug
        }
        #stdout用于调试，正式使用可以去掉
    }
}

filebeat读取日志的写法：

filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/access.log
  tags: ["newvp"]
  fields:
    type: newvp
  fields_under_root: true
output.redis:
  hosts: ["192.168.1.109"]
  key: "test"
  datatype: list

kibana展示：
logstash对nginx日志进行解析