Fluentd(td-agent)+Nginx 搭建日志收集系统

Fluentd是一个日志收集系统，可以指定多种日志来源，并且配置处理规则，到最后可以输出到不同的持久化系统。EFK就是其典型的应用场景，将日志收集并输出到ElasticSearch中。
本文的目的在于搭建一套收集Nginx产生的日志，并配置td-agent配置source type为tail的方式，从日志文件中实时收集数据，最后将日志信息经过特定的处理发送到Microsoft的EventHub消息中间件中，供后续的数据处理。
不用fluentd自带的http，原因是nginx的功能更丰富，并且解耦

1. Nginx环境搭建及参数配置

环境搭建
更新一下安装方式：yum安装nginx，

首先安装nginx的依赖环境

在/etc/yum.repos.d 目录下创建 nginx.repo文件

输入命令：touch nginx.repo

上官网 http://nginx.org/en/linux_packages.html#stable 拷贝对应linux版本的yum源

本服务器是centos7.4,所以对应的yum如下，将下面这段拷贝到创建的 nginx.repo 中

[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/mainline/centos/7/$basearch/
gpgcheck=0
enabled=1
输入yum list | grep nginx 可以查看yum版本

执行yum install nginx 安装完毕。

1 wget -c https://nginx.org/download/nginx-1.10.1.tar.gz   
2 yum install gcc-c++   
3 yum install -y pcre pcre-devel   
4 yum install -y zlib zlib-devel   
5 yum install -y openssl openssl-devel   
6 tar -zxvf nginx-1.10.1.tar.gz   
7 cd nginx-1.10.1
8 make
9 make install

参数配置

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    #开启下划线识别，并且中划线转下划线
    underscores_in_headers on;

    #设置body达到256k时写入临时文件，默认为两个系统页大小(4096*2)
    client_body_buffer_size 1m;
    client_max_body_size 1m;
    #client_body_in_single_buffer on;
    #client_body_in_file_only on;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    log_format  unimod  '$remote_addr [$time_local] "$request" $request_length $status $http_content_type $http_content_encoding "$request_body" '
                        '1:$http_row_priority_with_crawltime 2:$http_row_priority_without_crawltime 3:$http_column_priority_with_crawltime 4:$http_column_priority_without_crawltime';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    #include /etc/nginx/conf.d/default.conf;
    server {
        listen       80;
        server_name  10.0.0.8;

        #charset koi8-r;
        #if ($request_method !~* POST) {
        #    return 403;
        #}
        proxy_ignore_client_abort on;

        access_log  /var/log/nginx/host.access.log  unimod;

        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }

        location ~ ^/v[1-9]*/ingest/(qa|weibo|weixin)/(article|document)$ {
            if ($request_method !~* POST) {
                return 403;
            }
            #ngx_http_read_client_request_body();
            proxy_pass http://127.0.0.1:10086;
        }

        #err