ELKF简易配置备忘

发布于：2016-01-26

准备

环境基于Ubuntu Server 14.04、JRE 1.8.7，使用的软件版本如下

组件安装

除了 Kibana，ELF 均为 deb 包，直接使用 dpkg 安装

Elasticsearch

plaintext
UTF-8|3 Lines|
dpkg -i elasticsearch-2.1.1.deb
service elasticsearch start
curl -I http://localhost:9200

Logstash

plaintext
UTF-8|1 Line|
dpkg -i logstash_2.1.1-1_all.deb

Filebeat

plaintext
UTF-8|1 Line|
dpkg -i filebeat_1.0.1_amd64.de

Kibana

plaintext
UTF-8|3 Lines|
tar zxf kibana-4.3.1-linux-x64.tar.gz -C /usr/local/
cd /usr/local/
mv kibana-4.3.1-linux-x64 kibana

配置

以收集 NGINX 的 access_log 为例

NGINX 将 NGINX 的日志格式改为

plaintext
UTF-8|7 Lines|
log_format logstash '$http_host '
                    '$remote_addr [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    '$request_time '
                    '$upstream_response_time';
access_log /var/log/nginx/access.log logstash;

重新载入 NGINX 配置

plaintext
UTF-8|1 Line|
service nginx reload

Logstash 创建 patterns 目录并新建 NGINX pattern

plaintext
UTF-8|3 Lines|
mkdir /opt/logstash/patterns
cd /opt/logstash/patterns
vim nginx

使用如下规则匹配 access_log

plaintext
UTF-8|4 Lines|
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float} %{NUMBER:upstream_time:float}
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float}

Logstash 的 filter 使用的是 grok 语法，相关介绍可以参考此处

新建 Logstash 的 Filebeat Source 配置

plaintext
UTF-8|1 Line|
vim /etc/logstash/conf.d/filebeat.conf

内容

plaintext
UTF-8|28 Lines|
input {
  beats {
    port => 5044
  }
}

filter{
    if[type] == "nginx-access" {
        grok {
            match => {"message" => "%{NGINXACCESS}"}
            add_field => [ "received_at", "%{@timestamp}" ]
        }
        geoip {
                source => "clientip"
        }
        date {
                match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
        }
    }
}

output {
  elasticsearch {
          hosts => ["127.0.0.1:9200"]
          index => "logstash-beat-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
          document_type => "%{[@metadata][type]}"
  }
}

保存后

plaintext
UTF-8|1 Line|
service logstash restart

Filebeat

plaintext
UTF-8|1 Line|
vim /etc/filebeat/filebeat.yml

修改为

plaintext
UTF-8|15 Lines|
filebeat:
  prospectors:
    -
      paths:
        - /usr/local/nginx/logs/access.log # 你 NGINX 日志的位置
      input_type: log
      document_type: nginx-access
  registry_file: /var/lib/filebeat/registry
output:
  logstash:
    hosts: ["localhost:5044"]
    index: filebeat
shipper:
logging:
  files:

保存后重启 Filebeat

plaintext
UTF-8|1 Line|
service filebeat restart

Kibana 无需配置，直接使用 Kibana 内置的 Web server 启动（如果你想修改配置可以修改/usr/local/kibana/config/kibana.yml

plaintext
UTF-8|1 Line|
/usr/local/kibana/bin/kibana

收工

访问 http://localhost:5601

到此为止只是简单的跑起来，ELK的配置还有很多地方可以修改，有兴趣可以去 elastic.co 上看看