一、filebeat

1.filebeat收集日志保存到文件

2.filebeat收集日志保存到ES

3.修改ES保存日志的格式为json

1.日志格式必须为json格式，格式写法有固定要求
2.修改日志写法之后，要清空日志，重启nginx
3.要删除原来的索引

4.修改索引名字

1.修改索引名字
output.elasticsearch:
  hosts: []
  index: ""
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false

二、filebeat收集单个日志到redis

1.配置

[root@m01 ~]# vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.redis:
  hosts: ["172.16.1.91:6379"]
  key: "nginx_json_log"
  db: 0
  password: 123

2.重启

3.访问nginx页面查看redis

[root@redis01 ~]# redis-cli 
127.0.0.1:6379> keys *
1) "nginx_json_log"

127.0.0.1:6379> LLEN nginx_json_log
(integer) 9
127.0.0.1:6379> LRANGE nginx_json_log 0 -1

4.配置logstash将redis数据取出放到ES

[root@m01 ~]# vim /etc/logstash/conf.d/beats_redis_logstash_es.conf
input {
  redis {
    data_type => "list"
    host => ["172.16.1.91"]
    port => 6379
    key => "nginx_json_log"
    db => "0"
    codec => "json"
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.91:9200"]
    index => "redis-%{+YYYY-MM-dd}"
  }
}

5.查看ES页面数据

三、filebeat收集单个日志到logstash

1.配置

[root@m01 ~]# vim /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.logstash:
  hosts: ["10.0.0.61:6666"]

2.配置logstash将filebeat传来的数据保存到ES

[root@m01 ~]# vim /etc/logstash/conf.d/beats_logstash_es.conf 
input {
  beats {
    port => 6666
    codec => "json"
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.91:9200"]
    index => "filebeat-%{+YYYY-MM-dd}"
  }
}

3.到ES查看数据

四、filebeat收集多个日志到ES

1.配置

[root@m01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log

output.elasticsearch:
  hosts: ["10.0.0.91:9200"]
  indices:
    - index: "nginx_json-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        source: "/var/log/nginx/access_json.log"
    - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        source: "/var/log/nginx/access.log"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false

2.重启，访问测试

3.方法二：

[root@m01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access_json.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["json"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  tags: ["access"]

output.elasticsearch:
  hosts: ["10.0.0.91:9200"]
  indices:
    - index: "nginx_json-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "json"
    - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "access"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false

五、收集java报错

1.配置

[root@m01 ~]# cat /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access_json.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["json"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  tags: ["access"]
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ["10.0.0.91:9200"]
  indices:
    - index: "nginx_json-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "json"
    - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "access"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false

2.重启，导入报错日志查看

六、kibana画图

1.写一个logstash获取日志的配置

[root@m01 ~]# vim /etc/logstash/conf.d/nginx.conf 
input {
  file {
    path => "/var/log/nginx/access_json.log"
    type => "nginx_access_log"
    start_position => "end"
    codec => "json"
  }
}
output {
  elasticsearch {
    hosts => ["10.0.0.91:9200"]
    index => "nginx_json_log_%{+YYYY.MM.dd}"
  }
}

#启动
[root@m01 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx.conf

2.模拟日志写入日志文件

[root@m01 ~]# cat 1.txt > /var/log/nginx/access_json.log

3.ES查看数据

4.画图

七、使用地图统计客户端IP

1.安装geoip

#上传包
[root@m01 logstash]# rz ingest-geoip-6.6.0.zip

#解压包
[root@m01 logstash]# unzip ingest-geoip-6.6.0.zip

2.配置

[root@m01 ~]# cat /etc/logstash/conf.d/geoip.conf
input {
  file {
    path => "/var/log/nginx/access_json.log"
    type => "nginx_access_log"
    start_position => "end"
    codec => "json"
  }
}

filter {
  json {
    source => "message"
    remove_field => ["message"]
  }
  geoip {
    source => "clientip"
    target => "geoip"
    database => "/etc/logstash/config/GeoLite2-City.mmdb"
    add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
    add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
  }
  mutate {
    convert => [ "[geoip][coordinates]", "float"]
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.91:9200"]
    index => "logstash-%{type}-%{+YYYY.MM.dd}"
  }
}

作业：

1.将多数据存储到redis
2.logstash提取redis中的数据存储到ES不同的索引
3.导入一些数据，画些好看点的图