filebeat+redis+elk

centos7.1 x86_64
filebeat  6.2.3
elk 6.2.3
1.架构
filebeat——>redis——>logstash——>elasticsearch——>kibana
redis+elk部署在192.168.10.225
lyw项目：双机分别部署在192.168.10.11和10.12，日志进行合并
yjy项目：单机部署在10.11
spring项目：单机部署在10.11
bus项目：单机部署在10.12
2.安装软件
2.1导入验证文件
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2.2创建yum repo文件：
# vim /etc/yum.repos.d/elasticsearch-6x.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
2.3安装
# yum install logstash elasticsearch kibana -y
3.配置filebeat
# yum install filebeat -y
3.1 10.11主机，lyw、yjy和spring项目
# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/tomcat_zxwweb/logs/catalina.out
  fields:
    type: lyw
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /var/log/filebeat/filebeat
  fields:
    type: yjy
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /var/log/filebeat/filebeat
  fields:
    type: spring-user
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /var/log/filebeat/filebeat
  fields:
    type: spring-order
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /var/log/filebeat/filebeat
  fields:
    type: spring-product
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /var/log/filebeat/filebeat
  fields:
    type: spring-pay
  fields_under_root: true

output.redis:
    hosts: ["192.168.10.225"]
    port: "6379"
    password: "wang.123"
    key: "203"
    db: 0
    timeout: 5
3.2 10.12主机，bus和yjy项目
# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /data/trip-introtec-cloud/auth.log 
    - /data/trip-introtec-cloud/task.log 
  fields:
    type: bus
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /var/log/filebeat/filebeat
  fields:
    type: yjy
  fields_under_root: true

output.redis:
    hosts: ["192.168.10.225"]
    port: "6379"
    password: "wang.123"
    key: "203"

    db: 0

    timeout: 5

4.配置logstash

4.1 bus项目
# vim /etc/logstash/conf.d/bus.conf 
input {
  redis {
    host => "192.168.10.225"
    port => "6379"
    password => "wang.123"
    data_type => "list"
    db => 0
    key => "203"
    threads => 1
  }
}

output {
  if [type] == "bus" {
    elasticsearch {
      hosts => [ "192.168.10.225:9200"]
      index => "bus-%{+YYYY.MM.dd}"
    }
  }
}

4.2 lyw项目
# vim /etc/logstash/conf.d/lyw.conf 
input {
  redis {
    host => "192.168.10.225"
    port => "6379"
    password => "wang.123"
    data_type => "list"
    db => 0
    key => "203"
    threads => 1
  }
}

output {
  if [type] == "lyw" {
    elasticsearch {
      hosts => [ "192.168.10.225:9200"]
      index => "lyw-%{+YYYY.MM.dd}"
    }
  }
}

4.3 yjy项目
# vim /etc/logstash/conf.d/yjy.conf 
input {
  redis {
    host => "192.168.10.225"
    port => "6379"
    password => "wang.123"
    data_type => "list"
    db => 0
    key => "203"
    threads => 1
  }
}

output {
  if [type] == "yjy" {
    elasticsearch {
      hosts => [ "192.168.10.225:9200"]
      index => "yjy-%{+YYYY.MM.dd}"
    }
  }
}
4.4 spring项目
# vim /etc/logstash/conf.d/spring.conf 
input {
  redis {
    host => "192.168.10.225"
    port => "6379"
    password => "wang.123"
    data_type => "list"
    db => 0
    key => "203"
    threads => 1
  }
}

output {
  if [type] == "spring-user" {
    elasticsearch {
      hosts => [ "192.168.10.225:9200"]
      index => "spring-user-%{+YYYY.MM.dd}"
    }
  }
  if [type] == "spring-order" {
    elasticsearch {
      hosts => [ "192.168.10.225:9200"]
      index => "spring-order-%{+YYYY.MM.dd}"
    }
  }
  if [type] == "spring-product" {
    elasticsearch {
      hosts => [ "192.168.10.225:9200"]
      index => "spring-product-%{+YYYY.MM.dd}"
    }
  }
  if [type] == "spring-pay" {
    elasticsearch {
      hosts => [ "192.168.10.225:9200"]
      index => "spring-pay-%{+YYYY.MM.dd}"
    }
  }
}
4.5 测试配置文件
# /usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/*.conf
Configuration OK

[INFO ] 2019-04-22 11:42:19.285 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

5.elasticsearch配置
# sed -e '/^#/d' -e '/^$/d' /etc/elasticsearch/elasticsearch.yml 
cluster.name: ych-ELK
node.name: ELK-01
path.data: /data/elasticsearch
path.logs: /data/logs/elasticsearch
network.host: 0.0.0.0
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"

# sed -e '/^#/d' -e '/^$/d' /etc/kibana/kibana.yml 
server.port: 5601
server.host: 192.168.10.225
elasticsearch.url: "http://192.168.10.225:9200"

7.启动
systemctl start redis
systemctl start filebeat
systemctl start kibana
/etc/init.d/elasticsearch start

评论列表