Elasticsearch,Filebeat,Kibana部署,添加图表及elastalert报警

服务端安装

Elasticsearch和Kibana（需要安装openjdk1.8以上）

安装方法：https://www.elastic.co
以Ubuntu为例：

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
apt-get update
apt-get install elasticsearch
apt-get install kibana

elasticsearch配置

cat /etc/elasticsearch/elasticsearch.yml

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.16.80.28
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.16.80.28","172.16.80.155"]

附件有文件elasticsearch.map

Kibana配置

cat /etc/kibana/kibana.yml

server.port: 5601
server.host: "172.16.80.28"
elasticsearch.url: "http://172.16.80.28:9200"

Elastalert（需要Python2.7）

下载 https://github.com/Yelp/elastalert/releases

进入elastalert目录

pip install -r requirements.txt
python setup.py install
elastalert-create-index

前台运行elastalert

python -m elastalert.elastalert --verbose --rule rules/my_rule.yaml

安装参考:
https://elastalert.readthedocs.io/en/latest/running_elastalert.html#downloading-and-configuring
https://blog.csdn.net/df007df/article/details/54773391

elastalert配置

cat /etc/elastalert/rules/my_rule.yaml

es_host: 172.16.80.28
es_port: 9200
name: filebeat rules
type: frequency
index: filebeat*
num_events: 5
timeframe:
  hours: 1
filter:
- query:
    query_string:
      query: "message: *error*"
alert:
- "email"
email:
- "aaa@qq.com"
- "bbb@qq.com"
alert_text: "Ref Log http://172.16.80.28:5601/app/kibana"
smtp_host: smtp.exmail.qq.com
smtp_port: 25
smtp_ssl: false
smtp_auth_file: /etc/elastalert/rules/smtp_auth_file.yaml
from_addr:  aaa@qq.com

cat /etc/elastalert/rules/smtp_auth_file.yaml

user: "aaa@qq.com"
password: "邮箱密码"  

cat /etc/elastalert/config.yaml

rules_folder: /etc/elastalert/rules
run_every:
  minutes: 1
buffer_time:
  minutes: 15
es_host: 172.16.80.28
es_port: 9200
s_url_prefix: elasticsearch
writeback_index: elastalert_status
alert_time_limit:
  days: 2  

elastalert配置注解

rules_folder：用来加载下一阶段rule的设置，默认是example_rules
run_every：用来设置定时向elasticsearch发送请求
buffer_time：用来设置请求里时间字段的范围，默认是45分钟
es_host：elasticsearch的host地址
es_port：elasticsearch 对应的端口号
use_ssl：可选的，选择是否用SSL连接es，true或者false
es_username：es认证的username
es_password：es认证的password
writeback_index：elastalert产生的日志在elasticsearch中的创建的索引
alert_time_limit：失败重试的时间限制

告警参考
https://blog.csdn.net/gamer_gyt/article/details/52917116
https://elastalert.readthedocs.io/en/latest/ruletypes.html

客户端安装

Filebeat安装

以Ubuntu为例：

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
apt-get update
apt-get install filebeat

filebeat配置

cat /etc/filebeat/filebeat.yml

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/*.log
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "172.16.80.28:5601"
output.elasticsearch:
  hosts: ["172.16.80.28:9200"]

EFK备注

日志位置 /var/lib/elasticsearch/nodes/
日志大小 暂时为345Mb
日志轮循 暂时不知道在哪设置

elasticsearch.map内容：

# uint mapping
{
   "settings" : {
      "index" : {
         "number_of_replicas" : 1,
         "number_of_shards" : 5
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "type" : "long"
            }
         }
      }
   }
}
 
# dbl mapping
{
   "settings" : {
      "index" : {
         "number_of_replicas" : 1,
         "number_of_shards" : 5
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "type" : "double"
            }
         }
      }
   }
}
 
# str mapping
{
   "settings" : {
      "index" : {
         "number_of_replicas" : 1,
         "number_of_shards" : 5
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "fields" : {
                  "analyzed" : {
                     "index" : true,
                     "type" : "text",
                     "analyzer" : "standard"
                  }
               },
               "index" : false,
               "type" : "text"
            }
         }
      }
   }
}
 
# text mapping
{
   "settings" : {
      "index" : {
         "number_of_replicas" : 1,
         "number_of_shards" : 5
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "fields" : {
                  "analyzed" : {
                     "index" : true,
                     "type" : "text",
                     "analyzer" : "standard"
                  }
               },
               "index" : false,
               "type" : "text"
            }
         }
      }
   }
}
 
# log mapping
{
   "settings" : {
      "index" : {
         "number_of_replicas" : 1,
         "number_of_shards" : 5
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "fields" : {
                  "analyzed" : {
                     "index" : true,
                     "type" : "text",
                     "analyzer" : "standard"
                  }
               },
               "index" : false,
               "type" : "text"
            }
         }
      }
   }
}