ELKstack 简介
- ELK是Elasticsearch、Logstash、Kibana三个开源软件的组合。在实时数据检索和分析场合，三者通常是配合使用，而且又都先后归于 Elastic.co 公司名下，故有此简称。
- Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。
- Logstash是一个完全开源的工具，它可以对你的日志进行收集、分析，并将其存储供以后使用。
- kibana 是一个开源和免费的工具，它可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。
流程
在需要收集日志的所有服务上部署logstash，作为logstash agent（logstash shipper）用于监控并过滤收集日志，将过滤后的内容发送到Redis，然后logstash indexer将日志收集在一起交给全文搜索服务ElasticSearch，可以用ElasticSearch进行自定义搜索通过Kibana 来结合自定义搜索进行页面展示。
二、ELK安装及配置
1、系统及软件版本介绍：
系统：CentOS6.5_64
elasticsearch：elasticsearch-5.6.3.tar.gz
logstash：logstash-2.3.4.tar.gz
kibana：kibana-4.5.4-linux-x64.tar.gz
redis：redis-3.2.tar.gz
JDK：jdk-8u73-linux-x64.tar.gz
2、服务器规划 在两台服务器安装ELK组件
A-client（需要分析的nginx日志服务器）:安装logstash（logstash agent）
B-master（ELK服务端）：安装elasticsearch、logstash（logstash index）、kibana、redis
软件包安装目录：/data/elk
3、创建用户
# groupadd app
useradd -g app -d /data/elk elk4、安装及配置JDK
logstash及elasticsearch需要JDK支持
# su - elk $ tar zxf jdk-8u73-linux-x64.tar.gz $ vim .bash_profile （添加及修改如下内容） JAVA_HOME=/data/elk/jdk1.8.0_73 PATH=${JAVA_HOME}/bin:$PATH:$HOME/bin export PATH JAVA_HOME $ . .bash_profile
5. ansible 一键安装 A服务器安装及配置logstash client
#目录结构如下
# tree ansible-logstash-playbook ansible-logstash-playbook ├── hosts ├── jdk.retry ├── jdk.yml └── roles ├── jdk │ ├── defaults │ ├── files │ │ └── jdk-8u131-linux-x64.tar.gz │ ├── handlers │ ├── meta │ ├── tasks │ │ └── main.yml │ ├── templates │ └── vars └── logstash ├── defaults ├── files │ └── logstash-2.3.4.tar.gz ├── handlers ├── meta ├── tasks │ └── main.yml ├── templates │ └── logstash_agent.conf.j2 └── vars └── main.yml
ansible一键安装jdk
# cat ansible-logstash-playbook/roles/jdk/tasks/main.yml - name: create group group: name=app system=yes - name: create logstash-user user: name=elk group=app home=/data/elk system=yes - name: create a diretory if it doesn't exist file: path: /data/elk/soft state: directory mode: 0755 owner: elk group: app - name: create a diretory if it doesn't exist file: path: /data/elk/scripts state: directory mode: 0755 owner: elk group: app - name: jdk file to dest host copy: src: /data/soft/jdk-8u131-linux-x64.tar.gz dest: /data/elk/soft/ owner: elk group: app - name: tar jdk-8u131-linux-x64.tar.gz shell: chdir=/data/elk/soft tar zxf jdk-8u131-linux-x64.tar.gz && chown -R elk.app /data/elk/ - name: java_profile config shell: /bin/echo {{ item }} >> /data/elk/.bash_profile && source /data/elk/.bash_profile with_items: - "export JAVA_HOME=/data/elk/soft/jdk1.8.0_131" - PATH=${JAVA_HOME}/bin:$PATH:$HOME/bin - export PATH
absible 一键安装logstash client
# cat ansible-logstash-playbook/roles/logstash/tasks/main.yml
- name: logstash file to dest host copy: src: logstash-2.3.4.tar.gz dest: /data/elk/soft/ owner: elk group: app - name: tar logstash-2.3.4.tar.gz shell: chdir=/data/elk/soft/ tar zxf logstash-2.3.4.tar.gz - name: mv logstash-2.3.4 shell: mv /data/elk/soft/logstash-2.3.4 /data/elk/logstash && chown -R elk.app /data/elk - name: touch conf file: path=/data/elk/logstash/conf owner=elk group=app state=directory - name: logstash conf file to dest host template: src: logstash_agent.conf.j2 dest: /data/elk/logstash/conf/logstash_agent.conf owner: elk group: app - name: start logstash client shell: su - elk -c "nohup /data/elk/logstash/bin/logstash agent -f /data/elk/logstash/conf/logstash_agent.conf &"
# cat ansible-logstash-playbook/roles/logstash/templates/logstash_agent.conf.j2 input { file { type => "tomcat log" add_field => {"host"=> "{{IP}}" } #{{IP }} <==> vars/main.yml path => ["/data/tomcat/apache-tomcat-8088/logs/catalina.out"] #tomcat日志路径 } } output { redis { host => "10.19.182.215" #redis server IP port => "6379" #redis server port data_type => "list" #redis作为队列服务器，key的类型为list key => "tomcat:redis" #key的名称，可以自定义 } }
# cat ansible-logstash-playbook/roles/logstash/vars/main.yml IP: "{{ ansible_eth0['ipv4']['address'] }}"
ansible一键执行部署
# cat jdk.yml - hosts: logstash user: root roles: - jdk - logstash # ansible-playbook jdk.yml -i hosts
6.服务端： 安装及配置elasticsearch && redis
$ unzip elasticsearch-5.6.3.zip $ mv elasticsearch-5.6.3 elasticsearch $ mkdir elasticsearch/{logs,data} $ vim elasticsearch/config/elasticsearch.yml #修改如下内容 cluster.name: server node.name: node-1 path.data: /data1/elk/elasticsearch-5.6.3/data path.logs: /data1/elk/elasticsearch-5.6.3/logs network.host: 10.19.86.42 http.port: 9200 http.cors.enabled: true #开启跨域访问 http.cors.allow-origin: "*" #跨域访问，安装elasticsearch-head需要 discovery.zen.ping.unicast.hosts: ["10.19.33.42", "10.19.22.215","10.19.11.184"] discovery.zen.minimum_master_nodes: 3 #调整使用内存大小；默认为2G；建议调整物理机内存一半；最大内存使用32G； $ vim elasticsearch-5.6.3/config/jvm.options -Xms20g -Xmx20g
$ echo 511 > /proc/sys/net/core/somaxconn # cat /etc/security/limits.d/90-nproc.conf # Default limit for number of user's processes to prevent # accidental fork bombs. # See rhbz #432903 for reasoning. * soft nproc 102400 root soft nproc unlimited echo 262144 > /proc/sys/vm/max_map_count
$ ./bin/elasticsearch -d
7. 服务端配置logstash indexer
[elk@10-19-86-42 ~]$ cat logstash/conf/logstash_tomcat.conf input { redis { host => "10.19.86.42" port => "6379" data_type => "list" key => "tomcat:redis" type => "redis-input" } } filter { grok { match => { "message" => "^%{TIMESTAMP_ISO8601:[@metadata][timestamp]}s+ s+%{LOGLEVEL:level}s+%{GREEDYDATA:class}s+-s+%{GREEDYDATA:msg}" } #match => { "message" => "(^.+Exception:.+)|(^s+at .+)|(^s+... d+ more)|(^s*Caused by:.+)" } #匹配堆信息 } mutate { split => { "fieldname" => "," } } date { match => [ "timestamp","dd/MMM/yyyy:HH:mm:ss Z" ] remove_field => [ "timestamp" ] } } output { elasticsearch { hosts => ["10.19.86.42:9200","10.19.77.184:9200","10.19.182.215:9200"] workers =>2 #必须设置线程数 flush_size => 50000 idle_flush_time => 1 index => "catalina-%{+YYYY.MM.dd}" } stdout {codec => rubydebug} }
supervisor 启动logstash
$ cat /etc/supervisor/conf.d/logstash_tomcat.conf # start logstash [program:logstash-tomcat] environment= JAVA_HOME= /data/elk/jdk1.8.0_131 directory=/data/elk/logstash command=/data/elk/logstash/bin/logstash -w 24 -b 5000 -f /data/elk/logstash/conf/logstash_tomcat.conf autostart = true startsecs = 5 user = elk group = app stdout_logfile = /data/elk/logs/logstash_tomcat.log stderr_logfile = /data/elk/logs/logstash_tomcat_err.log
注：启动多个logstash indexer服务来消费redis队列;
8. 安装kibana并配置
$ tar kibana-5.6.3-linux-x86_64.tar.gz $ vim kibana-5.6.3-linux-x86_64/config/kibana.yml server.port: 5601 server.host: "10.19.11.42" elasticsearch.url: "http://10.19.11.42:9200"
kiana启动
$ cat logstash_kinana.conf # start logstash [program:kinana] environment= JAVA_HOME= /data/elk/jdk1.8.0_131 directory=/data/elk/kibana-5.6.3-linux-x86_64 command=/data/elk/kibana-5.6.3-linux-x86_64/bin/kibana autostart = true startsecs = 5 user = elk group = app stdout_logfile = /data/elk/logs/kibana.log stderr_logfile = /data/elk/logs/kibana_err.log
补充：
1. 安装elasticsearch-head
# yum install -y git grunt-cli grunt npm # git clone git://github.com/mobz/elasticsearch-head.git vim elasticsearch-head/Gruntfile.js connect: { server: { options: { hostname: '10.19.86.42', port: 9100, base: '.', keepalive: true } } }
启动elasticsearch-head
$ cd /data/elk/elasticsearch-5.6.3/elasticsearch-head/node_modules/grunt/bin && nohup ./grunt server &