ELK6.6.0+filebeat6.6.0部署

elastic不能用root用户去启动，否则会报错，所以创建elastic用户ES集群部署

1.创建elastic用户

$ useradd elastic
$ passwd elastic

2..部署JDK环境

$ tar xvf jdk-8u191-linux-x64.tar.gz -C /usr/local/
$ mv /usr/local/jdk1.8.0_191/ /usr/local/java
$ vim /etc/profile.d/elk.sh 
export JAVA_HOME=/usr/local/java
export PATH=$JAVA_HOME:$PATH
$ source /etc/profile

3.下载elastic源码包

$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.tar.gz

4.解压elastic

$ tar xvf elasticsearch-6.6.0.tar.gz -C /usr/local/
$ mv /usr/local/elasticsearch-6.6.0/ /usr/local/elastic
$ chown -R elastic /usr/local/elastic/

5.修改elastic内存配置

elasticsearch6.6.0默认内存需要1G，如果没有1G内存可能会报错，如果内存不够则需要修改配置文件

$ vim /usr/local/elastic/config/jvm.options
-Xms512m
-Xmx512m

6.修改elastic配置文件

$ mkdir /data/es-data -p
$ mkdir /var/log/elastic/
$ vim /usr/local/elastic/config/elasticsearch.yml 
# 组名自定义，但是同一个组，组名必须一致
cluster.name: my-application
# 节点名称，建议和主机名一致
node.name: elastic
# 数据存放目录
path.data: /data/es-data
# 日志存放路径
path.logs: /var/log/elastic
# 锁住内存，bubei 使用到交换分区去
bootstrap.memory_lock: true
# 由于只部署两个节点，因此设置为1，否则当master宕机，将无法重新选取master
discovery.zen.minimum_master_nodes: 1
​
# 网络设置
network.host: 0.0.0.0
# 端口
http.port: 9200
​
# 从节点配置
# 关闭多播
discovery.zen.ping.unicast.enabled: false
# 发单播，ip地址是master和自己
discovery.zen.ping.unicast.hosts: ["192.168.1.131", "192.168.1.164"]

6.启动elastic

$ chown -R elastic /data/
$ chown -R elastic /var/log/elastic/
$ su - elastic
$ /usr/local/elastic/bin/elasticsearch

7.测试

8.报错

[1]: max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]

#切换回root
$ vim /etc/security/limits.conf 
# elastic是用户
elastic soft nofile 65536
elastic hard nofile 65536
# 登录elastic查看
$ ulimit -Hn
65536

[2]: memory locking requested for elasticsearch process but memory is not locked

$ vim /etc/security/limits.conf 
elastic         -        memlock         unlimited

[3]: max number of threads [3802] for user [elastic] is too low, increase to at least [4096]

$ vim /etc/security/limits.d/20-nproc.conf
elastic    -       nproc     4096

[4]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

$ vim /etc/sysctl.conf
vm.max_map_count=655360
$ sysctl -p

插件安装

elastic5.0之后，head插件需要独立安装

1.head插件

# 安装NodeJS
$ wget https://npm.taobao.org/mirrors/node/latest-v4.x/node-v4.5.0-linux-x64.tar.gz
$ tar -zxvf node-v4.5.0-linux-x64.tar.gz -C /usr/local/
$ mv /usr/local/nodenode-v4.5.0-linux-x64 /usr/local/node
$ vim /etc/profile
    export NODE_HOME=/usr/local/node
    export PATH=$PATH:$NODE_HOME/bin/
    export NODE_PATH=$NODE_HOME/lib/node_modules
$ source /etc/profile
​
# 安装npm
$ npm install -g cnpm --registry=https://registry.npm.taobao.org
​
# 安装grunt
$ npm install -g grunt
$ npm install -g grunt-cli --registry=https://registry.npm.taobao.org --no-proxy
​
# 确认版本
node -v
v9.5.0
$ npm -v
5.6.0
$ grunt -version
grunt-cli v1.3.2
grunt v1.0.1
​
​
$ wget https://github.com/mobz/elasticsearch-head/archive/master.zip
$ unzip master.zip
$ cd elasticsearch-head-master/
# npm install -g cnpm --registry=https://registry.npm.taobao.org
$ npm install
​
# 修改es的配置文件
# head插件可以访问es
$ vim /usr/local/elasticsearch-6.6.0/config/elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: "*"
​
# 修改head插件配置文件
$ vim Gruntfile.js
# 增加一行hostname
               connect: {
                        server: {
                                options: {
                                        hostname: '0.0.0.0',
                                        port: 9100,
                                        base: '.',
                                        keepalive: true
                                }
                        }
                }
​
$ vim elasticsearch-head-master/_site/app.js
# 修改localhost为es的ip地址
# this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://localhost:9200";
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://192.168.1.126:9200";
# 重启es
$ grunt server 

　　

 

 

Filebeat+Logstash部署

1.Filebeat部署

a.下载filebeat源码包

$ wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.6.0-linux-x86_64.tar.gz

b.解压源码包

$ tar xvf filebeat-6.6.0-linux-x86_64.tar.gz -C /usr/local/
$ mv /usr/local/filebeat-6.6.0-linux-x86_64/ /usr/local/filebeat

c.修改配置文件

$ vim /usr/local/filebeat/filebeat.yml
#=========================== Filebeat inputs =============================
​
filebeat.inputs:
- type: log
  # Change to true to enable this input configuration.
  enabled: true
​
  # Paths that should be crawled and fetched. Glob based paths.
  # 指定读取文件的位置
  paths:
    - /var/log/*.log
  # 只发送包含ERR,WARN字样的日志
  # include_lines: ['^ERR', '^WARN']
  # 不发送包含OK字样的日志
  # exclude_lines: ["^OK"]
  # 定义写到ES时的type值
  # document_type: "test" 
​
# 输出的位置，直接输出到elastic的话，选第一个，输出到logstash的话，选第二个 
  #output.elasticsearch:
    # Array of hosts to connect to.
    #hosts: ["localhost:9200"]
  
  output.logstash:
    # The Logstash hosts
    hosts: ["localhost:5044"]

　　

d.启动filebeat

$ vim /etc/profile.d/elk.sh
export PATH=$PATH:/usr/local/filebeat/
$ source /etc/profile
$ filebeat -e -c /usr/local/filebeat/filebeat.yml

2.Logstash部署

a.部署JDK环境

$ tar xvf jdk-8u191-linux-x64.tar.gz -C /usr/local/
$ mv /usr/local/jdk1.8.0_191/ /usr/local/java
$ vim /etc/profile.d/elk.sh 
export JAVA_HOME=/usr/local/java
export PATH=$JAVA_HOME:$PATH:/usr/local/filebeat/
$ source /etc/profile

b.下载Logstash源码包

$ wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.0.tar.gz

c.解压源码包

$ tar xvf logstash-6.6.0.tar.gz -C /usr/local/
$ mv /usr/local/logstash-6.6.0/ /usr/local/logstash

d.修改配置文件

input {
  beats {
    port => 5044
  }
}
output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => ["http://192.168.1.126:9200"]
    index => "test"
  }
}

Kibana部署

1. 下载Kibana

$ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.6.0-linux-x86_64.tar.gz

2. 解压源码包

$ tar xvf kibana-6.6.0-linux-x86_64.tar.gz -C /usr/local/
$ mv /usr/local/kibana-6.6.0-linux-x86_64/ /usr/local/kibana

3. 修改配置文件

$ vim /usr/local/kibana/config/kibana.yml
server.host: "192.168.1.130"
elasticsearch.hosts: ["http://192.168.1.126:9200"]

4.启动Kibana

$ /usr/local/kibana/bin/kibana

访问 192.168.1.130:5601

 

PS：如有错误，欢迎指正