zoukankan      html  css  js  c++  java
  • ELK日志分析平台搭建

    1.下载安装JDK,并配置环境变量

    vim /etc/profile

    将下面的内容添加至文件末尾:

    JAVA_HOME=/usr/local/jdk1.8.0_111
    JRE_HOME=/usr/local/jdk1.8.0_111/jre
    CLASSPATH=.:$JAVA_HOME/lib:/dt.jar:$JAVA_HOME/lib/tools.jar
    PATH=$PATH:$JAVA_HOME/bin
    export  JAVA_HOME
    export  JRE_HOME

    ulimit -u 4096

    source /etc/profile

    配置limit相关参数

    vim /etc/security/limits.conf

    添加以下内容

    * soft nproc 65536
    * hard nproc 65536
    * soft nofile 65536
    * hard nofile 65536

    关闭防火墙:

    iptables -F

    创建运行ELK的用户

    groupadd elk
    useradd -g elk elk

    创建ELK运行目录

    mkdir /elk
    chown -R elk:elk /elk

    2.准备ELK安装包

    下载ELK安装包:https://www.elastic.co/downloads,并上传到服务器且解压,解压命令:tar -xzvf 包名

    下载kibana5.2.2 (下载地址:https://artifacts.elastic.co/downloads/kibana/kibana-5.2.2-linux-x86_64.tar.gz)

    wget https://artifacts.elastic.co/downloads/kibana/kibana-5.2.2-linux-x86_64.tar.gz

    下载elasticsearch (下载地址:https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.2.2.tar.gz)

    wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.2.2.tar.gz

    下载logstash (下载地址:https://artifacts.elastic.co/downloads/logstash/logstash-5.2.2.tar.gz)

    wget https://artifacts.elastic.co/downloads/logstash/logstash-5.2.2.tar.gz

    3.配置

    3.1 配置Elasticsearch

    vim config/elasticsearch.yml

    修改如下内容:

    保存退出

    启动Elasticsearch

    bin/elasticsearch &

    查看是否启动成功

    netstat -ant

    用浏览器访问:http://192.168.199.179:9200/

    Elasticsearch安装完毕

    3.2 安装配置logstash 

    简单说明:

    logstash的配置文件须包含三个内容:

    input{}:此模块是负责收集日志,可以从文件读取、从redis /kafka读取或者开启端口让产生日志的业务系统直接写入到logstash

    filter{}:此模块是负责过滤收集到的日志,并根据过滤后对日志定义显示字段

    output{}:此模块是负责将过滤后的日志输出到elasticsearch或者文件、redis等

    mkdir config.d
    vim nginx_accss.conf

    配置文件中添加下面内容:

    input {
        file {
            type => "app-log"
            path => [ "/home/youlan/software/taskSchedule/logs/taskSchedule-log*.log" ]
            codec => multiline {
                    pattern => "^["
                    negate => true
                    what => "previous"
            }
            start_position => "beginning"
        }
    }
     
    filter {
        grok {
            match => [
                "message","%{TIMESTAMP_ISO8601}s*%[{USER}] [%{USERNAME}] [%{USERNAME}] - %{NOTSPACE}, IP:%{IP:ip}"
            ]
        }
        grok {
            match => [
                "message","%{TIMESTAMP_ISO8601:date1}s*%[{USER:level}] [%{USERNAME}] [%{USERNAME}] - %{NOTSPACE:request}"
            ]
        }
        date {
            locale => "en"
            match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss.Z"]
     
       }
    }
    output {
        elasticsearch {
            hosts => ["192.168.199.179:9200"]
            index => "%{type}-%{+YYYY.MM.dd}"
            action => "index"
            template_name => "%{type}"
        }
        stdout {codec => rubydebug}
    }
    View Code

    测试配置文件是否有问题:

    /usr/local/elk/logstash-5.2.2/bin/logstash -t -f /usr/local/elk/logstash-5.2.2/config.d/nginx_accss.conf

    启动 logstash:

    nohup /usr/local/elk/logstash-5.2.2/bin/logstash -f /usr/local/elk/logstash-5.2.2/config.d/nginx_accss.conf &

    查看是否启动成功

    tail -f nohup.out 

     

    出现以上内容表示启动成功

    3.3 安装配置kibana

    vim kibana.yml

    保存退出

    启动kibana

    bin/kibana &

    访问kibana:http://192.168.199.179:9988

  • 相关阅读:
    pip或easy_install安装库报错:SSL: CERTIFICATE_VERIFY_FAILED
    js 闭包
    php 安装xdebug进行调试(phpstorm)
    Linux下一个最简单的不依赖第三库的的C程序(2)
    Linux下一个最简单的不依赖第三库的的C程序(1)
    windbg .net 程序的死锁检测 常用方法(个人备份笔记)
    自定义经纬度索引(非RTree、Morton Code[z order curve]、Geohash的方式)
    通过经纬度坐标计算距离的方法(经纬度距离计算)
    根据2个经纬度点,计算这2个经纬度点之间的距离(通过经度纬度得到距离)
    The version of SOS does not match the version of CLR you are debugging; SOS.dll版本不匹配; Dump文件不同环境mscordacwks.dll版本问题
  • 原文地址:https://www.cnblogs.com/it-davidchen/p/11002670.html
Copyright © 2011-2022 走看看