zoukankan      html  css  js  c++  java
  • MySQL日志收集之Filebeat和Logstsh的一键安装配置(ELK架构)

    关于ELK是什么、做什么用,我们不在此讨论。本文重点在如何实现快速方便地安装logstash和filebeat组件,特别是在近千台DB Server的环境下(为了安全保守,公司DB Server 目前尚未部署saltstack一类的管控软件)。在尽可能标准化的条件下,希望可以实现一键化安装。下面是我们功能实现的一些尝试,我们把手动一步步操作打包提炼到一个sh文档中,安装部署时只要执行sh文件即可。部署安装logstash和filebeat组件由原来的10分钟缩减到目前的1分钟左右,并且减少了因手动部署带来的误操作。

    1.logstash和filebeat安装包所在指定路径下

    logstash的安装包logstash-7.6.0.zip所在路径

    /data/logstash/logstash-7.6.0.zip

    filebeat的安装包filebeat-7.4.2-linux-x86_64.tar.gz所在路径

     2.上传经过标准化的程序配置文件

    上传程序的配置文件filebeat.service、filebeat.yml、logstash.conf、startup.options到指定位置,这些文件是格式化后的,不是解压的默认文件,目的是方便替换安装。

    如何想直接使用disposelogcollectot.sh文件,上传的路径一定要是/tmp/

    3.编写一键安装的可执行文件disposelogcollectot.sh

    #!/bin/bash
    
    # The version is defined V.001
    # Version   ModifyTime                ModifyBy              Desc
    # Ver001    2018-03-25            Carson.Xu             Create the Scripts File
    # Desc: This file is used to despose filebeat  logstash in order to  collect slow log and error log from mysqld.
    
    
    #### step 1 判断 需要上传的文件是否已上传
    cd /tmp/
    if [ -f "filebeat.service" -a -f "filebeat.yml" -a -f "logstash.conf" -a -f "startup.options" ]
    then
        echo 'step 1 安装过程需要的文件已到位,上传文件项检查通过....'
    else
        echo "step 1 安装过程中需要的filebeat.service、 filebeat.yml、 logstash.conf、 startup.options,没有到位,不能继续安装,安装进程退出!!!"
        exit
    fi
    
    #### step 2 解压指定文件
    cd /data/logstash/
    unzip logstash-7.6.0.zip
    echo 'step 2 解压logstash项工作完成....'
    sleep 3
    
    #### step 3 删除解压后的指定文件
    cd logstash-7.6.0/config/
    rm -rf startup.options
    echo 'step 3 删除解压后的指定文件startup.options工作完成....'
    sleep 3
    
    #### step 4 转移上传的文件
    mv /tmp/logstash.conf /tmp/startup.options -t /data/logstash/logstash-7.6.0/config/
    echo 'step 4 转移文件logstash.conf的工作完成....'
    sleep 2
    
    #### step 5 修改log上传的ES 索引[必做 建议用业务名称替换,例如qq/weixin/rewu]
    read  -p "请输入业务名称:" product
    echo -e "
    "
    echo "用户名为:$product"
    sed -i "s/qqweixinface/$product/" /data/logstash/logstash-7.6.0/config/logstash.conf
    
    echo 'step 5 删除解压后的指定文件startup.options工作完成....'
    sleep 2
    ##### step 6 安装logstash 服务
    /data/logstash/logstash-7.6.0/bin/system-install
    echo 'step 6 安装logstash 服务工作完成....'
    sleep 3
    
    ##### step 7 解压缩filebeat文件
    cd /data/filebeat/
    tar -zxvf filebeat-7.4.2-linux-x86_64.tar.gz
    echo 'step 7 解压缩filebeat文件工作完成....'
    sleep 3
    
    #### step 8 转移上传的filebeat.yml,允许覆盖掉生成默认配置文件
    rm -rf /data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat.yml
    mv /tmp/filebeat.yml /data/filebeat/filebeat-7.4.2-linux-x86_64/
    echo 'step 8 转移上传的filebeat.yml,允许覆盖掉生成默认配置文件工作完成....'
    sleep 2
    
    #### step 9 权限调整
    cd /data/filebeat/filebeat-7.4.2-linux-x86_64
    chown -R root:root filebeat.yml
    chmod 600 filebeat.yml
    echo 'step 9 调整filebeat文件权限的工作完成....'
    sleep 2
    
    ##### step 10 获取 Server IP
    ip=$(ip a|awk -F "inet|/"  '/inet.*brd/ {print $2}'|head -n 1)
    serverid=$(echo $ip) #去除左右空格
    echo $serverid
    echo 'step 10 获取Server IP的工作完成....'
    echo '获取Server IP的为:' $serverid
    sleep 1
    
    ###### step 11 调整host配置[必做 IP替换]
    sed -i "s/119.119.119.119/$serverid/" /data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat.yml
    echo 'step 11 替换配置文件中的Server IP工作完成....'
    sleep 1
    
    ###### step 12 转移服务文件
    mv /tmp/filebeat.service /etc/systemd/system/
    echo 'step 12 将filebeat服务的文件移动到指定位置的工作完成....'
    sleep 1
    
    ###### step 13 服务设置及启动
    systemctl enable logstash.service
    systemctl enable filebeat.service
    
    echo 'step 13 将服务设置为自启动的工作完成....'
    
    systemctl start logstash.service
    
    sleep 20
    
    systemctl start filebeat.service
    
    sleep 10
    
    ##### step 14 检查服务是否已正常启动
    logstashservice_check_result=`systemctl status logstash.service | grep "active (running)"| wc -l`
    if [ "$logstashservice_check_result" == "1" ]
    then
        echo 'step 14 检查logstash.service已启动....'
    else
        echo "step 14 检查logstash.service未正常启动....,安装进程退出!!!"
        exit
    fi
    
    sleep 3
    
    filebeatservice_check_result=`systemctl status filebeat.service | grep "active (running)"| wc -l`
    if [ "$filebeatservice_check_result" == "1" ]
    then
        echo 'step 14 检查filebeat.service已启动....'
    else
        echo "step 14 检查filebeat.service未正常启动....,安装进程退出!!!"
        exit
    fi
    
    ###### step 15 安装过程结束
    
    echo 'step 15 安装过程结束'

    4.执行

    chmod 755 disposelogcollectot.sh
    sh -x disposelogcollectot.sh

    5.附录

    在附录这一部分,介绍刚刚上传的文件--filebeat.service、filebeat.yml、logstash.conf、startup.options

    5.1 文件startup.options

    这一个文件主要描述了logstash程序的启动配置

    ################################################################################
    # These settings are ONLY used by $LS_HOME/bin/system-install to create a custom
    # startup script for Logstash and is not used by Logstash itself. It should
    # automagically use the init system (systemd, upstart, sysv, etc.) that your
    # Linux distribution uses.
    #
    # After changing anything here, you need to re-run $LS_HOME/bin/system-install
    # as root to push the changes to the init script.
    ################################################################################
    
    # Override Java location
    #JAVACMD=/usr/bin/java
    
    # Set a home directory
    LS_HOME=/data/logstash/logstash-7.6.0
    
    # logstash settings directory, the path which contains logstash.yml
    LS_SETTINGS_DIR=/data/logstash/logstash-7.6.0/config
    
    # Arguments to pass to logstash
    LS_OPTS="--path.settings ${LS_SETTINGS_DIR} -f /data/logstash/logstash-7.6.0/config/logstash.conf"
    
    # Arguments to pass to java
    LS_JAVA_OPTS=""
    
    # pidfiles aren't used the same way for upstart and systemd; this is for sysv users.
    LS_PIDFILE=/var/run/logstash.pid
    
    # user and group id to be invoked as
    LS_USER=root
    LS_GROUP=root
    
    # Enable GC logging by uncommenting the appropriate lines in the GC logging
    # section in jvm.options
    LS_GC_LOG_FILE=/var/log/logstash/gc.log
    
    # Open file limit
    LS_OPEN_FILES=16384
    
    # Nice level
    LS_NICE=19
    
    # Change these to have the init script named and described differently
    # This is useful when running multiple instances of Logstash on the same
    # physical box or vm
    SERVICE_NAME="logstash"
    SERVICE_DESCRIPTION="logstash"
    
    # If you need to run a command or script before launching Logstash, put it
    # between the lines beginning with `read` and `EOM`, and uncomment those lines.
    ###
    ## read -r -d '' PRESTART << EOM
    ## EOM

     5.2 附件logstash.conf

    这个文件主要说明的是格式化读取的数据 以及 如何保存到elasticsearch中

    # Sample Logstash configuration for creating a simple
    # Beats -> Logstash -> Elasticsearch pipeline.
    
    input {
      beats {
        port => 5044
      }
    }
    
    
    filter {
        if [fields][log_type] == "mysql-slow" {
            grok {
                match => ["message", "(?m)^#s+Time:s+%{TIMESTAMP_ISO8601}s*#s+User@Host:s+(?<user>.*)[%{USERNAME:user}?]s*@s*%{IPORHOST:client}?s*[%{IPORHOST:client}?]s+Id:s+%{BASE10NUM}s*#s+Query_time:s+%{BASE10NUM:query_time}s+Lock_time:s+%{BASE10NUM:lock_time}s+Rows_sent:s+%{BASE10NUM:rows_sent}s+Rows_examined:s+%{BASE10NUM:rows_examined}s*(uses+%{DATA:database};s*)?SETs+timestamp=%{BASE10NUM:timestamp};s*%{GREEDYDATA:sql_stmt}$"]
                keep_empty_captures => true
            }
            date {
                match => ["timestamp", "UNIX"]
                remove_field => ["timestamp"]
            }
            mutate {
                convert => {
                    "query_time" => "float"
                    "lock_time" => "float"
                    "rows_sent" => "integer"
                    "rows_examined" => "integer"
                }
                remove_field => ["@version", "beat", "host", "input", "log", "offset", "prospector", "source", "tags"]
            }
        }
        if [fields][log_type] == "mysql-error" {
            grok {
                match => ["message", "(?m)^%{TIMESTAMP_ISO8601:timestamp} %{BASE10NUM} [%{WORD:error_level}] %{GREEDYDATA:error_msg}$"]
            }
            date {
                match=> ["timestamp", "ISO8601"]
                remove_field => ["timestamp"]
            }
            mutate {
                remove_field => ["@version", "beat", "host", "input", "log", "offset", "prospector", "source", "tags"]
            }
        }
    }
    
    
    output {
      elasticsearch {
        hosts => ["http://110.110.110.110:10192"]
        #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        index => "%{[fields][log_type]}-qqweixinface-%{+YYYY.MM.dd}"
        user => "qquid_es"
        password => "xiang_ni_123+yidiandian"
      }
    }

     5.3 附件filebeat.yml

    这个文件主要说明了filebeat读取什么log,已经对读取的数据如何处理

    ###################### Filebeat Configuration Example #########################
    
    # This file is an example configuration file highlighting only the most common
    # options. The filebeat.reference.yml file from the same directory contains all the
    # supported options with more comments. You can use it as a reference.
    #
    # You can find the full configuration reference here:
    # https://www.elastic.co/guide/en/beats/filebeat/index.html
    
    # For more available modules and options, please see the filebeat.reference.yml sample
    # configuration file.
    
    #=========================== Filebeat inputs =============================
    
    filebeat.inputs:
    
    # Each - is an input. Most options can be set at the input level, so
    # you can use different inputs for various configurations.
    # Below are the input specific configurations.
    
    - type: log
    
      # Change to true to enable this input configuration.
      #enabled: false
    
      # Paths that should be crawled and fetched. Glob based paths.
      #paths:
        #- /var/log/*.log
        #- c:programdataelasticsearchlogs*
    
    
      paths:
        - /data/mysql/data/slow.log
      fields:
        log_type: mysql-slow
        db_host: 119.119.119.119
        db_port: 3306
      multiline.pattern: "^# Time:"
      multiline.negate: true
      multiline.match: after
    
    
    - type: log
      paths:
        - /data/mysql/data/error.log
      fields:
        log_type: mysql-error
        db_host: 119.119.119.119
        db_port: 3306
      multiline.pattern: ^20d{2}-d{2}-d{2}T
      multiline.negate: true
      multiline.match: after
    
    
      # Exclude lines. A list of regular expressions to match. It drops the lines that are
      # matching any regular expression from the list.
      #exclude_lines: ['^DBG']
    
      # Include lines. A list of regular expressions to match. It exports the lines that are
      # matching any regular expression from the list.
      #include_lines: ['^ERR', '^WARN']
    
      # Exclude files. A list of regular expressions to match. Filebeat drops the files that
      # are matching any regular expression from the list. By default, no files are dropped.
      #exclude_files: ['.gz$']
    
      # Optional additional fields. These fields can be freely picked
      # to add additional information to the crawled log files for filtering
      #fields:
      #  level: debug
      #  review: 1
    
      ### Multiline options
    
      # Multiline can be used for log messages spanning multiple lines. This is common
      # for Java Stack Traces or C-Line Continuation
    
      # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
      #multiline.pattern: ^[
    
      # Defines if the pattern set under pattern should be negated or not. Default is false.
      #multiline.negate: false
    
      # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
      # that was (not) matched before or after or as long as a pattern is not matched based on negate.
      # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
      #multiline.match: after
    
    
    #============================= Filebeat modules ===============================
    
    filebeat.config.modules:
      # Glob pattern for configuration loading
      path: ${path.config}/modules.d/*.yml
    
      # Set to true to enable config reloading
      reload.enabled: false
    
      # Period on which files under path should be checked for changes
      #reload.period: 10s
    
    #==================== Elasticsearch template setting ==========================
    
    setup.template.settings:
      index.number_of_shards: 1
      #index.codec: best_compression
      #_source.enabled: false
    
    #================================ General =====================================
    
    # The name of the shipper that publishes the network data. It can be used to group
    # all the transactions sent by a single shipper in the web interface.
    #name:
    
    # The tags of the shipper are included in their own field with each
    # transaction published.
    #tags: ["service-X", "web-tier"]
    
    # Optional fields that you can specify to add additional information to the
    # output.
    #fields:
    #  env: staging
    
    
    #============================== Dashboards =====================================
    # These settings control loading the sample dashboards to the Kibana index. Loading
    # the dashboards is disabled by default and can be enabled either by setting the
    # options here or by using the `setup` command.
    #setup.dashboards.enabled: false
    
    # The URL from where to download the dashboards archive. By default this URL
    # has a value which is computed based on the Beat name and version. For released
    # versions, this URL points to the dashboard archive on the artifacts.elastic.co
    # website.
    #setup.dashboards.url:
    
    #============================== Kibana =====================================
    
    # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
    # This requires a Kibana endpoint configuration.
    setup.kibana:
    
      # Kibana Host
      # Scheme and port can be left out and will be set to the default (http and 5601)
      # In case you specify and additional path, the scheme is required: http://localhost:5601/path
      # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
      #host: "localhost:5601"
    
      # Kibana Space ID
      # ID of the Kibana Space into which the dashboards should be loaded. By default,
      # the Default Space will be used.
      #space.id:
    
    #============================= Elastic Cloud ==================================
    
    # These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
    
    # The cloud.id setting overwrites the `output.elasticsearch.hosts` and
    # `setup.kibana.host` options.
    # You can find the `cloud.id` in the Elastic Cloud web UI.
    #cloud.id:
    
    # The cloud.auth setting overwrites the `output.elasticsearch.username` and
    # `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
    #cloud.auth:
    
    #================================ Outputs =====================================
    
    # Configure what output to use when sending the data collected by the beat.
    
    #-------------------------- Elasticsearch output ------------------------------
    #output.elasticsearch:
      # Array of hosts to connect to.
      #hosts: ["localhost:9200"]
    
      # Optional protocol and basic auth credentials.
      #protocol: "https"
      #username: "elastic"
      #password: "changeme"
    
    #----------------------------- Logstash output --------------------------------
    output.logstash:
      # The Logstash hosts
      hosts: ["localhost:5044"]
    
      # Optional SSL. By default is off.
      # List of root certificates for HTTPS server verifications
      #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
    
      # Certificate for SSL client authentication
      #ssl.certificate: "/etc/pki/client/cert.pem"
    
      # Client Certificate Key
      #ssl.key: "/etc/pki/client/cert.key"
    
    #================================ Processors =====================================
    
    # Configure processors to enhance or manipulate events generated by the beat.
    
    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~
    
    #================================ Logging =====================================
    
    # Sets log level. The default log level is info.
    # Available log levels are: error, warning, info, debug
    #logging.level: debug
    
    # At debug level, you can selectively enable logging only for some components.
    # To enable all selectors use ["*"]. Examples of other selectors are "beat",
    # "publish", "service".
    #logging.selectors: ["*"]
    
    #============================== X-Pack Monitoring ===============================
    # filebeat can export internal metrics to a central Elasticsearch monitoring
    # cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
    # reporting is disabled by default.
    
    # Set to true to enable the monitoring reporter.
    #monitoring.enabled: false
    
    # Sets the UUID of the Elasticsearch cluster under which monitoring data for this
    # Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
    # is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
    #monitoring.cluster_uuid:
    
    # Uncomment to send the metrics to Elasticsearch. Most settings from the
    # Elasticsearch output are accepted here as well.
    # Note that the settings should point to your Elasticsearch *monitoring* cluster.
    # Any setting that is not set is automatically inherited from the Elasticsearch
    # output configuration, so if you have the Elasticsearch output configured such
    # that it is pointing to your Elasticsearch monitoring cluster, you can simply
    # uncomment the following line.
    #monitoring.elasticsearch:
    
    #================================= Migration ==================================
    
    # This allows to enable 6.7 migration aliases
    #migration.6_to_7.enabled: true

     5.4.附件filebeat.service

    这个文件是关于filebeat.service的定义

    [Unit]
    Description=filebeat.service
    [Service]
    User=root
    ExecStart=/data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat -e -c /data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat.yml
    [Install]
    WantedBy=multi-user.target

     

    我的博客即将同步至腾讯云+社区,邀请大家一同入驻:https://cloud.tencent.com/developer/support-plan?invite_code=3opj47skjx4ws

  • 相关阅读:
    UOJ #455 [UER #8]雪灾与外卖 (贪心、模拟费用流)
    Codeforces 482E ELCA (LCT)
    Codeforces 798D Mike and distribution (构造)
    AtCoder AGC017C Snuke and Spells
    HDU 6089 Rikka with Terrorist (线段树)
    HDU 6136 Death Podracing (堆)
    AtCoder AGC032D Rotation Sort (DP)
    jenkins+python+kubectl实现批量更新k8s镜像
    Linux 下载最新kubectl版本的命令:
    jenkins X 和k8s CI/CD
  • 原文地址:https://www.cnblogs.com/xuliuzai/p/14486122.html
Copyright © 2011-2022 走看看