zoukankan      html  css  js  c++  java
  • Elasticsearch日志收集

    Install pip if necessary

    curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
    python get-pip.py


    Install Curator for Elasticsearch

    Elasticsearch Curator helps you curate, or manage, your Elasticsearch indices and snapshots by:

    • Obtaining the full list of indices (or snapshots) from the cluster, as the actionable list
    • Iterate through a list of user-defined filters to progressively remove indices (or snapshots) from this actionable list as needed.
    • Perform various actions on the items which remain in the actionable list.
    pip install elasticsearch-curator
    pip install click==6.7

    Configure curator

    mkdir -p /var/log/elastic
    touch /var/log/elastic/curator.log
    mkdir ~/.curator
    vi ~/.curator/curator.yml
    curator.yml
    # Remember, leave a key empty if there is no value. None will be a string,
    ## not a Python "NoneType"
    client:
    hosts: [Elasticsearch Server IP]
    port: 9200
    url_prefix:
    use_ssl: False
    certificate:
    client_cert:
    client_key:
    ssl_no_validate: False
    http_auth:
    timeout: 30
    master_only: False
     
    logging:
    loglevel: INFO
    logfile: /var/log/elastic/curator.log
    logformat: default
    blacklist: ['elasticsearch', 'urllib3']


    Have a test, now you can get the indices list
    curator_cli show_indices

    Create repository

    Configure elasticseach.yml default in /etc/elasticsearch/elasticsearch.yml

    elasticsearch.yml
    path.repo:  /u01/elasticsearch/backup
    http.max_header_size: 16kb

    Restart elasticsearch service (service elasticsearch restart) to make the configurations work.

    Create repository elasticsearch. Ensure location points to a valid path which is configured in path.repo, accesable from all nodes.

    curl -XPUT http://localhost:9200/_snapshot/es_backup -H "Content-Type: application/json" -d @repository.json
    repository.json
    {
       "type""fs",
       "settings": {
          "compress"true,
          "location""/u01/elasticsearch/backup"
       }
    }

    Have a test

    curl -XGET 'localhost:9200/_snapshot/_all?pretty=true'


    Create curator yaml action files

    daily_backup.yml

    Customize the snapshot name in name option
    action 1: backup all indices before today to repository elasticsearch with specified snapshot name
    action 2: delete indices older than 185 days

    daily_backup.yml
    ---
    actions:
      1:
        action: snapshot
        description: >-
          Snapshot selected all indices to repository 'elasticsearch' with the snapshot name
        options:
          repository: es_backup
          name: '<c4cert-{now/d-1d}>'
          wait_for_completion: True
          max_wait: 4800
          wait_interval: 30
        filters:
        - filtertype: age
          source: name
          direction: older
          unit: days
          unit_count: 1
          timestring: "%Y.%m.%d"
     
     
      2:
        action: delete_indices
        description: >-
          Delete indices which is older than 185 days
        filters:
        - filtertype: age
          source: name
          direction: older
          unit: days
          unit_count: 185
          timestring: "%Y.%m.%d"

    del_snapshot.yml
    action 1: Delete snapshots from repository elasticsearch which is older than 185 days

    del_snapshot.yml
    ---
     
    actions:
      1:
        action: delete_snapshots
        description: >-
          Delete snapshots from repository which is older than 185 days
        options:
          repository: es_backup
          retry_interval: 120
          retry_count: 3
        filters:
        - filtertype: age
          source: creation_date
          direction: older
          unit: days
          unit_count: 185

    restore.yml
    action 1: Restore all indices in the most recent snapshot with state SUCCESS.

    restore.yml
    ---
     
    actions:
      1:
        action: restore
        description: >-
          Restore all indices in the most recent snapshot with state SUCCESS.  Wait
          for the restore to complete before continuing.  Do not skip the repository
          filesystem access check.  Use the other options to define the index/shard
          settings for the restore.
        options:
          repository: es_backup
          # If name is blank, the most recent snapshot by age will be selected
          name:
          # If indices is blank, all indices in the snapshot will be restored
          indices:
          wait_for_completion: True
          max_wait: 3600
          wait_interval: 10
        filters:
        - filtertype: state
          state: SUCCESS

    Note: use --dry-run option to verify your action without any change. Find the dry run results in log path.
    Curator --dry-run daily_backup.yml

    Shell script and crontab

    run.sh
    #!/bin/sh
    curator /u01/curator/del_snapshot.yml
    curator /u01/curator/daily_backup.yml

    crontab -e

    Here configured the job run on every 3 AM

    crontab
    0 3 * * * /bin/sh /u01/curator/run.sh

    Restore

    Curator restore.yml

    Tested OK in CERT env.

    Some useful API 

    # get all repositories
    curl -XGET 'localhost:9200/_snapshot/_all?pretty=true'
     
    # delete repository
    curl -XDELETE 'localhost:9200/_snapshot/es-snapshot?pretty=true'
     
    # show snapshots
    curator_cli show_snapshots --repository es_backup
     
    # show indices
    curator_cli show_indices
  • 相关阅读:
    解决:npm中 下载速度慢 和(无法将“nrm”项识别为 cmdlet、函数、脚本文件或可运行程序的名称。请检查名称的拼写,如果包括路径,请确保路径正确, 然后再试一次)。
    maven(一) maven到底是个啥玩意~
    Luogu3959 NOIP2017宝藏(状压dp)
    Luogu3953 NOIP2017逛公园(最短路+拓扑排序+动态规划)
    Luogu3952 NOIP2017时间复杂度
    BZOJ4753 JSOI2016最佳团体(分数规划+树形dp)
    BZOJ1975 SDOI2010魔法猪学院(启发式搜索+最短路+堆)
    BZOJ4105 THUSC2015平方运算(线段树)
    BZOJ5109 CodePlus 2017大吉大利,晚上吃鸡!(最短路+拓扑排序+bitset)
    Luogu3731 HAOI2017新型城市化(二分图匹配+强连通分量)
  • 原文地址:https://www.cnblogs.com/ryansunyu/p/9944566.html
Copyright © 2011-2022 走看看