es定期删除数据

es定期删除数据

1、定期删除索引

使用sentinl报警后，会产生大量如下索引，虽然不占空间，但时间久了也不好，故写个脚本定期删除

脚本如下：

1 #!/bin/bash
2 #只保留5天内的日志索引
3 LAST_DATA=`date -d "-5 days" "+%Y.%m.%d"`
4 #删除上个月份所有的索引
5 curl -XDELETE 'http://10.139.xx.xx:9200/*-'${LAST_DATA}''

再在设置一个定时策略即可

1 0 1 * * * /data1/elk/scripts/clear-index.sh

2、定期删除索引

Curator 是elasticsearch 官方的一个索引管理工具，可以删除、创建、关闭、段合并等等功能

安装

参考官网：https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html

pip install elasticsearch-curator

安装完如果curator 和curator_cli说明安装成功

curator核心在于俩个配置文件，配置文件名称随意无要求：

配置文件config.yml：配置要连接的ES地址、日志配置、日志级别等；

执行文件action.yml: 配置要执行的操作(可批量）、配置索引的格式（前缀匹配、正则匹配方式等）

config.yml样例

具体参数解析见官网：https://www.elastic.co/guide/en/elasticsearch/client/curator/4.2/configfile.html

client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False
 
logging:
  loglevel: INFO
  logfile: /var/log/elasticsearch-curator.log
  logformat: default
  blacklist: []

action.yml样例(删除3天前的数据)：

参数具体意思参见官网：https://www.elastic.co/guide/en/elasticsearch/client/curator/4.2/actionfile.html

actions:
  1:
    action: delete_indices
    description: >-
      Delete metric indices older than 3 days (based on index name), for
      zou_data-2018-05-01
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options :
      ignore_empty_list: True
      disable_action: True
    filters:
      - filtertype: pattern
             kind: regex
             value: '^(zou_data-).*$'
      - filtertype: age
             source: name
             direction: older
             timestring: '%Y-%m-%d'
             unit: days
             unit_count: 3

运行curator

单次运行

curator --config config.yml action.yml 

定时任务运行

0 0 */1 * * curator --config /opt/elasticsearch-curator/config.yml /opt/elasticsearch-curator/action.yml

3、定期删除索引内的数据

#!/bin/bash

indexs=` curl -X GET 'http://10.10.10.10:9200/_cat/indices?v' | awk '{print $3}' | grep -vE '(kibana|index|watcher|monitoring)'`

for index in $indexs
  do
     curl  -X POST "10.139.34.129:9200/$index/_delete_by_query?pretty" -H 'Content-Type:application/json' -d '
      {
         "query": { 
              "bool": {
                   "must": [
                      {
                        "range": {
                           "@timestamp": {
                               "gte": "now-7d",
                               "lte": "now",
                               "format": "epoch_millis"
                              }
                         }
                     }
                   ],
                   "must_not": []
              }  
         }  
     }'
     echo "已清除$index 索引内七天前数据~"
   done

0 1 * * * /data1/elk/scripts/clear-data.sh

ES的删除操作，不会立即生效，跟更新操作类似。只是会被标记为已删除状态，ES后期会自动删除。

es启动脚本

#!/bin/bash
#set -x
cd `dirname $0`
data_dir=/data1/elk/elasticsearch

if [ ! -d $data_dir/data ]; then
    mkdir $data_dir/data && chown -R dev.dev $data_dir
fi

bin_dir=$data_dir/bin

PID=`ps -ef | grep elasticsearch | grep -v grep  | grep root | grep -v bash |awk '{print $2}'`

if [ -n "$PID" ]
        then kill -9 $PID
        echo "before: $PID"
        cd $bin_dir &&  nohup su - dev -c "$bin_dir/elasticsearch"  >> /dev/null 2>&1 &
        sleep 3
        P=`ps -ef | grep elasticsearch | grep -v grep |  grep root | grep -v bash |awk '{print $2}'`
        echo "now   : $P"
else 
        echo "starting"
        cd $bin_dir &&  nohup su - dev -c "$bin_dir/elasticsearch"  >> /dev/null 2>&1 &
        P=`ps -ef | grep elasticsearch | grep -v grep  |  grep root | grep -v bash |awk '{print $2}'`
        echo "now   : $P"
fi

kinaba启动脚本

#!/bin/bash
#set -x
cd `dirname $0`
data_dir=/data1/elk/kibana

if [ ! -d $data_dir/data ]; then
    mkdir $data_dir/data && chown -R dev.dev $data_dir
fi

bin_dir=$data_dir/bin

PID=`netstat -nlpt | grep 5601 | awk '{print $7}' | cut -d / -f1`

if [ -n "$PID" ]
        then kill -9 $PID
        echo "before: $PID"
        cd $bin_dir &&  nohup su - dev -c "$bin_dir/kibana"  >> $data_dir/logs/kibana.log 2>&1 &
        sleep 3
        P=`netstat -nlpt | grep 5601 | awk '{print $7}' | cut -d / -f1`
        echo "now   : $P"
else 
        echo "starting"
        cd $bin_dir &&  nohup su - dev -c "$bin_dir/kibana"  >> $data_dir/logs/kibana.log 2>&1 &
        P=`netstat -nlpt | grep 5601 | awk '{print $7}' | cut -d / -f1`
        echo "now   : $P"
fi