ELK批量删除索引

一、存在问题

用了一段时间elk发现如果索引长时间不删除，elk会越来越慢，重启elasticsearch服务器节点之前同步时间也会很长

二、解决方法（定期删除索引）

1.在elasticsearch节点上使用curl -XGET 'http://192.168.X.XX:9200/_cat/shards'查看索引

[root@192-168-x-x scripts]# curl -XGET 'http://192.168.x.x:9200/_cat/shards' | more
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  0   0   0  0   0   0   0 --:--:-- --:--:-- --:--:--     0freetrip-2019.01.29   2 r STARTED   34  59kb 192.168.x.x node3-192.168.x.x
freetrip-2019.01.29                2 p STARTED     34    59kb 192.168.x.x node1-192.168.x.x
freetrip-2019.01.29                1 p STARTED     47 122.7kb 192.168.x.x node3-192.168.x.x
freetrip-2019.01.29                1 r STARTED     47 122.7kb 192.168.x.x node1-192.168.x.x
freetrip-2019.01.29                3 p STARTED     58  90.5kb 192.168.x.x node2-192.168.x.x
freetrip-2019.01.29                3 r STARTED     58  90.5kb 192.168.x.x node1-192.168.x.x
freetrip-2019.01.29                4 r STARTED     41  45.5kb 192.168.x.x node2-192.168.x.x
freetrip-2019.01.29                4 p STARTED     41  45.5kb 192.168.x.x node3-192.168.x.x
freetrip-2019.01.29                0 p STARTED     36  97.9kb 192.168.x.x node2-192.168.x.x
freetrip-2019.01.29                0 r STARTED     36  97.9kb 192.168.x.x node3-192.168.x.x

2.将过滤出来要删除的索引存到临时文件夹里面

[root@192-168-x-x ~]# curl -XGET 'http://192.168.x.x:9200/_cat/shards' | grep 192.168.x.x | grep 2018 | awk '{print $1}' |uniq > elk-index.tmp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 81832  100 81832    0     0   142k      0 --:--:-- --:--:-- --:--:--  142k
[root@192-168-3-163 ~]# cat elk-index.tmp 
otter-canal-195-2018.12.10
logstash-nginx-chm-accesslog-2018.11.20
logstash-nginx-oms-accesslog-2018.10.14
laravel-chm248-online-2018.12.10
otter-canal-195-2018.11.20
logstash-nginx-oms-accesslog-2018.11.20
otter-communication-195-2018.11.23
otter-communication-195-2018.12.10
otter-node-195-2018.12.10
logstash-nginx-chvisa-accesslog-2018.10.14
wanmei219-online-2018.10.14
logstash-nginx-chm-accesslog-2018.11.19
otter-canal-195-2018.11.23
otter-canal-195-2018.11.19
logstash-nginx-oms-accesslog-2018.11.19
laravel-chm248-online-2018.11.19
logstash-nginx-oms-accesslog-2018.12.10
otter-manager-195-2018.12.10
logstash-nginx-chvisa-accesslog-2018.11.20
wanmei219-online-2018.12.10
logstash-nginx-chvisa-accesslog-2018.11.19
otter-communication-195-2018.11.19
otter-communication-195-2018.11.20
laravel-chm248-online-2018.11.20
laravel-chm248-online-2018.10.14
logstash-nginx-chm-accesslog-2018.12.10
wanmei219-online-2018.11.19
logstash-nginx-chvisa-accesslog-2018.12.10
wanmei219-online-2018.11.20
logstash-nginx-chm-accesslog-2018.10.14

3. 删除过滤出来的索引文件

for i in `cat elk-index.tmp`
do 
      curl -XDELETE  http://192.168.3.163:9200/$i 
done

4. 使用脚本加定时任务，每一天删除前三天的索引

[root@192-168-x-x ~]# cat /home/scripts/del_elasticseatch_index.sh 
#!/bin/bash
curl -XGET 'http://192.168.x.x:9200/_cat/shards' |grep 192.168.x.x | awk '{print $1}' |grep `date -d "5 days ago" +%Y.%m.%d` | uniq > /tmp/index_name.tmp

for index_name in `cat /tmp/index_name.tmp`
do
   echo $index_name
    curl -XDELETE  http://192.168.x.x:9200/$index_name
    echo "${index_name} delete success" >> /home/scripts/del_elasticseatch_index.log
done

5. 定时任务

[root@192-168-x-x ~]# crontab -l
0 3 * * * bash /home/scripts/del_elasticseatch_index.sh