elastchsearch基于kibana Index lifecycle Managent管理模块做生命周期管理

Index lifecycle Managent 是ES6.6后才提供基于X-Pack Basic License（可以免费使用的一个功能）

大致作用如下入,可以根据时间,条目数,数据块大小进行数据生命周期管理,具体好处还很多,这里记录下用法。

 一.创建生命周期的规则,可以通过代码进行创建,也可以通过kibana界面进行创建。

　　1.界面模式

　　可以填入控制的条件来制定周期规则

 

 　　2.代码模式

　　eg:其实和代码和界面模式是一致的，代码创建完依然可以kibana中进行查看和更改

PUT /_ilm/policy/log_ilm_policy2
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_docs": 5
          }
        }
      },
      "warm": {
        "min_age": "10s",
        "actions": {
          "allocate": {
            "include": {
              "box_type": "warm"
            }
          }
        }
      },
     "delete": {
        "min_age": "60s",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

 二.设置索引模板 对所有lm_index-模板索引都做一个设定

PUT /_template/log_ilm_template
{
  "index_patterns": [
    "ilm_index-*"
    ],
    "settings": {
      "index":{
        "lifecycle":{
          "name":"log_ilm_policy2",
          "rollover_alias":"ilm_alias"#这个别名记得和索引别名用的要一致不然会报illegal_argument_exception: index.lifecycle.rollover_alias [ilm_alias] does not point to index [ilm_index-000001]

        },
        "routing":{
          "allocation":{
            "include":{
              "box_type":"hot"
            }
          }
        },
        "number_of_shards":"1",
        "number_of_replicas":"0"
      }
    },
    "mappings": {
         "_source": {
            "enabled": true
         },
         "properties": {
            
        }
    }
}

三.建立索引

#创建索引
#别名为 myindex
#允许索引被写入数据
PUT ilm_index-000001 #索引名字必须以数字结尾不然会报illegal_argument_exception: index name [ilm_index-hot] does not match pattern '^.*-d+$'
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"aliases": {
"ilm_alias":{ #这里的名字要和上面的模板一致
"is_write_index": true
}
}

}

#插入测试数据

POST /ilm_index-000002/_doc
{
"log":"something"
}

#1.默认是10分钟
PUT _cluster/settings
{
"persistent": {
"indices.lifecycle.poll_interval": "1s"
}
}

到此只要条件达到更换生命周期和kibana一旦触发,数据就会进行转移。比如满足60条或者时间超过60秒

 

上面是自测通过,以下是网上查找亲测的可用模板

Creating an alias with the lifecycle policy is a 3 step process.

Elastic provides a great tutorial

In short:

Create a lifecycle policy that defines the appropriate phases and actions.
Create an index template to apply the policy to each new index.
Bootstrap an index as the initial write index.
I think you are creating the template AFTER you create the first index. You should first create the ilm, after that the template where you specify what ilm policy you want to use for the indexes and finally create the first index (bootstrap).
var indexName = "index_name";
var indexPattern = $"{indexName}-*";
var aliasName = $"{indexName}-alias";
var policyName = $"{indexName}-policy";
var firstIndexName = $"{indexName}-000001";

PUT _ilm/policy/index_name-policy
{
    "policy": {
        "phases": {
            "hot": {
                "actions": {
                    "rollover": {
                        "max_size": "5gb",
                        "max_docs": 10000,
                        "max_age":"2d"
                    }
                }
            },
            "warm": {
                "min_age": "5d",
                "actions": {
                }
            },
            "delete": {
                "min_age": "10d",
                "actions": {
                    "delete": {}
                }
            }
        }
    }
}

PUT _template/index_name-template
{
    "index_patterns": ["{{.IndexPattern}}"],
    "settings": {
        "index.number_of_shards": "1",
        "index.number_of_replicas": "1",
        "index.lifecycle.name": "{{.PolicyName}}", 
        "index.lifecycle.rollover_alias": "{{.AliasName}}" 
    },
    "mappings": {
         "_source": {
            "enabled": true
         },
         "properties": {
            {{Properties}}
        }
    }
}

PUT index_name-000001
{
   "aliases": {
        "{{.AliasName}}":{
            "is_write_index": true 
        }
    }
}

　　

模板解说：

PUT /_ilm/policy/fact_terminal_phone_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "300s"
}
}
},
"warm": {
"min_age": "120s",
"actions": {
"allocate": {
"include": {
"box_type": "warm"
}
}
}
},
"delete": {
"min_age": "180s",
"actions": {
"delete": {}
}
}
}
}
}

第一个300秒会进行新的索引创建,第二个重新计算开始120秒会进行索引迁移到冷节点，数据迁移到warm节点（这个120秒和前面300秒不重叠），最后在经过60秒删除。这个是和120重叠的，所以要比120大。