Create rolling monthly, weekly and daily Logstash indices

在刚刚开始接触ELK的时候我们习惯把每一个index都按照day来切割。但是我们会发现我们的shards 会很多。

其实我们一该把那些小的index按照一周或者一个月来rolling，来减少我们的shards数。

我使用的是logstash5，这里我们每启动一个logstash的进程的时候我们会发现我们的jvm 参数的-Xmx1g -Xms1g

但是我么一些可以设置小一点，比如512m 甚至是256m 来减少资源分配

具体的操作方案：

原文来自于：http://www.atechref.com/blog/elk/elk-stack-logstash-mutate-and-monthly-index-setting/

How often should a new log index be created? Once a day, Once a week, Once a month? A simple search in Google would return various responses each arguing the pros and cons of creating indexes daily or weekly. Lets look at how to do that with logstash.

My take on that is “once a month” index is the best option.  The following is my reasoning for this.

Pros:

Easier back up with a monthly index
Simple to create snapshots and restore
One index to backup externally on NAS or other storage outside of ELK stack
Works well where the retention policy for active logs is 30 days or 60 days etc
Allows complete logs for the whole month to be restored in one go.
Cons:

 Potential for large index sizes
Have to restore the whole index to search for a specific day of the month
Backup and restore of these indexes can take some time in slower systems or single node instances
Creating a monthly rolling index file

In order to create a new index each month automatically ensure you have the following setting in your logstash config file for e.g. devlogstash.conf.

input{
  ….
  }
  filter{
   ….
  }
  output{
elasticsearch{

            hosts => [“192.168.0.1:9200”]
            index => “dev-logstash-%{+YYYY.MM}”

}

}
Creating a weekly rolling index file

The weekly name format would be YYYY.ww as in 2017.01 for the first week of the year in number.  Config setting would be as shown below.

input{
  ….
  }
  filter{
   ….
  }
  output{
elasticsearch{

            hosts => [“192.168.0.1:9200”]
             index => “dev-logstash-%{+YYYY.ww}”

}

}
Creating a daily rolling index file

Just add MM.DD instead of WW to the setting above to create a daily rolling index as shown below.

input{
  ….
  }
  filter{
   ….
  }
  output{
elasticsearch{

         hosts => [“192.168.0.1:9200”]
         index => “dev-logstash-%{+YYYY.MM.DD}”

}

}
Creating a Year, month and week rolling index file

And that could be defined as YYYY.MM.ww to create a weekly rolling index as shown below.

input{
  ….
  }
  filter{
   ….
  }
  output{
elasticsearch{

         hosts => [“192.168.0.1:9200”]
         index => “dev-logstash-%{+YYYY.MM.ww}”

}

}
 Restart logstash for these changes to take effect.