Rotating Indexes in ElasticSearch with Curator

Published on Author gryzli

At certain point in time, you will want to rotate (delete) your old indexes in ElasticSearch.

Most of the time people are using time based naming convention for their index names like: index_name-Year-Month-Day or index_name-Year.Month.Day.

Tools like Filebeat/Logstash can also use such naming conventions.

Before you go and write your own script that will do the rotation, you better consider curator !

Curator is maintained by Elastic guys and looks like the perfect tool for doing index rotation.

 

How-To Install Elasticsearch Curator On Centos 7

You can install curator both from YUM (epel repo) or manually by using python pip. I suggest you to use the latter (pip install) as a better way of doing the installation.

By installing directly from pip, you will get the latest version of curator (5.x right now).

If you go with YUM, be warned that the EPEL version is too old (3.2.3 as of wrting this).

 

Easily Install curator by using python pip

You are done !

Now you will be able to execute the following commands curator and curator_cli

 

Examples on Using Curator For Index Rotation (Deleting Old Indices)

 

Step 1) Create curator-config.yml config file

Before going further, you need to have some general curator-config.yml file.

In this example I’m going to run curator on the same server where my ElasticSearch instance is installed. That’s why the hosts configuration is ‘localhost’.

 

vim curator-config.yml 

 

 

Step 2) Create curator-action.yml config file

In this file, we must define what kind of actions we require curator to take.

In this example I have indexes named like this:

Using curator_cli to list elasticsearch indices

As shown, my index format is shcpu-stats-%Y-%m-%d.

Let say I want to delete all indexes older than 30 days, I have to create the following action config file:

vim curator-action.yml 

Pay attention, that I have defined my index format under timestring: section of curator-action.yml file.

If your indexes look like : some_index-2019.11.22 , you may want to use timestring: '%Y.%m.%d'

 

 

Step 3) Executing curator

Now it is time to run curator in dry-run mode and make sure it is going to delete the correct indexes we want.

curator curator-action.yml --config curator-config.yml --dry-run

Running curator in dry-run mode
Running curator in dry-run mode

If you are happy with the result from the dry-run, you can finally run the tool without dry-run option and see it work.