Rotating Indexes in ElasticSearch with Curator

At certain point in time, you will want to rotate (delete) your old indexes in ElasticSearch.

Most of the time people are using time based naming convention for their index names like: index_name-Year-Month-Day or index_name-Year.Month.Day.

Tools like Filebeat/Logstash can also use such naming conventions.

Before you go and write your own script that will do the rotation, you better consider curator !

Curator is maintained by Elastic guys and looks like the perfect tool for doing index rotation.

 

How-To Install Elasticsearch Curator On Centos 7

You can install curator both from YUM (epel repo) or manually by using python pip. I suggest you to use the latter (pip install) as a better way of doing the installation.

By installing directly from pip, you will get the latest version of curator (5.x right now).

If you go with YUM, be warned that the EPEL version is too old (3.2.3 as of wrting this).

 

Easily Install curator by using python pip

yum install python2-pip
pip install elasticsearch-curator

You are done !

Now you will be able to execute the following commands curator and curator_cli

 

Examples on Using Curator For Index Rotation (Deleting Old Indices)

 

Step 1) Create curator-config.yml config file

Before going further, you need to have some general curator-config.yml file.

In this example I’m going to run curator on the same server where my ElasticSearch instance is installed. That’s why the hosts configuration is ‘localhost’.

 

vim curator-config.yml 

client:
  hosts:
    - localhost
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  aws_key:
  aws_secret_key:
  aws_region:
  ssl_no_validate: False
  http_auth:
  timeout: 100
  master_only: False
logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch']

 

 

Step 2) Create curator-action.yml config file

In this file, we must define what kind of actions we require curator to take.

In this example I have indexes named like this:

As shown, my index format is shcpu-stats-%Y-%m-%d.

Let say I want to delete all indexes older than 30 days, I have to create the following action config file:

vim curator-action.yml 

actions:
  1:
    action: delete_indices
    description: >-
      Delete shcpu-stats indexes older than 30 days 
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: shcpu-stats-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 30
      exclude:

Pay attention, that I have defined my index format under timestring: section of curator-action.yml file.

If your indexes look like : some_index-2019.11.22 , you may want to use timestring: '%Y.%m.%d'

 

 

Step 3) Executing curator

Now it is time to run curator in dry-run mode and make sure it is going to delete the correct indexes we want.

curator curator-action.yml --config curator-config.yml --dry-run

If you are happy with the result from the dry-run, you can finally run the tool without dry-run option and see it work.