Table of Contents
At certain point in time, you will want to rotate (delete) your old indexes in ElasticSearch.
Most of the time people are using time based naming convention for their index names like: index_name-Year-Month-Day or index_name-Year.Month.Day.
Tools like Filebeat/Logstash can also use such naming conventions.
Before you go and write your own script that will do the rotation, you better consider curator !
Curator is maintained by Elastic guys and looks like the perfect tool for doing index rotation.
How-To Install Elasticsearch Curator On Centos 7
You can install curator both from YUM (epel repo) or manually by using python pip. I suggest you to use the latter (pip install) as a better way of doing the installation.
By installing directly from pip, you will get the latest version of curator (5.x right now).
If you go with YUM, be warned that the EPEL version is too old (3.2.3 as of wrting this).
Easily Install curator by using python pip
yum install python2-pip pip install elasticsearch-curator
You are done !
Now you will be able to execute the following commands
Examples on Using Curator For Index Rotation (Deleting Old Indices)
Step 1) Create curator-config.yml config file
Before going further, you need to have some general curator-config.yml file.
In this example I’m going to run curator on the same server where my ElasticSearch instance is installed. That’s why the hosts configuration is ‘localhost’.
client: hosts: - localhost port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: aws_key: aws_secret_key: aws_region: ssl_no_validate: False http_auth: timeout: 100 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: ['elasticsearch']
Step 2) Create curator-action.yml config file
In this file, we must define what kind of actions we require curator to take.
In this example I have indexes named like this:
As shown, my index format is shcpu-stats-%Y-%m-%d.
Let say I want to delete all indexes older than 30 days, I have to create the following action config file:
actions: 1: action: delete_indices description: >- Delete shcpu-stats indexes older than 30 days options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: shcpu-stats- exclude: - filtertype: age source: name direction: older timestring: '%Y-%m-%d' unit: days unit_count: 30 exclude:
Pay attention, that I have defined my index format under
timestring: section of curator-action.yml file.
If your indexes look like : some_index-2019.11.22 , you may want to use
Step 3) Executing curator
Now it is time to run curator in dry-run mode and make sure it is going to delete the correct indexes we want.
curator curator-action.yml --config curator-config.yml --dry-run
If you are happy with the result from the dry-run, you can finally run the tool without dry-run option and see it work.