ElasticSearch(ELK Stack) Security HowTo

Published on Author gryzli

Table of Contents

Security is one of the major “missing” things from the free ELK Stack compilation, so let’s talk about achieving it!

Soon or later there will come time, when you may want some more “Secure” ElasticSearch cluster, and by “Secure” I mean some of the following :

  • Encrypted communication between cluster nodes
  • Encrypted communication between “Indexing application/servers” and Elastic nodes
  • Encrypted communication between Kibana and ElasticSearch
  • Authentication – having different users/passwords for connecting to Elastic
  • Authorization   – having different user permissions and roles , things like:
    • Access of a given user to a specific Index
    • Different type of access level (write access to one index, read access to another) and so on….etc

 

Current Solutions For Achieving ElasticSearch / ELK Stack Security

 

1) ElasticStack X-Pack Security

I have never tried X-Pack Security, but considering the fact it is made by the elastic guys, it should be pretty handy of achieving all kind of Elastic related security.

By reading the documentation, seems like it provides almost anything you may need about successfully  securing your cluster. If you or your company are already paying for X-Pack, this may perfectly work for you.

Because X-Pack Security is NOT FREE , I will go on to different solutions for achieving security.

 

 

2) ReadOnlyREST

This is another solution which I have not tried still, but looks really promising.

The plugin is recommended in the ElasticSearch “Security Plugins” documentation section.

They also have all the blink blink features like encryption, authentication, authorization plus they have FREE TO USE version , but also paid plans.

Here you can find some more information about this plugin:

https://github.com/beshu-tech/readonlyrest-docs/blob/master/elasticsearch.md

 

3) Search-Guard

This is the solution which I have tried (and still using) and for now suites best my needs.

They are also selling this product, but having community FREE TO USE version as well (the one I use).

More information about what you can use in the free version, could be seen here:

https://search-guard.com/product/

 

The only one thing I don’t like about search guard, is that the documentation is terrible. That’s why I will try to write deep walk-through of the installation and configuration process, hoping to make some lives easier than main..

 

Installing and Configuring Search-Guard On Existing ElasticSearch Cluster

Here I assume you already have working ElasticSearch cluster. In the examples that follow, I’m using ES cluster consisting of 2 nodes (both being data and master eligible nodes).

Also the nodes are using ElasticSearch version 6.5.1 .

My current configuration is as follows:

 

 

1) Generating certificates for Search-Guard

The first thing to do is to prepare your SSL certificates for Search-Guard, you will need the following certificates:

For this tutorial I’m going to use search-guard tlstool for doing the following:

  1. Generate my own Certificate Signing Authority
  2. Generate/Sign self-signed signing certificate
  3. Generate CSR’s for all node certificates
  4. Sign all nodes certificates

 

(HINT) In this step you could actually proceed differently based on your opinion: 

  • You can use real CA signed certificates (or maybe LetsEncrypt or something like that)
  • You may use one certificate for both Data/HTTP channel

 

So let’s generate the certificates by using Search-Guard provided tools, which is really easy to do.

 

First you have to download search-guard tlstool from here:

https://search.maven.org/search?q=a:search-guard-tlstool

 

Extract the files 

 

Create configuration file for the tlstool

(Update the information for certificates according to your setup )

 

Now generate all the needed certificates

 

After executing the commands above, you should have directory “out” create in your current dir (tools)

 

Uploading necessary certificates to Elastic cluster nodes 

The next step is to upload the certificates each node will need to the appropriate cluster nodes.

 

For both nodes node1/node2 , you should do the following:

 

Fix permissions for the newly uploaded certificate files, here I assume your ElasticSearch is installed in the default (for Centos) directory (/etc/elasticsearch) and also that your local user is named “elasticsearch”.

Execute the following command on both elastic nodes:

 

Now we are ready to proceed with installing search-guard plugin for ElasticSearch and configure it futher.

 

2) Installing and configuring Search-Guard plugin for ElasticSearch

 

1) Disable cluster shard allocation 

 

2) Check which search-guard plugin version you need to install

Before installing the plugin, you need to make sure, what exact version you should use, based on your ElasticSearch cluster version. This could be checked from here:

https://docs.search-guard.com/latest/search-guard-versions

For my current example I’m using ElasticSearch 6.5.1 and the corresponding plugin version is: com.floragunn:search-guard-6:6.5.1-23.2

 

3) Stop ElasticSearch server on your cluster nodes 

 

4) Install search-guard plugin on both Node1/Node2

5) Add search-guard configuration to elasticsearch.yml  on Node1

We are going to first configure Node1 , from where we will re-enable cluster shard allocation and then initialize search-guard index, which is needed for storing search-guard roles/permissions and other related data.

At this point ElasticSearch is still stopped on both nodes !

5.1) Add search-guard configuration to Node1->elasticsearch.yml

Open the config and add the following to the end of the configuration file.

If your ElasticSearch comes with X-Pack integrated (default for Centos6/7 elasticsearch.rpm install) , make sure to DISABLE X-PACK SECURITY !

 

5.2) Start ElasticSearch server on Node1

 

5.3) Re-enable cluster shard allocation by using sgadmin tool

 

5.4) (Optional) Change the default admin password before initializing search-guard index

By default search-guard comes with pre-defined users and passwords, which are described in the following configuration file:

/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_internal_users.yml

If your cluster is publicly accessible, you may want to first change the default passwords for the users or maybe comment out some of the pre-defined users as well.

Here is how to change the default admin password, before starting Elastic and importing Search-Guard index.

1) Generate hash for the new password by using search-guard hash.sh

2) Copy the hash and replace it inside the user configuration file:

/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_internal_users.yml

 

This is everything you should do ,later after initializing the SG index, your passwords will be changed.

 

 

5.4) Initialize search-guard index by using sgadmin tool

 

At this point, your search-guard plugin is initially configured !

 

From now on, you should always use “https://” when communicating with the HTTP channel and also, you will need to provide basic auth user and password, to communicate with the cluster .

 

The default search-guard configuration is stored here:

/usr/share/elasticsearch/plugins/search-guard-6/sgconfig

 

The default admin credentials  are  user:admin and password:admin 

5.5) Validate cluster status, by using HTTPS and admin/admin credentials

(Here I’m piping the output of curl to “jq” which prettyfies JSON, if you don;t have it installed, you can remove it from the pipe, or just install it )

 

5.6) Add search-guard configuration settings to Node2 and start ElasticSearch

I assume you have already uploaded node2* + root-ca.pem certificates on Node2 (inside /etc/elasticsearch/ssl).

The only thing left is to add configuration to elasticsearch.yml and start ElasticSearch.

 

Add the following to your: /etc/elasticsearch/elasticsearch.yml on Node2 : 

 

Finally start elastic there:

 

5.7) Update the configuration of all your elasticsearch clients (logstash, kibana, scripts ..etc) 

After enabling search-guard, all your existing clients connecting through plain connection to ElasticSearch, would be unable to connect anymore.

You will need to either create separate users/user roles/ or use existing ones and configure your clients apropriate.

Authentication from clients happens by using HTTP Basic-Auth , so you should basically just update your Elastic URL’s, to look like this :

 

3) (Optional) Installing Search-Guard Kibana Plugin

By default search-guard comes with pre-created user for Kibana (if you have used the default sgconf before initializing your SG index), credentials for which are:

 

Also you are able to install search-guard module for Kibana, which will give you the following benefits:

  • GUI Interface for managing your SG roles / users and mappings
  • SG Login page for accessing Kibana through HTTP

Also keep in mind, that if you are using X-Pack->Monitoring (which is free) , it won’t work unless you install Kibana search-guard plugin (at least it didn’t worked for me before that – I was getting constant redirects).

 

1) Downloading Kibana Search-Guard plugin

The first thing to do is to download the correct version of search-guard plugin for Kibana.

You should search the plugin from here:

https://search.maven.org/search?q=g:com.floragunn%20AND%20a:search-guard-kibana-plugin&core=gav

 

I’m going to download and install the plugin on Node1 which in the current scenario would be used for hosting Kibana as well.

Download the plugin:

 

Make sure Kibana is stopped

 

Install the plugin (The optimizing and caching browser bundles part may take few minutes to complete)

 

Now add the following configuration to your kibana.yml (usually /etc/kibana/kibana.yml)

 

Finally start Kibana and go to your login URL , where you should see something like this:

Kibana Search Guard login screen
Kibana Search Guard login screen

For login you should use the default admin credentials (user: admin , pass: admin).

 

 

4) (Optional) Configure Logstash to authenticate in ElasticSearch + Search-Guard

If you already using Logstash, you should add some additional configuration to it in order to be able to communicate with the Elastic cluster after installing Search-Guard.

The steps you should take are as follows.

 

4.1) Upload root-ca.pem to your Logstash server

You could upload it to “/etc/logstash/root-ca.pem”

 

4.2) Update the configuration of all pipeline files, which use “elasticsearch” output

Your output definition should look like this:

I assume you are using the default search-guard configuration, where there is already user for logstash created, that has the following creds: 

 

4.3) Restart your Logstash and make sure it is connecting to your cluster (by observing /var/log/logstash/logstash-plain.log )

 

5) (BONUS) Creating Write Only User for ElasticSearch with Search-Guard

This was my first intention at the beginning I started to search for authentication/authorization plugins for ElasticSearch.

Understanding how roles_mappings and roles work in SG could be difficult, so I’m going to explain it here.

 

The Task

Let’s assume we have the following task

1) Generate password hash for our new user  (writeonly)

2) Create user called “writeonly”

3) Give write-only permissions to user “writeonly” and indexes matching the pattern “write_only*”

4) Apply the permissions to our ES cluster

 

Doing it

1) Generate password hash

First we need to generate hash for our password, which could be done by using search-guard hash.sh:

 

2) Add user information to sg_internal_users.yml

Here I’m using the Hash generated in the previous step.

As you may recall, here the role name is written as “write_only” instead of “sg_write_only”. That is how search-guard works (“sg_” is stripped from role names, when referring).

 

3) Create backend role granting write only access to our index “write_only”

The role name should be prefixed with “sg_”  , which is stripped during referring the role from another files (don’t know why this is like that).

 

4) Create role mapping to connect our user with our newly created backend role 

Here again, “sg_” is stripped when referring backend role.

 

5) Apply the new search-guard settings 

Finally we need to re-apply our search-guard configuration, by executing the following :

 

6) Test our user 

 

Try to index document inside “write_only” index:

The request is successfull.

 

Now try to read a document from Index (write_only)  (this should fail)

The request has failed – as expected.