Securely and Programmatically Accessing Elasticsearch with curl and Python

In this blog post, you will learn how to securely and programmatically access Elasticsearch with Curl and Python. When you install the current version of Elasticsearch, which is 8.11.1, the security is enabled by default. To get started securely working with Elasticsearch you must specify the Certificate Authority (CA) certificate, and you must authenticate to the service, which means you need to know the default user, which is ‘elastic’ and the password.

I assume that your elasticsearch.yml configuration file has the default settings, and it looks similar to Figure 1. Notice all the security features are enabled.

Figure 1

Elasticsearch.yml with security features enabled.

Specifying the CA certificate when using Curl

The default CA certificate is stored in the <Elasticsearch home directory>/config/certs/http_ca.crt

Assuming that you have Elasticsearch up and running and the Elasticsearch home directory is in your current working directory, you can open up a terminal and run the command:

curl --cacert elasticsearch-8.11.1/config/certs/http_ca.crt -XGET "https://localhost:9200?pretty"

You will see the output as seen in Figure 2 (“[Elasticsearch 8.3] Problem with the SSL encryption in the master node”, 2022).

Figure 2

Invoking Elasticsearch with curl and specifying the CA cert.

You will notice that the response contains JavaScript Object Notation (JSON) that contains an error message, which informs us that we are not authenticated (“What is JSON”, n.d.).

Authenticating to Elasticsearch with Curl

The first time you ran Elasticsearch; you should have seen a screen similar to Figure 3, which states the password for the ‘elastic’ user. If you did not jot down the password when you first ran Elasticsearch, then you’ll have to reset it.

Figure 3

Screen after first run of Elasticsearch from the terminal, which contains password for ‘elastic’ user.

From the screen, I can see that the password for the user ‘elastic’ is C3iJZCogfZCxFrQE2E6t, which means I will use the following command to invoke elasticsearch with curl.

curl -u elastic:C3iJZCogfZCxFrQE2E6t --cacert elasticsearch-8.11.1/config/certs/http_ca.crt "https://localhost:9200"

The -u switch specifies the username and password delimited by a colon.

Resetting the Password (if you need to)

If you didn’t catch the password for the user ‘elastic’ when you first ran elasticsearch, you can reset the password. All you need to do is run the elasticsearch-reset-password utility in the bin directory of the Elasticsearch home directory. If you run it with the -u switch, which specifies the user you want to reset the password for, it will reset the password for that user and display it as seen in Figure 4.

Figure 4

Running the elasticsearch-reset-password utility from the bin directory in elasticsearch home directory (“Missing authentication credentials for REST request”, 2022).

Once you can successfully invoke elasticsearch with the security features enabled, whenever you invoke the elasticsearch service with Curl, you need to specify the username, password, and CA cert.

The following curl command will insert the following JSON document into an index called rss:

{
  “fulltext” : “the quick brown fox jumped over the lazy dog”
}

curl -u elastic:fL1F3_72R7tD8KwlvjQC --cacert ../config/certs/http_ca.crt -H "Content-Type: application/json" "https://localhost:9200/rss/_doc" -d '{"fulltext" : "the quick brown fox jumped over the lazy dog"}'

The -H switch adds the Content-Type header, which is application/json. The -d specifies the content of the request, which is the JSON document to be inserted.

Securely and Programmatically Accessing Elasticsearch with Python

In order to securely and programmatically access elasticsearch with python, in your project directory, run the following commands in the terminal:

$ pipenv shell
$ pip install elasticsearch

In the same directory, create a python file for your python code, and open it in your favorite Integrated Development Environment (IDE). Write the following code:

from elasticsearch import Elasticsearch
from dotenv import load_dotenv
import os

def main():
    load_dotenv()
    es_password = os.environ.get("ELASTIC_PASSWORD")

    es = Elasticsearch("https://localhost:9200",
        ca_certs="/Users/gcdrocella/tmp/elasticsearch-8.11.1/config/certs/http_ca.crt", 
        basic_auth=("elastic", es_password))

    print(str(es.info()))

main()

The above script makes use of the load_dotenv function, which expects there to be a .env file in the directory of the script with a variable called ELASTIC_PASSWORD that is set to the Elasticsearch authentication password. Calling load_dotenv will load the ELASTIC_PASSWORD as an environment variable.

When instantiating the Elasticsearch object, you pass in the argument “https://localhost:9200”, which specifies the host, port, and to use the HyperText Transfer Protocol Secure (HTTPS), which will encrypt the channel (“Connecting”, n.d.). Also, when instantiating the Elasticsearch object, the keyword argument ca_certs is set to the absolute file path to the http_ca.crt, and basic_auth is set to a tuple where the first value is the username ‘elastic’ and the second value is the password for the ‘elastic’ user. After the Elasticsearch object is instantiated, any of the Application Programming Interface (API) can be invoked as usual.

In this blog post, you learned how to securely and programmatically access Elasticsearch locally with curl and python by configuring the CA certificate and HTTP basic authentication username and password.

Thanks for reading! For more blog posts just like this, subscribe to my blog, and buy me a coffee.

References

Connecting. (n.d.). Elastic.Co. Retrieved November 29, 2023, from https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html

[Elasticsearch 8.3] Problem with the SSL encryption in the master node. (2022, August 5). Discuss the Elastic Stack. https://discuss.elastic.co/t/elasticsearch-8-3-problem-with-the-ssl-encryption-in-the-master-node/311499

Missing authentication credentials for REST request. (2022, August 3). Discuss the Elastic Stack. https://discuss.elastic.co/t/missing-authentication-credentials-for-rest-request/311338What is JSON. (n.d.). W3schools.com. Retrieved November 29, 2023, from https://www.w3schools.com/whatis/whatis_json.asp

What is JSON. (n.d.). W3schools.com. Retrieved November 29, 2023, from https://www.w3schools.com/whatis/whatis_json.asp