Elasticsearch Service

    The Elasticsearch Service in LocalStack lets you create one or more single-node Elasticsearch/OpenSearch cluster that behaves like the Amazon Elasticsearch Service. This service is, like its AWS counterpart, heavily linked with the . Any cluster created with the Elasticsearch Service will show up in the OpenSearch Service and vice versa.

    You can go ahead and use awslocal to create a new elasticsearch domain via the command.

    Note: Unless you use the Elasticsearch default version, the first time you create a cluster with a specific version, the Elasticsearch binary is downloaded, which may take a while to download.

    Note: The default Elasticsearch version used is 7.10.0. This is a slight deviation from the default version used in AWS (Elasticsearch 1.5), which is not supported in LocalStack.

    In the LocalStack log you will see something like the following, where you can see the cluster starting up in the background.

    1. 2021-11-08T16:29:28:INFO:localstack.services.es.cluster: starting elasticsearch: /opt/code/localstack/localstack/localstack/infra/elasticsearch/bin/elasticsearch -E http.port=57705 -E http.publish_port=57705 -E transport.port=0 -E network.host=127.0.0.1 -E http.compression=false -E path.data="/var/lib/localstack/lib//elasticsearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/data" -E path.repo="/var/lib/localstack/lib//elasticsearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/backup" -E xpack.ml.enabled=false with env {'ES_JAVA_OPTS': '-Xms200m -Xmx600m', 'ES_TMPDIR': '/var/lib/localstack/lib//elasticsearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/tmp'}
    2. 2021-11-08T16:29:28:INFO:localstack.services.es.cluster: registering an endpoint proxy for http://my-domain.us-east-1.es.localhost.localstack.cloud:4566 => http://127.0.0.1:57705
    3. 2021-11-08T16:29:30:INFO:localstack.services.es.cluster: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
    4. 2021-11-08T16:29:32:INFO:localstack.services.es.cluster: [2021-11-08T16:29:32,502][INFO ][o.e.n.Node               ] [noctua] version[7.10.0], pid[22403], build[default/tar/51e9d6f22758d0374a0f3f5c6e8f3a7997850f96/2020-11-09T21:30:33.964949Z], OS[Linux/5.4.0-89-generic/amd64], JVM[Ubuntu/OpenJDK 64-Bit Server VM/11.0.11/11.0.11+9-Ubuntu-0ubuntu2.20.04]
    5. 2021-11-08T16:29:32:INFO:localstack.services.es.cluster: [2021-11-08T16:29:32,510][INFO ][o.e.n.Node               ] [noctua] JVM home [/usr/lib/jvm/java-11-openjdk-amd64], using bundled JDK [false]
    6. 2021-11-08T16:29:32:INFO:localstack.services.es.cluster: [2021-11-08T16:29:32,511][INFO ][o.e.n.Node               ] [noctua] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/var/lib/localstack/lib//elasticsearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms200m, -Xmx600m, -XX:MaxDirectMemorySize=314572800, -Des.path.home=/opt/code/localstack/localstack/localstack/infra/elasticsearch, -Des.path.conf=/opt/code/localstack/localstack/localstack/infra/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
    7. 2021-11-08T16:29:36:INFO:localstack.services.es.cluster: [2021-11-08T16:29:36,258][INFO ][o.e.p.PluginsService     ] [noctua] loaded module [aggs-matrix-stats]
    8. 2021-11-08T16:29:36:INFO:localstack.services.es.cluster: [2021-11-08T16:29:36,259][INFO ][o.e.p.PluginsService     ] [noctua] loaded module [analysis-common]
    9. 2021-11-08T16:29:36:INFO:localstack.services.es.cluster: [2021-11-08T16:29:36,260][INFO ][o.e.p.PluginsService     ] [noctua] loaded module [constant-keyword]
    10. ...

    and after some time, you should see that the Processing state of the domain is set to false:

    1. $ awslocal es describe-elasticsearch-domain --domain-name my-domain | jq ".DomainStatus.Processing"
    2. false

    You can now interact with the cluster at the cluster API endpoint for the domain, in this case http://my-domain.us-east-1.es.localhost.localstack.cloud:4566.

    1. $ curl http://my-domain.us-east-1.es.localhost.localstack.cloud:4566
    2. {
    3.   "name" : "localstack",
    4.   "cluster_name" : "elasticsearch",
    5.   "cluster_uuid" : "IC7E9daNSiepRBB9Ksul7w",
    6.   "version" : {
    7.     "number" : "7.10.0",
    8.     "build_flavor" : "default",
    9.     "build_type" : "tar",
    10.     "build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
    11.     "build_snapshot" : false,
    12.     "lucene_version" : "8.7.0",
    13.     "minimum_wire_compatibility_version" : "6.8.0",
    14.     "minimum_index_compatibility_version" : "6.0.0-beta1"
    15.   },
    16.   "tagline" : "You Know, for Search"
    17. }

    Or the health endpoint:

    There are three configurable strategies that govern how domain endpoints are created, and can be configured via the OPENSEARCH_ENDPOINT_STRATEGY (previously ES_ENDPOINT_STRATEGY) environment variable.

    Regardless of the service from which the clusters were created, the domain of the cluster always corresponds to the engine type (OpenSearch or Elasticsearch) of the cluster. OpenSearch cluster therefore have opensearch in their domain (e.g. my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566) and Elasticsearch clusters have es in their domain (e.g. my-domain.us-east-1.es.localhost.localstack.cloud:4566)

    Custom Endpoints

    LocalStack allows you to set arbitrary custom endpoints for your clusters in the domain endpoint options. This can be used to overwrite the behavior of the endpoint strategies described above. You can also choose custom domains, however it is important to add the edge port (80/443 or by default 4566).

    1. $ awslocal es create-elasticsearch-domain --domain-name my-domain \
    2.     --elasticsearch-version 7.10 \
    3.     --domain-endpoint-options '{ "CustomEndpoint": "http://localhost:4566/my-custom-endpoint", "CustomEndpointEnabled": true }'

    Once the domain processing is complete, you can access the cluster:

    1. $ curl http://localhost:4566/my-custom-endpoint/_cluster/health

    In some cases, you may not want to create a new cluster instance for each domain, for example when you are only interested in testing API interactions instead of actual Elasticsearch functionality. In this case, you can set OPENSEARCH_MULTI_CLUSTER=0 (previously ES_MULTI_CLUSTER). This will multiplex all domains to the same cluster, or return the same port every time when using the port endpoint strategy. This can however lead to unexpected behavior when persisting data into Elasticsearch, or creating clusters with different versions, so we do not recommend it.

    Elasticsearch will be organized in your state directory as follows:

    1. localstack@machine % tree -L 4 volume/state
    2. .
    3. ├── elasticsearch
    4. │   └── arn:aws:es:us-east-1:000000000000:domain
    5. │       ├── my-cluster-1
    6. │       │   ├── backup
    7. │       │   ├── data
    8. │       │   └── tmp
    9. │       ├── my-cluster-2
    10. │       │   ├── backup
    11. │       │   ├── data
    12. │       │   └── tmp

    To customize the elasticsearch backend, you can your own elasticsearch cluster locally and point localstack to it using the OPENSEARCH_CUSTOM_BACKEND (previously ES_CUSTOM_BACKEND) environment variable. Note that only a single backend can be configured, meaning that you will get a similar behavior as when you re-use a single cluster instance.

    The following shows a sample docker-compose file that contains a single-noded elasticsearch cluster and a basic localstack setp.

    1. Run docker compose:

      1. $ docker-compose up -d

    2. Create the Elasticsearch domain:

      1. $ awslocal es create-elasticsearch-domain \
      2.     --domain-name mylogs-2 \
      4.     --elasticsearch-cluster-config '{ "InstanceType": "m3.xlarge.elasticsearch", "InstanceCount": 4, "DedicatedMasterEnabled": true, "ZoneAwarenessEnabled": true, "DedicatedMasterType": "m3.xlarge.elasticsearch", "DedicatedMasterCount": 3}'
      5. {
      6.     "DomainStatus": {
      7.         "DomainName": "mylogs-2",
      8.         "ARN": "arn:aws:es:us-east-1:000000000000:domain/mylogs-2",
      9.         "Created": true,
      10.         "Deleted": false,
      11.         "Endpoint": "mylogs-2.us-east-1.es.localhost.localstack.cloud:4566",
      12.         "Processing": true,
      13.         "ElasticsearchVersion": "7.10",
      14.         "ElasticsearchClusterConfig": {
      15.             "InstanceType": "m3.xlarge.elasticsearch",
      16.             "InstanceCount": 4,
      17.             "DedicatedMasterEnabled": true,
      18.             "ZoneAwarenessEnabled": true,
      19.             "DedicatedMasterType": "m3.xlarge.elasticsearch",
      20.             "DedicatedMasterCount": 3
      21.         },
      22.         "EBSOptions": {
      23.             "EBSEnabled": true,
      24.             "VolumeType": "gp2",
      25.             "VolumeSize": 10,
      26.             "Iops": 0
      27.         },
      28.         "CognitoOptions": {
      29.             "Enabled": false
      30.         }
      31.     }
      32. }

    3. If the Processing status is true, it means that the cluster is not yet healthy. You can run describe-elasticsearch-domain to receive the status:

      1. $ awslocal es describe-elasticsearch-domain --domain-name mylogs-2

    4. Check the cluster health endpoint and create indices:

    5. Create an example index:

      1. $ curl -X PUT mylogs-2.us-east-1.es.localhost.localstack.cloud:4566/my-index
      2. {"acknowledged":true,"shards_acknowledged":true,"index":"my-index"}
    • The CustomEndpointOptions allows arbitrary endpoint URLs, which is not allowed in AWS