OpenSearch Service

    The OpenSearch Service in LocalStack lets you create one or more single-node OpenSearch clusters that behave like the Amazon OpenSearch Service. This service is, like its AWS counterpart, heavily linked with the . Any cluster created with the OpenSearch Service will show up in the Elasticsearch Service and vice versa.

    You can go ahead and use awslocal to create a new OpenSearch domain via the command.

    Note: Every time when you create a cluster with a version of OpenSearch you haven’t used before, the OpenSearch binary for the respective version needs to be downloaded, which may take a while.

    Note: The default OpenSearch version used is 1.1.0.

    In the LocalStack log you will see something like, where you can see the cluster starting up in the background.

    1. 2022-01-13T10:36:29.436:INFO:localstack.services.opensearch.cluster: starting opensearch: /var/lib/localstack/libs/opensearch/1.1.0/bin/opensearch -E http.port=35403 -E http.publish_port=35403 -E transport.port=0 -E network.host=127.0.0.1 -E http.compression=false -E path.data="/var/lib/localstack/opensearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/data" -E path.repo="/var/lib/localstack/opensearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/backup" -E plugins.security.disabled=true with env {'OPENSEARCH_JAVA_OPTS': '-Xms200m -Xmx600m', 'OPENSEARCH_TMPDIR': '/var/lib/localstack/opensearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/tmp'}
    2. 2022-01-13T10:36:29.437:INFO:localstack.services.opensearch.cluster: registering an endpoint proxy for http://my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566 => http://127.0.0.1:35403
    3. 2022-01-13T10:36:32.803:INFO:localstack.services.opensearch.cluster: [2022-01-13T10:36:32,800][INFO ][o.o.n.Node               ] [host-pc] version[1.1.0], pid[231895], build[tar/15e9f137622d878b79103df8f82d78d782b686a1/2021-10-04T21:29:03.079792Z], OS[Linux/5.11.0-46-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
    4. 2022-01-13T10:36:32.805:INFO:localstack.services.opensearch.cluster: [2022-01-13T10:36:32,805][INFO ][o.o.n.Node               ] [host-pc] JVM home [/var/lib/localstack/libs/opensearch/1.1.0/jdk], using bundled JDK [true]
    5. 2022-01-13T10:36:32.806:INFO:localstack.services.opensearch.cluster: [2022-01-13T10:36:32,805][INFO ][o.o.n.Node               ] [host-pc] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/lib/localstack/opensearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms200m, -Xmx600m, -XX:MaxDirectMemorySize=314572800, -Dopensearch.path.home=/var/lib/localstack/libs/opensearch/1.1.0, -Dopensearch.path.conf=/var/lib/localstack/libs/opensearch/1.1.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
    6. ...

    and after some time, you should see that the Processing state of the domain is set to false:

    1. $ awslocal opensearch describe-domain --domain-name my-domain | jq ".DomainStatus.Processing"
    2. false

    Like in AWS, the OpenSearch service can create Elasticsearch clusters and manage them. To do so, you can use and select an Elasticsearch version with the --engine-version parameter of the awslocal opensearch create-domain command. For an overview of existing Elasticsearch versions you can use awslocal opensearch list-versions.

    Interact with the cluster

    You can now interact with the cluster at the cluster API endpoint for the domain, in this case http://my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566.

    1. $ curl http://my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566
    2. {
    3.   "name" : "host-pc",
    4.   "cluster_name" : "opensearch",
    5.   "cluster_uuid" : "DMN-2TlwRkuhMH4aRRqrkA",
    6.   "version" : {
    7.     "distribution" : "opensearch",
    8.     "number" : "1.1.0",
    9.     "build_type" : "tar",
    10.     "build_hash" : "15e9f137622d878b79103df8f82d78d782b686a1",
    11.     "build_date" : "2021-10-04T21:29:03.079792Z",
    12.     "build_snapshot" : false,
    13.     "lucene_version" : "8.9.0",
    14.     "minimum_wire_compatibility_version" : "6.8.0",
    15.     "minimum_index_compatibility_version" : "6.0.0-beta1"
    16.   },
    17.   "tagline" : "The OpenSearch Project: https://opensearch.org/"
    18. }

    Or the health endpoint:

    There are two configurable strategies that govern how domain endpoints are created. The strategy can be configured via the OPENSEARCH_ENDPOINT_STRATEGY environment variable.

    Regardless of the service from which the clusters were created, the domain of the cluster always corresponds to the engine type (OpenSearch or Elasticsearch) of the cluster. OpenSearch cluster therefore have opensearch in their domain (e.g. my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566) and Elasticsearch clusters have es in their domain (e.g. my-domain.us-east-1.es.localhost.localstack.cloud:4566)

    Custom Endpoints

    LocalStack allows you to set arbitrary custom endpoints for your clusters in the domain endpoint options. This can be used to overwrite the behavior of the endpoint strategies described above. You can also choose custom domains, however it is important to add the edge port (80/443 or by default 4566).

    1. $ awslocal opensearch create-domain --domain-name my-domain \

    Once the domain processing is complete, you can access the cluster:

    1. $ curl http://localhost:4566/my-custom-endpoint/_cluster/health

    In some cases, you may not want to create a new cluster instance for each domain, for example when you are only interested in testing API interactions instead of actual OpenSearch functionality. In this case, you can set OPENSEARCH_MULTI_CLUSTER=0, which will multiplex all domains to the same cluster. This can however lead to unexpected behavior when persisting data into OpenSearch, or creating clusters with different versions, so we do not recommend it.

    OpenSearch will be organized in your state directory as follows:

    1. localstack@machine % tree -L 4 ./volume/state
    2. ./volume/state
    3. ├── opensearch
    4. │       ├── my-cluster-1
    5. │       │   ├── backup
    6. │       │   ├── data
    7. │       │   └── tmp
    8. │       ├── my-cluster-2
    9. │       │   ├── backup
    10. │       │   ├── data
    11. │       │   └── tmp

    Custom OpenSearch backends

    LocalStack downloads OpenSearch asynchronously the first time you run the aws opensearch create-domain, so you will get the response from LocalStack first and then (after download/install) you will have your OpenSearch cluster running locally. You may not want this, and instead use your already running OpenSearch cluster. This can also be useful when you want to run a cluster with a custom configuration that LocalStack does not support.

    The following shows a sample docker-compose.yaml file that contains a single-node OpenSearch cluster and a basic LocalStack setup.

    1. Run docker compose:

      1. $ docker-compose up -d

    2. Create the OpenSearch domain:

      1. $ awslocal opensearch create-domain --domain-name my-domain
      2. {
      3.     "DomainStatus": {
      4.         "DomainId": "000000000000/my-domain",
      5.         "DomainName": "my-domain",
      6.         "ARN": "arn:aws:es:us-east-1:000000000000:domain/my-domain",
      7.         "Created": true,
      8.         "Deleted": false,
      9.         "Endpoint": "my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566",
      10.         "Processing": false,
      11.         "UpgradeProcessing": false,
      12.         "EngineVersion": "OpenSearch_1.1",
      13.         "ClusterConfig": {
      14.             "InstanceType": "m3.medium.search",
      15.             "InstanceCount": 1,
      16.             "DedicatedMasterEnabled": true,
      17.             "ZoneAwarenessEnabled": false,
      18.             "DedicatedMasterType": "m3.medium.search",
      19.             "DedicatedMasterCount": 1,
      20.             "WarmEnabled": false,
      21.             "ColdStorageOptions": {
      22.                 "Enabled": false
      23.             }
      24.         },
      25.         "EBSOptions": {
      26.             "EBSEnabled": true,
      27.             "VolumeType": "gp2",
      28.             "VolumeSize": 10,
      29.             "Iops": 0
      30.         },
      32.         "SnapshotOptions": {
      33.             "AutomatedSnapshotStartHour": 0
      34.         "CognitoOptions": {
      35.             "Enabled": false
      36.         },
      37.         "EncryptionAtRestOptions": {
      38.             "Enabled": false
      39.         },
      40.         "NodeToNodeEncryptionOptions": {
      41.             "Enabled": false
      42.         },
      43.         "AdvancedOptions": {
      44.             "override_main_response_version": "false",
      45.             "rest.action.multi.allow_explicit_index": "true"
      46.         },
      47.         "ServiceSoftwareOptions": {
      48.             "CurrentVersion": "",
      49.             "NewVersion": "",
      50.             "UpdateAvailable": false,
      51.             "Cancellable": false,
      52.             "UpdateStatus": "COMPLETED",
      53.             "Description": "There is no software update available for this domain.",
      54.             "AutomatedUpdateDate": 0.0,
      55.             "OptionalDeployment": true
      56.         },
      57.         "DomainEndpointOptions": {
      58.             "EnforceHTTPS": false,
      59.             "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07",
      60.             "CustomEndpointEnabled": false
      61.         },
      62.         "AdvancedSecurityOptions": {
      63.             "Enabled": false,
      64.             "InternalUserDatabaseEnabled": false
      65.         },
      66.         "AutoTuneOptions": {
      67.             "State": "ENABLE_IN_PROGRESS"
      68.         }
      69.     }
      70. }

    3. If the Processing status is true, it means that the cluster is not yet healthy. You can run decribe-domain to receive the status:

      1. $ awslocal opensearch describe-domain --domain-name my-domain

    4. Check the cluster health endpoint and create indices:

    5. Create an example index:

      1. $ curl -X PUT my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566/my-index
      2. {"acknowledged":true,"shards_acknowledged":true,"index":"my-index"}
    • By default, AWS only sets the Endpoint attribute of the cluster status once the cluster is up. LocalStack will return the endpoint immediately, but keep Processing = "true" until the cluster has been started.

    Troubleshooting

    If you are using the OPENSEARCH_ENDPOINT_STRATEGY=domain (which is the default) and are having issues with resolving the subdomains, please check if your DNS blocks rebind queries.