Use OpenSearch

Adding the service

1. Go to your application, choose an environment, and open the Services tab.
2. Add a new service and select OpenSearch X.
3. Give it a prefix. The default is DEFAULT. A different prefix is needed if the environment already has another search service.

4. Provision the service, then deploy the environment.

The service is only visible to your application after a deployment. Until then it shows as Pending attachment: it exists and is usable, but its environment variables have not been injected into your application yet.

Use one service per environment. Test and Live must never share an index prefix.

For more about how services are added, provisioned, attached, and removed, see the Services documentation.

For the environment variables provided by the service, see OpenSearch configuration.

Local development

Run OpenSearch locally with Docker rather than connecting your local application to the cloud service. Add this to your docker-compose.yml:

services:
  opensearch:
    image: opensearchproject/opensearch:3.7.0
    container_name: opensearch-local
    environment:
      - discovery.type=single-node
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      # Local development only: no TLS, no authentication.
      - DISABLE_INSTALL_DEMO_CONFIG=true
      - DISABLE_SECURITY_PLUGIN=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    ports:
      - "9200:9200"
    volumes:
      - opensearch-data:/usr/share/opensearch/data

volumes:
  opensearch-data:

Use the same version locally as the one provided by the service, which you can read from DEFAULT_SEARCH_VERSION.

On Linux hosts you may also need to run sysctl -w vm.max_map_count=262144 for the container to start.

The local cluster uses plain HTTP with no authentication and no index prefix. The examples below fall back to http://opensearch:9200 when DEFAULT_SEARCH_AUTH_SCHEME is not set, so the same settings file works locally and in the cloud.

Configuring your application

Whatever your language and framework, three rules apply:

• Use an official OpenSearch client. Elasticsearch clients do not work against the service, see Troubleshooting.
• Sign every request with SigV4. Most OpenSearch clients have built-in support for it.
• Keep every index name inside your prefix, and build the name without the trailing *.

Clients are available for Python, JavaScript, PHP, Java, Go, Ruby and .NET. See the OpenSearch clients documentation for the full list.

Python

Use opensearch-py together with boto3 for the request signing. The following helper works both locally and in the cloud:

import os
from urllib.parse import urlparse

import boto3
from opensearchpy import AWSV4SignerAuth, OpenSearch, RequestsHttpConnection

# Fall back to the local docker-compose cluster when no service is attached.
dsn = os.getenv("DEFAULT_SEARCH_DSN", "http://opensearch:9200")
parsed = urlparse(dsn)

aws_auth = os.getenv("DEFAULT_SEARCH_AUTH_SCHEME") == "aws-v4-auth"

# Managed AWS domains are HTTPS only. Do not rely on the DSN scheme for this:
# it can be prefixed (for example "es+https+aws://").
scheme = "https" if aws_auth else ("https" if "https" in parsed.scheme.split("+") else "http")
port = f":{parsed.port}" if parsed.port else ""
cluster_url = f"{scheme}://{parsed.hostname}{port}"

# INDEX_PREFIX is a pattern ending in "*". Index names cannot contain "*",
# so derive a concrete name inside the allowed prefix.
prefix = os.getenv("DEFAULT_SEARCH_INDEX_PREFIX", "").rstrip("*")
index_name = f"{prefix}default" if prefix else "myindex"


def aws_region_from_host(hostname):
    """search-<domain>-<hash>.<region>.es.amazonaws.com -> <region>"""
    if hostname.endswith((".es.amazonaws.com", ".aos.on.aws")):
        return hostname.split(".")[-4]
    raise ValueError(f"Cannot derive an AWS region from {hostname!r}")


def get_client():
    if not aws_auth:
        # Local development: plain HTTP, no authentication.
        return OpenSearch(hosts=[cluster_url], use_ssl=False)

    session = boto3.Session(
        aws_access_key_id=os.environ["DEFAULT_SEARCH_ACCESS_KEY_ID"],
        aws_secret_access_key=os.environ["DEFAULT_SEARCH_SECRET_ACCESS_KEY"],
    )
    return OpenSearch(
        hosts=[cluster_url],
        # AWSV4SignerAuth is a requests auth handler, so it must be paired
        # with RequestsHttpConnection.
        http_auth=AWSV4SignerAuth(
            session.get_credentials(),
            aws_region_from_host(parsed.hostname),
            service="es",
        ),
        connection_class=RequestsHttpConnection,
        use_ssl=True,
        verify_certs=True,
    )

Django

Django applications usually use django-haystack as an abstraction over the search engine. Use the django-haystack-opensearch backend, which is built on opensearch-py.

info

If your application currently uses django-haystack with an elasticsearch backend, and you cannot meet the requirements below, the migration guide describes a temporary alternative that does not require a Django upgrade.

Configuring django-haystack with OpenSearch

django-haystack-opensearch is a native OpenSearch backend for Haystack, built on opensearch-py. It supports OpenSearch 1.x to 3.x and needs no wrapper code of your own.

Requirements: Python >= 3.11, Django >= 5.2, django-haystack >= 3.3.0, opensearch-py >= 3.1.0.

Dependencies: remove elasticsearch and elasticsearch-dsl, then add django-haystack-opensearch, opensearch-py and boto3 (for SigV4).

warning

Two fixes are required on top of version 1.0.0 of the package. We found both while testing against our cluster, and both fail quietly:

• update() calls the bulk helper without an index, which sends the request to the cluster-level /_bulk endpoint. Your credentials are scoped to your index prefix, so that returns HTTP 403 and nothing is ever indexed. With Haystack's default SILENTLY_FAIL, rebuild_index still exits with code 0.
• the mapping helper drops the analyzer on every text field. EdgeNgramField and NgramField then match nothing, so autocomplete returns empty results, and the document field loses its snowball analyzer, which changes relevance.

Put this next to your settings, in myapp/backends.py:

from django_haystack_opensearch.haystack import (
    OpenSearchSearchBackend,
    OpenSearchSearchEngine,
)
from opensearchpy.helpers import bulk


class PatchedOpenSearchBackend(OpenSearchSearchBackend):
    def update(self, index, iterable, commit=True):
        # Upstream omits index=, which targets the cluster-level /_bulk
        # endpoint. A prefix-scoped policy denies that with a 403.
        if not self.setup_complete:
            self.setup()
        prepped_docs = self._prepare_documents_for_bulk(index, iterable)
        if prepped_docs:
            bulk(self.conn, prepped_docs, index=self.index_name)
        if commit:
            self.conn.indices.refresh(index=self.index_name)

    def _add_keyword_and_exact_subfields(self, props, unified_index):
        # Upstream replaces the whole definition for text fields, dropping
        # "analyzer" and with it EdgeNgram/Ngram matching.
        new_props = {}
        for field, definition in props.items():
            new_props[field] = dict(definition)
            if definition.get("type") == "text":
                subfields = dict(new_props[field].get("fields", {}))
                subfields["keyword"] = {"type": "keyword", "ignore_above": 256}
                new_props[field]["fields"] = subfields
        return new_props


class PatchedOpenSearchEngine(OpenSearchSearchEngine):
    backend = PatchedOpenSearchBackend

And in your settings.py:

import os
from urllib.parse import urlparse

import boto3
from opensearchpy import AWSV4SignerAuth, RequestsHttpConnection

# Fall back to the local docker-compose cluster when no service is attached.
dsn = os.getenv("DEFAULT_SEARCH_DSN", "http://opensearch:9200")
parsed = urlparse(dsn)

aws_auth = os.getenv("DEFAULT_SEARCH_AUTH_SCHEME") == "aws-v4-auth"

# Managed AWS domains are HTTPS only. Do not rely on the DSN scheme for this:
# it can be prefixed (for example "es+https+aws://").
scheme = "https" if aws_auth else ("https" if "https" in parsed.scheme.split("+") else "http")
port = f":{parsed.port}" if parsed.port else ""
cluster_url = f"{scheme}://{parsed.hostname}{port}"

# INDEX_PREFIX is a pattern ending in "*". Index names cannot contain "*",
# so derive a concrete name inside the allowed prefix.
prefix = os.getenv("DEFAULT_SEARCH_INDEX_PREFIX", "").rstrip("*")
index_name = f"{prefix}default" if prefix else "haystack"


def aws_region_from_host(hostname):
    """search-<domain>-<hash>.<region>.es.amazonaws.com -> <region>"""
    if hostname.endswith((".es.amazonaws.com", ".aos.on.aws")):
        return hostname.split(".")[-4]
    raise ValueError(f"Cannot derive an AWS region from {hostname!r}")


if aws_auth:
    session = boto3.Session(
        aws_access_key_id=os.environ["DEFAULT_SEARCH_ACCESS_KEY_ID"],
        aws_secret_access_key=os.environ["DEFAULT_SEARCH_SECRET_ACCESS_KEY"],
    )
    opensearch_kwargs = {
        # AWSV4SignerAuth is a requests auth handler, so it must be paired
        # with RequestsHttpConnection.
        "http_auth": AWSV4SignerAuth(
            session.get_credentials(),
            aws_region_from_host(parsed.hostname),
            service="es",
        ),
        "connection_class": RequestsHttpConnection,
        "use_ssl": True,
        "verify_certs": True,
    }
else:
    # Local development: plain HTTP, no authentication.
    opensearch_kwargs = {"use_ssl": False}

HAYSTACK_CONNECTIONS = {
    "default": {
        # The patched engine from myapp/backends.py, not the package's own.
        "ENGINE": "myapp.backends.PatchedOpenSearchEngine",
        "URL": cluster_url,
        "INDEX_NAME": index_name,
        "KWARGS": opensearch_kwargs,
    },
}

Everything in KWARGS is passed straight to the opensearch-py client constructor. Do not put timeout there, the backend already passes it. Use Haystack's own TIMEOUT key instead.

After this, the usual Haystack management commands work as expected: rebuild_index, update_index, clear_index.

Building and rebuilding indexes

The search index is derived from your primary database, so it can always be rebuilt. There is nothing to back up, and nothing is copied when you add a new service: a new index starts empty.

For Haystack projects, python manage.py rebuild_index builds it. You can run this over SSH, or as part of your release commands.

warning

Haystack's SILENTLY_FAIL defaults to true, so indexing errors are logged and swallowed instead of raised. An empty index looks exactly like a successful rebuild. Always check the document count after a rebuild, and set "SILENTLY_FAIL": False on the connection while you are testing.

For environment variables, authentication details, index naming rules, cluster limitations, and troubleshooting, see OpenSearch configuration.