How to migrate from Elasticsearch to self-service OpenSearch

This guide is only about the 2026 migration to self-service

This page covers the one-time move from our legacy Elasticsearch clusters to the new self-service OpenSearch service in 2026. It will be removed once the legacy clusters are gone.

If you are setting up search on a new application, or you are looking for the reference documentation of the service (environment variables, authentication, index naming, limitations, configuration examples), read Search with OpenSearch instead.

Divio is replacing the legacy Elasticsearch clusters (Elasticsearch 2.3 and 7.10) with a single managed OpenSearch 3 cluster.

Two things change for you:

• Search becomes self-service. You can add, provision, and remove the search service yourself from the Services tab of your environment. You no longer need to ask our support team to provision it.
• The connection details change. The new service provides a new set of environment variables, and authentication is now AWS SigV4 instead of a username and password in a URL.

Your application code needs to be updated before the old clusters are switched off. This guide walks you through it.

warning

The legacy Elasticsearch clusters will be decommissioned on November 6, 2026. After that date, applications still pointing at them will stop returning search results. Please start with your Test environment as soon as the new service is available.

Before you start

A few things that are useful to know before you plan the work:

• There is no data migration. Search indexes are built from your primary database, so you do not need to export or restore any snapshot. The migration is: update your code, point it at the new service, rebuild the index.
• The two clusters can run side by side. You can add the OpenSearch service while your application still uses the old Elasticsearch one, so you can prepare and test without downtime.
• Search will be unavailable until you rebuild the index. The new index starts empty. Plan the rebuild right after the deployment that switches the connection.
• Do it per environment. Add one OpenSearch service to each environment (Test, Live). Never share an index between environments.

If you are coming from Elasticsearch 2.3, expect real code changes: mapping types, several query constructs, and the client library have all changed since then. If you are coming from Elasticsearch 7.10, the main changes are the client library and the authentication.

Step 1: Add the OpenSearch service

info

The OpenSearch service requires a search subscription. The service is listed in the Services tab, but it cannot be added or provisioned without an active subscription. Contact our support team if you are not sure whether you have one.

1. Go to your application, choose an environment, and open the Services tab.
2. Add a new service and select OpenSearch 3.
3. Give it a prefix. The default is DEFAULT. If you already have another search service in that environment, use a different prefix, for example NEW.

4. Provision the service.

At this point the service exists and is usable, but your application does not see it yet. It shows as Pending attachment. The environment variables are only injected into your application after the next deployment.

For more details about how services are added, provisioned, attached, and removed, see the Services documentation.

Step 2: Look at your new environment variables

Open the service details (the "..." button, then Details) to see the variables it provides. With the default prefix DEFAULT, your application receives values like the following:

# Backend configuration
DEFAULT_SEARCH_BACKEND=opensearch-multi-index
DEFAULT_SEARCH_AUTH_SCHEME=aws-v4-auth
DEFAULT_SEARCH_VERSION=3.7.0

# Connection details
DEFAULT_SEARCH_HOSTNAME=search-cluster-psckcd7m2b6d5iiijunwsyvo34.eu-central-1.es.amazonaws.com
DEFAULT_SEARCH_PORT=443
DEFAULT_SEARCH_ACCESS_KEY_ID=AKIA...
DEFAULT_SEARCH_SECRET_ACCESS_KEY=SECRET...

# Full connection string and Haystack URL (credentials are URL-encoded)
DEFAULT_SEARCH_DSN=https://AKIA...:SECRET...@search-cluster-psckcd7m2b6d5iiijunwsyvo34.eu-central-1.es.amazonaws.com:443/myapp-live-7303e23cd7e9497aa6b25789c5a6e6-be706cf-*
DEFAULT_SEARCH_HAYSTACK_URL=es+https+aws://AKIA...:SECRET...@search-cluster-psckcd7m2b6d5iiijunwsyvo34.eu-central-1.es.amazonaws.com:443/myapp-live-7303e23cd7e9497aa6b25789c5a6e6-be706cf-*

# Index prefix constraint
DEFAULT_SEARCH_INDEX_PREFIX=myapp-live-7303e23cd7e9497aa6b25789c5a6e6-be706cf-*

If you used a different prefix, replace DEFAULT_ with your own prefix.

info

The old DEFAULT_HAYSTACK_URL variable is not provided by the new service. Use DEFAULT_SEARCH_HAYSTACK_URL, or better, build the connection from the individual variables as shown below.

Ensure you delete the DEFAULT_HAYSTACK_URL environment variable once the migration is complete!

Authentication is AWS SigV4

DEFAULT_SEARCH_ACCESS_KEY_ID and DEFAULT_SEARCH_SECRET_ACCESS_KEY are AWS IAM credentials. They are used to sign each request. You cannot pass them as a username and password, even though they appear in the DSN.

Index names must start with your prefix

DEFAULT_SEARCH_INDEX_PREFIX is a pattern, not an index name. It ends with * and describes every index your application is allowed to touch. Index names cannot contain *, so your application has to build a real name by removing the wildcard and adding its own suffix:

prefix = os.environ["DEFAULT_SEARCH_INDEX_PREFIX"].rstrip("*")
index_name = f"{prefix}default"   # myapp-live-...-be706cf-default

Your credentials only allow index names matching that pattern:

Index name	Result
myapp-live-7303e23cd7e9497aa6b25789c5a6e6-be706cf-default	allowed
myapp-live-7303e23cd7e9497aa6b25789c5a6e6-be706cf-another-index	allowed
default	denied
myapp-test-7303e23cd7e9497aa6b25789c5a6e6-ab3121a-index	denied
anotherapp-live-123afde4de3321afe45610eabce234-ab3121a-default	denied

Step 3: Test locally first

Run OpenSearch 3 on your machine and get your application working against it before you touch the platform. Add this to your docker-compose.yml:

services:
  opensearch:
    image: opensearchproject/opensearch:3.7.0
    container_name: opensearch-local
    environment:
      - discovery.type=single-node
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      # Local development only: no TLS, no authentication.
      - DISABLE_INSTALL_DEMO_CONFIG=true
      - DISABLE_SECURITY_PLUGIN=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    ports:
      - "9200:9200"
    volumes:
      - opensearch-data:/usr/share/opensearch/data

volumes:
  opensearch-data:

On Linux hosts you may also need to run sysctl -w vm.max_map_count=262144 for the container to start.

The local cluster uses plain HTTP with no authentication. The code examples below fall back to http://opensearch:9200 when DEFAULT_SEARCH_AUTH_SCHEME is not set, so the same settings file works locally and in the cloud.

Step 4: Update your code

What actually breaks

	Coming from Elasticsearch 7.10	Coming from Elasticsearch 2.3
Client library	move to opensearch-py	move to opensearch-py
Authentication	new: requests must be SigV4-signed	new: requests must be SigV4-signed
Mapping types (doc_type)	already gone in your current setup	must be removed
Query DSL	mostly unchanged	filtered queries, _all, "index": "not_analyzed", "type": "string", and the nGram / edgeNGram filter names no longer exist
Sorting and faceting	unchanged	sorting or faceting on an analyzed text field now returns HTTP 400
Index rebuild	required	required

Three points worth noting:

1. Mapping types. They were removed in Elasticsearch 7, and OpenSearch never supported them. If you are on Elasticsearch 2.3 today, anything that sends a custom doc_type (in a URL path or in a mapping body) will return HTTP 400.
2. elasticsearch-py does not work against the new cluster. Version 7.14 and later run a product check on first use and raise UnsupportedProductError against a non-Elasticsearch server, and 8.x fails outright. On our cluster that check happens to be inconclusive, because the client cannot read GET /, so a 7.x client may look like it connects. That is a side effect of an access policy we may tighten at any time, and the 7.x line is end of life anyway. Move to opensearch-py.
3. Sorting and faceting on analyzed text. On Elasticsearch 2.3 you could sort or facet on an analyzed string field. Since Elasticsearch 5 that is rejected. In Haystack terms: declare the field with faceted=True (which gives you a <field>_exact keyword field) and sort or facet on that, or on a date or integer field.

Which path applies to you

Most of our clients use django-haystack with elasticsearch on Django. Pick the path that matches your stack:

Your stack	Path
Django >= 5.2 and Python >= 3.11	Option A: django-haystack-opensearch
Anything older	Option B: keep django-haystack, swap the transport for opensearch-py

Option A is where you want to end up. Option B is a supported bridge that lets you migrate without upgrading Django first.

If you do not use Django or Haystack, the rules are the same: use an OpenSearch client for your language, sign your requests with SigV4, and keep all index names inside your prefix.

Option A is also documented on the Search with OpenSearch reference page, together with a generic opensearch-py example and a troubleshooting table. Option B is described here only: it is a migration bridge, and we do not recommend it as a long term setup.

Option A: use django-haystack-opensearch

django-haystack-opensearch is a native OpenSearch backend for Haystack, built on opensearch-py. It supports OpenSearch 1.x to 3.x and needs no wrapper code of your own.

Requirements: Python >= 3.11, Django >= 5.2, django-haystack >= 3.3.0, opensearch-py >= 3.1.0. If you cannot meet these, use Option B.

Dependencies: remove elasticsearch and elasticsearch-dsl, then add django-haystack-opensearch, opensearch-py and boto3 (for SigV4).

Two fixes are required on top of version 1.0.0 of the package. We found both while testing against our cluster, and both fail quietly:

• update() calls the bulk helper without an index, which sends the request to the cluster-level /_bulk endpoint. Your credentials are scoped to your index prefix, so that returns HTTP 403 and nothing is ever indexed. With Haystack's default SILENTLY_FAIL, rebuild_index still exits with code 0.
• the mapping helper drops the analyzer on every text field. EdgeNgramField and NgramField then match nothing, so autocomplete returns empty results, and the document field loses its snowball analyzer, which changes relevance.

Put this next to your settings, in myapp/backends.py:

from django_haystack_opensearch.haystack import (
    OpenSearchSearchBackend,
    OpenSearchSearchEngine,
)
from opensearchpy.helpers import bulk


class PatchedOpenSearchBackend(OpenSearchSearchBackend):
    def update(self, index, iterable, commit=True):
        # Upstream omits index=, which targets the cluster-level /_bulk
        # endpoint. A prefix-scoped policy denies that with a 403.
        if not self.setup_complete:
            self.setup()
        prepped_docs = self._prepare_documents_for_bulk(index, iterable)
        if prepped_docs:
            bulk(self.conn, prepped_docs, index=self.index_name)
        if commit:
            self.conn.indices.refresh(index=self.index_name)

    def _add_keyword_and_exact_subfields(self, props, unified_index):
        # Upstream replaces the whole definition for text fields, dropping
        # "analyzer" and with it EdgeNgram/Ngram matching.
        new_props = {}
        for field, definition in props.items():
            new_props[field] = dict(definition)
            if definition.get("type") == "text":
                subfields = dict(new_props[field].get("fields", {}))
                subfields["keyword"] = {"type": "keyword", "ignore_above": 256}
                new_props[field]["fields"] = subfields
        return new_props


class PatchedOpenSearchEngine(OpenSearchSearchEngine):
    backend = PatchedOpenSearchBackend

And in your settings.py:

import os
from urllib.parse import urlparse

import boto3
from opensearchpy import AWSV4SignerAuth, RequestsHttpConnection

# Fall back to the local docker-compose cluster when no service is attached.
dsn = os.getenv("DEFAULT_SEARCH_DSN", "http://opensearch:9200")
parsed = urlparse(dsn)

aws_auth = os.getenv("DEFAULT_SEARCH_AUTH_SCHEME") == "aws-v4-auth"

# Managed AWS domains are HTTPS only. Do not rely on the DSN scheme for this:
# it can be prefixed (for example "es+https+aws://").
scheme = "https" if aws_auth else ("https" if "https" in parsed.scheme.split("+") else "http")
port = f":{parsed.port}" if parsed.port else ""
cluster_url = f"{scheme}://{parsed.hostname}{port}"

# INDEX_PREFIX is a pattern ending in "*". Index names cannot contain "*",
# so derive a concrete name inside the allowed prefix.
prefix = os.getenv("DEFAULT_SEARCH_INDEX_PREFIX", "").rstrip("*")
index_name = f"{prefix}default" if prefix else "haystack"


def aws_region_from_host(hostname):
    """search-<domain>-<hash>.<region>.es.amazonaws.com -> <region>"""
    if hostname.endswith((".es.amazonaws.com", ".aos.on.aws")):
        return hostname.split(".")[-4]
    raise ValueError(f"Cannot derive an AWS region from {hostname!r}")


if aws_auth:
    session = boto3.Session(
        aws_access_key_id=os.environ["DEFAULT_SEARCH_ACCESS_KEY_ID"],
        aws_secret_access_key=os.environ["DEFAULT_SEARCH_SECRET_ACCESS_KEY"],
    )
    opensearch_kwargs = {
        # AWSV4SignerAuth is a requests auth handler, so it must be paired
        # with RequestsHttpConnection.
        "http_auth": AWSV4SignerAuth(
            session.get_credentials(),
            aws_region_from_host(parsed.hostname),
            service="es",
        ),
        "connection_class": RequestsHttpConnection,
        "use_ssl": True,
        "verify_certs": True,
    }
else:
    # Local development: plain HTTP, no authentication.
    opensearch_kwargs = {"use_ssl": False}

HAYSTACK_CONNECTIONS = {
    "default": {
        # The patched engine from myapp/backends.py, not the package's own.
        "ENGINE": "myapp.backends.PatchedOpenSearchEngine",
        "URL": cluster_url,
        "INDEX_NAME": index_name,
        "KWARGS": opensearch_kwargs,
    },
}

Everything in KWARGS is passed straight to the opensearch-py client constructor. Do not put timeout there, the backend already passes it. Use Haystack's own TIMEOUT key instead.

Option B: keep django-haystack and swap the transport

Use this if you cannot yet meet Option A's Django and Python requirements. The idea: keep Haystack's elasticsearch7 backend, which already produces OpenSearch-compatible mappings and never sends a doc_type, but replace its transport with opensearch-py.

Dependencies:

• opensearch-py: use >=3.1 on Python >= 3.10, or >=2.8,<3.1 on Python 3.8 and 3.9
• boto3
• django-haystack >= 3.2, which is where elasticsearch7_backend lives. If you are on Elasticsearch 2.3 today you are probably using elasticsearch2_backend. That module was removed in django-haystack 3.4.0, so this move is unavoidable. Pick the newest django-haystack your Django version supports.
• elasticsearch>=7.0,<8 must stay installed. Haystack's elasticsearch7_backend refuses to import without it. It is no longer used to talk to the cluster.

In myapp/backends.py:

"""OpenSearch 3.x transport for django-haystack's elasticsearch7 backend.

Haystack imports its bulk helpers and exception classes at module scope, so
swapping the client alone is not enough: the helpers and the exception classes
have to be redirected to opensearch-py as well. This is a bridge until the
project can move to django-haystack-opensearch.
"""

import os
from urllib.parse import urlparse

import boto3
import elasticsearch
import opensearchpy
from haystack.backends import elasticsearch7_backend as es7_module
from haystack.backends import elasticsearch_backend as es_base_module
from opensearchpy import AWSV4SignerAuth, OpenSearch, RequestsHttpConnection
from opensearchpy.helpers import bulk as opensearch_bulk
from opensearchpy.helpers import scan as opensearch_scan


class _ElasticsearchModuleShim:
    """Stands in for the `elasticsearch` module inside Haystack's backends.

    Haystack catches `elasticsearch.TransportError`; opensearch-py raises its
    own class, which would otherwise sail past those handlers and break
    SILENTLY_FAIL. `Elasticsearch` is kept because Haystack's __init__ builds a
    client before we get the chance to replace it.
    """

    Elasticsearch = elasticsearch.Elasticsearch
    TransportError = opensearchpy.TransportError


for _module in (es_base_module, es7_module):
    _module.elasticsearch = _ElasticsearchModuleShim
    _module.bulk = opensearch_bulk

es_base_module.NotFoundError = opensearchpy.NotFoundError
es7_module.scan = opensearch_scan


def aws_region_from_host(hostname):
    """search-<domain>-<hash>.<region>.es.amazonaws.com -> <region>"""
    if hostname.endswith((".es.amazonaws.com", ".aos.on.aws")):
        return hostname.split(".")[-4]
    raise ValueError(f"Cannot derive an AWS region from {hostname!r}")


class OpenSearchBackend(es7_module.Elasticsearch7SearchBackend):
    def __init__(self, connection_alias, **connection_options):
        # Keep KWARGS away from super(): it would feed them to the
        # elasticsearch client, which does not understand them.
        kwargs = dict(connection_options.pop("KWARGS", {}))
        super().__init__(connection_alias, **connection_options)

        url = connection_options["URL"]

        if os.getenv("DEFAULT_SEARCH_AUTH_SCHEME") == "aws-v4-auth":
            hostname = urlparse(url).hostname
            session = boto3.Session(
                aws_access_key_id=os.environ["DEFAULT_SEARCH_ACCESS_KEY_ID"],
                aws_secret_access_key=os.environ["DEFAULT_SEARCH_SECRET_ACCESS_KEY"],
            )
            kwargs.update(
                # AWSV4SignerAuth is a requests auth handler, hence
                # RequestsHttpConnection.
                http_auth=AWSV4SignerAuth(
                    session.get_credentials(),
                    aws_region_from_host(hostname),
                    service="es",
                ),
                connection_class=RequestsHttpConnection,
                use_ssl=True,
                verify_certs=True,
            )

        # Replace the elasticsearch client Haystack built in super().__init__().
        self.conn = OpenSearch(hosts=[url], timeout=self.timeout, **kwargs)


class OpenSearchEngine(es7_module.Elasticsearch7SearchEngine):
    backend = OpenSearchBackend

And in your settings.py:

import os
from urllib.parse import urlparse

dsn = os.getenv("DEFAULT_SEARCH_DSN", "http://opensearch:9200")
parsed = urlparse(dsn)

aws_auth = os.getenv("DEFAULT_SEARCH_AUTH_SCHEME") == "aws-v4-auth"
scheme = "https" if aws_auth else ("https" if "https" in parsed.scheme.split("+") else "http")
port = f":{parsed.port}" if parsed.port else ""

prefix = os.getenv("DEFAULT_SEARCH_INDEX_PREFIX", "").rstrip("*")

HAYSTACK_CONNECTIONS = {
    "default": {
        "ENGINE": "myapp.backends.OpenSearchEngine",
        "URL": f"{scheme}://{parsed.hostname}{port}",
        "INDEX_NAME": f"{prefix}default" if prefix else "haystack",
    },
}

Two things to be aware of with Option B:

• The module-level redirection applies to the whole process. If the same project keeps another Haystack connection pointing at a real Elasticsearch cluster, do not use this approach.
• Test rebuild_index, update_index --remove, clear_index and a representative search against your local OpenSearch container before you rely on it. Those four paths cover the bulk helpers, the scroll helper, and the exception handling that this shim replaces.

Step 5: Deploy and rebuild the index

Do this one environment at a time, starting with Test.

1. Push the code changes to the branch used by that environment.
2. Deploy the environment. The OpenSearch service moves to Attached and your application receives the new environment variables.
3. Rebuild the index, for example python manage.py rebuild_index for Haystack projects. You can run this over SSH, or as part of your release commands.
4. Check the document count in the index and test a real search on the site. Haystack swallows indexing errors by default, so an empty index can look like a successful rebuild. See the checklist below.
5. Once search works, delete the DEFAULT_HAYSTACK_URL environment variable and redeploy your environment.

Repeat for each environment. Do the Live environment last.

Known limitations of the shared cluster

info

Those are not new limitations, they have always been enforced.

Your credentials are scoped to your index prefix, so cluster-level API paths are not available to your application:

• GET / is denied, and so are /_cluster/health and /_cat/indices. Anything that reads cluster info or cluster health as a readiness check will get HTTP 403. Point health checks at your own index instead, for example a _count on it.
• Searches must name an index. POST /_search without an index, and /_all/_search, are denied. Haystack always addresses your index by name, so this only affects hand-written queries.
• the scroll API (/_search/scroll) is not available. Instead, use the sort parameter with the search_after parameter to scroll responses for user queries.

Migration checklist

• Test locally against the OpenSearch container: connection, indexing commands, search views.
• Derive a concrete index name from DEFAULT_SEARCH_INDEX_PREFIX by removing the trailing *. An index name containing * is rejected by OpenSearch.
• Rebuild your indexes. Nothing is copied from the old cluster.
• Audit raw queries. If you bypass SearchQuerySet with .raw() or hand-written JSON, look for filtered queries, _all, _type / doc_type, "type": "string" and "index": "not_analyzed". None of these exist in OpenSearch.
• Check custom SearchIndex field types if you use EdgeNgramField or NgramField: the analyzer names changed with the backend.
• Confirm SigV4 authentication works from your environment, and that TLS verification is enabled in production.
• Check the document count after the rebuild. Haystack's SILENTLY_FAIL defaults to true, so indexing errors are logged and swallowed instead of raised. Set "SILENTLY_FAIL": False on the connection while testing, and confirm the index really holds the number of documents you expect.
• Re-test autocomplete if you use EdgeNgramField or NgramField. Empty results are the usual symptom of a mapping that lost its analyzer.
• Use one service per environment. Test and Live must not share an index prefix.

What we tested

Both options above were run end to end against a real Amazon OpenSearch domain with prefix-scoped IAM credentials: index creation and mapping, rebuild_index, update_index, update_index --remove, single document removal, clear_index, keyword search, edge-ngram autocomplete, faceting, and sorting.

• Option A was tested on Django 5.2 with django-haystack 3.4.0 and opensearch-py 3.2.0.
• Option B was tested on Django 4.2 with django-haystack 3.3.0, elasticsearch-py 7.17.13 and opensearch-py 3.2.0.

Need help?

If something does not work as described here, or your stack is not covered, contact us.

See also

• Search with OpenSearch: understand how OpenSearch works on Divio
• Use OpenSearch: add the service, configure your application, and set up local development
• OpenSearch configuration: environment variables, authentication, index naming rules, cluster limitations, and troubleshooting
• Services: how services are added, provisioned, attached, and removed