Skip to content

Updated Is "Migration Right for you?" page and some dependencies #10158

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
+94 −79
Open

Updated Is "Migration Right for you?" page and some dependencies #10158

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
df3f743
Restructured "Is Migration Assistant Right for you?" page.
sumobrian Jun 29, 2025
c62887a
Broke down limitation by component. Minor editorial changes.
sumobrian Jun 29, 2025
b58adc4
Removed "Chossing migration approach" since it will be revised and in…
sumobrian Jun 29, 2025
3e364fa
Updated requirements around VPC connectivity.
sumobrian Jul 2, 2025
38b6d73
Updated list of Migration Assistant products to vale github action pr…
sumobrian Jul 7, 2025
27dd744
Added further breakdown of networking requirements
sumobrian Jul 7, 2025
a365541
Moved limitations and assumptions
sumobrian Jul 7, 2025
a67a2be
Removed Github issue reference for Kibana
sumobrian Jul 7, 2025
200024d
Identify services that are only Capture-and-Replay
sumobrian Jul 7, 2025
0a801a8
Added ALB to list of services. Updated verbiage for ELB.
sumobrian Jul 8, 2025
dcffac5
Updated based on style-job suggestions.
sumobrian Jul 8, 2025
242887c
Updated S3 verbiage
sumobrian Jul 8, 2025
70e22e5
Remove duplicate compression bullet point.
sumobrian Jul 8, 2025
7178b11
Indicate that secrets manager is only used for basic auth creds.
sumobrian Jul 8, 2025
45ceabf
Updated SSM verbiage replacing EC2 w/ ECS
sumobrian Jul 8, 2025
6e475d6
Add writer edits
Naarcha-AWS Jul 10, 2025
1339b1b
Apply suggestions from code review
Naarcha-AWS Jul 10, 2025
b9c53f9
Apply suggestions from code review
Naarcha-AWS Jul 10, 2025
1b225ac
Apply suggestions from code review
natebower Jul 10, 2025
61321f1
Merge branch 'main' into ma_phases
Naarcha-AWS Jul 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/vale/styles/Vocab/OpenSearch/Products/accept.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Anthropic Claude
Apache Kafka
Auditbeat
AWS Cloud
Capture and Replay
Cohere Command
Cohere Embed
Cohere Embed English
Expand Down Expand Up @@ -61,8 +62,10 @@ Log4j
Logstash
Lucene
macOS
Metadata-Migration-Tool
Metricbeat
Microsoft Power BI
Migration Assistant
Minikube
Nagios
Okta
Expand Down Expand Up @@ -93,6 +96,7 @@ PyTorch
Querqy
Query Workbench
RankLib
Reindex-from-Snapshot
RCF Summarize
RPM Package Manager
Ruby
Expand All @@ -104,6 +108,8 @@ Titan
Titan Multimodal Embeddings
Titan Text Embeddings
TorchScript
Traffic Capture Proxy
Traffic Replayer
Tribuo
VisBuilder
Winlogbeat
Expand Down
16 changes: 2 additions & 14 deletions ...ation-assistant/deploying-migration-assistant/getting-started-data-migration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,22 +12,10 @@ redirect_from:

This quickstart outlines how to deploy Migration Assistant for OpenSearch and execute an existing data migration using `Reindex-from-Snapshot` (RFS). It uses AWS for illustrative purposes. However, the steps can be modified for use with other cloud providers.

## Prerequisites and assumptions
Before using this quickstart, make sure you review [Is Migration Assistant right for you?]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths).

Before using this quickstart, make sure you fulfill the following prerequisites:

* Verify that your migration path [is supported]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths). Note that we test with the exact versions specified, but you should be able to migrate data on alternative minor versions as long as the major version is supported.
* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin.
* The target cluster must be deployed.
* Verify that the `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about how to bootstrap your AWS account in the required AWS Region, see [the CDKToolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html).

The steps in this guide assume the following:

* In this guide, a snapshot will be taken and stored in Amazon S3; the following assumptions are made about this snapshot:
* The `_source` flag is enabled on all indexes to be migrated.
* The snapshot includes the global cluster state (`include_global_state` is `true`).
* Shard sizes of up to approximately 80 GB are supported. Larger shards cannot be migrated. If this presents challenges for your migration, contact the [migration team](https://opensearch.slack.com/archives/C054JQ6UJFK).
* Migration Assistant will be installed in the same AWS Region and have access to both the source snapshot and target cluster.
Because this guide uses [AWS Cloud Development Kit (AWS CDK)](https://aws.amazon.com/cdk/), make sure that the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html).

---

Expand Down
6 changes: 3 additions & 3 deletions _migration-assistant/overview/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ has_children: true
has_toc: false
permalink: /migration-assistant/overview/
items:
- heading: "Is Migration Assistant right for you?"
description: "Evaluate whether Migration Assistant is right for your use case."
link: "/migration-assistant/overview/is-migration-assistant-right-for-you/"
- heading: "Key components"
description: "Get familiar with the key components of Migration Assistant."
link: "/migration-assistant/overview/key-components/"
- heading: "Architecture"
description: "Understand how Migration Assistant integrates into your infrastructure."
link: "/migration-assistant/overview/architecture/"
- heading: "Is Migration Assistant right for you?"
description: "Evaluate whether Migration Assistant is right for your use case."
link: "/migration-assistant/overview/is-migration-assistant-right-for-you/"
---

# Migration Assistant overview
Expand Down
145 changes: 83 additions & 62 deletions _migration-assistant/overview/is-migration-assistant-right-for-you.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,11 @@

# Is Migration Assistant right for you?

Deciding whether to use Migration Assistant depends on your specific upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant is right for your use case—or whether another tool might be a better fit.
Whether Migration Assistant is right for you depends on your upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant fits your use case.

Migration Assistant was built to fill important gaps in common migration strategies. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant lets you do this in a single step. Other methods, like rolling upgrades or snapshot restores, require you to upgrade through each major version, often reindexing your data at every step.
Migration Assistant addresses key limitations in traditional migration approaches. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant enables you to complete the process in a single step. Other methods, like rolling upgrades or snapshot restores, require upgrading through each major version, often with reindexing at every stage.

Migration Assistant also supports live traffic replication, allowing for zero-downtime migrations. This makes it a strong choice for production environments, where minimizing service disruption is critical.

If your migration is limited to a static cluster configuration (like index templates and aliases), or if you're not concerned about downtime, simpler tools may be sufficient. But for complex migrations involving real-time traffic or major version jumps, Migration Assistant offers robust, flexible capabilities.
Migration Assistant also supports live traffic replication, enabling zero-downtime migrations. This makes it a strong fit for environments where minimizing service disruption is critical.

## Supported migration paths

Expand All @@ -34,7 +32,7 @@
<table class="migration-matrix">
<thead>
<tr>
<th></th>
<th>Source Version</th>

Check failure on line 35 in _migration-assistant/overview/is-migration-assistant-right-for-you.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.TableHeadings] 'Source Version' is a table heading and should be in sentence case.

Raw Output:
{"message": "[OpenSearch.TableHeadings] 'Source Version' is a table heading and should be in sentence case.", "location": {"path": "_migration-assistant/overview/is-migration-assistant-right-for-you.md", "range": {"start": {"line": 35, "column": 11}}}, "severity": "ERROR"}
{% for target in unique_targets %}
<th>{{ target }}</th>
{% endfor %}
Expand All @@ -59,95 +57,118 @@
**Source and target platforms**

- Self-managed (on premises or hosted by a cloud provider)
- Amazon OpenSearch Service


**AWS Regions**

Migration Assistant is supported in the following AWS Regions:

- US East (N. Virginia, Ohio)
- US West (Oregon, N. California)
- Europe (Frankfurt, Ireland, London)
- Asia Pacific (Tokyo, Singapore, Sydney)
- AWS GovCloud (US-East, US-West)[^1]
- Amazon OpenSearch Service (Amazon OpenSearch Serverless collections are not supported)

[^1]: In AWS GovCloud (US), `reindex-from-snapshot` (RFS) is limited to shard sizes of 80 GiB or smaller.
**Supported AWS Regions**

Refer to [Supported AWS Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the full list of supported Regions.

## Supported features

## Supported components
Before starting an upgrade or migration, consider the cluster feature to be included. The following table lists what can be migrated using Migration Assistant, whether it is currently supported, and recommendations for how to handle each component.

Before starting a migration, consider the scope of the components involved. The following table outlines components that should potentially be migrated, indicates whether they are supported by Migration Assistant, and provides recommendations.

| Component | Supported | Recommendations |
| Feature | Supported | Recommendations |
| :--- |:--- | :--- |
| **Documents** | Yes | Migrate existing data with RFS and live traffic with capture and replay. |
| **Documents** | Yes | Migrate existing data with RFS and live traffic with Capture and Replay. |
| **Index settings** | Yes | Migrate with the `Metadata-Migration-Tool`. |
| **Index mappings** | Yes | Migrate with the `Metadata-Migration-Tool`. |
| **Index templates** | Yes | Migrate with the `Metadata-Migration-Tool`. |
| **Component templates** | Yes | Migrate with the `Metadata-Migration-Tool`. |
| **Aliases** | Yes | Migrate with the `Metadata-Migration-Tool`. |
| **Index State Management (ISM) policies** | Expected in 2025 | Manually migrate using an API. For more information about ISM support, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). |
| **Elasticsearch Kibana[^2] dashboards** | Expected in 2025 | This tool is only needed when migrating from Elasticsearch Kibana dashboards to OpenSearch Dashboards. Start by exporting JSON files from Kibana and importing them into OpenSearch Dashboards. For Elasticsearch versions 7.10.2 to 7.17, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool before importing X-Pack visualizations like Canvas and Lens in Kibana dashboards, as they may require recreation for compatibility with OpenSearch.|
| **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage AWS Identity and Access Management (IAM) for enhanced security management. |
| **Elasticsearch Kibana dashboards** | Expected in 2025 | This tool is only needed when migrating from Elasticsearch Kibana dashboards to OpenSearch Dashboards. Start by exporting JSON files from Kibana and importing them into OpenSearch Dashboards. For Elasticsearch versions 7.10.2 to 7.17, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool before importing X-Pack visualizations like Canvas and Lens into Kibana dashboards, as they may require recreation for compatibility with OpenSearch.|
| **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, use AWS Identity and Access Management (IAM) for enhanced security management. |
| **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct OpenSearch equivalents. |

[^2]: Support for Kibana 5.0 through 7.10.2 migration paths to OpenSearch Dashboards will be added in a future version. Kibana 8 and later are not supported. For more information, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944).
## Checklist

## Choosing your migration approach
Use this checklist to determine whether Migration Assistant is the right fit for your migration:

Use the following checklist to determine which Migration Assistant components best fit your use case.
- Are you migrating across one or more major versions—for example, from Elasticsearch 5 to OpenSearch 3—in a single step?
- Are you upgrading but want the ability to safely back out, reducing the risk of data loss or service disruption?
- Do you need to maintain high service availability with minimal or zero downtime?
- Do you need to validate a new OpenSearch cluster before switching over---with rollback capabilities?
- Is your environment self-managed or running on Amazon OpenSearch Service?
- Are you looking for tooling to migrate index settings and other metadata?
- Do you need to reconfigure your target cluster—for example, by changing the sharding strategy and reindexing?
- Are you migrating across Regions, from on premises, or from another cloud provider?
- Do you need a high-performance backfill solution that can reliably reindex documents—with support for pause, resume, or checkpoint recovery?

### Metadata migration
If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration.

Use [metadata migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) if:
## Migration Assistant assumptions and limitations

- You need to migrate while mitigating breaking changes between the source and target clusters, such as differences in mappings, settings, aliases, or component templates.
- You want a relatively consistent configuration between the source and target clusters.
Before using Migration Assistant, review the following assumptions and limitations.

### Backfill migration
### Networking and environment

Use [backfill migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/) if:
Connectivity to AWS services and outbound internet access are required in order to build and deploy Migration Assistant. Requirements differ based on whether you're deploying to a new or existing virtual private cloud (VPC).

- You need to move historical data without disrupting live traffic.
- You want to backfill indexes from a specific point in time without impacting the source cluster.
- You want to verify historical data in the target cluster before switching over.
- You want to backfill using an existing or incremental snapshot.
- You need the fastest backfill option that includes reindexing.
- You want the ability to pause and resume migration.
#### Source and target connectivity

### RFS
To meet connectivity requirements, ensure the following:

Use [RFS]({{site.url}}{{site.baseurl}}/migration-assistant/deploying-migration-assistant/getting-started-data-migration/) if:
- You must establish connectivity between:
- The source cluster and/or Amazon Simple Storage Service (Amazon S3) (for snapshots, which may only require updating the bucket policy) and Migration Assistant.
- The target cluster and Migration Assistant.
- If the source or target resides in a private VPC without internet access, use one of the following to connect:
- VPC endpoints
- VPC peering
- AWS Transit Gateway

- You already use OpenSearch snapshots for backups.
- You need to migrate documents at scale in parallel, such as with Amazon Elastic Container Service (Amazon ECS).
- You require a data migration path as part of a zero-downtime migration.
- Your AWS Region supports RFS and your shard sizes are within supported limits.
#### Deploying to a new VPC

### Combination of all three
When deploying to a new VPC, consider the following:

Use a combination of all three migration types if:
- Migration Assistant provisions a new VPC with required components (for example, NAT gateway, subnets).

Check failure on line 123 in _migration-assistant/overview/is-migration-assistant-right-for-you.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: subnets. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: subnets. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migration-assistant/overview/is-migration-assistant-right-for-you.md", "range": {"start": {"line": 123, "column": 96}}}, "severity": "ERROR"}
- You must establish network access from this VPC to both the source and target clusters.

- You're performing a complex, multi-version migration.
- You require zero downtime and full validation of the target environment.
- You want end-to-end tooling for metadata, data movement, and cluster behavior comparison.
- You're cloning an existing cluster and changing the source's configuration.
- You're setting up disaster recovery.
#### Deploying to an existing VPC

## Checklist
When deploying to an existing VPC, consider the following:

Use this checklist to decide whether Migration Assistant is right for you:
- If deploying Migration Assistant to an existing VPC (for example, the same VPC as the source or target), you may need to configure connectivity to any cluster external to the target VPC.
- For example, if Migration Assistant is deployed to the source VPC, you may need VPC endpoints or peering to reach the target.

- Are you migrating across one or more major versions?
- Ensure all required AWS services are reachable by Migration Assistant components.

- Do you need to maintain service availability with zero downtime?
- If the VPC has outbound access using private subnets with a NAT gateway or public subnets with an internet gateway, then VPC interface endpoints are not required.

Check failure on line 135 in _migration-assistant/overview/is-migration-assistant-right-for-you.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: subnets. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: subnets. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migration-assistant/overview/is-migration-assistant-right-for-you.md", "range": {"start": {"line": 135, "column": 87}}}, "severity": "ERROR"}

Check failure on line 135 in _migration-assistant/overview/is-migration-assistant-right-for-you.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: subnets. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: subnets. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migration-assistant/overview/is-migration-assistant-right-for-you.md", "range": {"start": {"line": 135, "column": 50}}}, "severity": "ERROR"}

- Do you need to validate a new OpenSearch cluster before switching over?

- Is your environment self-managed or running on Amazon OpenSearch Service?
- If using isolated subnets with no outbound access, you must configure VPC interface endpoints or routing to the following services:

Check failure on line 138 in _migration-assistant/overview/is-migration-assistant-right-for-you.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: subnets. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: subnets. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migration-assistant/overview/is-migration-assistant-right-for-you.md", "range": {"start": {"line": 138, "column": 23}}}, "severity": "ERROR"}

- Are you looking for tooling that can automate metadata migration and performance comparison?
- **Application Load Balancer** (Capture and Replay only) – Used to optionally reroute client traffic from the source to the target during migration.
- **Amazon CloudWatch** – Publishes migration metrics.
- **Amazon CloudWatch Logs** – Ingests Amazon Elastic Container Service (Amazon ECS) task logs.
- **Amazon Elastic Compute Cloud (Amazon EC2)** – Used to bootstrap Migration Assistant. When deploying using AWS CloudFormation, the bootstrap EC2 instance requires outbound internet access (using NAT gateway or an internet gateway) to download the latest version from GitHub.
- **Amazon Elastic Block Store (Amazon EBS)** – Provides temporary disk storage.
- **Amazon Elastic Container Registry (Amazon ECR)** – Pulls container images.
- **Amazon ECS** – Orchestrates container workloads.
- **Amazon Elastic File System (Amazon EFS)** – Stores persistent logs.
- **Amazon Managed Streaming for Apache Kafka (Amazon MSK)** (Capture and Replay only) – Used as durable storage to capture and replay live HTTP traffic.

Check failure on line 148 in _migration-assistant/overview/is-migration-assistant-right-for-you.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[Vale.Terms] Use 'Capture and Replay' instead of 'capture and replay'.

Raw Output:
{"message": "[Vale.Terms] Use 'Capture and Replay' instead of 'capture and replay'.", "location": {"path": "_migration-assistant/overview/is-migration-assistant-right-for-you.md", "range": {"start": {"line": 148, "column": 121}}}, "severity": "ERROR"}
- **Amazon S3** – Stores and retrieves snapshots and artifacts.
- **Elastic Load Balancing** (Capture and Replay only) – Used by the migration console to connect to the Application Load Balancer.
- **AWS Secrets Manager** – Securely stores credentials when using basic authentication on the source or target.
- **AWS Systems Manager Parameter Store** – Stores configuration parameters.
- **AWS Systems Manager Session Manager** – Enables secure shell access to ECS tasks such as the migration console.
- **AWS X-Ray** – Supports distributed tracing.
- **Amazon Virtual Private Cloud (Amazon VPC)** – Ensures proper routing, DNS resolution, and endpoint configuration.

If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration.
### Reindex-from-Snapshot

To use `Reindex-from-Snapshot` (RFS), ensure the following:

- The `_source` field must be enabled on all indexes to be migrated. See [Source]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/).
- The source cluster must have the Amazon S3 plugin installed.
- If you choose to bring your own snapshot (that is, one not created by Migration Assistant), the following settings must be applied when creating the snapshot:
- `include_global_state: true` – Ensures that global cluster state is included.
- `compress: false` – Disables metadata compression, which is required for compatibility with RFS.
- Shards of up to **80 GiB** are supported by default. Larger shard sizes can be configured, **except in AWS GovCloud (US)**, where 80 GiB is the maximum.

### Capture and Replay

Capture and Replay has the following requirements:

- The Traffic Capture Proxy must be deployed to intercept client traffic.
- Live capture is recommended only for workloads with **< 4 TB/day** of incoming traffic to the source cluster.
- Automatically generated document IDs are **not preserved** during replay. Clients must explicitly provide document IDs for `index` and `update` operations.
Loading