From df3f74366323f635ce1a913d454d0cbcd32a4558 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Sun, 29 Jun 2025 12:04:21 -0500 Subject: [PATCH 01/19] Restructured "Is Migration Assistant Right for you?" page. Signed-off-by: Brian Presley --- .../getting-started-data-migration.md | 17 +------- _migration-assistant/overview/index.md | 6 +-- .../is-migration-assistant-right-for-you.md | 43 ++++++++++--------- 3 files changed, 27 insertions(+), 39 deletions(-) diff --git a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md index 856d2e53aa5..5e0ac5dcf59 100644 --- a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md +++ b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md @@ -12,22 +12,7 @@ redirect_from: This quickstart outlines how to deploy Migration Assistant for OpenSearch and execute an existing data migration using `Reindex-from-Snapshot` (RFS). It uses AWS for illustrative purposes. However, the steps can be modified for use with other cloud providers. -## Prerequisites and assumptions - -Before using this quickstart, make sure you fulfill the following prerequisites: - -* Verify that your migration path [is supported]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths). Note that we test with the exact versions specified, but you should be able to migrate data on alternative minor versions as long as the major version is supported. -* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin. -* The target cluster must be deployed. -* Verify that the `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about how to bootstrap your AWS account in the required AWS Region, see [the CDKToolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). - -The steps in this guide assume the following: - -* In this guide, a snapshot will be taken and stored in Amazon S3; the following assumptions are made about this snapshot: - * The `_source` flag is enabled on all indexes to be migrated. - * The snapshot includes the global cluster state (`include_global_state` is `true`). - * Shard sizes of up to approximately 80 GB are supported. Larger shards cannot be migrated. If this presents challenges for your migration, contact the [migration team](https://opensearch.slack.com/archives/C054JQ6UJFK). -* Migration Assistant will be installed in the same AWS Region and have access to both the source snapshot and target cluster. +Before using this quickstart, make sure you review ["Is Migration Assistant right for you?"]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths). --- diff --git a/_migration-assistant/overview/index.md b/_migration-assistant/overview/index.md index 7962fe5e85d..11c7d716ef5 100644 --- a/_migration-assistant/overview/index.md +++ b/_migration-assistant/overview/index.md @@ -6,15 +6,15 @@ has_children: true has_toc: false permalink: /migration-assistant/overview/ items: + - heading: "Is Migration Assistant right for you?" + description: "Evaluate whether Migration Assistant is right for your use case." + link: "/migration-assistant/overview/is-migration-assistant-right-for-you/" - heading: "Key components" description: "Get familiar with the key components of Migration Assistant." link: "/migration-assistant/overview/key-components/" - heading: "Architecture" description: "Understand how Migration Assistant integrates into your infrastructure." link: "/migration-assistant/overview/architecture/" - - heading: "Is Migration Assistant right for you?" - description: "Evaluate whether Migration Assistant is right for your use case." - link: "/migration-assistant/overview/is-migration-assistant-right-for-you/" --- # Migration Assistant overview diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 2d9a4a20508..308e461de3c 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -9,13 +9,26 @@ redirect_from: # Is Migration Assistant right for you? -Deciding whether to use Migration Assistant depends on your specific upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant is right for your use case—or whether another tool might be a better fit. +Deciding whether to use Migration Assistant depends on your specific upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant is right for your case. -Migration Assistant was built to fill important gaps in common migration strategies. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant lets you do this in a single step. Other methods, like rolling upgrades or snapshot restores, require you to upgrade through each major version, often reindexing your data at every step. +Migration Assistant was built to address limitations in common migration strategies. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant lets you do this in a single step. Other methods, like rolling upgrades or snapshot restores, require you to upgrade through each major version, often reindexing your data at every step. -Migration Assistant also supports live traffic replication, allowing for zero-downtime migrations. This makes it a strong choice for production environments, where minimizing service disruption is critical. +Migration Assistant also supports live traffic replication, allowing for zero-downtime migrations. This makes it a strong choice for environments where minimizing service disruption is critical. -If your migration is limited to a static cluster configuration (like index templates and aliases), or if you're not concerned about downtime, simpler tools may be sufficient. But for complex migrations involving real-time traffic or major version jumps, Migration Assistant offers robust, flexible capabilities. +## Migration Assistant Assumptions and Limitations + +Before using Migration Assistant, review the following assumptions and limitations. If a limitation only, applies to a specific component (i.e., Capture-and-Replay or Reindex-from-Snapshot) it is specified. + +* If deploying to AWS, the Idendity used to deploy Migration Assistant must have permissions to install all resources used by Migration Assistant. For a full list of resource refer to the AWS documentation, [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). +* *Reindex-from-Snapshot only.* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin for Reindex-from-Snapshot. +* The target cluster must be deployed and reachable by Migration Assistant by Reindex-from-Snapshot resources for backfill and the Traffic Replayer for live capture. +* *Capture-and-Replay.* The Traffic Capture Proxy must be deployed to intercept client traffic relevant for migration. The proxy must. +* *Reindex-from-Snapshot.* The snapshot includes the global cluster state (`include_global_state` is `true`). +* By default shards up to 80GB are supported but larger shards are permitted if configured. There is, however, a hard limitation of 80GB shards if the source is in an AWS GovCloud region. +* *Capture-and-Replay.* Live capture should be liitated to 4TB/day of network traffic. +* *Capture-and-Replay.* Currently, for index requests, automatically generated document IDs will not be preserved during the migration. Clients must specify document IDs when writing or updating indecing if using Capture-and-Replay. +* The `_source` flag is enabled on all indexes to be migrated. Refer to [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for details. +* If deploying to AWS, verify `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about how to bootstrap your AWS account in the required AWS Region, see [the CDKToolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). ## Supported migration paths @@ -34,7 +47,7 @@ The following matrix shows which source versions can be directly migrated to whi - + {% for target in unique_targets %} {% endfor %} @@ -59,28 +72,18 @@ The following matrix shows which source versions can be directly migrated to whi **Source and target platforms** - Self-managed (on premises or hosted by a cloud provider) -- Amazon OpenSearch Service - - -**AWS Regions** - -Migration Assistant is supported in the following AWS Regions: - -- US East (N. Virginia, Ohio) -- US West (Oregon, N. California) -- Europe (Frankfurt, Ireland, London) -- Asia Pacific (Tokyo, Singapore, Sydney) -- AWS GovCloud (US-East, US-West)[^1] +- Amazon OpenSearch Service. Amazon OpenSearch Serverless Collections are not supported. -[^1]: In AWS GovCloud (US), `reindex-from-snapshot` (RFS) is limited to shard sizes of 80 GiB or smaller. +**Supported AWS Regions** +Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the official list of supported regions. -## Supported components +## Supported features Before starting a migration, consider the scope of the components involved. The following table outlines components that should potentially be migrated, indicates whether they are supported by Migration Assistant, and provides recommendations. -| Component | Supported | Recommendations | +| Feature | Supported | Recommendations | | :--- |:--- | :--- | | **Documents** | Yes | Migrate existing data with RFS and live traffic with capture and replay. | | **Index settings** | Yes | Migrate with the `Metadata-Migration-Tool`. | From c62887a1e3bb9d11a0f687de68c1f39ab26ab460 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Sun, 29 Jun 2025 12:38:06 -0500 Subject: [PATCH 02/19] Broke down limitation by component. Minor editorial changes. Signed-off-by: Brian Presley --- .../is-migration-assistant-right-for-you.md | 50 +++++++++++-------- 1 file changed, 30 insertions(+), 20 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 308e461de3c..6e530a9dfc9 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -9,30 +9,41 @@ redirect_from: # Is Migration Assistant right for you? -Deciding whether to use Migration Assistant depends on your specific upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant is right for your case. +Whether Migration Assistant is right for you depends on your upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant fits your use case. -Migration Assistant was built to address limitations in common migration strategies. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant lets you do this in a single step. Other methods, like rolling upgrades or snapshot restores, require you to upgrade through each major version, often reindexing your data at every step. +Migration Assistant addresses key limitations in traditional migration approaches. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant enables you to complete the process in a single step. Other methods, like rolling upgrades or snapshot restores, require upgrading through each major version, often with reindexing at every stage. -Migration Assistant also supports live traffic replication, allowing for zero-downtime migrations. This makes it a strong choice for environments where minimizing service disruption is critical. +Migration Assistant also supports live traffic replication, enabling zero-downtime migrations. This makes it a strong fit for environments where minimizing service disruption is critical. -## Migration Assistant Assumptions and Limitations +## Migration Assistant assumptions and limitations -Before using Migration Assistant, review the following assumptions and limitations. If a limitation only, applies to a specific component (i.e., Capture-and-Replay or Reindex-from-Snapshot) it is specified. +Before using Migration Assistant, review the following assumptions and limitations. They are grouped by scope to clarify which apply generally and which are specific to components. -* If deploying to AWS, the Idendity used to deploy Migration Assistant must have permissions to install all resources used by Migration Assistant. For a full list of resource refer to the AWS documentation, [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). -* *Reindex-from-Snapshot only.* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin for Reindex-from-Snapshot. -* The target cluster must be deployed and reachable by Migration Assistant by Reindex-from-Snapshot resources for backfill and the Traffic Replayer for live capture. -* *Capture-and-Replay.* The Traffic Capture Proxy must be deployed to intercept client traffic relevant for migration. The proxy must. -* *Reindex-from-Snapshot.* The snapshot includes the global cluster state (`include_global_state` is `true`). -* By default shards up to 80GB are supported but larger shards are permitted if configured. There is, however, a hard limitation of 80GB shards if the source is in an AWS GovCloud region. -* *Capture-and-Replay.* Live capture should be liitated to 4TB/day of network traffic. -* *Capture-and-Replay.* Currently, for index requests, automatically generated document IDs will not be preserved during the migration. Clients must specify document IDs when writing or updating indecing if using Capture-and-Replay. -* The `_source` flag is enabled on all indexes to be migrated. Refer to [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for details. -* If deploying to AWS, verify `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about how to bootstrap your AWS account in the required AWS Region, see [the CDKToolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). +### General + +- If deploying on AWS, the identity used to deploy Migration Assistant must have permission to install all required resources. For a full list of services, see [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). +- The target cluster must be deployed and reachable by Migration Assistant components, including the Migration Console, Reindex-from-Snapshot, and Traffic Replayer, depending on which features are used. +- The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for more information. +- If deploying to AWS, ensure the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). + +### Reindex-from-Snapshot + +- The source cluster must have the Amazon S3 plugin installed. +- Snapshots must include global cluster state (`include_global_state` is `true`). +- Shards up to 80 GiB are supported by default. Larger shards can be configured, except in AWS GovCloud, where 80 GiB is the maximum supported size. + +### Capture-and-Replay + +- The Traffic Capture Proxy must be deployed to intercept relevant client traffic. +- Live capture is recommended only for environments with less than 4 TB/day of incoming traffic to the source cluster. +- Automatically generated document IDs are not preserved for index requests. Clients must explicitly provide document IDs when indexing or updating documents. + +--- ## Supported migration paths -The following matrix shows which source versions can be directly migrated to which OpenSearch target versions: +The matrix below shows which source versions can be directly migrated to which OpenSearch target versions: + {% comment %}First, collect all unique target versions{% endcomment %} @@ -72,16 +83,15 @@ The following matrix shows which source versions can be directly migrated to whi **Source and target platforms** - Self-managed (on premises or hosted by a cloud provider) -- Amazon OpenSearch Service. Amazon OpenSearch Serverless Collections are not supported. - +- Amazon OpenSearch Service (OpenSearch Serverless Collections are not supported) **Supported AWS Regions** -Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the official list of supported regions. +Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the for the full list of supported regions. ## Supported features -Before starting a migration, consider the scope of the components involved. The following table outlines components that should potentially be migrated, indicates whether they are supported by Migration Assistant, and provides recommendations. +Before starting an upgrade or migration, consider the cluster feature to be included. The table below lists what can be migrated using Migration Assistant, whether it is currently supported, and recommendations for how to handle each component. | Feature | Supported | Recommendations | | :--- |:--- | :--- | From b58adc45081075915648f9f8f18f4434a7f30e14 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Sun, 29 Jun 2025 12:58:32 -0500 Subject: [PATCH 03/19] Removed "Chossing migration approach" since it will be revised and included in a seperate migration patterns page. Updated checklist. Signed-off-by: Brian Presley --- .../is-migration-assistant-right-for-you.md | 58 +++---------------- 1 file changed, 9 insertions(+), 49 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 6e530a9dfc9..11d2a3cf10e 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -108,59 +108,19 @@ Before starting an upgrade or migration, consider the cluster feature to be incl [^2]: Support for Kibana 5.0 through 7.10.2 migration paths to OpenSearch Dashboards will be added in a future version. Kibana 8 and later are not supported. For more information, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). -## Choosing your migration approach - -Use the following checklist to determine which Migration Assistant components best fit your use case. - -### Metadata migration - -Use [metadata migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) if: - -- You need to migrate while mitigating breaking changes between the source and target clusters, such as differences in mappings, settings, aliases, or component templates. -- You want a relatively consistent configuration between the source and target clusters. - -### Backfill migration - -Use [backfill migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/) if: - -- You need to move historical data without disrupting live traffic. -- You want to backfill indexes from a specific point in time without impacting the source cluster. -- You want to verify historical data in the target cluster before switching over. -- You want to backfill using an existing or incremental snapshot. -- You need the fastest backfill option that includes reindexing. -- You want the ability to pause and resume migration. - -### RFS - -Use [RFS]({{site.url}}{{site.baseurl}}/migration-assistant/deploying-migration-assistant/getting-started-data-migration/) if: - -- You already use OpenSearch snapshots for backups. -- You need to migrate documents at scale in parallel, such as with Amazon Elastic Container Service (Amazon ECS). -- You require a data migration path as part of a zero-downtime migration. -- Your AWS Region supports RFS and your shard sizes are within supported limits. - -### Combination of all three - -Use a combination of all three migration types if: - -- You're performing a complex, multi-version migration. -- You require zero downtime and full validation of the target environment. -- You want end-to-end tooling for metadata, data movement, and cluster behavior comparison. -- You're cloning an existing cluster and changing the source's configuration. -- You're setting up disaster recovery. ## Checklist -Use this checklist to decide whether Migration Assistant is right for you: - -- Are you migrating across one or more major versions? - -- Do you need to maintain service availability with zero downtime? - -- Do you need to validate a new OpenSearch cluster before switching over? +Use this checklist to determine whether Migration Assistant is the right fit for your migration: +- Are you migrating across one or more major versions—for example, from Elasticsearch 5 to OpenSearch 3—in a single step? +- Are you upgrading but want the ability to safely back out, reducing the risk of data loss or service disruption? +- Do you need to maintain high service availability with minimal or zero downtime? +- Do you need to validate a new OpenSearch cluster before switching over, with rollback capabilities? - Is your environment self-managed or running on Amazon OpenSearch Service? - -- Are you looking for tooling that can automate metadata migration and performance comparison? +- Are you looking for tooling to migrate index settings and other metadata? +- Do you need to reconfigure your target cluster—for example, by changing the sharding strategy and reindexing? +- Are you migrating across regions, from on-premises, or from another cloud provider? +- Do you need a high-performance backfill solution that can reliably reindex documents—with support for pause, resume, or checkpoint recovery? If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration. From 3e364faf9eb3fab5221ab06b9e53b1d7cc1e9fe0 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Wed, 2 Jul 2025 18:22:32 -0500 Subject: [PATCH 04/19] Updated requirements around VPC connectivity. Signed-off-by: Brian Presley --- .../is-migration-assistant-right-for-you.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 11d2a3cf10e..0cda220dc7b 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -21,10 +21,25 @@ Before using Migration Assistant, review the following assumptions and limitatio ### General + - If deploying on AWS, the identity used to deploy Migration Assistant must have permission to install all required resources. For a full list of services, see [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). - The target cluster must be deployed and reachable by Migration Assistant components, including the Migration Console, Reindex-from-Snapshot, and Traffic Replayer, depending on which features are used. - The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for more information. - If deploying to AWS, ensure the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). +- If deploying Migration Assistant into an existing AWS VPC, you must configure VPC interface endpoints and IAM permissions for the following AWS services: + - **Amazon CloudWatch Logs** – for log ingestion from ECS tasks. + - **Amazon CloudWatch** – for metrics published during migration + - **Amazon Elastic Container Registry (ECR)** – for pulling container images. + - **Amazon Elastic Container Service (ECS)** – for task orchestration and migration. + - **Elastic Load Balancing (ELB)** – for routing traffic to Capture Proxy. + - **AWS Secrets Manager** – for storing credentials. + - **AWS Systems Manager Parameter Store** – for storing and accessing parameter values. + - **AWS Systems Manager Session Manager** – for secure shell access to EC2 (i.e., bootstrap instance). + - **Amazon EC2** – for launching the bootstrap instance responsible for build and deployment. + - **Amazon Elastic Block Store (EBS)** – for disk storage during migration. + - **Amazon Virtual Private Cloud (VPC)** – for private networking and VPC endpoints. + - **AWS X-Ray** – for distributed tracing across components. + - **Amazon Elastic File System (EFS)** – for persistent logging. ### Reindex-from-Snapshot From 38b6d733fff381bbf51f7dece506ef5f636cedab Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Mon, 7 Jul 2025 15:49:51 -0500 Subject: [PATCH 05/19] Updated list of Migration Assistant products to vale github action products list. --- .github/vale/styles/Vocab/OpenSearch/Products/accept.txt | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/vale/styles/Vocab/OpenSearch/Products/accept.txt b/.github/vale/styles/Vocab/OpenSearch/Products/accept.txt index a308212b734..b77aa68b1a1 100644 --- a/.github/vale/styles/Vocab/OpenSearch/Products/accept.txt +++ b/.github/vale/styles/Vocab/OpenSearch/Products/accept.txt @@ -13,6 +13,7 @@ Anthropic Claude Apache Kafka Auditbeat AWS Cloud +Capture-and-Replay Cohere Command Cohere Embed Cohere Embed English @@ -61,8 +62,10 @@ Log4j Logstash Lucene macOS +Metadata-Migration-Tool Metricbeat Microsoft Power BI +Migration Assistant Minikube Nagios Okta @@ -93,6 +96,7 @@ PyTorch Querqy Query Workbench RankLib +Reindex-from-Snapshot RCF Summarize RPM Package Manager Ruby @@ -104,6 +108,8 @@ Titan Titan Multimodal Embeddings Titan Text Embeddings TorchScript +Traffic Capture Proxy +Traffic Replayer Tribuo VisBuilder Winlogbeat From 27dd7445d82b4eb343ce6a7385e008061e4eec17 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Mon, 7 Jul 2025 18:29:30 -0500 Subject: [PATCH 06/19] Added further breakdown of networking requirements Signed-off-by: Brian Presley --- .../getting-started-data-migration.md | 3 + .../is-migration-assistant-right-for-you.md | 91 +++++++++++++------ 2 files changed, 64 insertions(+), 30 deletions(-) diff --git a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md index 5e0ac5dcf59..2f1970b27db 100644 --- a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md +++ b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md @@ -14,6 +14,9 @@ This quickstart outlines how to deploy Migration Assistant for OpenSearch and ex Before using this quickstart, make sure you review ["Is Migration Assistant right for you?"]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths). + + Since this guide uses [AWS Cloud Development Kit](https://aws.amazon.com/cdk/), ensure the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). + --- ## Step 1: Install Bootstrap on an Amazon EC2 instance (~10 minutes) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 0cda220dc7b..07b6fbc9f1d 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -17,43 +17,74 @@ Migration Assistant also supports live traffic replication, enabling zero-downti ## Migration Assistant assumptions and limitations -Before using Migration Assistant, review the following assumptions and limitations. They are grouped by scope to clarify which apply generally and which are specific to components. - -### General - - -- If deploying on AWS, the identity used to deploy Migration Assistant must have permission to install all required resources. For a full list of services, see [AWS Services in this Solution](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/architecture-details.html#aws-services-in-this-solution). -- The target cluster must be deployed and reachable by Migration Assistant components, including the Migration Console, Reindex-from-Snapshot, and Traffic Replayer, depending on which features are used. -- The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/) for more information. -- If deploying to AWS, ensure the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). -- If deploying Migration Assistant into an existing AWS VPC, you must configure VPC interface endpoints and IAM permissions for the following AWS services: - - **Amazon CloudWatch Logs** – for log ingestion from ECS tasks. - - **Amazon CloudWatch** – for metrics published during migration - - **Amazon Elastic Container Registry (ECR)** – for pulling container images. - - **Amazon Elastic Container Service (ECS)** – for task orchestration and migration. - - **Elastic Load Balancing (ELB)** – for routing traffic to Capture Proxy. - - **AWS Secrets Manager** – for storing credentials. - - **AWS Systems Manager Parameter Store** – for storing and accessing parameter values. - - **AWS Systems Manager Session Manager** – for secure shell access to EC2 (i.e., bootstrap instance). - - **Amazon EC2** – for launching the bootstrap instance responsible for build and deployment. - - **Amazon Elastic Block Store (EBS)** – for disk storage during migration. - - **Amazon Virtual Private Cloud (VPC)** – for private networking and VPC endpoints. - - **AWS X-Ray** – for distributed tracing across components. - - **Amazon Elastic File System (EFS)** – for persistent logging. +Before using Migration Assistant, review the following assumptions and limitations. + +### Networking and Environment + +Migration Assistant requires network connectivity to AWS services and outbound internet access to build and deploy. Requirements differ based on whether you're deploying into a new or existing VPC. + +#### Source and Target Connectivity + +- You must establish connectivity between: + - The source cluster and/or Amazon S3 (for snapshots) and Migration Assistant + - The target cluster and Migration Assistant +- If the source or target resides in a private VPC without internet access, use one of the following to connect: + - VPC endpoints + - VPC peering + - AWS Transit Gateway + +#### Deploying into a New VPC + +- Migration Assistant provisions a new VPC with required components (e.g., NAT gateway, subnets). +- You must establish network access from this VPC to both the source (or S3) and target clusters. + +#### Deploying into an Existing VPC + +- If installing Migration Assistant into an existing VPC (e.g., same VPC as the source or target), you may need to configure connectivity to any cluster external to the target VPC. + - For example, if installed in the source VPC, you may need VPC endpoints or peering to reach the target. + +- Ensure all required AWS services are reachable by Migration Assistant components. + + - If the VPC has outbound access via: + - Private subnets with a NAT gateway, or + - Public subnets with an Internet Gateway + + then VPC interface endpoints are not required. + + - If using isolated subnets with no outbound access, you must configure VPC interface endpoints or routing to the following services: + + - **Amazon CloudWatch** – Publishes migration metrics. + - **Amazon CloudWatch Logs** – Ingests ECS task logs. + - **Amazon EC2** – Launches the bootstrap instance. + - **Amazon Elastic Block Store (Amazon EBS)** – Provides temporary disk storage. + - **Amazon Elastic Container Registry (Amazon ECR)** – Pulls container images. + - **Amazon Elastic Container Service (Amazon ECS)** – Orchestrates container workloads. + - **Amazon Elastic File System (Amazon EFS)** – Stores persistent logs. + - **Amazon Managed Streaming for Apache Kafka** – Used by `Capture-and-Replay` for HTTP event streaming. + - **Amazon S3** – Stores and retrieves snapshots and artifacts. + - **AWS Elastic Load Balancing (ELB)** – Routes traffic to the Capture Proxy. + - **AWS Secrets Manager** – Stores credentials securely. + - **AWS Systems Manager Parameter Store** – Holds configuration parameters. + - **AWS Systems Manager Session Manager** – Enables secure EC2 shell access. + - **AWS X-Ray** – Supports distributed tracing. + - **Amazon VPC** – Ensure proper routing, DNS resolution, and endpoint configuration. + +#### Bootstrap Instance (Build Host) + +- When deploying via CloudFormation, the bootstrap EC2 instance requires outbound internet access (via NAT gateway or IGW) to download the latest release from GitHub. ### Reindex-from-Snapshot +- The `_source` field must be enabled on all indices to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/). - The source cluster must have the Amazon S3 plugin installed. -- Snapshots must include global cluster state (`include_global_state` is `true`). -- Shards up to 80 GiB are supported by default. Larger shards can be configured, except in AWS GovCloud, where 80 GiB is the maximum supported size. +- Snapshots must include global cluster state (`include_global_state: true`). +- Shards up to **80 GiB** are supported by default. Larger shard sizes can be configured, except in **AWS GovCloud**, where 80 GiB is the maximum. ### Capture-and-Replay -- The Traffic Capture Proxy must be deployed to intercept relevant client traffic. -- Live capture is recommended only for environments with less than 4 TB/day of incoming traffic to the source cluster. -- Automatically generated document IDs are not preserved for index requests. Clients must explicitly provide document IDs when indexing or updating documents. - ---- +- The Traffic Capture Proxy must be deployed to intercept client traffic. +- Live capture is recommended only for workloads with **< 4 TB/day** of incoming traffic to the source cluster. +- Automatically generated document IDs are **not preserved** during replay. Clients must explicitly provide document IDs for `index` and `update` operations. ## Supported migration paths From a36554195639ae83f5943f4d0e3dc5f1566c5c60 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Mon, 7 Jul 2025 18:30:49 -0500 Subject: [PATCH 07/19] Moved limitations and assumptions Signed-off-by: Brian Presley --- .../is-migration-assistant-right-for-you.md | 142 +++++++++--------- 1 file changed, 71 insertions(+), 71 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 07b6fbc9f1d..cf1e53a22a9 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -15,77 +15,6 @@ Migration Assistant addresses key limitations in traditional migration approache Migration Assistant also supports live traffic replication, enabling zero-downtime migrations. This makes it a strong fit for environments where minimizing service disruption is critical. -## Migration Assistant assumptions and limitations - -Before using Migration Assistant, review the following assumptions and limitations. - -### Networking and Environment - -Migration Assistant requires network connectivity to AWS services and outbound internet access to build and deploy. Requirements differ based on whether you're deploying into a new or existing VPC. - -#### Source and Target Connectivity - -- You must establish connectivity between: - - The source cluster and/or Amazon S3 (for snapshots) and Migration Assistant - - The target cluster and Migration Assistant -- If the source or target resides in a private VPC without internet access, use one of the following to connect: - - VPC endpoints - - VPC peering - - AWS Transit Gateway - -#### Deploying into a New VPC - -- Migration Assistant provisions a new VPC with required components (e.g., NAT gateway, subnets). -- You must establish network access from this VPC to both the source (or S3) and target clusters. - -#### Deploying into an Existing VPC - -- If installing Migration Assistant into an existing VPC (e.g., same VPC as the source or target), you may need to configure connectivity to any cluster external to the target VPC. - - For example, if installed in the source VPC, you may need VPC endpoints or peering to reach the target. - -- Ensure all required AWS services are reachable by Migration Assistant components. - - - If the VPC has outbound access via: - - Private subnets with a NAT gateway, or - - Public subnets with an Internet Gateway - - then VPC interface endpoints are not required. - - - If using isolated subnets with no outbound access, you must configure VPC interface endpoints or routing to the following services: - - - **Amazon CloudWatch** – Publishes migration metrics. - - **Amazon CloudWatch Logs** – Ingests ECS task logs. - - **Amazon EC2** – Launches the bootstrap instance. - - **Amazon Elastic Block Store (Amazon EBS)** – Provides temporary disk storage. - - **Amazon Elastic Container Registry (Amazon ECR)** – Pulls container images. - - **Amazon Elastic Container Service (Amazon ECS)** – Orchestrates container workloads. - - **Amazon Elastic File System (Amazon EFS)** – Stores persistent logs. - - **Amazon Managed Streaming for Apache Kafka** – Used by `Capture-and-Replay` for HTTP event streaming. - - **Amazon S3** – Stores and retrieves snapshots and artifacts. - - **AWS Elastic Load Balancing (ELB)** – Routes traffic to the Capture Proxy. - - **AWS Secrets Manager** – Stores credentials securely. - - **AWS Systems Manager Parameter Store** – Holds configuration parameters. - - **AWS Systems Manager Session Manager** – Enables secure EC2 shell access. - - **AWS X-Ray** – Supports distributed tracing. - - **Amazon VPC** – Ensure proper routing, DNS resolution, and endpoint configuration. - -#### Bootstrap Instance (Build Host) - -- When deploying via CloudFormation, the bootstrap EC2 instance requires outbound internet access (via NAT gateway or IGW) to download the latest release from GitHub. - -### Reindex-from-Snapshot - -- The `_source` field must be enabled on all indices to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/). -- The source cluster must have the Amazon S3 plugin installed. -- Snapshots must include global cluster state (`include_global_state: true`). -- Shards up to **80 GiB** are supported by default. Larger shard sizes can be configured, except in **AWS GovCloud**, where 80 GiB is the maximum. - -### Capture-and-Replay - -- The Traffic Capture Proxy must be deployed to intercept client traffic. -- Live capture is recommended only for workloads with **< 4 TB/day** of incoming traffic to the source cluster. -- Automatically generated document IDs are **not preserved** during replay. Clients must explicitly provide document IDs for `index` and `update` operations. - ## Supported migration paths The matrix below shows which source versions can be directly migrated to which OpenSearch target versions: @@ -170,3 +99,74 @@ Use this checklist to determine whether Migration Assistant is the right fit for - Do you need a high-performance backfill solution that can reliably reindex documents—with support for pause, resume, or checkpoint recovery? If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration. + +## Migration Assistant assumptions and limitations + +Before using Migration Assistant, review the following assumptions and limitations. + +### Networking and Environment + +Migration Assistant requires network connectivity to AWS services and outbound internet access to build and deploy. Requirements differ based on whether you're deploying into a new or existing VPC. + +#### Source and Target Connectivity + +- You must establish connectivity between: + - The source cluster and/or Amazon S3 (for snapshots) and Migration Assistant + - The target cluster and Migration Assistant +- If the source or target resides in a private VPC without internet access, use one of the following to connect: + - VPC endpoints + - VPC peering + - AWS Transit Gateway + +#### Deploying into a New VPC + +- Migration Assistant provisions a new VPC with required components (e.g., NAT gateway, subnets). +- You must establish network access from this VPC to both the source (or S3) and target clusters. + +#### Deploying into an Existing VPC + +- If installing Migration Assistant into an existing VPC (e.g., same VPC as the source or target), you may need to configure connectivity to any cluster external to the target VPC. + - For example, if installed in the source VPC, you may need VPC endpoints or peering to reach the target. + +- Ensure all required AWS services are reachable by Migration Assistant components. + + - If the VPC has outbound access via: + - Private subnets with a NAT gateway, or + - Public subnets with an Internet Gateway + + then VPC interface endpoints are not required. + + - If using isolated subnets with no outbound access, you must configure VPC interface endpoints or routing to the following services: + + - **Amazon CloudWatch** – Publishes migration metrics. + - **Amazon CloudWatch Logs** – Ingests ECS task logs. + - **Amazon EC2** – Launches the bootstrap instance. + - **Amazon Elastic Block Store (Amazon EBS)** – Provides temporary disk storage. + - **Amazon Elastic Container Registry (Amazon ECR)** – Pulls container images. + - **Amazon Elastic Container Service (Amazon ECS)** – Orchestrates container workloads. + - **Amazon Elastic File System (Amazon EFS)** – Stores persistent logs. + - **Amazon Managed Streaming for Apache Kafka** – Used by `Capture-and-Replay` for HTTP event streaming. + - **Amazon S3** – Stores and retrieves snapshots and artifacts. + - **AWS Elastic Load Balancing (ELB)** – Routes traffic to the Capture Proxy. + - **AWS Secrets Manager** – Stores credentials securely. + - **AWS Systems Manager Parameter Store** – Holds configuration parameters. + - **AWS Systems Manager Session Manager** – Enables secure EC2 shell access. + - **AWS X-Ray** – Supports distributed tracing. + - **Amazon VPC** – Ensure proper routing, DNS resolution, and endpoint configuration. + +#### Bootstrap Instance (Build Host) + +- When deploying via CloudFormation, the bootstrap EC2 instance requires outbound internet access (via NAT gateway or IGW) to download the latest release from GitHub. + +### Reindex-from-Snapshot + +- The `_source` field must be enabled on all indices to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/). +- The source cluster must have the Amazon S3 plugin installed. +- Snapshots must include global cluster state (`include_global_state: true`). +- Shards up to **80 GiB** are supported by default. Larger shard sizes can be configured, except in **AWS GovCloud**, where 80 GiB is the maximum. + +### Capture-and-Replay + +- The Traffic Capture Proxy must be deployed to intercept client traffic. +- Live capture is recommended only for workloads with **< 4 TB/day** of incoming traffic to the source cluster. +- Automatically generated document IDs are **not preserved** during replay. Clients must explicitly provide document IDs for `index` and `update` operations. From a67a2be3166a19de40d3b73379c5af556acf266c Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Mon, 7 Jul 2025 18:32:50 -0500 Subject: [PATCH 08/19] Removed Github issue reference for Kibana Signed-off-by: Brian Presley --- .../overview/is-migration-assistant-right-for-you.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index cf1e53a22a9..46659a0a767 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -77,12 +77,10 @@ Before starting an upgrade or migration, consider the cluster feature to be incl | **Component templates** | Yes | Migrate with the `Metadata-Migration-Tool`. | | **Aliases** | Yes | Migrate with the `Metadata-Migration-Tool`. | | **Index State Management (ISM) policies** | Expected in 2025 | Manually migrate using an API. For more information about ISM support, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). | -| **Elasticsearch Kibana[^2] dashboards** | Expected in 2025 | This tool is only needed when migrating from Elasticsearch Kibana dashboards to OpenSearch Dashboards. Start by exporting JSON files from Kibana and importing them into OpenSearch Dashboards. For Elasticsearch versions 7.10.2 to 7.17, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool before importing X-Pack visualizations like Canvas and Lens in Kibana dashboards, as they may require recreation for compatibility with OpenSearch.| +| **Elasticsearch Kibana dashboards** | Expected in 2025 | This tool is only needed when migrating from Elasticsearch Kibana dashboards to OpenSearch Dashboards. Start by exporting JSON files from Kibana and importing them into OpenSearch Dashboards. For Elasticsearch versions 7.10.2 to 7.17, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool before importing X-Pack visualizations like Canvas and Lens in Kibana dashboards, as they may require recreation for compatibility with OpenSearch.| | **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage AWS Identity and Access Management (IAM) for enhanced security management. | | **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct OpenSearch equivalents. | -[^2]: Support for Kibana 5.0 through 7.10.2 migration paths to OpenSearch Dashboards will be added in a future version. Kibana 8 and later are not supported. For more information, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). - ## Checklist From 200024d899efcf0bb180139f19a3004180f8c270 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Mon, 7 Jul 2025 18:42:25 -0500 Subject: [PATCH 09/19] Identify services that are only Capture-and-Replay Signed-off-by: Brian Presley --- .../overview/is-migration-assistant-right-for-you.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 46659a0a767..c210e1f96da 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -143,9 +143,9 @@ Migration Assistant requires network connectivity to AWS services and outbound i - **Amazon Elastic Container Registry (Amazon ECR)** – Pulls container images. - **Amazon Elastic Container Service (Amazon ECS)** – Orchestrates container workloads. - **Amazon Elastic File System (Amazon EFS)** – Stores persistent logs. - - **Amazon Managed Streaming for Apache Kafka** – Used by `Capture-and-Replay` for HTTP event streaming. + - **Amazon Managed Streaming for Apache Kafka** – (`Capture-and-Replay` only) - Used as a durable storage to capture and replay HTTP live traffic. - **Amazon S3** – Stores and retrieves snapshots and artifacts. - - **AWS Elastic Load Balancing (ELB)** – Routes traffic to the Capture Proxy. + - **AWS Elastic Load Balancing (ELB)** – (`Capture-and-Replay` only) - Routes traffic to the Capture Proxy. - **AWS Secrets Manager** – Stores credentials securely. - **AWS Systems Manager Parameter Store** – Holds configuration parameters. - **AWS Systems Manager Session Manager** – Enables secure EC2 shell access. From 0a801a892896808a2d099c42194a1bc6efabbbd7 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Tue, 8 Jul 2025 08:49:47 -0500 Subject: [PATCH 10/19] Added ALB to list of services. Updated verbiage for ELB. Signed-off-by: Brian Presley --- .../overview/is-migration-assistant-right-for-you.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index c210e1f96da..b4828f0a278 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -136,6 +136,7 @@ Migration Assistant requires network connectivity to AWS services and outbound i - If using isolated subnets with no outbound access, you must configure VPC interface endpoints or routing to the following services: + - **Amazon Application Load Balancer (ALB)** – *(`Capture-and-Replay` only)* - Used to optionally reroute client traffic from the source to the target during migration. - **Amazon CloudWatch** – Publishes migration metrics. - **Amazon CloudWatch Logs** – Ingests ECS task logs. - **Amazon EC2** – Launches the bootstrap instance. @@ -143,9 +144,9 @@ Migration Assistant requires network connectivity to AWS services and outbound i - **Amazon Elastic Container Registry (Amazon ECR)** – Pulls container images. - **Amazon Elastic Container Service (Amazon ECS)** – Orchestrates container workloads. - **Amazon Elastic File System (Amazon EFS)** – Stores persistent logs. - - **Amazon Managed Streaming for Apache Kafka** – (`Capture-and-Replay` only) - Used as a durable storage to capture and replay HTTP live traffic. + - **Amazon Managed Streaming for Apache Kafka** – *(`Capture-and-Replay` only)* - Used as a durable storage to capture and replay HTTP live traffic. - **Amazon S3** – Stores and retrieves snapshots and artifacts. - - **AWS Elastic Load Balancing (ELB)** – (`Capture-and-Replay` only) - Routes traffic to the Capture Proxy. + - **AWS Elastic Load Balancing (ELB)** – *(Capture-and-Replay only)* – Used by the Migration Console to connect to the Application Load Balancer (ALB). - **AWS Secrets Manager** – Stores credentials securely. - **AWS Systems Manager Parameter Store** – Holds configuration parameters. - **AWS Systems Manager Session Manager** – Enables secure EC2 shell access. From dcffac5ded57bcb7eb0bc660812a99195c08ffaa Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Tue, 8 Jul 2025 09:12:53 -0500 Subject: [PATCH 11/19] Updated based on style-job suggestions. Signed-off-by: Brian Presley --- .../is-migration-assistant-right-for-you.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index b4828f0a278..268c1b13d90 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -17,7 +17,7 @@ Migration Assistant also supports live traffic replication, enabling zero-downti ## Supported migration paths -The matrix below shows which source versions can be directly migrated to which OpenSearch target versions: +The following matrix shows which source versions can be directly migrated to which OpenSearch target versions: @@ -66,7 +66,7 @@ Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/mi ## Supported features -Before starting an upgrade or migration, consider the cluster feature to be included. The table below lists what can be migrated using Migration Assistant, whether it is currently supported, and recommendations for how to handle each component. +Before starting an upgrade or migration, consider the cluster feature to be included. The following table lists what can be migrated using Migration Assistant, whether it is currently supported, and recommendations for how to handle each component. | Feature | Supported | Recommendations | | :--- |:--- | :--- | @@ -128,7 +128,7 @@ Migration Assistant requires network connectivity to AWS services and outbound i - Ensure all required AWS services are reachable by Migration Assistant components. - - If the VPC has outbound access via: + - If the VPC has outbound access using: - Private subnets with a NAT gateway, or - Public subnets with an Internet Gateway @@ -139,7 +139,7 @@ Migration Assistant requires network connectivity to AWS services and outbound i - **Amazon Application Load Balancer (ALB)** – *(`Capture-and-Replay` only)* - Used to optionally reroute client traffic from the source to the target during migration. - **Amazon CloudWatch** – Publishes migration metrics. - **Amazon CloudWatch Logs** – Ingests ECS task logs. - - **Amazon EC2** – Launches the bootstrap instance. + - **Amazon EC2** – Used to bootstrap Migration Assistant. When deploying using CloudFormation, the bootstrap EC2 instance requires outbound internet access (using NAT gateway or IGW) to download the latest release from GitHub. - **Amazon Elastic Block Store (Amazon EBS)** – Provides temporary disk storage. - **Amazon Elastic Container Registry (Amazon ECR)** – Pulls container images. - **Amazon Elastic Container Service (Amazon ECS)** – Orchestrates container workloads. @@ -153,13 +153,9 @@ Migration Assistant requires network connectivity to AWS services and outbound i - **AWS X-Ray** – Supports distributed tracing. - **Amazon VPC** – Ensure proper routing, DNS resolution, and endpoint configuration. -#### Bootstrap Instance (Build Host) - -- When deploying via CloudFormation, the bootstrap EC2 instance requires outbound internet access (via NAT gateway or IGW) to download the latest release from GitHub. - ### Reindex-from-Snapshot -- The `_source` field must be enabled on all indices to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/). +- The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/). - The source cluster must have the Amazon S3 plugin installed. - Snapshots must include global cluster state (`include_global_state: true`). - Shards up to **80 GiB** are supported by default. Larger shard sizes can be configured, except in **AWS GovCloud**, where 80 GiB is the maximum. From 242887c3da01ac75996167b61f1fc194b3c56840 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Tue, 8 Jul 2025 12:41:48 -0500 Subject: [PATCH 12/19] Updated S3 verbiage Signed-off-by: Brian Presley --- .../overview/is-migration-assistant-right-for-you.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 268c1b13d90..e817e3ebaa6 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -109,7 +109,7 @@ Migration Assistant requires network connectivity to AWS services and outbound i #### Source and Target Connectivity - You must establish connectivity between: - - The source cluster and/or Amazon S3 (for snapshots) and Migration Assistant + - The source cluster and/or Amazon S3 (for snapshots, which may only require updating the bucket policy) and Migration Assistant - The target cluster and Migration Assistant - If the source or target resides in a private VPC without internet access, use one of the following to connect: - VPC endpoints @@ -119,7 +119,7 @@ Migration Assistant requires network connectivity to AWS services and outbound i #### Deploying into a New VPC - Migration Assistant provisions a new VPC with required components (e.g., NAT gateway, subnets). -- You must establish network access from this VPC to both the source (or S3) and target clusters. +- You must establish network access from this VPC to both the source and target clusters. #### Deploying into an Existing VPC @@ -146,7 +146,7 @@ Migration Assistant requires network connectivity to AWS services and outbound i - **Amazon Elastic File System (Amazon EFS)** – Stores persistent logs. - **Amazon Managed Streaming for Apache Kafka** – *(`Capture-and-Replay` only)* - Used as a durable storage to capture and replay HTTP live traffic. - **Amazon S3** – Stores and retrieves snapshots and artifacts. - - **AWS Elastic Load Balancing (ELB)** – *(Capture-and-Replay only)* – Used by the Migration Console to connect to the Application Load Balancer (ALB). + - **AWS Elastic Load Balancing (ELB)** – *(Capture-and-Replay only)* – Used by the Migration console to connect to the Application Load Balancer. - **AWS Secrets Manager** – Stores credentials securely. - **AWS Systems Manager Parameter Store** – Holds configuration parameters. - **AWS Systems Manager Session Manager** – Enables secure EC2 shell access. From 70e22e555cdffbfae9af8f3bfcd785f57569cc50 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Tue, 8 Jul 2025 17:07:36 -0500 Subject: [PATCH 13/19] Remove duplicate compression bullet point. Signed-off-by: Brian Presley --- .../overview/is-migration-assistant-right-for-you.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index e817e3ebaa6..c47d83671a6 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -157,7 +157,9 @@ Migration Assistant requires network connectivity to AWS services and outbound i - The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/). - The source cluster must have the Amazon S3 plugin installed. -- Snapshots must include global cluster state (`include_global_state: true`). +- If you choose to bring your own snapshot (i.e., one not created by Migration Assistant), the following settings must be applied when creating the snapshot: + - `include_global_state: true` – Ensures that global cluster state is included. + - `compress: false` – Disables metadata compression, which is required for compatibility with Reindex-from-Snapshot. - Shards up to **80 GiB** are supported by default. Larger shard sizes can be configured, except in **AWS GovCloud**, where 80 GiB is the maximum. ### Capture-and-Replay From 7178b11f0e626ddf85505ac4d5e39e2521fcc29a Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Tue, 8 Jul 2025 17:12:32 -0500 Subject: [PATCH 14/19] Indicate that secrets manager is only used for basic auth creds. Signed-off-by: Brian Presley --- .../overview/is-migration-assistant-right-for-you.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index c47d83671a6..9a1900b37da 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -147,7 +147,7 @@ Migration Assistant requires network connectivity to AWS services and outbound i - **Amazon Managed Streaming for Apache Kafka** – *(`Capture-and-Replay` only)* - Used as a durable storage to capture and replay HTTP live traffic. - **Amazon S3** – Stores and retrieves snapshots and artifacts. - **AWS Elastic Load Balancing (ELB)** – *(Capture-and-Replay only)* – Used by the Migration console to connect to the Application Load Balancer. - - **AWS Secrets Manager** – Stores credentials securely. + - **AWS Secrets Manager** – Securely stores credentials when using basic authentication on source or target. - **AWS Systems Manager Parameter Store** – Holds configuration parameters. - **AWS Systems Manager Session Manager** – Enables secure EC2 shell access. - **AWS X-Ray** – Supports distributed tracing. From 45ceabf74c811cbc938569e5ddb7099ce7bb42e9 Mon Sep 17 00:00:00 2001 From: Brian Presley Date: Tue, 8 Jul 2025 17:18:44 -0500 Subject: [PATCH 15/19] Updated SSM verbiage replacing EC2 w/ ECS Signed-off-by: Brian Presley --- .../overview/is-migration-assistant-right-for-you.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 9a1900b37da..248afb4cc64 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -149,7 +149,7 @@ Migration Assistant requires network connectivity to AWS services and outbound i - **AWS Elastic Load Balancing (ELB)** – *(Capture-and-Replay only)* – Used by the Migration console to connect to the Application Load Balancer. - **AWS Secrets Manager** – Securely stores credentials when using basic authentication on source or target. - **AWS Systems Manager Parameter Store** – Holds configuration parameters. - - **AWS Systems Manager Session Manager** – Enables secure EC2 shell access. + - **AWS Systems Manager Session Manager** – Enables secure shell access to Elastic Container Service tasks such as the Migration console. - **AWS X-Ray** – Supports distributed tracing. - **Amazon VPC** – Ensure proper routing, DNS resolution, and endpoint configuration. From 6e475d66f820e0369b09798c1868ee54cf820d30 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Thu, 10 Jul 2025 11:16:45 -0500 Subject: [PATCH 16/19] Add writer edits Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../is-migration-assistant-right-for-you.md | 32 ++++++++++++------- 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 248afb4cc64..f2162319a34 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -19,7 +19,6 @@ Migration Assistant also supports live traffic replication, enabling zero-downti The following matrix shows which source versions can be directly migrated to which OpenSearch target versions: - {% comment %}First, collect all unique target versions{% endcomment %} {% assign all_targets = "" | split: "" %} @@ -62,7 +61,7 @@ The following matrix shows which source versions can be directly migrated to whi **Supported AWS Regions** -Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the for the full list of supported regions. +Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the full list of supported regions. ## Supported features @@ -78,10 +77,9 @@ Before starting an upgrade or migration, consider the cluster feature to be incl | **Aliases** | Yes | Migrate with the `Metadata-Migration-Tool`. | | **Index State Management (ISM) policies** | Expected in 2025 | Manually migrate using an API. For more information about ISM support, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). | | **Elasticsearch Kibana dashboards** | Expected in 2025 | This tool is only needed when migrating from Elasticsearch Kibana dashboards to OpenSearch Dashboards. Start by exporting JSON files from Kibana and importing them into OpenSearch Dashboards. For Elasticsearch versions 7.10.2 to 7.17, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool before importing X-Pack visualizations like Canvas and Lens in Kibana dashboards, as they may require recreation for compatibility with OpenSearch.| -| **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage AWS Identity and Access Management (IAM) for enhanced security management. | +| **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, use AWS Identity and Access Management (IAM) for enhanced security management. | | **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct OpenSearch equivalents. | - ## Checklist Use this checklist to determine whether Migration Assistant is the right fit for your migration: @@ -108,6 +106,8 @@ Migration Assistant requires network connectivity to AWS services and outbound i #### Source and Target Connectivity +To meet connectivity requirements, ensure the following: + - You must establish connectivity between: - The source cluster and/or Amazon S3 (for snapshots, which may only require updating the bucket policy) and Migration Assistant - The target cluster and Migration Assistant @@ -118,14 +118,18 @@ Migration Assistant requires network connectivity to AWS services and outbound i #### Deploying into a New VPC -- Migration Assistant provisions a new VPC with required components (e.g., NAT gateway, subnets). +When deploying into a new VPC, consider the following: + +- Migration Assistant provisions a new VPC with required components (for example, NAT gateway, subnets). - You must establish network access from this VPC to both the source and target clusters. #### Deploying into an Existing VPC -- If installing Migration Assistant into an existing VPC (e.g., same VPC as the source or target), you may need to configure connectivity to any cluster external to the target VPC. +When deploying into an existing VPC, follow these considerations: + +- If installing Migration Assistant into an existing VPC (for example, same VPC as the source or target), you may need to configure connectivity to any cluster external to the target VPC. - For example, if installed in the source VPC, you may need VPC endpoints or peering to reach the target. - + - Ensure all required AWS services are reachable by Migration Assistant components. - If the VPC has outbound access using: @@ -136,7 +140,7 @@ Migration Assistant requires network connectivity to AWS services and outbound i - If using isolated subnets with no outbound access, you must configure VPC interface endpoints or routing to the following services: - - **Amazon Application Load Balancer (ALB)** – *(`Capture-and-Replay` only)* - Used to optionally reroute client traffic from the source to the target during migration. + - **Amazon Application Load Balancer (ALB)** – *(Capture-and-Replay only)* – Used to optionally reroute client traffic from the source to the target during migration. - **Amazon CloudWatch** – Publishes migration metrics. - **Amazon CloudWatch Logs** – Ingests ECS task logs. - **Amazon EC2** – Used to bootstrap Migration Assistant. When deploying using CloudFormation, the bootstrap EC2 instance requires outbound internet access (using NAT gateway or IGW) to download the latest release from GitHub. @@ -144,7 +148,7 @@ Migration Assistant requires network connectivity to AWS services and outbound i - **Amazon Elastic Container Registry (Amazon ECR)** – Pulls container images. - **Amazon Elastic Container Service (Amazon ECS)** – Orchestrates container workloads. - **Amazon Elastic File System (Amazon EFS)** – Stores persistent logs. - - **Amazon Managed Streaming for Apache Kafka** – *(`Capture-and-Replay` only)* - Used as a durable storage to capture and replay HTTP live traffic. + - **Amazon Managed Streaming for Apache Kafka** – *(Capture-and-Replay only)* – Used as a durable storage to capture and replay HTTP live traffic. - **Amazon S3** – Stores and retrieves snapshots and artifacts. - **AWS Elastic Load Balancing (ELB)** – *(Capture-and-Replay only)* – Used by the Migration console to connect to the Application Load Balancer. - **AWS Secrets Manager** – Securely stores credentials when using basic authentication on source or target. @@ -153,16 +157,20 @@ Migration Assistant requires network connectivity to AWS services and outbound i - **AWS X-Ray** – Supports distributed tracing. - **Amazon VPC** – Ensure proper routing, DNS resolution, and endpoint configuration. -### Reindex-from-Snapshot +### Reindex-from-snapshot + +To use Reindex-from-Snapshot, ensure the following: - The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/). - The source cluster must have the Amazon S3 plugin installed. -- If you choose to bring your own snapshot (i.e., one not created by Migration Assistant), the following settings must be applied when creating the snapshot: +- If you choose to bring your own snapshot (that is, one not created by Migration Assistant), the following settings must be applied when creating the snapshot: - `include_global_state: true` – Ensures that global cluster state is included. - `compress: false` – Disables metadata compression, which is required for compatibility with Reindex-from-Snapshot. - Shards up to **80 GiB** are supported by default. Larger shard sizes can be configured, except in **AWS GovCloud**, where 80 GiB is the maximum. -### Capture-and-Replay +### Capture-and-replay + +Capture-and-Replay has the following requirements: - The Traffic Capture Proxy must be deployed to intercept client traffic. - Live capture is recommended only for workloads with **< 4 TB/day** of incoming traffic to the source cluster. From 1339b1be855e00ce5d78f6f10a92c6f54b1d629b Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Thu, 10 Jul 2025 11:18:18 -0500 Subject: [PATCH 17/19] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../getting-started-data-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md index 2f1970b27db..df963506d26 100644 --- a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md +++ b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md @@ -15,7 +15,7 @@ This quickstart outlines how to deploy Migration Assistant for OpenSearch and ex Before using this quickstart, make sure you review ["Is Migration Assistant right for you?"]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths). - Since this guide uses [AWS Cloud Development Kit](https://aws.amazon.com/cdk/), ensure the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). +Because this guide uses [AWS Cloud Development Kit](https://aws.amazon.com/cdk/), make sure that the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). --- From b9c53f9dc042a59a0e81edf62f9b09ffc0e7ec7b Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Thu, 10 Jul 2025 11:19:51 -0500 Subject: [PATCH 18/19] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../overview/is-migration-assistant-right-for-you.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index f2162319a34..0de2fc2a897 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -100,11 +100,11 @@ If you answered "yes" to most of these questions, Migration Assistant is likely Before using Migration Assistant, review the following assumptions and limitations. -### Networking and Environment +### Networking and environment Migration Assistant requires network connectivity to AWS services and outbound internet access to build and deploy. Requirements differ based on whether you're deploying into a new or existing VPC. -#### Source and Target Connectivity +#### Source and target connectivity To meet connectivity requirements, ensure the following: @@ -116,14 +116,14 @@ To meet connectivity requirements, ensure the following: - VPC peering - AWS Transit Gateway -#### Deploying into a New VPC +#### Deploying into a new VPC When deploying into a new VPC, consider the following: - Migration Assistant provisions a new VPC with required components (for example, NAT gateway, subnets). - You must establish network access from this VPC to both the source and target clusters. -#### Deploying into an Existing VPC +#### Deploying into an existing VPC When deploying into an existing VPC, follow these considerations: From 1b225aca5ec102a2b35921722a91463d61a6c445 Mon Sep 17 00:00:00 2001 From: Nathan Bower Date: Thu, 10 Jul 2025 13:16:19 -0400 Subject: [PATCH 19/19] Apply suggestions from code review Signed-off-by: Nathan Bower --- .../Vocab/OpenSearch/Products/accept.txt | 2 +- .../getting-started-data-migration.md | 4 +- .../is-migration-assistant-right-for-you.md | 69 +++++++++---------- 3 files changed, 36 insertions(+), 39 deletions(-) diff --git a/.github/vale/styles/Vocab/OpenSearch/Products/accept.txt b/.github/vale/styles/Vocab/OpenSearch/Products/accept.txt index b77aa68b1a1..0e76eff113d 100644 --- a/.github/vale/styles/Vocab/OpenSearch/Products/accept.txt +++ b/.github/vale/styles/Vocab/OpenSearch/Products/accept.txt @@ -13,7 +13,7 @@ Anthropic Claude Apache Kafka Auditbeat AWS Cloud -Capture-and-Replay +Capture and Replay Cohere Command Cohere Embed Cohere Embed English diff --git a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md index df963506d26..59639248312 100644 --- a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md +++ b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md @@ -12,10 +12,10 @@ redirect_from: This quickstart outlines how to deploy Migration Assistant for OpenSearch and execute an existing data migration using `Reindex-from-Snapshot` (RFS). It uses AWS for illustrative purposes. However, the steps can be modified for use with other cloud providers. -Before using this quickstart, make sure you review ["Is Migration Assistant right for you?"]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths). +Before using this quickstart, make sure you review [Is Migration Assistant right for you?]({{site.url}}{{site.baseurl}}/migration-assistant/overview/is-migration-assistant-right-for-you/#supported-migration-paths). -Because this guide uses [AWS Cloud Development Kit](https://aws.amazon.com/cdk/), make sure that the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). +Because this guide uses [AWS Cloud Development Kit (AWS CDK)](https://aws.amazon.com/cdk/), make sure that the `CDKToolkit` stack exists and is in the `CREATE_COMPLETE` state. For setup instructions, see the [CDK Toolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). --- diff --git a/_migration-assistant/overview/is-migration-assistant-right-for-you.md b/_migration-assistant/overview/is-migration-assistant-right-for-you.md index 0de2fc2a897..b7e84dadc33 100644 --- a/_migration-assistant/overview/is-migration-assistant-right-for-you.md +++ b/_migration-assistant/overview/is-migration-assistant-right-for-you.md @@ -57,11 +57,11 @@ The following matrix shows which source versions can be directly migrated to whi **Source and target platforms** - Self-managed (on premises or hosted by a cloud provider) -- Amazon OpenSearch Service (OpenSearch Serverless Collections are not supported) +- Amazon OpenSearch Service (Amazon OpenSearch Serverless collections are not supported) **Supported AWS Regions** -Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the full list of supported regions. +Refer to [Supported AWS Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the full list of supported Regions. ## Supported features @@ -69,14 +69,14 @@ Before starting an upgrade or migration, consider the cluster feature to be incl | Feature | Supported | Recommendations | | :--- |:--- | :--- | -| **Documents** | Yes | Migrate existing data with RFS and live traffic with capture and replay. | +| **Documents** | Yes | Migrate existing data with RFS and live traffic with Capture and Replay. | | **Index settings** | Yes | Migrate with the `Metadata-Migration-Tool`. | | **Index mappings** | Yes | Migrate with the `Metadata-Migration-Tool`. | | **Index templates** | Yes | Migrate with the `Metadata-Migration-Tool`. | | **Component templates** | Yes | Migrate with the `Metadata-Migration-Tool`. | | **Aliases** | Yes | Migrate with the `Metadata-Migration-Tool`. | | **Index State Management (ISM) policies** | Expected in 2025 | Manually migrate using an API. For more information about ISM support, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). | -| **Elasticsearch Kibana dashboards** | Expected in 2025 | This tool is only needed when migrating from Elasticsearch Kibana dashboards to OpenSearch Dashboards. Start by exporting JSON files from Kibana and importing them into OpenSearch Dashboards. For Elasticsearch versions 7.10.2 to 7.17, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool before importing X-Pack visualizations like Canvas and Lens in Kibana dashboards, as they may require recreation for compatibility with OpenSearch.| +| **Elasticsearch Kibana dashboards** | Expected in 2025 | This tool is only needed when migrating from Elasticsearch Kibana dashboards to OpenSearch Dashboards. Start by exporting JSON files from Kibana and importing them into OpenSearch Dashboards. For Elasticsearch versions 7.10.2 to 7.17, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool before importing X-Pack visualizations like Canvas and Lens into Kibana dashboards, as they may require recreation for compatibility with OpenSearch.| | **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, use AWS Identity and Access Management (IAM) for enhanced security management. | | **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct OpenSearch equivalents. | @@ -87,11 +87,11 @@ Use this checklist to determine whether Migration Assistant is the right fit for - Are you migrating across one or more major versions—for example, from Elasticsearch 5 to OpenSearch 3—in a single step? - Are you upgrading but want the ability to safely back out, reducing the risk of data loss or service disruption? - Do you need to maintain high service availability with minimal or zero downtime? -- Do you need to validate a new OpenSearch cluster before switching over, with rollback capabilities? +- Do you need to validate a new OpenSearch cluster before switching over---with rollback capabilities? - Is your environment self-managed or running on Amazon OpenSearch Service? - Are you looking for tooling to migrate index settings and other metadata? - Do you need to reconfigure your target cluster—for example, by changing the sharding strategy and reindexing? -- Are you migrating across regions, from on-premises, or from another cloud provider? +- Are you migrating across Regions, from on premises, or from another cloud provider? - Do you need a high-performance backfill solution that can reliably reindex documents—with support for pause, resume, or checkpoint recovery? If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration. @@ -102,75 +102,72 @@ Before using Migration Assistant, review the following assumptions and limitatio ### Networking and environment -Migration Assistant requires network connectivity to AWS services and outbound internet access to build and deploy. Requirements differ based on whether you're deploying into a new or existing VPC. +Connectivity to AWS services and outbound internet access are required in order to build and deploy Migration Assistant. Requirements differ based on whether you're deploying to a new or existing virtual private cloud (VPC). #### Source and target connectivity To meet connectivity requirements, ensure the following: - You must establish connectivity between: - - The source cluster and/or Amazon S3 (for snapshots, which may only require updating the bucket policy) and Migration Assistant - - The target cluster and Migration Assistant + - The source cluster and/or Amazon Simple Storage Service (Amazon S3) (for snapshots, which may only require updating the bucket policy) and Migration Assistant. + - The target cluster and Migration Assistant. - If the source or target resides in a private VPC without internet access, use one of the following to connect: - VPC endpoints - VPC peering - AWS Transit Gateway -#### Deploying into a new VPC +#### Deploying to a new VPC -When deploying into a new VPC, consider the following: +When deploying to a new VPC, consider the following: - Migration Assistant provisions a new VPC with required components (for example, NAT gateway, subnets). - You must establish network access from this VPC to both the source and target clusters. -#### Deploying into an existing VPC +#### Deploying to an existing VPC -When deploying into an existing VPC, follow these considerations: +When deploying to an existing VPC, consider the following: -- If installing Migration Assistant into an existing VPC (for example, same VPC as the source or target), you may need to configure connectivity to any cluster external to the target VPC. - - For example, if installed in the source VPC, you may need VPC endpoints or peering to reach the target. +- If deploying Migration Assistant to an existing VPC (for example, the same VPC as the source or target), you may need to configure connectivity to any cluster external to the target VPC. + - For example, if Migration Assistant is deployed to the source VPC, you may need VPC endpoints or peering to reach the target. - Ensure all required AWS services are reachable by Migration Assistant components. - - If the VPC has outbound access using: - - Private subnets with a NAT gateway, or - - Public subnets with an Internet Gateway + - If the VPC has outbound access using private subnets with a NAT gateway or public subnets with an internet gateway, then VPC interface endpoints are not required. - then VPC interface endpoints are not required. - If using isolated subnets with no outbound access, you must configure VPC interface endpoints or routing to the following services: - - **Amazon Application Load Balancer (ALB)** – *(Capture-and-Replay only)* – Used to optionally reroute client traffic from the source to the target during migration. + - **Application Load Balancer** (Capture and Replay only) – Used to optionally reroute client traffic from the source to the target during migration. - **Amazon CloudWatch** – Publishes migration metrics. - - **Amazon CloudWatch Logs** – Ingests ECS task logs. - - **Amazon EC2** – Used to bootstrap Migration Assistant. When deploying using CloudFormation, the bootstrap EC2 instance requires outbound internet access (using NAT gateway or IGW) to download the latest release from GitHub. + - **Amazon CloudWatch Logs** – Ingests Amazon Elastic Container Service (Amazon ECS) task logs. + - **Amazon Elastic Compute Cloud (Amazon EC2)** – Used to bootstrap Migration Assistant. When deploying using AWS CloudFormation, the bootstrap EC2 instance requires outbound internet access (using NAT gateway or an internet gateway) to download the latest version from GitHub. - **Amazon Elastic Block Store (Amazon EBS)** – Provides temporary disk storage. - **Amazon Elastic Container Registry (Amazon ECR)** – Pulls container images. - - **Amazon Elastic Container Service (Amazon ECS)** – Orchestrates container workloads. + - **Amazon ECS** – Orchestrates container workloads. - **Amazon Elastic File System (Amazon EFS)** – Stores persistent logs. - - **Amazon Managed Streaming for Apache Kafka** – *(Capture-and-Replay only)* – Used as a durable storage to capture and replay HTTP live traffic. + - **Amazon Managed Streaming for Apache Kafka (Amazon MSK)** (Capture and Replay only) – Used as durable storage to capture and replay live HTTP traffic. - **Amazon S3** – Stores and retrieves snapshots and artifacts. - - **AWS Elastic Load Balancing (ELB)** – *(Capture-and-Replay only)* – Used by the Migration console to connect to the Application Load Balancer. - - **AWS Secrets Manager** – Securely stores credentials when using basic authentication on source or target. - - **AWS Systems Manager Parameter Store** – Holds configuration parameters. - - **AWS Systems Manager Session Manager** – Enables secure shell access to Elastic Container Service tasks such as the Migration console. + - **Elastic Load Balancing** (Capture and Replay only) – Used by the migration console to connect to the Application Load Balancer. + - **AWS Secrets Manager** – Securely stores credentials when using basic authentication on the source or target. + - **AWS Systems Manager Parameter Store** – Stores configuration parameters. + - **AWS Systems Manager Session Manager** – Enables secure shell access to ECS tasks such as the migration console. - **AWS X-Ray** – Supports distributed tracing. - - **Amazon VPC** – Ensure proper routing, DNS resolution, and endpoint configuration. + - **Amazon Virtual Private Cloud (Amazon VPC)** – Ensures proper routing, DNS resolution, and endpoint configuration. -### Reindex-from-snapshot +### Reindex-from-Snapshot -To use Reindex-from-Snapshot, ensure the following: +To use `Reindex-from-Snapshot` (RFS), ensure the following: -- The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/). +- The `_source` field must be enabled on all indexes to be migrated. See [Source]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/). - The source cluster must have the Amazon S3 plugin installed. - If you choose to bring your own snapshot (that is, one not created by Migration Assistant), the following settings must be applied when creating the snapshot: - `include_global_state: true` – Ensures that global cluster state is included. - - `compress: false` – Disables metadata compression, which is required for compatibility with Reindex-from-Snapshot. -- Shards up to **80 GiB** are supported by default. Larger shard sizes can be configured, except in **AWS GovCloud**, where 80 GiB is the maximum. + - `compress: false` – Disables metadata compression, which is required for compatibility with RFS. +- Shards of up to **80 GiB** are supported by default. Larger shard sizes can be configured, **except in AWS GovCloud (US)**, where 80 GiB is the maximum. -### Capture-and-replay +### Capture and Replay -Capture-and-Replay has the following requirements: +Capture and Replay has the following requirements: - The Traffic Capture Proxy must be deployed to intercept client traffic. - Live capture is recommended only for workloads with **< 4 TB/day** of incoming traffic to the source cluster.
Source Version{{ target }}