-
Notifications
You must be signed in to change notification settings - Fork 578
Updated Is "Migration Right for you?" page and some dependencies #10158
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 17 commits
df3f743
c62887a
b58adc4
3e364fa
38b6d73
27dd744
a365541
a67a2be
200024d
0a801a8
dcffac5
242887c
70e22e5
7178b11
45ceabf
6e475d6
1339b1b
b9c53f9
1b225ac
61321f1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -9,13 +9,11 @@ | |||||
|
||||||
# Is Migration Assistant right for you? | ||||||
|
||||||
Deciding whether to use Migration Assistant depends on your specific upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant is right for your use case—or whether another tool might be a better fit. | ||||||
Whether Migration Assistant is right for you depends on your upgrade path, infrastructure complexity, and operational goals. This page will help you evaluate whether Migration Assistant fits your use case. | ||||||
|
||||||
Migration Assistant was built to fill important gaps in common migration strategies. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant lets you do this in a single step. Other methods, like rolling upgrades or snapshot restores, require you to upgrade through each major version, often reindexing your data at every step. | ||||||
Migration Assistant addresses key limitations in traditional migration approaches. For example, if you're upgrading across multiple major versions—such as from Elasticsearch 6.8 to OpenSearch 2.19—Migration Assistant enables you to complete the process in a single step. Other methods, like rolling upgrades or snapshot restores, require upgrading through each major version, often with reindexing at every stage. | ||||||
|
||||||
Migration Assistant also supports live traffic replication, allowing for zero-downtime migrations. This makes it a strong choice for production environments, where minimizing service disruption is critical. | ||||||
|
||||||
If your migration is limited to a static cluster configuration (like index templates and aliases), or if you're not concerned about downtime, simpler tools may be sufficient. But for complex migrations involving real-time traffic or major version jumps, Migration Assistant offers robust, flexible capabilities. | ||||||
Migration Assistant also supports live traffic replication, enabling zero-downtime migrations. This makes it a strong fit for environments where minimizing service disruption is critical. | ||||||
|
||||||
## Supported migration paths | ||||||
|
||||||
|
@@ -34,7 +32,7 @@ | |||||
<table class="migration-matrix"> | ||||||
<thead> | ||||||
<tr> | ||||||
<th></th> | ||||||
<th>Source Version</th> | ||||||
Check failure on line 35 in _migration-assistant/overview/is-migration-assistant-right-for-you.md
|
||||||
{% for target in unique_targets %} | ||||||
<th>{{ target }}</th> | ||||||
{% endfor %} | ||||||
|
@@ -59,28 +57,17 @@ | |||||
**Source and target platforms** | ||||||
|
||||||
- Self-managed (on premises or hosted by a cloud provider) | ||||||
- Amazon OpenSearch Service | ||||||
|
||||||
|
||||||
**AWS Regions** | ||||||
|
||||||
Migration Assistant is supported in the following AWS Regions: | ||||||
|
||||||
- US East (N. Virginia, Ohio) | ||||||
- US West (Oregon, N. California) | ||||||
- Europe (Frankfurt, Ireland, London) | ||||||
- Asia Pacific (Tokyo, Singapore, Sydney) | ||||||
- AWS GovCloud (US-East, US-West)[^1] | ||||||
- Amazon OpenSearch Service (OpenSearch Serverless Collections are not supported) | ||||||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
[^1]: In AWS GovCloud (US), `reindex-from-snapshot` (RFS) is limited to shard sizes of 80 GiB or smaller. | ||||||
**Supported AWS Regions** | ||||||
|
||||||
Refer to [AWS Supported Regions](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html#supported-aws-regions) for the full list of supported regions. | ||||||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
## Supported features | ||||||
|
||||||
## Supported components | ||||||
Before starting an upgrade or migration, consider the cluster feature to be included. The following table lists what can be migrated using Migration Assistant, whether it is currently supported, and recommendations for how to handle each component. | ||||||
|
||||||
Before starting a migration, consider the scope of the components involved. The following table outlines components that should potentially be migrated, indicates whether they are supported by Migration Assistant, and provides recommendations. | ||||||
|
||||||
| Component | Supported | Recommendations | | ||||||
| Feature | Supported | Recommendations | | ||||||
| :--- |:--- | :--- | | ||||||
| **Documents** | Yes | Migrate existing data with RFS and live traffic with capture and replay. | | ||||||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| **Index settings** | Yes | Migrate with the `Metadata-Migration-Tool`. | | ||||||
|
@@ -89,65 +76,102 @@ | |||||
| **Component templates** | Yes | Migrate with the `Metadata-Migration-Tool`. | | ||||||
| **Aliases** | Yes | Migrate with the `Metadata-Migration-Tool`. | | ||||||
| **Index State Management (ISM) policies** | Expected in 2025 | Manually migrate using an API. For more information about ISM support, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). | | ||||||
| **Elasticsearch Kibana[^2] dashboards** | Expected in 2025 | This tool is only needed when migrating from Elasticsearch Kibana dashboards to OpenSearch Dashboards. Start by exporting JSON files from Kibana and importing them into OpenSearch Dashboards. For Elasticsearch versions 7.10.2 to 7.17, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool before importing X-Pack visualizations like Canvas and Lens in Kibana dashboards, as they may require recreation for compatibility with OpenSearch.| | ||||||
| **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage AWS Identity and Access Management (IAM) for enhanced security management. | | ||||||
| **Elasticsearch Kibana dashboards** | Expected in 2025 | This tool is only needed when migrating from Elasticsearch Kibana dashboards to OpenSearch Dashboards. Start by exporting JSON files from Kibana and importing them into OpenSearch Dashboards. For Elasticsearch versions 7.10.2 to 7.17, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool before importing X-Pack visualizations like Canvas and Lens in Kibana dashboards, as they may require recreation for compatibility with OpenSearch.| | ||||||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, use AWS Identity and Access Management (IAM) for enhanced security management. | | ||||||
| **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct OpenSearch equivalents. | | ||||||
|
||||||
[^2]: Support for Kibana 5.0 through 7.10.2 migration paths to OpenSearch Dashboards will be added in a future version. Kibana 8 and later are not supported. For more information, see [issue #944](https://github.com/opensearch-project/opensearch-migrations/issues/944). | ||||||
## Checklist | ||||||
|
||||||
## Choosing your migration approach | ||||||
Use this checklist to determine whether Migration Assistant is the right fit for your migration: | ||||||
|
||||||
Use the following checklist to determine which Migration Assistant components best fit your use case. | ||||||
- Are you migrating across one or more major versions—for example, from Elasticsearch 5 to OpenSearch 3—in a single step? | ||||||
- Are you upgrading but want the ability to safely back out, reducing the risk of data loss or service disruption? | ||||||
- Do you need to maintain high service availability with minimal or zero downtime? | ||||||
- Do you need to validate a new OpenSearch cluster before switching over, with rollback capabilities? | ||||||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
- Is your environment self-managed or running on Amazon OpenSearch Service? | ||||||
- Are you looking for tooling to migrate index settings and other metadata? | ||||||
- Do you need to reconfigure your target cluster—for example, by changing the sharding strategy and reindexing? | ||||||
- Are you migrating across regions, from on-premises, or from another cloud provider? | ||||||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
- Do you need a high-performance backfill solution that can reliably reindex documents—with support for pause, resume, or checkpoint recovery? | ||||||
|
||||||
### Metadata migration | ||||||
If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration. | ||||||
|
||||||
Use [metadata migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) if: | ||||||
## Migration Assistant assumptions and limitations | ||||||
|
||||||
- You need to migrate while mitigating breaking changes between the source and target clusters, such as differences in mappings, settings, aliases, or component templates. | ||||||
- You want a relatively consistent configuration between the source and target clusters. | ||||||
Before using Migration Assistant, review the following assumptions and limitations. | ||||||
|
||||||
### Backfill migration | ||||||
### Networking and Environment | ||||||
Check failure on line 103 in _migration-assistant/overview/is-migration-assistant-right-for-you.md
|
||||||
Naarcha-AWS marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
Use [backfill migration]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/) if: | ||||||
Migration Assistant requires network connectivity to AWS services and outbound internet access to build and deploy. Requirements differ based on whether you're deploying into a new or existing VPC. | ||||||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
- You need to move historical data without disrupting live traffic. | ||||||
- You want to backfill indexes from a specific point in time without impacting the source cluster. | ||||||
- You want to verify historical data in the target cluster before switching over. | ||||||
- You want to backfill using an existing or incremental snapshot. | ||||||
- You need the fastest backfill option that includes reindexing. | ||||||
- You want the ability to pause and resume migration. | ||||||
#### Source and Target Connectivity | ||||||
Check failure on line 107 in _migration-assistant/overview/is-migration-assistant-right-for-you.md
|
||||||
Naarcha-AWS marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
### RFS | ||||||
To meet connectivity requirements, ensure the following: | ||||||
|
||||||
Use [RFS]({{site.url}}{{site.baseurl}}/migration-assistant/deploying-migration-assistant/getting-started-data-migration/) if: | ||||||
- You must establish connectivity between: | ||||||
- The source cluster and/or Amazon S3 (for snapshots, which may only require updating the bucket policy) and Migration Assistant | ||||||
sumobrian marked this conversation as resolved.
Show resolved
Hide resolved
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
- The target cluster and Migration Assistant | ||||||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
- If the source or target resides in a private VPC without internet access, use one of the following to connect: | ||||||
- VPC endpoints | ||||||
- VPC peering | ||||||
- AWS Transit Gateway | ||||||
|
||||||
- You already use OpenSearch snapshots for backups. | ||||||
- You need to migrate documents at scale in parallel, such as with Amazon Elastic Container Service (Amazon ECS). | ||||||
- You require a data migration path as part of a zero-downtime migration. | ||||||
- Your AWS Region supports RFS and your shard sizes are within supported limits. | ||||||
#### Deploying into a New VPC | ||||||
Naarcha-AWS marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
### Combination of all three | ||||||
When deploying into a new VPC, consider the following: | ||||||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
Use a combination of all three migration types if: | ||||||
- Migration Assistant provisions a new VPC with required components (for example, NAT gateway, subnets). | ||||||
Check failure on line 123 in _migration-assistant/overview/is-migration-assistant-right-for-you.md
|
||||||
- You must establish network access from this VPC to both the source and target clusters. | ||||||
|
||||||
- You're performing a complex, multi-version migration. | ||||||
- You require zero downtime and full validation of the target environment. | ||||||
sumobrian marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
- You want end-to-end tooling for metadata, data movement, and cluster behavior comparison. | ||||||
- You're cloning an existing cluster and changing the source's configuration. | ||||||
- You're setting up disaster recovery. | ||||||
#### Deploying into an Existing VPC | ||||||
Naarcha-AWS marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
## Checklist | ||||||
When deploying into an existing VPC, follow these considerations: | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
Use this checklist to decide whether Migration Assistant is right for you: | ||||||
- If installing Migration Assistant into an existing VPC (for example, same VPC as the source or target), you may need to configure connectivity to any cluster external to the target VPC. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- For example, if installed in the source VPC, you may need VPC endpoints or peering to reach the target. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
- Are you migrating across one or more major versions? | ||||||
- Ensure all required AWS services are reachable by Migration Assistant components. | ||||||
|
||||||
- Do you need to maintain service availability with zero downtime? | ||||||
- If the VPC has outbound access using: | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- Private subnets with a NAT gateway, or | ||||||
Check failure on line 136 in _migration-assistant/overview/is-migration-assistant-right-for-you.md
|
||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- Public subnets with an Internet Gateway | ||||||
Check failure on line 137 in _migration-assistant/overview/is-migration-assistant-right-for-you.md
|
||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
- Do you need to validate a new OpenSearch cluster before switching over? | ||||||
then VPC interface endpoints are not required. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
- Is your environment self-managed or running on Amazon OpenSearch Service? | ||||||
- If using isolated subnets with no outbound access, you must configure VPC interface endpoints or routing to the following services: | ||||||
Check failure on line 141 in _migration-assistant/overview/is-migration-assistant-right-for-you.md
|
||||||
|
||||||
- Are you looking for tooling that can automate metadata migration and performance comparison? | ||||||
- **Amazon Application Load Balancer (ALB)** – *(Capture-and-Replay only)* – Used to optionally reroute client traffic from the source to the target during migration. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- **Amazon CloudWatch** – Publishes migration metrics. | ||||||
- **Amazon CloudWatch Logs** – Ingests ECS task logs. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- **Amazon EC2** – Used to bootstrap Migration Assistant. When deploying using CloudFormation, the bootstrap EC2 instance requires outbound internet access (using NAT gateway or IGW) to download the latest release from GitHub. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- **Amazon Elastic Block Store (Amazon EBS)** – Provides temporary disk storage. | ||||||
- **Amazon Elastic Container Registry (Amazon ECR)** – Pulls container images. | ||||||
- **Amazon Elastic Container Service (Amazon ECS)** – Orchestrates container workloads. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- **Amazon Elastic File System (Amazon EFS)** – Stores persistent logs. | ||||||
- **Amazon Managed Streaming for Apache Kafka** – *(Capture-and-Replay only)* – Used as a durable storage to capture and replay HTTP live traffic. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- **Amazon S3** – Stores and retrieves snapshots and artifacts. | ||||||
- **AWS Elastic Load Balancing (ELB)** – *(Capture-and-Replay only)* – Used by the Migration console to connect to the Application Load Balancer. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- **AWS Secrets Manager** – Securely stores credentials when using basic authentication on source or target. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- **AWS Systems Manager Parameter Store** – Holds configuration parameters. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- **AWS Systems Manager Session Manager** – Enables secure shell access to Elastic Container Service tasks such as the Migration console. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- **AWS X-Ray** – Supports distributed tracing. | ||||||
- **Amazon VPC** – Ensure proper routing, DNS resolution, and endpoint configuration. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
If you answered "yes" to most of these questions, Migration Assistant is likely the right solution for your migration. | ||||||
### Reindex-from-snapshot | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
To use Reindex-from-Snapshot, ensure the following: | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
- The `_source` field must be enabled on all indexes to be migrated. See [Source documentation]({{site.url}}{{site.baseurl}}/field-types/metadata-fields/source/). | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- The source cluster must have the Amazon S3 plugin installed. | ||||||
- If you choose to bring your own snapshot (that is, one not created by Migration Assistant), the following settings must be applied when creating the snapshot: | ||||||
- `include_global_state: true` – Ensures that global cluster state is included. | ||||||
- `compress: false` – Disables metadata compression, which is required for compatibility with Reindex-from-Snapshot. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- Shards up to **80 GiB** are supported by default. Larger shard sizes can be configured, except in **AWS GovCloud**, where 80 GiB is the maximum. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
### Capture-and-replay | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
Capture-and-Replay has the following requirements: | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
- The Traffic Capture Proxy must be deployed to intercept client traffic. | ||||||
- Live capture is recommended only for workloads with **< 4 TB/day** of incoming traffic to the source cluster. | ||||||
- Automatically generated document IDs are **not preserved** during replay. Clients must explicitly provide document IDs for `index` and `update` operations. |
Uh oh!
There was an error while loading. Please reload this page.