Skip to content

Add Replication APIs and automation #9904

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add Replication APIs and automation #9904

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fabd8e6
Add Replication APIs
Naarcha-AWS May 14, 2025
e526516
Edits for intro
Naarcha-AWS May 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions _api-reference/replication-apis/create-replication-rule.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
layout: default
title: Create Replication Rule API
parent: Replication APIs
nav_order: 55
---

# Create Replication Rule API

Check failure on line 8 in _api-reference/replication-apis/create-replication-rule.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _api-reference/replication-apis/create-replication-rule.md#L8 

[OpenSearch.HeadingCapitalization] 'Create Replication Rule API' is a heading and should be in sentence case.
Raw output 
{"message": "[OpenSearch.HeadingCapitalization] 'Create Replication Rule API' is a heading and should be in sentence case.", "location": {"path": "_api-reference/replication-apis/create-replication-rule.md", "range": {"start": {"line": 8, "column": 3}}}, "severity": "ERROR"}
Introduced 1.0
{: .label .label-purple }

The Create Replication Rule API establishes automatic replication patterns (auto-follow rules) that monitor for new indexes matching specified patterns and automatically configure them for replication. This allows for automated replication management without requiring manual setup for each new index.

<!-- spec_insert_start
api: replication.create_replication_rule
component: endpoints
-->
## Endpoints
```json
POST /_plugins/_replication/_autofollow
```
<!-- spec_insert_end -->

## Request body fields

The following table lists the available request body fields.

| Field | Data type | Description |
| :--- | :--- | :--- |
| `leader_alias` | String | The name of an index alias to use when referring to the leader index. |
| `name` | String | The name of the replication rule. Required. |
| `pattern` | String | The pattern used to match indexes for automatic replication. Required. |
| `use_roles` | Object | Object containing custom roles for the replication connection. |

<details markdown="block">
<summary>
Request body fields: <code>use_roles</code>
</summary>
{: .text-delta}

`use_roles` is a JSON object with the following fields:

| Field | Data type | Description |
| :--- | :--- | :--- |
| `follower_cluster_role` | String | The role used for the follower cluster during replication. |
| `leader_cluster_role` | String | The role used for the leader cluster during replication. |
</details>

## Example request

The following example creates a replication rule named "logs-rule" that automatically replicates any index matching the "logs-*" pattern:

```json
POST /_plugins/_replication/_autofollow
{
"name": "logs-rule",
"pattern": "logs-*",
"leader_alias": "logs-primary"
}
```
{% include copy-curl.html %}

The following example creates a replication rule with custom roles:

```json
POST /_plugins/_replication/_autofollow
{
"name": "metrics-rule",
"pattern": "metrics-*",
"leader_alias": "metrics-primary",
"use_roles": {
"leader_cluster_role": "replication_leader_role",
"follower_cluster_role": "replication_follower_role"
}
}
```
{% include copy-curl.html %}

## Example response

The following example shows a successful API response:

```json
{
"acknowledged": true
}
```
56 changes: 56 additions & 0 deletions _api-reference/replication-apis/delete-replication-rule.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
---
layout: default
title: Delete Replication Rule API
parent: Replication APIs
nav_order: 60
---

# Delete Replication Rule API

Check failure on line 8 in _api-reference/replication-apis/delete-replication-rule.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _api-reference/replication-apis/delete-replication-rule.md#L8 

[OpenSearch.HeadingCapitalization] 'Delete Replication Rule API' is a heading and should be in sentence case.
Raw output 
{"message": "[OpenSearch.HeadingCapitalization] 'Delete Replication Rule API' is a heading and should be in sentence case.", "location": {"path": "_api-reference/replication-apis/delete-replication-rule.md", "range": {"start": {"line": 8, "column": 3}}}, "severity": "ERROR"}
Introduced 1.0
{: .label .label-purple }

The Delete Replication Rule API removes an existing auto-follow replication rule. When deleted, the rule will no longer trigger automatic replication for new indexes that match its pattern. However, existing replication relationships that were established by the rule will continue to operate until explicitly stopped.

<!-- spec_insert_start
api: replication.delete_replication_rule
component: endpoints
-->
## Endpoints
```json
DELETE /_plugins/_replication/_autofollow
```
<!-- spec_insert_end -->

<!-- spec_insert_start
api: replication.delete_replication_rule
component: request_body_parameters
-->
## Request body fields

The request body is __required__. It is a JSON object with the following fields.

| Property | Data type | Description |
| :--- | :--- | :--- |
| `leader_alias` | String | The name of an index alias. |
| `name` | String | The name of the replication rule to delete. |

<!-- spec_insert_end -->

## Example request

The following example deletes a replication rule named "logs-rule":

```json
DELETE /_plugins/_replication/_autofollow?name=logs-rule
```
{% include copy-curl.html %}

## Example response

The following example shows a successful API response:

```json
{
"acknowledged": true
}
```
95 changes: 95 additions & 0 deletions _api-reference/replication-apis/get-autofollow-stats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
layout: default
title: Get Auto-follow Stats API
parent: Replication APIs
nav_order: 45
---

# Get Auto-follow Stats API

Check failure on line 8 in _api-reference/replication-apis/get-autofollow-stats.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _api-reference/replication-apis/get-autofollow-stats.md#L8 

[OpenSearch.HeadingCapitalization] 'Get Auto-follow Stats API' is a heading and should be in sentence case.
Raw output 
{"message": "[OpenSearch.HeadingCapitalization] 'Get Auto-follow Stats API' is a heading and should be in sentence case.", "location": {"path": "_api-reference/replication-apis/get-autofollow-stats.md", "range": {"start": {"line": 8, "column": 3}}}, "severity": "ERROR"}
Introduced 1.0
{: .label .label-purple }

The Get Auto-follow Stats API retrieves statistics about auto-follow patterns and their replication activities. This API helps you monitor the performance and status of automatic index replication configured through auto-follow patterns.

<!-- spec_insert_start
api: replication.autofollow_stats
component: endpoints
-->
## Endpoints
```json
GET /_plugins/_replication/autofollow_stats
```
<!-- spec_insert_end -->

## Example request

The following example gets statistics for all configured auto-follow patterns:

```json
GET /_plugins/_replication/autofollow_stats
```
{% include copy-curl.html %}

## Example response

The following example shows a successful API response with statistics for two auto-follow patterns:

```json
{
"num_success_start_replication": 12,
"num_failed_start_replication": 1,
"num_failed_leader_calls": 0,
"failed_indices": ["logs-2023-error"],
"autofollow_stats": [
{
"name": "logs-pattern",
"pattern": "logs-*",
"num_success_start_replication": 10,
"num_failed_start_replication": 1,
"num_failed_leader_calls": 0,
"last_execution_time": 1645729365423,
"failed_indices": ["logs-2023-error"]
},
{
"name": "metrics-pattern",
"pattern": "metrics-*",
"num_success_start_replication": 2,
"num_failed_start_replication": 0,
"num_failed_leader_calls": 0,
"last_execution_time": 1645729245895,
"failed_indices": []
}
]
}
```

## Response body fields

The following table lists all response body fields.

| Field | Data type | Description |
| :--- | :--- | :--- |
| `autofollow_stats` | Array of Objects | A list of statistics for each auto-follow pattern. |
| `failed_indices` | Array of Strings | The list of indexes that failed to replicate across all patterns. |
| `num_failed_leader_calls` | Float | The number of failed calls to the leader cluster across all patterns. |
| `num_failed_start_replication` | Float | The number of failed replication starts across all patterns. |
| `num_success_start_replication` | Float | The number of successful replication starts across all patterns. |

<details markdown="block">
<summary>
Response body fields: <code>autofollow_stats</code>
</summary>
{: .text-delta}

`autofollow_stats` is an array of JSON objects. Each object represents a single auto-follow pattern and has the following fields:

| Field | Data type | Description |
| :--- | :--- | :--- |
| `failed_indices` | Array of Strings | The list of indexes that failed to replicate for this pattern. |
| `last_execution_time` | Float | When the last execution of this auto-follow pattern occurred. |
| `name` | String | The name of the auto-follow pattern. |
| `num_failed_leader_calls` | Float | The number of failed calls to the leader cluster for this pattern. |
| `num_failed_start_replication` | Float | The number of failed replication starts for this pattern. |
| `num_success_start_replication` | Float | The number of successful replication starts for this pattern. |
| `pattern` | String | The pattern used for auto-following indexes. |
</details>
119 changes: 119 additions & 0 deletions _api-reference/replication-apis/get-follower-stats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
---
layout: default
title: Get Follower Stats API
parent: Replication APIs
nav_order: 40
---

# Get Follower Stats API

Check failure on line 8 in _api-reference/replication-apis/get-follower-stats.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _api-reference/replication-apis/get-follower-stats.md#L8 

[OpenSearch.HeadingCapitalization] 'Get Follower Stats API' is a heading and should be in sentence case.
Raw output 
{"message": "[OpenSearch.HeadingCapitalization] 'Get Follower Stats API' is a heading and should be in sentence case.", "location": {"path": "_api-reference/replication-apis/get-follower-stats.md", "range": {"start": {"line": 8, "column": 3}}}, "severity": "ERROR"}
Introduced 1.0
{: .label .label-purple }

The Get Follower Stats API retrieves statistics about follower indexes in the replication process. This API provides detailed metrics about operations, progress, and performance of follower indexes during replication activities.

## Endpoints

```json
GET /_plugins/_replication/follower_stats
```

## Example request

The following example gets statistics about all follower indexes in the replication process:

```json
GET /_plugins/_replication/follower_stats
```
{% include copy-curl.html %}

## Example response

The following example shows a successful API response with statistics for two follower indexes:

```json
{
"num_shard_tasks": 7,
"num_index_tasks": 2,
"num_bootstrapping_indices": 0,
"num_syncing_indices": 2,
"num_paused_indices": 0,
"num_failed_indices": 0,
"operations_read": 1254,
"operations_written": 1254,
"failed_read_requests": 0,
"failed_write_requests": 0,
"throttled_read_requests": 5,
"throttled_write_requests": 2,
"leader_checkpoint": 42,
"follower_checkpoint": 40,
"total_write_time_millis": 1850,
"index_stats": {
"customer-data": {
"operations_read": 854,
"operations_written": 854,
"failed_read_requests": 0,
"failed_write_requests": 0,
"throttled_read_requests": 3,
"throttled_write_requests": 1,
"leader_checkpoint": 27,
"follower_checkpoint": 25,
"total_write_time_millis": 1250
},
"product-data": {
"operations_read": 400,
"operations_written": 400,
"failed_read_requests": 0,
"failed_write_requests": 0,
"throttled_read_requests": 2,
"throttled_write_requests": 1,
"leader_checkpoint": 15,
"follower_checkpoint": 15,
"total_write_time_millis": 600
}
}
}
```

## Response body fields

The following table lists all response body fields.

| Field | Data type | Description |
| :--- | :--- | :--- |
| `failed_read_requests` | Float | The number of failed read requests during replication. |
| `failed_write_requests` | Float | The number of failed write requests during replication. |
| `follower_checkpoint` | Float | The current checkpoint of the follower index. |
| `index_stats` | Object | Per-index statistics for all indexes being replicated. |
| `leader_checkpoint` | Float | The current checkpoint of the leader index. |
| `num_bootstrapping_indices` | Float | The number of indexes currently bootstrapping. |
| `num_failed_indices` | Float | The number of indexes that have failed replication. |
| `num_index_tasks` | Float | The number of active index-level replication tasks. |
| `num_paused_indices` | Float | The number of indexes currently paused. |
| `num_shard_tasks` | Float | The number of active shard-level replication tasks. |
| `num_syncing_indices` | Float | The number of indexes currently syncing. |
| `operations_read` | Float | The total number of operations read during replication. |
| `operations_written` | Float | The total number of operations written during replication. |
| `throttled_read_requests` | Float | The number of throttled read requests during replication. |
| `throttled_write_requests` | Float | The number of throttled write requests during replication. |
| `total_write_time_millis` | Integer or String | The total time in milliseconds spent writing operations. |

<details markdown="block">
<summary>
Response body fields: <code>index_stats</code>
</summary>
{: .text-delta}

`index_stats` is a JSON object with index names as keys. Each index entry contains the following fields:

| Field | Data type | Description |
| :--- | :--- | :--- |
| `failed_read_requests` | Float | The number of failed read requests for this index during replication. |
| `failed_write_requests` | Float | The number of failed write requests for this index during replication. |
| `follower_checkpoint` | Float | The current checkpoint of this follower index. |
| `leader_checkpoint` | Float | The current checkpoint of the leader index for this follower. |
| `operations_read` | Float | The number of operations read for this index during replication. |
| `operations_written` | Float | The number of operations written for this index during replication. |
| `throttled_read_requests` | Float | The number of throttled read requests for this index during replication. |
| `throttled_write_requests` | Float | The number of throttled write requests for this index during replication. |
| `total_write_time_millis` | Integer or String | The total time in milliseconds spent writing operations for this index. |
</details>
Loading
Loading