Added the English version of the docs for asynchronous replication (#9609)

anton-bobkov · CyberROFL · blinkov · web-flow · commit b429de0e4203 · 2024-09-23T15:40:14.000+03:00
Co-authored-by: Ilnaz Nizametdinov &lt;i.nizametdinov@gmail.com&gt;
Co-authored-by: Ivan Blinkov &lt;ivan@blinkov.ru&gt;
diff --git a/ydb/docs/en/core/concepts/async-replication.md b/ydb/docs/en/core/concepts/async-replication.md
@@ -0,0 +1,142 @@
+# Asynchronous replication
+
+Asynchronous replication allows for synchronizing data between {{ ydb-short-name }} [databases](glossary.md#database) in near real time. It can also be used for data migration between databases with minimal downtime for applications interacting with these databases. Such databases can be located in the same {{ ydb-short-name }} [cluster](glossary.md#cluster) as well as in different clusters.
+
+## Overview {#how-it-works}
+
+Asynchronous replication is based on [Change Data Capture](cdc.md) and operates on logical data. The following diagram illustrates the replication process:
+
+```mermaid
+sequenceDiagram
+  participant dst as Source
+  participant src as Target
+
+  dst-->src: Initialization
+  dst->>dst: Creating an asynchronous replication instance
+  dst->>src: Creating changefeeds
+  dst->>dst: Creating replica objects
+
+  dst-->src: Initial table scan
+  loop
+    dst->>src: Request to get data
+    src->>dst: Data
+  end
+
+  dst-->src: Change data replication
+  loop
+    dst->>src: Request to get data
+    src->>dst: Data
+  end
+```
+
+As shown in the diagram above, asynchronous replication involves two databases:
+
+1. **Source**. A database with [replicated objects](glossary.md#replicated-object).
+2. **Target**. A database where an [asynchronous replication instance](glossary.md#async-replication-instance) and [replica objects](glossary.md#replica-object) will be created.
+
+Asynchronous replication consists of the following stages:
+
+* [Initialization](#init)
+* [Initial table scan](#initial-scan)
+* [Change data replication](#replication-of-changes)
+
+### Initialization {#init}
+
+Initialization of asynchronous replication includes the following steps:
+
+* Creating an asynchronous replication instance on the target database using the [CREATE ASYNC REPLICATION](../yql/reference/syntax/create-async-replication.md) YQL expression.
+* Establishing a connection with the source database. The target database connects to the source using the [connection parameters](../yql/reference/syntax/create-async-replication.md#params) specified during the creation of the asynchronous replication instance.
+
+{% note info %}
+
+The user account that is used to connect to the source database must have the following [permissions](../security/short-access-control-notation.md#access-rights):
+
+* Read permissions for schema objects and directory objects
+* Create, update, delete, and read permissions for changefeeds
+
+{% endnote %}
+
+* The following objects are created for replicated objects on the source:
+  * [changefeeds](glossary.md#changefeed) on the source
+  * [replica objects](glossary.md#replica-object) on the target
+
+{% note info %}
+
+Replicas are created under the user account that was used to create the asynchronous replication instance.
+
+{% endnote %}
+
+### Initial table scan {#initial-scan}
+
+During the [initial table scan](cdc.md#initial-scan) the source data is exported to changefeeds. The target runs [consumers](topic.md#consumer) that read the source data from the changefeeds and write it to replicas.
+
+You can get the progress of the initial table scan from the [description](../reference/ydb-cli/commands/scheme-describe.md) of the asynchronous replication instance.
+
+### Change data replication {#replication-of-changes}
+
+After the initial table scan is completed, the consumers read the change data and write it to replicas.
+
+Each change data block has its *creation time* ($created\_at$). Consumers track the *reception time* of the change data ($received\_at$). Thus, you can use the following formula to calculate the *replication lag*:
+
+$$
+replication\_lag = received\_at - created\_at
+$$
+
+You can also get the replication lag from the [description](../reference/ydb-cli/commands/scheme-describe.md) of the asynchronous replication instance.
+
+## Restrictions {#restrictions}
+
+* The set of replicated objects is immutable and is generated when {{ ydb-short-name }} creates an asynchronous replication instance.
+* {{ ydb-short-name }} supports the following types of replicated objects:
+
+  * [row-based tables](datamodel/table.md#row-oriented-tables)
+  * [directories](datamodel/dir.md)
+
+    {{ ydb-short-name }} will replicate all row-based tables that are located in the given directories and subdirectories at the time the asynchronous replication instance is created.
+
+* During asynchronous replication, you cannot [add or delete columns](../yql/reference/syntax/alter_table/columns.md) in the source tables.
+* During asynchronous replication, replicas are available only for reading.
+
+## Error handling during asynchronous replication {#error-handling}
+
+Possible errors during asynchronous replication can be grouped into the following classes:
+
+* **Temporary failures**, such as transport errors, system overload, etc. Requests will be resent until they are processed successfully.
+* **Critical errors**, such as access violation errors, schema errors, etc. Replication will be aborted, and the [description](../reference/ydb-cli/commands/scheme-describe.md) of the asynchronous replication instance will include the text of the error.
+
+{% note warning %}
+
+Currently, asynchronous replication that is aborted due to a critical error cannot be resumed. In this case, you must [drop](../yql/reference/syntax/drop-async-replication.md) and [create](../yql/reference/syntax/create-async-replication.md) a new asynchronous replication instance.
+
+{% endnote %}
+
+For more information about error classes and how to address them, refer to [Error handling](../reference/ydb-sdk/error_handling.md).
+
+## Asynchronous replication completion {#done}
+
+Completion of asynchronous replication might be an end goal of data migration from one database to another. In this case the client stops writing data to the source, waits for the zero replication lag, and completes replication. After the replication process is completed, replicas become available both for reading and writing. Then you can switch the load from the source database to the target database and complete data migration.
+
+{% note info %}
+
+You cannot resume completed asynchronous replication.
+
+{% endnote %}
+
+{% note warning %}
+
+{{ ydb-short-name }} currently supports only **forced** completion of asynchronous replication, when no additional checks are performed for data consistency, replication lag, etc.
+
+{% endnote %}
+
+To complete asynchronous replication, use the [ALTER ASYNC REPLICATION](../yql/reference/syntax/alter-async-replication.md) YQL expression.
+
+## Dropping an asynchronous replication instance {#drop}
+
+When you drop an asynchronous replication instance:
+
+* Changefeeds are deleted in the source tables.
+* The source tables are unlocked, and you can add and delete columns again.
+* Optionally, all replicas are deleted.
+* Asynchronous replication instance is deleted.
+
+To drop an asynchronous replication instance, use the [DROP ASYNC REPLICATION](../yql/reference/syntax/drop-async-replication.md) YQL expression.
diff --git a/ydb/docs/en/core/concepts/glossary.md b/ydb/docs/en/core/concepts/glossary.md
@@ -181,7 +181,23 @@ A **consumer** is an entity that reads messages from a topic.
 
 ### Change data capture {#cdc}
 
-**Change data capture** or **CDC** is a mechanism that allows subscribing to a stream of changes to a given [table](#table). Technically, it is implemented on top of [topics](#topic). It is described in more detail in a separate article [{#T}](cdc.md).
+**Change data capture** or **CDC** is a mechanism that allows subscribing to a **stream of changes** to a given [table](#table). Technically, it is implemented on top of [topics](#topic). It is described in more detail in a separate article [{#T}](cdc.md).
+
+#### Changefeed {#changefeed}
+
+**Changefeed** or **stream of changes** is an ordered list of changes in a given [table](#table) published via a [topic](#topic).
+
+### Asynchronous replication instance {#async-replication-instance}
+
+**Asynchronous replication instance** is a named entity that stores [asynchronous replication](async-replication.md) settings (connection properties, a list of replicated objects, etc.) It can also be used to retrieve the status of asynchronous replication, such as the [initial synchronization process](async-replication.md#initial-scan), [replication lag](async-replication.md#replication-of-changes), [errors](async-replication.md#error-handling), and more.
+
+#### Replicated object {#replicated-object}
+
+**Replicated object** is an object, for example, a table, that is asynchronously replicated to the target database.
+
+#### Replica object {#replica-object}
+
+**Replica object** is a mirror copy of the replicated object, automatically created by an [asynchronous replication instance](#async-replication-instance). Replica objects are typically read-only.
 
 ### YQL {#yql}
 
diff --git a/ydb/docs/en/core/concepts/toc_i.yaml b/ydb/docs/en/core/concepts/toc_i.yaml
@@ -22,6 +22,7 @@ items:
     href: cluster/common_scheme_ydb.md
   - name: Disk subsystem of a cluster
     href: cluster/distributed_storage.md
+- { name: Asynchronous replication,               href: async-replication.md, when: feature_async_replication }
 - name: Federated query
   include: { path: federated_query/toc_p.yaml, mode: link }
 - name: Query optimizer
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/alter-async-replication.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/alter-async-replication.md
@@ -0,0 +1,35 @@
+# ALTER ASYNC REPLICATION
+
+The `ALTER ASYNC REPLICATION` statement modifies the status and parameters of an [asynchronous replication instance](../../../concepts/async-replication.md).
+
+## Syntax {#syntax}
+
+```yql
+ALTER ASYNC REPLICATION <name> SET (option = value [, ...])
+```
+
+### Parameters {#params}
+
+* `name` — a name of the asynchronous replication instance.
+* `SET (option = value [, ...])` — asynchronous replication parameters:
+
+    * `STATE` — the state of asynchronous replication. This parameter can only be used in combination with the `FAILOVER_MODE` parameter (see below). Valid values are:
+
+        * `DONE` — [completion of the asynchronous replication process](../../../concepts/async-replication.md#done).
+
+    * `FAILOVER_MODE` — the mode for changing the replication state. This parameter can only be used in combination with the `STATE` parameter. Valid values are:
+
+        * `FORCE` — forced failover.
+
+## Examples {#examples}
+
+The following statement forces the asynchronous replication process to complete:
+
+```yql
+ALTER ASYNC REPLICATION my_replication SET (STATE = "DONE", FAILOVER_MODE = "FORCE");
+```
+
+## See also
+
+* [CREATE ASYNC REPLICATION](create-async-replication.md)
+* [DROP ASYNC REPLICATION](drop-async-replication.md)
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/create-async-replication.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/create-async-replication.md
@@ -0,0 +1,90 @@
+# CREATE ASYNC REPLICATION
+
+The `CREATE ASYNC REPLICATION` statement creates an [asynchronous replication instance](../../../concepts/async-replication.md).
+
+## Syntax {#syntax}
+
+```yql
+CREATE ASYNC REPLICATION <name>
+FOR <remote_path> AS <local_path> [, <another_remote_path> AS <another_local_path>]
+WITH (option = value [, ...])
+```
+
+### Parameters {#params}
+
+* `name` — a name of the asynchronous replication instance.
+* `remote_path` — a relative or absolute path to a table or directory in the source database.
+* `local_path` — a relative or absolute path to a target table or directory in the local database.
+* `WITH (option = value [, ...])` — asynchronous replication parameters:
+
+
+    * `CONNECTION_STRING` — a [connection string](../../../concepts/connect.md#connection_string) for the source database (mandatory).
+    * Authentication details for the source database (mandatory) depending on the authentication method:
+
+        * [Access token](../../../recipes/ydb-sdk/auth-access-token.md):
+
+            * `TOKEN_SECRET_NAME` — the name of the [secret](../../../concepts/datamodel/secrets.md) that contains the token.
+
+        * [Login and password](../../../recipes/ydb-sdk/auth-static.md):
+
+            * `USER` — a database user name.
+            * `PASSWORD_SECRET_NAME` — the name of the [secret](../../../concepts/datamodel/secrets.md) that contains the password for the source database user.
+
+## Examples {#examples}
+
+{% note tip %}
+
+Before creating an asynchronous replication instance, you must [create](create-object-type-secret.md) a secret with authentication credentials for the source database or ensure that you have access to an existing secret.
+
+{% endnote %}
+
+The following statement creates an asynchronous replication instance to synchronize the `original_table` source table in the `/Root/another_database` database to the `replica_table` target table in the local database:
+
+```yql
+CREATE ASYNC REPLICATION my_replication_for_single_table
+FOR original_table AS replica_table
+WITH (
+    CONNECTION_STRING = 'grpcs://example.com:2135/?database=/Root/another_database',
+    TOKEN_SECRET_NAME = 'my_secret'
+);
+```
+
+The statement above uses the token from the `my_secret` secret for authentication and the `grpcs://example.com:2135` [endpoint](../../../concepts/connect.md#endpoint) to connect to the `/Root/another_database` database.
+
+The following statement creates an asynchronous replication instance to replicate the source tables `original_table_1` and `original_table_2` to the target tables `replica_table_1` and `replica_table_2`:
+
+```yql
+CREATE ASYNC REPLICATION my_replication_for_multiple_tables
+FOR original_table_1 AS replica_table_1, original_table_2 AS replica_table_2
+WITH (
+    CONNECTION_STRING = 'grpcs://example.com:2135/?database=/Root/another_database',
+    TOKEN_SECRET_NAME = 'my_secret'
+);
+```
+
+The following statement creates an asynchronous replication instance for the objects in the `original_dir` directory:
+
+```yql
+CREATE ASYNC REPLICATION my_replication_for_dir
+FOR original_dir AS replica_dir
+WITH (
+    CONNECTION_STRING = 'grpcs://example.com:2135/?database=/Root/another_database',
+    TOKEN_SECRET_NAME = 'my_secret'
+);
+```
+
+The following statement creates an asynchronous replication instance for the objects in the `/Root/another_database` database:
+
+```yql
+CREATE ASYNC REPLICATION my_replication_for_database
+FOR `/Root/another_database` AS `/Root/my_database`
+WITH (
+    CONNECTION_STRING = 'grpcs://example.com:2135/?database=/Root/another_database',
+    TOKEN_SECRET_NAME = 'my_secret'
+);
+```
+
+## See also
+
+* [ALTER ASYNC REPLICATION](alter-async-replication.md)
+* [DROP ASYNC REPLICATION](drop-async-replication.md)
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/create-object-type-secret.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/create-object-type-secret.md
@@ -0,0 +1,28 @@
+# CREATE OBJECT (TYPE SECRET)
+
+{% note warning %}
+
+The syntax for managing secrets will change in future {{ ydb-full-name }} releases.
+
+{% endnote %}
+
+The `CREATE OBJECT (TYPE SECRET)` statement creates a [secret](../../../concepts/datamodel/secrets.md).
+
+## Syntax {#syntax}
+
+```yql
+CREATE OBJECT <secret_name> (TYPE SECRET) WITH value="<secret_value>";
+```
+
+### Parameters
+
+* `secret_name` - the name of the secret.
+* `secret_value` - the contents of the secret.
+
+## Example {#examples}
+
+The following statement creates a secret named `MySecretName` with `MySecretData` as a value.
+
+```yql
+CREATE OBJECT MySecretName (TYPE SECRET) WITH value="MySecretData";
+```
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/drop-async-replication.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/drop-async-replication.md
@@ -0,0 +1,47 @@
+# DROP ASYNC REPLICATION
+
+The `DROP ASYNC REPLICATION` statement deletes an [asynchronous replication](../../../concepts/async-replication.md) instance. When an asynchronous replication instance is [deleted](../../../concepts/async-replication.md#drop), the following objects are also deleted:
+
+* automatically created [streams of changes](../../../concepts/glossary.md#changefeed)
+* [replicas](../../../concepts/glossary.md#replica-object) (optionally)
+
+## Syntax {#syntax}
+
+```yql
+DROP ASYNC REPLICATION <name> [CASCADE]
+```
+
+### Parameters
+
+* `name` — the name of the asynchronous replication instance.
+* `CASCADE` — cascaded deletion of the replicas that were created for a given asynchronous replication instance.
+
+## Examples {#examples}
+
+This section contains examples of YQL statements that drop the asynchronous replication instance created with the following expression:
+
+```yql
+CREATE ASYNC REPLICATION my_replication
+FOR original_table AS replica_table
+WITH (
+    CONNECTION_STRING = 'grpcs://example.com:2135/?database=/Root/another_database',
+    TOKEN_SECRET_NAME = 'my_secret'
+);
+```
+
+The following statement drops an asynchronous replication instance and the automatically created stream of changes for the `original_table` table, but the `replica_table` table is not deleted:
+
+```yql
+DROP ASYNC REPLICATION my_replication;
+```
+
+The following statement drops an asynchronous replication instance, the automatically created stream of changes for the `original_table` table, and the `replica_table` table:
+
+```yql
+DROP ASYNC REPLICATION my_replication CASCADE;
+```
+
+## See also
+
+* [CREATE ASYNC REPLICATION](create-async-replication.md)
+* [ALTER ASYNC REPLICATION](alter-async-replication.md)
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/index.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/index.md
@@ -97,3 +97,11 @@
 
 {% endif %}
 
+{% if feature_async_replication %}
+
+* [CREATE ASYNC REPLICATION](create-async-replication.md)
+* [ALTER ASYNC REPLICATION](alter-async-replication.md)
+* [DROP ASYNC REPLICATION](drop-async-replication.md)
+
+{% endif %}
+
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/toc_i.yaml b/ydb/docs/en/core/yql/reference/yql-core/syntax/toc_i.yaml
diff --git a/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-async-replication.md b/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-async-replication.md
diff --git a/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-object-type-secret.md b/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-object-type-secret.md
