autopartitioning-eng (#10532)

Co-authored-by: Ivan Blinkov <ivan@ydb.tech>

Author: ElenaAfina

ydb/docs/en/core/concepts/topic.md

@@ -19,18 +19,63 @@ Messages may contain user-defined attributes in "key-value" format. They are ret
 
 ## Partitioning {#partitioning}
 
-To enable horizontal scaling, a topic is divided into `partitions` that are units of parallelism. Each partition has a limited bandwidth. The recommended write speed is 1 MBps.
+To enable horizontal scaling, a topic is divided into `partitions` that are units of parallelism. Each partition has a limited throughput. The recommended write speed is 1 MBps.
 
 {% note info %}
 
 As for now, you can only reduce the number of partitions in a topic by deleting and recreating a topic with a smaller number of partitions.
 
 {% endnote %}
 
+Partitions can be:
+
+ - **Active.** By default, all partitions are active. Both read and write operations are allowed on an active partition.
+ - **Inactive.** An inactive partition is read-only. A partition becomes inactive after splitting for [autopartitioning](#autopartitioning). It is automatically deleted once all messages are removed due to the expiration of the retention period.
+
 ### Offset {#offset}
 
 All messages within a partition have a unique sequence number called an `offset` An offset monotonically increases as new messages are written.
 
+## Autopartitioning {#autopartitioning}
+
+Total topic throughput is determined by the number of partitions in the topic and the throughput of each partition. The number of partitions and the throughput of each partition are set at the time of topic creation. If the maximum required write speed for a topic is unknown at the creation time, autopartitioning allows the topic to be scaled automatically. If autopartitioning is enabled for a topic, the number of partitions will increase automatically as the write speed increases (see [Autopartitioning strategies](#autopartitioning_strategies)).
+
+### Guarantees {#autopartitioning_guarantee}
+
+1. The SDK and server provide an exactly-once guarantee in the case of writing during a partition split. This means that any message will be written either to the parent partition or to one of the child partitions but never to both simultaneously. Additionally, a message cannot be written to the same partition multiple times.
+2. The SDK and server maintain the reading order. Data is read from the parent partition first, followed by the child partitions.
+3. As a result, the exactly-once writing guarantee and the reading order guarantee are preserved for a specific [producer identifier](#producer-id).
+
+### Autopartitioning strategies {#autopartitioning_strategies}
+
+The following autopartitioning strategies are available for any topic:
+
+#### DISABLED
+
+Autopartitioning is disabled for this topic. The number of partitions remains constant, and there is no automatic scaling.
+
+The initial number of partitions is set during topic creation. If the partition count is manually adjusted, new partitions are added. Both previously existing and new partitions are active.
+
+#### UP
+
+Upwards autopartitioning is enabled for this topic. This means that if the write speed to the topic increases, the number of partitions will automatically increase. However, if the write speed decreases, the number of partitions remains unchanged.
+
+The partition count increase algorithm works as follows: if the write speed for a partition exceeds a defined threshold (as a percentage of the maximum write speed for that partition) during a specified period, the partition is split into two child partitions. The original partition becomes inactive, allowing only read operations. When the retention period expires, and all messages in the original partition are deleted, the partition itself is also deleted. The two new child partitions become active, allowing both read and write operations.
+
+#### PAUSED
+
+Autopartitioning is paused for this topic, meaning that the number of partitions does not increase automatically. If needed, you can re-enable autopartitioning for this topic.
+
+Examples of YQL queries for switching between different autopartitioning strategies can be found [here](../yql/reference/syntax/alter-topic.md#autopartitioning).
+
+### Autopartitioning constraints {#autopartitioning_constraints}
+
+The following constraints apply when using autopartitioning:
+
+1. Once autopartitioning is enabled for a topic, it cannot be stopped, only paused.
+2. When autopartitioning is enabled for a topic, it is impossible to read from or write to it using the [Kafka API](../reference/kafka-api/index.md).
+3. Autopartitioning can only be enabled on topics that use the reserved capacity mode.
+
 ## Message sources and groups {#producer-id}
 
 Messages are ordered using the `producer_id` and `message_group_id`. The order of written messages is maintained within pairs: `<producer ID, message group ID>`.

ydb/docs/en/core/reference/kafka-api/constraints.md

@@ -8,4 +8,5 @@
 4. Transactions are not supported.
 5. DDL operations are not supported. Use the [{{ ydb-short-name }} SDK](../ydb-sdk/index.md) or [{{ ydb-short-name }} CLI](../ydb-cli/index.md) to perform them.
 6. Data schema validation not supported.
-7. Kafka Connect is only supported in standalone mode.
+7. Kafka Connect is supported only in standalone mode.
+8. Kafka API can't be used to read from or write to topics with enabled autopartitioning.

ydb/docs/en/core/yql/reference/yql-core/syntax/alter-topic.md

@@ -141,3 +141,31 @@ ALTER TOPIC `my_topic` RESET (
 * `supported_codecs`: List of [codecs](../../../../concepts/topic#message-codec) supported by the topic. Value type: `String`.
 
 {% endif %}
+
+### Change autopartitioning strategies for the topic {#autopartitioning}
+
+The following command sets the [autopartitioning](../../../concepts/topic.md#autopartitioning) strategy to `UP`:
+
+```yql
+ALTER TOPIC `my_topic` SET (
+    min_active_partitions = 1,
+    max_active_partitions = 5,
+    auto_partitioning_strategy = 'scale_up'
+);
+```
+
+The following command pauses the topic [autopartitioning](../../../concepts/topic.md#autopartitioning):
+
+```yql
+ALTER TOPIC `my_topic` SET (
+    auto_partitioning_strategy = 'paused'
+);
+```
+
+The following command unpauses the topic [autopartitioning](../../../concepts/topic.md#autopartitioning):
+
+```yql
+ALTER TOPIC `my_topic` SET (
+    auto_partitioning_strategy = 'scale_up'
+);
+```