DOC-4067 added backpressure mech section to architecture page

Author: andy-stark-redis

content/integrate/redis-data-integration/ingest/_index.md

@@ -54,10 +54,10 @@ RDI provides enterprise-grade streaming data pipelines with the following featur
   [Prometheus](https://prometheus.io/) endpoints.
 - **Observability - logs** - RDI saves rotating logs to a single folder. They are in a JSON format,
   so you can collect and process them with your favorite observability tool.
-- **Back-pressure mechanism** - RDI is designed to backoff writing data when the cache gets
+- **Backpressure mechanism** - RDI is designed to backoff writing data when the cache gets
   disconnected, which prevents cascading failure. Since the change data is persisted in the source
   database and Redis is very fast, RDI can easily catch up with missed changes after a short period of
-  disconnection.
+  disconnection. See [Backpressure mechanism]({{< relref "/integrate/redis-data-integration/ingest/architecture #backpressure-mechanism">}}) for more information.
 - **Recovering from full failure** - If the cache fails or gets disconnected for a long time,
   RDI can reconstruct the cache data in Redis using a full snapshot of the defined dataset.
 - **High throughput** - Because RDI uses Redis for staging and writes to Redis as a target,

content/integrate/redis-data-integration/ingest/architecture.md

@@ -68,6 +68,23 @@ RDI automatically enters a second phase called *change streaming*, where
 changes in the data are captured as they happen. Changes are usually
 added to the target within a few seconds after capture.
 
+## Backpressure mechanism
+
+Sometimes, data records can get added to the streams faster than RDI can
+process them. This can happen if the target is slowed or disconnected
+or simply if the source generates a lot of change data quickly.
+If this continues then the streams will eventually occupy all the
+available memory. When RDI detects this situation, it applies a
+*backpressure* mechanism to slow or stop the flow of incoming data.
+Change data is held at the source until RDI clears the backlog and has
+enough free memory to resume streaming.
+
+{{<note>}}The Debezium log sometimes reports that RDI has run out
+of memory (usually while creating the initial snapshot). This is not
+an error, just an informative message to note that RDI has applied
+the backpressure mechanism.
+{{</note>}}
+
 ### Supported sources
 
 RDI supports the following database sources using [Debezium Server](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) connectors:

content/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines.md

@@ -495,10 +495,3 @@ completely. For example, you might want to apply a new transformation to all the
 data or refresh the dataset if RDI is disconnected from the
 source for a long time. In situations like these, you can *reset* the pipeline back
 to the snapshot phase. When this is complete, the pipeline continues with CDC as usual. 
-
-{{<note>}}During the snapshot phase and occasionally during CDC, Debezium
-will sometimes log a message saying that RDI is out of memory. This is just
-an information message, not an error condition, and is typically harmless. See the
-[Ingest FAQ]({{< relref "/integrate/redis-data-integration/ingest/faq#rdi-oom" >}})
-for more information.
-{{</note>}}

content/integrate/redis-data-integration/ingest/faq.md

@@ -66,13 +66,10 @@ new or renamed tables and columns.
 ## Should I be concerned when the log says RDI is out of memory? {#rdi-oom}
 
 Sometimes the Debezium log will contain a message saying that RDI is out of
-memory. This is actually an information message, not an error. It means that
-RDI has used all available memory for its data streams and must now concentrate
-on processing the streams to make space for new data from the collector.
-The message is typically logged during the
-snapshot phase of the
-[ingest pipeline lifecycle]({{< relref "/integrate/redis-data-integration/ingest/data-pipelines/data-pipelines#ingest-pipeline-lifecycle" >}})
-but you may also sometimes see it during the CDC phase.
+memory. This is not an error but an informative message to say that RDI
+is applying *backpressure* to Debezium. See
+[Backpressure mechanism]({{< relref "/integrate/redis-data-integration/ingest/architecture#backpressure-mechanism" >}})
+in the Architecture guide for more information.
 
 ## What happens when RDI can't write to the target Redis database?