Merge pull request #537 from redis/DOC-4067-rdi-oom-error-reassure

DOC-4067 added FAQ and note about harmless OOM message

Author: andy-stark-redis

content/integrate/redis-data-integration/ingest/_index.md

@@ -54,10 +54,10 @@ RDI provides enterprise-grade streaming data pipelines with the following featur
   [Prometheus](https://prometheus.io/) endpoints.
 - **Observability - logs** - RDI saves rotating logs to a single folder. They are in a JSON format,
   so you can collect and process them with your favorite observability tool.
-- **Back-pressure mechanism** - RDI is designed to backoff writing data when the cache gets
+- **Backpressure mechanism** - RDI is designed to backoff writing data when the cache gets
   disconnected, which prevents cascading failure. Since the change data is persisted in the source
   database and Redis is very fast, RDI can easily catch up with missed changes after a short period of
-  disconnection.
+  disconnection. See [Backpressure mechanism]({{< relref "/integrate/redis-data-integration/ingest/architecture #backpressure-mechanism">}}) for more information.
 - **Recovering from full failure** - If the cache fails or gets disconnected for a long time,
   RDI can reconstruct the cache data in Redis using a full snapshot of the defined dataset.
 - **High throughput** - Because RDI uses Redis for staging and writes to Redis as a target,

content/integrate/redis-data-integration/ingest/architecture.md

@@ -68,6 +68,23 @@ RDI automatically enters a second phase called *change streaming*, where
 changes in the data are captured as they happen. Changes are usually
 added to the target within a few seconds after capture.
 
+## Backpressure mechanism
+
+Sometimes, data records can get added to the streams faster than RDI can
+process them. This can happen if the target is slowed or disconnected
+or simply if the source quickly generates a lot of change data.
+If this continues, then the streams will eventually occupy all the
+available memory. When RDI detects this situation, it applies a
+*backpressure* mechanism to slow or stop the flow of incoming data.
+Change data is held at the source until RDI clears the backlog and has
+enough free memory to resume streaming.
+
+{{<note>}}The Debezium log sometimes reports that RDI has run out
+of memory (usually while creating the initial snapshot). This is not
+an error, just an informative message to note that RDI has applied
+the backpressure mechanism.
+{{</note>}}
+
 ### Supported sources
 
 RDI supports the following database sources using [Debezium Server](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) connectors:

content/integrate/redis-data-integration/ingest/faq.md

@@ -59,10 +59,18 @@ replica of an Active-Active replication setup or an Auto tiering database.
 
 If you don't configure RDI to capture a specific set of tables in the schema then it will
 detect any new tables when they are added. Similarly, RDI will capture new table columns
-and changes to column names unless you configure it for a specfic set of columns.
+and changes to column names unless you configure it for a specific set of columns.
 Bear in mind that the Redis keys in the target database will change to reflect the
 new or renamed tables and columns.
 
+## Should I be concerned when the log says RDI is out of memory? {#rdi-oom}
+
+Sometimes the Debezium log will contain a message saying that RDI is out of
+memory. This is not an error but an informative message to say that RDI
+is applying *backpressure* to Debezium. See
+[Backpressure mechanism]({{< relref "/integrate/redis-data-integration/ingest/architecture#backpressure-mechanism" >}})
+in the Architecture guide for more information.
+
 ## What happens when RDI can't write to the target Redis database?
 
 RDI will keep attempting to write the changes to the target and will also attempt
@@ -79,7 +87,7 @@ more space available.
 ## What does RDI do if the data is corrupted or invalid?
 
 The collector reports the data to RDI in a structured JSON format. If
-the structure of the JSON data is invalid or if there is a fatal bug in the tranformation
+the structure of the JSON data is invalid or if there is a fatal bug in the transformation
 job then RDI can't transform the data. When this happens, RDI will store the original data
 in a "dead letter queue" along with a message to say why it was rejected. The dead letter
 queue is stored as a capped stream in the RDI staging database. You can see its contents