Skip to content

Commit 6d94270

Browse files
DOC-5282 add processors section to config docs
1 parent 5d22c46 commit 6d94270

File tree

1 file changed

+28
-1
lines changed

1 file changed

+28
-1
lines changed

content/integrate/redis-data-integration/data-pipelines/pipeline-config.md

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,8 @@ processors:
158158

159159
## Sections
160160

161-
The main sections of the file configure [`sources`](#sources) and [`targets`](#targets).
161+
The main sections of the file configure [`sources`](#sources), [`targets`](#targets),
162+
and [`processors`](#processors).
162163

163164
### Sources
164165

@@ -235,3 +236,29 @@ sudo k3s kubectl delete nodes --all
235236
sudo service k3s restart
236237
```
237238
{{< /note >}}
239+
240+
### Processors
241+
242+
The `processors` section configures the behavior of the pipeline. The [example](#example)
243+
configuration above contains the following properties:
244+
245+
- `on_failed_retry_interval`: Number of seconds to wait before retrying a failed operation.
246+
The default is 5 seconds.
247+
- `read_batch_size`: Maximum number of records to read from the source database. RDI will
248+
wait for the batch to fill up to `read_batch_size` or for `duration` to elapse,
249+
whichever happens first. The default is 2000.
250+
- `duration`: Time (in ms) after which data will be read from the stream even if
251+
`read_batch_size` was not reached. The default is 100 ms.
252+
- `write_batch_size`: The batch size for writing data to the target Redis database. This should be
253+
less than or equal to the `read_batch_size`. The default is 200.
254+
- `dedup`: Boolean value to enable the deduplication mechanism. The default is `false`.
255+
- `dedup_max_size`: Maximum size of the deduplication set. The default is 1024.
256+
- `error_handling`: The strategy to use when an invalid record is encountered. The available
257+
strategies are `ignore` and `dlq` (store rejected messages in a dead letter queue).
258+
The default is `dlq`. See
259+
[What does RDI do if the data is corrupted or invalid?]({{< relref "/integrate/redis-data-integration/faq#what-does-rdi-do-if-the-data-is-corrupted-or-invalid" >}})
260+
for more information about the dead letter queue.
261+
262+
See also the
263+
[RDI configuration file reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}})
264+
for full details of the other available properties.

0 commit comments

Comments
 (0)