Skip to content

Commit 2c76ebf

Browse files
DOC-4549 implemented feedback
1 parent d3df8af commit 2c76ebf

File tree

1 file changed

+21
-19
lines changed

1 file changed

+21
-19
lines changed

content/integrate/redis-data-integration/reference/config-yaml-reference.md

Lines changed: 21 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ for more information about the role `config.yaml` plays in defining a pipeline.
1414

1515
## Note about fully-qualified table names
1616

17-
Throughout this document we use the format `<databaseName>.<tableName>` to refer to a fully-qualified table name. This format is actually the one used by MySQL, but for Oracle,
17+
Throughout this document we use the format `<databaseName>.<tableName>` to refer to a fully-qualified table name. This format is actually the one used by MySQL/MariaDB, but for Oracle,
1818
SQLServer, and PostgreSQL, you should use `<schemaName>`.`<tableName>` instead.
1919

2020
{{< note >}}You can specify the fully-qualified table name `<databaseName>.<tableName>` as
@@ -84,7 +84,7 @@ See the Debezium documentation for more information about the specific connector
8484
| -- | -- | -- | -- |
8585
| `host` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQLServer| The IP address of the database instance. |
8686
| `port` | `integer` | MariaDB, MySQL, Oracle, PostgreSQL, SQLServer | The port of the database instance. |
87-
| `database` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQLServer| The name of the database to capture changes from. For `SQL Server` you can define this as comma-separated list of database names. |
87+
| `database` | `string` | Oracle, PostgreSQL, SQLServer| The name of the database to capture changes from. For `SQL Server` you can define this as comma-separated list of database names. |
8888
| `database.pdb.name` | `string` | Oracle |The name of the [Oracle Pluggable Database](https://docs.oracle.com/en/database/oracle/oracle-database/19/riwin/about-pluggable-databases-in-oracle-rac.html) that the connector captures changes from. Do not specify this property for a non-CDB installation.<br/> Default: `"ORCLPDB1"` |
8989
| `database.encrypt` | `string` | SQL Server| If SSL is enabled for your SQL Server database, you should also enable SSL in RDI by setting the value of this property to `true`.<br/> Default: `false` |
9090
| `database.server.id` | `integer` | MySQL | Numeric ID of this database client, which must be unique across all currently-running database processes in the MySQL cluster.<br/> Default: 1|
@@ -104,15 +104,15 @@ See the Debezium documentation for more information about the specific connector
104104
| `redis.wait.enabled` | `string` | If Redis is configured with a replica shard, this lets you verify that the data has been written to the replica.<br/> Default: `false` |
105105
| `redis.wait.timeout.ms` | `integer` | Defines the timeout in milliseconds when waiting for the replica.<br/> Default: `1000` |
106106
| `redis.wait.retry.enabled` | `string` | Enables retry on wait for replica failure.<br/> Default: `false` |
107-
| `redis.wait.retry.delay.ms` | `integer` | Defines the delay for retry on wait for replica failure.<br/> Default: `1000` |
107+
| `redis.wait.retry.delay.ms` | `integer` | Defines the delay (in milliseconds) for retry on wait for replica failure.<br/> Default: `1000` |
108108
| `redis.retry.initial.delay.ms` | `integer` | Initial retry delay (in milliseconds) when encountering Redis connection or OOM issues. This value will be doubled upon every retry but won’t exceed `redis.retry.max.delay.ms`.<br/> Default: `300` |
109109
| `redis.retry.max.delay.ms` | `integer` | Maximum delay (in milliseconds) when encountering Redis connection or OOM issues.<br/> Default: `10000` |
110110

111111
#### `source`
112112

113113
| Name | Type | Source Databases | Description |
114114
|--|--|--|--|
115-
| `snapshot.mode` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQLServer | Specifies the mode that the connector uses to take snapshots of a captured table.<br/> Default: `"initial"` |
115+
| `snapshot.mode` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQLServer | Specifies the mode that the connector uses to take snapshots of a captured table. See the [Debezium documentation](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) for more details about the available options and configuration.<br/> Default: `"initial"` |
116116
| `topic.prefix` | `string` | MySQL, Oracle, PostgreSQL, SQLServer| A prefix for all topic names that receive events emitted by this connector.<br/>Default: `"rdi"` |
117117
| `database.exclude.list` | `string` | MariaDB, MySQL | An optional, comma-separated list of regular expressions that match the names of databases for which you do not want to capture changes. The connector captures changes in any database whose name is not included in `database.exclude.list`. Do not specify the `database` field in the `connection` configuration if you are using the `database.exclude.list` property to filter out databases. |
118118
| `schema.exclude.list` | `string` | Oracle, PostgreSQL, SQLServer | An optional, comma-separated list of regular expressions that match names of schemas for which you do not want to capture changes. The connector captures changes in any schema whose name is not included in `schema.exclude.list`. Do not specify the `schemas` section if you are using the `schema.exclude.list` property to filter out schemas. |
@@ -124,7 +124,7 @@ See the Debezium documentation for more information about the specific connector
124124

125125
### Using custom queries in the initial snapshot {#custom-initial-query}
126126

127-
{{< note >}}This section is relevant only for MySQL, Oracle, PostgreSQL, and SQLServer.
127+
{{< note >}}This section is relevant only for MySQL/MariaDB, Oracle, PostgreSQL, and SQLServer.
128128
{{< /note >}}
129129

130130
By default, the initial snapshot captures all rows from each table.
@@ -136,15 +136,15 @@ After the `snapshot.select.statement.overrides` list, you must then add another
136136
The format of the property name depends on the database you are using:
137137

138138
- For Oracle, SQLServer, and PostrgreSQL, use `snapshot.select.statement.overrides.<SCHEMA_NAME>.<TABLE_NAME>`
139-
- For MySQL, use: `snapshot.select.statement.overrides<DATABASE_NAME>.<TABLE_NAME>`
139+
- For MySQL and MariaDB, use: `snapshot.select.statement.overrides<DATABASE_NAME>.<TABLE_NAME>`
140140

141141
For example, with PostgreSQL, you would have a configuration like the following:
142142

143143
```yaml
144144
source:
145145
snapshot.select.statement.overrides: myschema.mytable
146146
snapshot.select.statement.overrides.myschema.mytable: |
147-
SELECT ...
147+
SELECT ...
148148
```
149149

150150
For MySQL, you would have:
@@ -153,36 +153,38 @@ For MySQL, you would have:
153153
source:
154154
snapshot.select.statement.overrides: mydatabase.mytable
155155
snapshot.select.statement.overrides.mydatabase.mytable: |
156-
SELECT ...
156+
SELECT ...
157157
```
158158

159159
You must also add the list of columns you want to include in the custom `SELECT` statement using fully-qualified names under "sources.tables". Specify each column in the configuration as shown below:
160160

161161
```yaml
162162
tables:
163-
schema_name.table_name: # For MySQL: use database_name.table_name
164-
columns:
165-
- column_name1 # Each column on a new line
166-
- column_name2
167-
- column_name3
163+
# For MySQL and MariaDB: use database_name.table_name
164+
schema_name.table_name:
165+
columns:
166+
- column_name1 # Each column on a new line
167+
- column_name2
168+
- column_name3
168169
```
169170

170171
If you want to capture all columns from a table, you can use empty curly braces `{}` instead of listing all the individual columns:
171172

172173
```yaml
173174
tables:
174-
schema_name.table_name: {} # Captures all columns
175+
# Captures all columns. For MySQL and MariaDB: use database_name.table_name.
176+
schema_name.table_name: {}
175177
```
176178

177179
The example configuration below selects the columns `CustomerId`, `FirstName` and `LastName` from the `customer` table and joins it with the `invoice` table to select customers with total invoices greater than 8000:
178180

179181
```yaml
180182
tables:
181183
chinook.customer:
182-
columns:
183-
- CustomerID
184-
- FirstName
185-
- LastName
184+
columns:
185+
- CustomerID
186+
- FirstName
187+
- LastName
186188
187189
advanced:
188190
source:
@@ -231,7 +233,7 @@ message.
231233
| `read_batch_size` |`integer`, `string`| Batch size for reading data from the source database.<br/>Default: `2000`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`|
232234
| `debezium_lob_encoded_placeholder` |`string`| Enable Debezium LOB placeholders.<br/>Default: `"X19kZWJleml1bV91bmF2YWlsYWJsZV92YWx1ZQ=="`|
233235
| `dedup` |`boolean`| Enable deduplication mechanism.<br/>Default: `false`<br/>||
234-
| `dedup_max_size` |`integer`| Maximum size of the deduplication set.<br/> Default: `1024`<br/>Minimum: `1`<br/>|
236+
| `dedup_max_size` |`integer`| Maximum number of items in the deduplication set.<br/> Default: `1024`<br/>Minimum: `1`<br/>|
235237
| `dedup_strategy` |`string`| Deduplication strategy: `reject` - reject messages (dlq), `ignore` \- ignore messages.<br/> (DEPRECATED)<br/>The property `dedup_strategy` is now deprecated. The only supported strategy is `ignore`. Please remove from the configuration.<br/>Default: `"ignore"`<br/>Enum: `"reject"`, `"ignore"`<br/>|
236238
| `duration` |`integer`, `string`| Time (in ms) after which data will be read from stream even if `read_batch_size` was not reached.<br/> Default: `100`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/>|
237239
| `write_batch_size` |`integer`, `string`| The batch size for writing data to target Redis database\. Should be less or equal to `read_batch_size`.<br/> Default: `200`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/>|

0 commit comments

Comments
 (0)