Skip to content

Commit d162840

Browse files
andy-stark-redisandy-stark-redis
authored
Merge pull request #1579 from redis/RDSC-3434_add_documentation_for_mongodb
RDSC-3434: Add documentation for MongoDB
2 parents e5b33a7 + 3246469 commit d162840

File tree

2 files changed

+133
-0
lines changed

2 files changed

+133
-0
lines changed

‎content/embeds/rdi-supported-source-versions.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
| :-- | :-- | :-- | :-- |
33
| Oracle | 12c, 19c, 21c | 19c, 21c | - |
44
| MariaDB | 10.5, 11.4.3 | 10.4 to 10.11, 11.4.3 | - |
5+
| MongoDB | 6.0, 7.0, 8.0 | - | - |
56
| MySQL | 5.7, 8.0.x, 8.2 | 8.0.x | 8.0 |
67
| PostgreSQL | 10, 11, 12, 13, 14, 15, 16 | 11, 12, 13, 14, 15, 16 | 15 |
78
| SQL Server | 2017, 2019, 2022 | 2016, 2017, 2019, 2022 | 2019 |

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
---
2+
Title: Prepare MongoDB for RDI
3+
alwaysopen: false
4+
categories:
5+
- docs
6+
- integrate
7+
- rs
8+
- rdi
9+
description: Prepare MongoDB databases to work with RDI
10+
group: di
11+
linkTitle: Prepare MongoDB
12+
summary: Redis Data Integration keeps Redis in sync with the primary database in near
13+
real time.
14+
type: integration
15+
weight: 2
16+
---
17+
18+
This guide describes the steps required to prepare a MongoDB database as a source for Redis Data Integration (RDI) pipelines.
19+
20+
## Prerequisites
21+
- **MongoDB version:** 6.0 or later (replica set, sharded cluster, or MongoDB Atlas).
22+
- **User privileges:** You must have a MongoDB user with sufficient privileges to read the oplog and collections, and to use change streams.
23+
- **Network access:** The RDI Collector must be able to connect to all MongoDB nodes in your deployment.
24+
25+
{{< note >}}The MongoDB connector is not capable of monitoring the changes of a standalone MongoDB server, since standalone servers do not have an oplog. The connector will work if the standalone server is converted to a replica set with one member.{{< /note >}}
26+
## Summary
27+
28+
The following table summarizes the steps to prepare a MongoDB database for RDI. The sections below explain the steps in more detail.
29+
30+
| Requirement | Description |
31+
|---------------------|-----------------------------------------------------------------------------|
32+
| MongoDB Topology | Replica Set, Sharded Cluster, or MongoDB Atlas |
33+
| User Roles | readAnyDatabase, clusterMonitor |
34+
| Oplog | Sufficient size for snapshot and streaming |
35+
| Pre/Post Images | Enable on collections **only if using a custom key** |
36+
| Connection String | Must include all hosts, replicaSet (if applicable), authSource, credentials |
37+
| MongoDB Atlas | **[SSL required](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-mongodb-ssl-enabled)**, provide root CA as `SOURCE_DB_CACERT` secret in RDI |
38+
| Network | RDI Collector must reach all MongoDB nodes on required ports |
39+
40+
## 1. Configure oplog size
41+
42+
The Debezium MongoDB connector relies on the [oplog](https://www.mongodb.com/docs/manual/core/replica-set-oplog/) to capture changes from a replica set. The oplog is a fixed-size, capped collection. When it reaches its maximum size, it overwrites the oldest entries. If the connector is stopped and restarted, it attempts to resume from its last recorded position in the oplog. If that position has been overwritten, the connector may fail to start and report an invalid resume token error.
43+
44+
To prevent this, ensure the oplog retains enough history for Debezium to resume streaming after interruptions. You can do this by:
45+
46+
- **Increasing the oplog size:** Set the oplog size based on your workload, ensuring it can store more than the peak number of oplog entries generated per hour.
47+
- **Setting a minimum oplog retention period (MongoDB 4.4+):** Configure MongoDB to retain oplog entries for a minimum number of hours, guaranteeing availability even if the oplog reaches its maximum size. This is generally preferred, but for high-throughput clusters nearing capacity, you may need to increase the oplog size instead.
48+
49+
For detailed guidance, see the Debezium [oplog configuration documentation](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-optimal-oplog-config).
50+
51+
## 2. Create a MongoDB user for RDI
52+
53+
Create a user with the following roles on the source database:
54+
- `readAnyDatabase` (optional) OR grant `read` for the specific database(s) you will use with RDI
55+
- `clusterMonitor`
56+
57+
Example:
58+
```javascript
59+
use admin;
60+
db.createUser({
61+
user: "rdi_user",
62+
pwd: "rdi_password",
63+
roles: [
64+
// You can have multiple read roles, one per database.
65+
{ role: "read", db: "your_database" },
66+
// Use the role below if you don't want to grant the `read` role for each database.
67+
// { role: "readAnyDatabase", db: "admin" },
68+
{ role: "clusterMonitor", db: "admin" }
69+
]
70+
});
71+
```
72+
73+
## 3. Connection string format
74+
75+
The RDI Collector requires a MongoDB connection string that includes all relevant hosts and authentication details.
76+
77+
Example (Replica Set):
78+
```
79+
mongodb://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@host1:27017,host2:27017,host3:27017/?replicaSet=rs0&authSource=admin
80+
```
81+
Example (Sharded Cluster):
82+
```
83+
mongodb://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@host:30000
84+
```
85+
- For Atlas, adjust the connection string accordingly (see example below).
86+
- Set `replicaSet` and `authSource` as appropriate for your deployment.
87+
88+
## 4. Enable change streams and pre/post images (only if using a custom key)
89+
90+
Change Streams are required only if you are using a custom key in your RDI pipeline. Change streams are available by default on replica sets, sharded clusters, and MongoDB Atlas.
91+
92+
If your RDI pipeline uses a custom key, you must enable pre- and post-images on the relevant collections to capture the document state before and after updates or deletes. This allows RDI to access both the previous and updated versions of documents during change events, ensuring accurate synchronization.
93+
94+
Use the command below to enable change streams and pre/post images:
95+
96+
```javascript
97+
db.runCommand({
98+
collMod: "your_collection",
99+
changeStreamPreAndPostImages: { enabled: true }
100+
});
101+
```
102+
103+
## 5. MongoDB Atlas specific requirements
104+
105+
MongoDB Atlas only supports secure connections via SSL.
106+
The root CA certificate for MongoDB Atlas must be added as a SOURCE_DB_CACERT secret in RDI.
107+
108+
- Download the MongoDB Atlas root CA certificate.
109+
- In RDI, add this certificate as a secret named SOURCE_DB_CACERT.
110+
- Ensure that the `mongodb.ssl.enabled: true` setting is present in your RDI configuration.
111+
112+
Example connection string for Atlas:
113+
```
114+
mongodb+srv://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@cluster0.mongodb.net/?authSource=admin
115+
```
116+
117+
## 6. Network and security
118+
119+
- Ensure the RDI Collector can connect to all MongoDB nodes on the required ports (default: 27017, or as provided by Atlas).
120+
- If using TLS/SSL, provide the necessary certificates and connection options in the connection string.
121+
122+
## 7. Configuration is complete
123+
Once you have followed the steps above, your MongoDB database is ready for Debezium to use.
124+
125+
## See also
126+
127+
- [MongoDB Replica Set Documentation](https://www.mongodb.com/docs/manual/replication/)
128+
- [MongoDB Sharded Cluster Documentation](https://www.mongodb.com/docs/manual/sharding/)
129+
- [MongoDB Change Streams](https://www.mongodb.com/docs/manual/changeStreams/)
130+
- [MongoDB User Management](https://www.mongodb.com/docs/manual/core/security-users/)
131+
- [Debezium MongoDB Connector Documentation](https://debezium.io/documentation/reference/stable/connectors/mongodb.html)
132+
- [MongoDB Atlas SSL Setup](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-in-the-cloud)

0 commit comments

Comments
 (0)