Skip to content

Commit 7637972

Browse files
vhristev-1vhristev-1
committed
RDSC-3434: Add documentation for MongoDB
1 parent d9235df commit 7637972

File tree

1 file changed

+120
-0
lines changed
  • content/integrate/redis-data-integration/data-pipelines/prepare-dbs

1 file changed

+120
-0
lines changed

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
---
2+
Title: Prepare MongoDB for RDI
3+
aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/sql-server/
4+
alwaysopen: false
5+
categories:
6+
- docs
7+
- integrate
8+
- rs
9+
- rdi
10+
description: Prepare MongoDB databases to work with RDI
11+
group: di
12+
linkTitle: Prepare MongoDB
13+
summary: Redis Data Integration keeps Redis in sync with the primary database in near
14+
real time.
15+
type: integration
16+
weight: 2
17+
---
18+
19+
This guide describes the steps required to prepare a MongoDB database as a source for Redis Data Integration (RDI) pipelines.
20+
21+
## Prerequisites
22+
- **MongoDB version:** 4.0 or later (standalone, replica set, sharded cluster, or MongoDB Atlas).
23+
- **User privileges:** You must have a MongoDB user with sufficient privileges to read the oplog and collections, and to use change streams (if required).
24+
- **Network access:** The RDI Collector must be able to connect to all MongoDB nodes in your deployment.
25+
26+
## 1. Configure Oplog Size
27+
Ensure the oplog is large enough to retain changes for the duration of your RDI pipeline's snapshot and streaming operations.
28+
29+
- Check oplog size:
30+
```javascript
31+
db.printReplicationInfo();
32+
```
33+
- Resize the oplog (run on the PRIMARY):
34+
```bash
35+
$ mongod --replSet <replSetName> --oplogSize <sizeInMB>
36+
```
37+
38+
## 2. Create a MongoDB User for RDI
39+
Create a user with the following roles on the source database:
40+
- read
41+
- readAnyDatabase
42+
- clusterMonitor
43+
- changeStream
44+
- readWrite (if you want to test writes)
45+
46+
Example:
47+
```javascript
48+
use admin;
49+
db.createUser({
50+
user: "rdi_user",
51+
pwd: "rdi_password",
52+
roles: [
53+
{ role: "read", db: "your_db" },
54+
{ role: "readAnyDatabase", db: "admin" },
55+
{ role: "clusterMonitor", db: "admin" },
56+
{ role: "changeStream", db: "admin" }
57+
]
58+
});
59+
```
60+
61+
## 3. Connection String Format
62+
The RDI Collector requires a MongoDB connection string that includes all relevant hosts and authentication details.
63+
64+
Example (Replica Set):
65+
```mongodb://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@host1:27017,host2:27017,host3:27017/?replicaSet=rs0&authSource=admin```
66+
- For standalone or Atlas, adjust the connection string accordingly.
67+
- Set replicaSet and authSource as appropriate for your deployment.
68+
69+
## 4. Enable Change Streams and Pre/Post Images (Only if Using a Custom Key)
70+
Change Streams: Required only if you are using a custom key in your RDI pipeline. Change streams are available by default on replica sets, sharded clusters, and MongoDB Atlas. For standalone deployments, ensure your MongoDB version supports change streams.
71+
Pre/Post Images: If you use a custom key and want to capture full document changes on updates and deletes, enable pre- and post-images on collections as needed:
72+
```javascript
73+
db.runCommand({
74+
collMod: "your_collection",
75+
changeStreamPreAndPostImages: { enabled: true }
76+
});
77+
```
78+
79+
## 5. MongoDB Atlas Specific Requirements
80+
MongoDB Atlas only supports secure connections via SSL.
81+
The root CA certificate for MongoDB Atlas must be added as a SOURCE_DB_CACERT secret in RDI.
82+
83+
- Download the MongoDB Atlas root CA certificate.
84+
- In RDI, add this certificate as a secret named SOURCE_DB_CACERT.
85+
- Ensure your connection string includes ssl=true or tls=true and references the CA certificate if required by your deployment.
86+
87+
Example connection string for Atlas:
88+
```mongodb+srv://rdi_user:rdi_password@cluster0.mongodb.net/?authSource=admin&tls=true```
89+
90+
## 6. Network and Security
91+
- Ensure the RDI Collector can connect to all MongoDB nodes on the required ports (default: 27017, or as provided by Atlas).
92+
- If using TLS/SSL, provide the necessary certificates and connection options in the connection string.
93+
94+
## 7. Test the Connection
95+
You can test the connection using the RDI CLI or API:
96+
```bash
97+
$ redis-di pipelines validate-source --db-type mongodb \
98+
--connection-string "mongodb+srv://rdi_user:rdi_password@cluster0.mongodb.net/?authSource=admin&tls=true"
99+
```
100+
Or via the API `/collector/connection/validate` endpoint.
101+
102+
## Additional Resources
103+
- [MongoDB Replica Set Documentation](https://www.mongodb.com/docs/manual/replication/)
104+
- [MongoDB Change Streams](https://www.mongodb.com/docs/manual/changeStreams/)
105+
- [MongoDB User Management](https://www.mongodb.com/docs/manual/core/security-users/)
106+
- [Debezium MongoDB Connector Documentation](https://debezium.io/documentation/reference/stable/connectors/mongodb.html)
107+
- [MongoDB Atlas SSL Setup](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-in-the-cloud)
108+
109+
## Summary Table
110+
| Requirement | Description |
111+
|---------------------|-----------------------------------------------------------------------------|
112+
| MongoDB Topology | Standalone, Replica Set, Sharded Cluster, or MongoDB Atlas |
113+
| User Roles | read, readAnyDatabase, clusterMonitor, changeStream |
114+
| Oplog | Sufficient size for snapshot and streaming |
115+
| Pre/Post Images | Enable on collections **only if using a custom key** |
116+
| Connection String | Must include all hosts, replicaSet (if applicable), authSource, credentials |
117+
| MongoDB Atlas | **[SSL required](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-mongodb-ssl-enabled)**, provide root CA as `SOURCE_DB_CACERT` secret in RDI |
118+
| Network | RDI Collector must reach all MongoDB nodes on required ports |
119+
120+
By following these steps, your MongoDB database—including MongoDB Atlas—will be ready to serve as a source for Redis Data Integration pipelines.

0 commit comments

Comments
 (0)