MySQL ClickPipe: Document generic MySQL and MariaDB

docs/integrations/data-ingestion/clickpipes/mysql/index.md

@@ -36,7 +36,11 @@ To get started, you first need to make sure that your MySQL database is set up c
 
 3. [Cloud SQL for MySQL](./mysql/source/gcp)
 
-4. [Amazon RDS MariaDB](./mysql/source/rds_maria)
+4. [Generic MySQL](./mysql/source/generic)
+
+5. [Amazon RDS MariaDB](./mysql/source/rds_maria)
+
+6. [Generic MariaDB](./mysql/source/generic_maria)
 
 Once your source MySQL database is set up, you can continue creating your ClickPipe.
 

docs/integrations/data-ingestion/clickpipes/mysql/source/generic.md

@@ -0,0 +1,139 @@
+---
+sidebar_label: 'Generic MySQL'
+description: 'Set up any MySQL instance as a source for ClickPipes'
+slug: /integrations/clickpipes/mysql/source/generic
+title: 'Generic MySQL source setup guide'
+---
+
+# Generic MySQL source setup guide
+
+:::info
+
+If you use one of the supported providers (in the sidebar), please refer to the specific guide for that provider.
+
+:::
+
+## Enable binary log retention {#enable-binlog-retention}
+
+Binary logs contain information about data modifications made to a MySQL server instance and are required for replication.
+
+### MySQL 8.x and newer {#binlog-v8-x}
+
+To enable binary logging on your MySQL instance, ensure that the following settings are configured:
+
+```sql
+log_bin = ON                        -- default value
+binlog_format = ROW                 -- default value
+binlog_row_image = FULL             -- default value
+binlog_row_metadata = FULL
+binlog_expire_logs_seconds = 86400  -- 1 day or higher; default is 30 days
+```
+
+To check these settings, run the following SQL commands:
+```sql
+SHOW VARIABLES LIKE 'log_bin';
+SHOW VARIABLES LIKE 'binlog_format';
+SHOW VARIABLES LIKE 'binlog_row_image';
+SHOW VARIABLES LIKE 'binlog_row_metadata';
+SHOW VARIABLES LIKE 'binlog_expire_logs_seconds';
+```
+
+If the values don't match, you can run the following SQL commands to set them:
+```sql
+SET PERSIST log_bin = ON;
+SET PERSIST binlog_format = ROW;
+SET PERSIST binlog_row_image = FULL;
+SET PERSIST binlog_row_metadata = FULL;
+SET PERSIST binlog_expire_logs_seconds = 86400;
+```
+
+If you have changed the `log_bin` setting, you NEED to RESTART the MySQL instance for the changes to take effect.
+
+After changing the settings, continue on with [configuring a database user](#configure-database-user).
+
+### MySQL 5.7 {#binlog-v5-x}
+
+To enable binary logging on your MySQL 5.7 instance, ensure that the following settings are configured:
+
+```sql
+server_id = 1            -- or greater; anything but 0
+log_bin = ON
+binlog_format = ROW      -- default value
+binlog_row_image = FULL  -- default value
+expire_logs_days = 1     -- or higher; 0 would mean logs are preserved forever
+```
+
+To check these settings, run the following SQL commands:
+```sql
+SHOW VARIABLES LIKE 'server_id';
+SHOW VARIABLES LIKE 'log_bin';
+SHOW VARIABLES LIKE 'binlog_format';
+SHOW VARIABLES LIKE 'binlog_row_image';
+SHOW VARIABLES LIKE 'expire_logs_days';
+```
+
+If the values don't match, you can set them in the config file (typically at `/etc/my.cnf` or `/etc/mysql/my.cnf`):
+```ini
+[mysqld]
+server_id = 1
+log_bin = ON
+binlog_format = ROW
+binlog_row_image = FULL
+expire_logs_days = 1
+```
+
+You NEED to RESTART the MySQL instance for the changes to take effect.
+
+:::note
+
+Column exclusion is not supported for MySQL 5.7 because the `binlog_row_metadata` setting wasn't yet introduced.
+
+:::
+
+## Configure a database user {#configure-database-user}
+
+Connect to your MySQL instance as the root user and execute the following commands:
+
+1. Create a dedicated user for ClickPipes:
+
+    ```sql
+    CREATE USER 'clickpipes_user'@'%' IDENTIFIED BY 'some_secure_password';
+    ```
+
+2. Grant schema permissions. The following example shows permissions for the `clickpipes` database. Repeat these commands for each database and host you want to replicate:
+
+    ```sql
+    GRANT SELECT ON `clickpipes`.* TO 'clickpipes_user'@'%';
+    ```
+
+3. Grant replication permissions to the user:
+
+    ```sql
+    GRANT REPLICATION CLIENT ON *.* TO 'clickpipes_user'@'%';
+    GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%';
+    ```
+
+:::note
+
+Make sure to replace `clickpipes_user` and `some_secure_password` with your desired username and password.
+
+:::
+
+## SSL/TLS configuration (recommended) {#ssl-tls-configuration}
+
+SSL certificates ensure secure connections to your MySQL database. Configuration depends on your certificate type:
+
+**Trusted Certificate Authority (DigiCert, Let's Encrypt, etc.)** - no additional configuration needed.
+
+**Internal Certificate Authority** - Obtain the root CA certificate file from your IT team. In the ClickPipes UI, upload it when creating a new MySQL ClickPipe.
+
+**Self-hosted MySQL** - Copy the CA certificate from your MySQL server (typically at `/var/lib/mysql/ca.pem`) and upload it in the UI when creating a new MySQL ClickPipe. Use the IP address of the server as the host.
+
+**Self-hosted MySQL without server access** - Contact your IT team for the certificate. As a last resort, use the "Skip Certificate Verification" toggle in ClickPipes UI (not recommended for security reasons).
+
+For more information on SSL/TLS options, check out our [FAQ](https://clickhouse.com/docs/integrations/clickpipes/mysql/faq#tls-certificate-validation-error).
+
+## What's next? {#whats-next}
+
+You can now [create your ClickPipe](../index.md) and start ingesting data from your MySQL instance into ClickHouse Cloud.
+Make sure to note down the connection details you used while setting up your MySQL instance as you will need them during the ClickPipe creation process.

docs/integrations/data-ingestion/clickpipes/mysql/source/generic_maria.md

@@ -0,0 +1,106 @@
+---
+sidebar_label: 'Generic MariaDB'
+description: 'Set up any MariaDB instance as a source for ClickPipes'
+slug: /integrations/clickpipes/mysql/source/generic_maria
+title: 'Generic MariaDB source setup guide'
+---
+
+# Generic MariaDB source setup guide
+
+:::info
+
+If you use one of the supported providers (in the sidebar), please refer to the specific guide for that provider.
+
+:::
+
+## Enable binary log retention {#enable-binlog-retention}
+
+Binary logs contain information about data modifications made to a MariaDB server instance and are required for replication.
+
+To enable binary logging on your MariaDB instance, ensure that the following settings are configured:
+
+```sql
+server_id = 1               -- or greater; anything but 0
+log_bin = ON
+binlog_format = ROW
+binlog_row_image = FULL
+binlog_row_metadata = FULL  -- introduced in 10.5.0
+expire_logs_days = 1        -- or higher; 0 would mean logs are preserved forever
+```
+
+To check these settings, run the following SQL commands:
+```sql
+SHOW VARIABLES LIKE 'server_id';
+SHOW VARIABLES LIKE 'log_bin';
+SHOW VARIABLES LIKE 'binlog_format';
+SHOW VARIABLES LIKE 'binlog_row_image';
+SHOW VARIABLES LIKE 'binlog_row_metadata';
+SHOW VARIABLES LIKE 'expire_logs_days';
+```
+
+If the values don't match, you can set them in the config file (typically at `/etc/my.cnf` or `/etc/my.cnf.d/mariadb-server.cnf`):
+```ini
+[mysqld]
+server_id = 1
+log_bin = ON
+binlog_format = ROW
+binlog_row_image = FULL
+binlog_row_metadata = FULL  ; only in 10.5.0 and newer
+expire_logs_days = 1
+```
+
+You NEED to RESTART the MariaDB instance for the changes to take effect.
+
+:::note
+
+Column exclusion is not supported for MariaDB \<= 10.4 because the `binlog_row_metadata` setting wasn't yet introduced.
+
+:::
+
+## Configure a database user {#configure-database-user}
+
+Connect to your MariaDB instance as the root user and execute the following commands:
+
+1. Create a dedicated user for ClickPipes:
+
+    ```sql
+    CREATE USER 'clickpipes_user'@'%' IDENTIFIED BY 'some_secure_password';
+    ```
+
+2. Grant schema permissions. The following example shows permissions for the `clickpipes` database. Repeat these commands for each database and host you want to replicate:
+
+    ```sql
+    GRANT SELECT ON `clickpipes`.* TO 'clickpipes_user'@'%';
+    ```
+
+3. Grant replication permissions to the user:
+
+    ```sql
+    GRANT REPLICATION CLIENT ON *.* TO 'clickpipes_user'@'%';
+    GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%';
+    ```
+
+:::note
+
+Make sure to replace `clickpipes_user` and `some_secure_password` with your desired username and password.
+
+:::
+
+## SSL/TLS configuration (recommended) {#ssl-tls-configuration}
+
+SSL certificates ensure secure connections to your MariaDB database. Configuration depends on your certificate type:
+
+**Trusted Certificate Authority (DigiCert, Let's Encrypt, etc.)** - no additional configuration needed.
+
+**Internal Certificate Authority** - Obtain the root CA certificate file from your IT team. In the ClickPipes UI, upload it when creating a new MariaDB ClickPipe.
+
+**Self-hosted MariaDB** - Copy the CA certificate from your MariaDB server (look up the path via the `ssl_ca` setting in your `my.cnf`). In the ClickPipes UI, upload it when creating a new MariaDB ClickPipe. Use the IP address of the server as the host.
+
+**Self-hosted MariaDB starting with 11.4** - If your server has `ssl_ca` set up, follow the option above. Otherwise, consult with your IT team to provision a proper certificate. As a last resort, use the "Skip Certificate Verification" toggle in ClickPipes UI (not recommended for security reasons).
+
+For more information on SSL/TLS options, check out our [FAQ](https://clickhouse.com/docs/integrations/clickpipes/mysql/faq#tls-certificate-validation-error).
+
+## What's next? {#whats-next}
+
+You can now [create your ClickPipe](../index.md) and start ingesting data from your MariaDB instance into ClickHouse Cloud.
+Make sure to note down the connection details you used while setting up your MariaDB instance as you will need them during the ClickPipe creation process.

scripts/aspell-ignore/en/aspell-dict.txt

@@ -296,6 +296,7 @@ DeprecatedBadge
 DestroyAggregatesThreads
 DestroyAggregatesThreadsActive
 DictCacheRequests
+DigiCert
 DiskAvailable
 DiskObjectStorage
 DiskObjectStorageAsyncThreads

sidebars.js

@@ -747,8 +747,10 @@ const sidebars = {
               items: [
                 "integrations/data-ingestion/clickpipes/mysql/source/rds",
                 "integrations/data-ingestion/clickpipes/mysql/source/aurora",
+                "integrations/data-ingestion/clickpipes/mysql/source/gcp",
+                "integrations/data-ingestion/clickpipes/mysql/source/generic",
                 "integrations/data-ingestion/clickpipes/mysql/source/rds_maria",
-                "integrations/data-ingestion/clickpipes/mysql/source/gcp"
+                "integrations/data-ingestion/clickpipes/mysql/source/generic_maria",
               ],
             },
             "integrations/data-ingestion/clickpipes/mysql/datatypes"