docs: add 5.1 migration guide (#53)

* docs: add confluent platform migration guide

* docs: add sensitive properties section to migration

* docs: add confluent cloud migration guide

* docs: add migration to content nav

* docs: move migration to reference section

* docs: refactor migration into separate section and split into deployments

* docs: use defined variables for version and product name

* docs: update migration navigation bar

* docs: update wording

Co-authored-by: Nicola Vitucci <nicola.vitucci@gmail.com>

* docs: move files into 5.1 specification section

* docs: fix example indentation

* Apply suggestions from code review

Co-authored-by: Nicola Vitucci <nicola.vitucci@gmail.com>

* docs: update navigation title

---------

Co-authored-by: Ali Ince <ali-ince@users.noreply.github.com>
Co-authored-by: Nicola Vitucci <nicola.vitucci@gmail.com>

Author: 3 people

modules/ROOT/content-nav.adoc

@@ -25,6 +25,11 @@
 * xref:sink/error-handling.adoc[Error handling]
 * xref:sink/configuration.adoc[Settings]
 
+* *Upgrades and migrations*
+* xref:migration/5.1/index.adoc[Migrate to 5.1]
+** xref:migration/5.1/migration-docker.adoc[Confluent Platform]
+** xref:migration/5.1/migration-confluent-cloud.adoc[Confluent Cloud]
+
 * *Reference*
 * xref::known-issues.adoc[]
 * xref::changelog.adoc[]

modules/ROOT/examples/docker-data/how-to-upgrade.migrated-config.log

@@ -0,0 +1,18 @@
+connect  | [2024-09-04 09:18:40,066] INFO The migrated settings for 5.1 version of Neo4j Source Connector 'Neo4jSourceConnectorAVRO' is: `{
+connect  |   "connector.class" : "org.neo4j.connectors.kafka.source.Neo4jConnector",
+connect  |   "neo4j.authentication.basic.password" : "",
+connect  |   "neo4j.uri" : "bolt://neo4j:7687",
+connect  |   "neo4j.query" : "MATCH (ts:TestSource) WHERE ts.timestamp > $lastCheck RETURN ts.name AS name, ts.surname AS surname, ts.timestamp AS timestamp",
+connect  |   "neo4j.query.streaming-property" : "timestamp",
+connect  |   "value.converter.schema.registry.url" : "http://schema-registry:8081",
+connect  |   "task.class" : "streams.kafka.connect.source.Neo4jSourceTask",
+connect  |   "neo4j.authentication.basic.username" : "neo4j",
+connect  |   "name" : "Neo4jSourceConnectorAVRO",
+connect  |   "neo4j.query.topic" : "my-topic",
+connect  |   "value.converter" : "io.confluent.connect.avro.AvroConverter",
+connect  |   "key.converter" : "io.confluent.connect.avro.AvroConverter",
+connect  |   "key.converter.schema.registry.url" : "http://schema-registry:8081",
+connect  |   "neo4j.query.poll-interval" : "5000ms",
+connect  |   "neo4j.start-from" : "USER_PROVIDED",
+connect  |   "neo4j.start-from.value" : 1725441505774
+connect  | }` (streams.kafka.connect.source.Neo4jSourceService)

modules/ROOT/images/migration/confluent-cloud-config-log.png

@@ -0,0 +1,27 @@
+[[connector-migration]]
+= Migrate from 5.0.x to 5.1
+
+When upgrading the {product-name} to 5.1, be aware that many of the configuration properties have been changed and updated.
+Read over xref:whats-new.adoc[] to familiarize with the recent changes, and make sure that the new configuration logically matches the previous configuration.
+
+For generic Kafka Connect plugin upgrade documentation see: https://docs.confluent.io/platform/current/connect/upgrade.html[Confluent Documentation -> Upgrade a Connector Plugin].
+
+== Deployment guides
+
+Follow the relevant instructions to configure and deploy the new connector.
+
+* Read the xref:migration/5.1/migration-docker.adoc[Confluent Platform migration] for a connector deployed to Confluent Platform as a self-managed connector.
+* Read the xref:migration/5.1/migration-confluent-cloud.adoc[Confluent Cloud migration] for a connector deployed as a Custom Connector on Confluent Cloud.
+
+== Troubleshooting
+
+[qanda]
+Failing to connect to Neo4j instance?::
+The configuration migration process does not migrate the password of the Neo4j instance or other sensitive properties.
+You need to enter these properties again.
+Once updated, restart the connector.
+
+Unexpected error::
+Looking at the logs of Kafka Connect may reveal other errors in configuration.
+. On Confluent Cloud, the logs are accessible through the *Logs* section on the Connector run page.
+. On Confluent Platform, the logs are accessible through the Docker command `docker-compose logs -f connect`.

modules/ROOT/images/migration/confluent-cloud-pause.png

@@ -0,0 +1,27 @@
+[[migration-confluent-cloud]]
+= Migration on Confluent Cloud
+
+include::partial$third-party.adoc[]
+
+IMPORTANT: Complete the following steps in full for each connector (source or sink) currently registered.
+
+This guide follows the connector xref:quickstart-confluent-cloud.adoc[] execution method.
+
+.Steps
+
+. Access the Confluent Cloud cluster associated with the relevant connectors.
+. Open the *Connectors* section and find the relevant source or sink connector for upgrading.
+. Open the *Settings* section and select *Pause* to pause the connector.
+. Open the *Logs* section and search for a log message starting with `The migrated settings for 5.1 version of Neo4j`.
++
+image::migration/confluent-cloud-config-log.png[width=800]
+
+. Copy the configuration shown in the log into a local file.
+
+include::partial$how-to-upgrade-validate-config.adoc[]
+
+. Follow the xref:quickstart-confluent-cloud.adoc[] to upload the new 5.1 Connector plugin.
+. Create a Source or Sink instance.
+. Copy the migrated configuration from the local file into the Custom Configuration JSON section, and check that it is correct.
+. Continue with the steps in the Quickstart guide.
+. Once the connector is up and running, validate it is successfully running.

modules/ROOT/pages/migration/5.1/index.adoc

@@ -0,0 +1,33 @@
+[[migration-docker]]
+= Migration on Confluent Platform
+
+This guide follows the xref:quickstart-docker.adoc[].
+
+.Steps
+. Open the Confluent Control Center instance at `http://localhost:9021/clusters` and find the registered source or sink connector.
+Once found, delete it.
+
+. Look at the logs for the `connect` container (`docker-compose logs -f connect`).
+A migrated configuration will be printed to the logs as part of the connector deletion/shutdown.
+Example:
++
+[source,shell]
+----
+include::example$docker-data/how-to-upgrade.migrated-config.log[]
+----
+
+. Create a new JSON file `(source/sink)_migrated.neo4j.json` and copy the migrated example config to this file along with the connector name.
+The configuration should contain a `name` property and the migrated configuration should be nested in the `config` key.
+
+include::partial$how-to-upgrade-validate-config.adoc[]
+
+. Download the new connector version following the xref:installation.adoc[] guide and remove the original 5.0.x connector version.
+Make sure to copy the new plugin into the `./plugins/` folder created during the Docker setup.
+
+. Restart the Kafka Connect Worker by running `docker-compose restart connect`.
+This allows the Kafka Connect platform to pick up the new plugin from the `./plugins/` folder.
+
+. Continue following the xref:quickstart-docker.adoc[] to deploy the plugin to Kafka Connect with the new configuration.
+
+. Once the connector is up and running, validate it is successfully running.
+

modules/ROOT/pages/migration/5.1/migration-confluent-cloud.adoc

@@ -0,0 +1,8 @@
+. Validate that the new configuration contains all the relevant keys compared with the previous version of the connector configuration.
+.. See xref:source/configuration.adoc[] and xref:sink/configuration.adoc[] for descriptions and examples of the new configuration options.
+.. If migrating the Source component, make note of the `neo4j.start-from.value` that has been set to the last checked offset.
+.. Replace the sensitive values.
+[WARNING]
+Sensitive values in the original configuration are not printed in the migrated configuration.
+These keys need to be filled in with the appropriate values.
+The affected configuration keys are `neo4j.authentication.basic.password` and `neo4j.authentication.kerberos.ticket`.