Skip to content

Commit 05859c5

Browse files
Add Strimzi CR and version 1.3.0
Signed-off-by: Andrew Schofield <andrew_schofield@uk.ibm.com>
1 parent 6540b46 commit 05859c5

File tree

7 files changed

+52
-7
lines changed

7 files changed

+52
-7
lines changed

Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ COPY --chown=esuser:esgroup --from=builder /opt/kafka/libs/ /opt/kafka/libs/
1010
COPY --chown=esuser:esgroup --from=builder /opt/kafka/config/connect-distributed.properties /opt/kafka/config/
1111
COPY --chown=esuser:esgroup --from=builder /opt/kafka/config/connect-log4j.properties /opt/kafka/config/
1212
RUN mkdir /opt/kafka/logs && chown esuser:esgroup /opt/kafka/logs
13-
COPY --chown=esuser:esgroup target/kafka-connect-mq-source-1.3.0-SNAPSHOT-jar-with-dependencies.jar /opt/kafka/libs/
13+
COPY --chown=esuser:esgroup target/kafka-connect-mq-source-1.3.0-jar-with-dependencies.jar /opt/kafka/libs/
1414

1515
WORKDIR /opt/kafka
1616

README.md

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ curl -X POST -H "Content-Type: application/json" http://localhost:8083/connector
8383

8484
## Running with Docker
8585

86-
This repository includes a Dockerfile to run Kafka Connect in distributed mode. It also adds in the MQ Source Connector as an available connector plugin. It uses the default `connect-distributed.properties` and `connect-log4j.properties` files.
86+
This repository includes a Dockerfile to run Kafka Connect in distributed mode. It also adds in the MQ source connector as an available connector plugin. It uses the default `connect-distributed.properties` and `connect-log4j.properties` files.
8787

8888
1. `mvn clean package`
8989
1. `docker build -t kafkaconnect-with-mq-source:0.0.1 .`
@@ -116,7 +116,33 @@ Create ConfigMap for Kafka Connect Log4j configuration:
116116

117117
1. Update the namespace in `kafka-connect.yaml`
118118
1. `kubectl -n <namespace> apply -f kafka-connect.yaml`
119-
1. `curl <serviceIP>:<servicePort>/connector-plugins` to see the MQ Source connector available to use
119+
1. `curl <serviceIP>:<servicePort>/connector-plugins` to see whether the MQ source connector is available to use
120+
121+
### Deploying to OpenShift using Strimzi
122+
123+
This repository includes a Kubernetes yaml file called `strimzi.kafkaconnect.yaml` for use with the [Strimzi](https://strimzi.io) operator. Strimzi provides a simplified way of running the Kafka Connect distributed worker, by defining either a KafkaConnect resource or a KafkaConnectS2I resource.
124+
125+
The KafkaConnectS2I resource provides a nice way to have OpenShift do all the work of building the Docker images for you. This works particularly nicely combined with the KafkaConnector resource that represents an individual connector.
126+
127+
The following instructions assume you are running on OpenShift and have Strimzi 0.16 or later installed.
128+
129+
#### Start a Kafka Connect cluster using KafkaConnectS2I
130+
1. Create a file called `kafkaconnect.yaml` containing the definition of a KafkaConnectS2I resource. Configure it with information it needs to connect to your Kafka cluster. You must include the annotation `strimzi.io/use-connector-resources: "true"` to configure it to use KafkaConnector resources so you can avoid needing to call the Kafka Connect REST API directly.
131+
1. `oc apply -f kafkaconnect.yaml` to create the cluster, which usually takes several minutes.
132+
133+
#### Add the MQ source connector to the cluster
134+
1. `mvn clean package` to build the connector JAR.
135+
1. `mkdir myplugins`
136+
1. `cp target/kafka-connect-mq-source-*-jar-with-dependencies.jar myplugins`
137+
1. `oc start-build <kafkaconnectClusterName>-connect --from-dir ./myplugins` to add the MQ source connector to the Kafka Connect distributed worker cluster. Wait for the build to complete, which usually takes a few minutes.
138+
1. `oc describe kafkaconnects2i <kafkaConnectClusterName>` to check that the MQ source connector is in the list of available connector plugins.
139+
140+
#### Start an instance of the MQ source connector using KafkaConnector
141+
1. `cp deploy/strimzi.kafkaconnector.yaml kafkaconnector.yaml`
142+
1. Update the `kafkaconnector.yaml` file to replace all of the values in `<>`, adding any additional configuration properties.
143+
1. `oc apply -f kafkaconnector.yaml` to start the connector.
144+
1. `oc get kafkaconnector` to list the connectors. You can use `oc describe` to get more details on the connector, such as its status.
145+
120146

121147
## Data formats
122148
Kafka Connect is very flexible but it's important to understand the way that it processes messages to end up with a reliable system. When the connector encounters a message that it cannot process, it stops rather than throwing the message away. Therefore, you need to make sure that the configuration you use can handle the messages the connector will process.
@@ -284,7 +310,7 @@ When configuring TLS connection to MQ, you may find that the queue manager rejec
284310

285311

286312
## Support
287-
A commercially supported version of this connector is available for customers with a support entitlement for [IBM Event Streams](https://www.ibm.com/cloud/event-streams).
313+
A commercially supported version of this connector is available for customers with a support entitlement for [IBM Event Streams](https://www.ibm.com/cloud/event-streams) or [IBM Cloud Pak for Integration](https://www.ibm.com/cloud/cloud-pak-for-integration).
288314

289315

290316
## Issues and contributions

config/mq-source.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@
66
"tasks.max": "1",
77
"topic": "<TOPIC>",
88

9-
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
9+
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
10+
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
1011

1112
"mq.queue.manager": "<QUEUE_MANAGER>",
1213
"mq.connection.name.list": "<CONNECTION_NAME_LIST>",
File renamed without changes.

deploy/strimzi.kafkaconnector.yaml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
apiVersion: kafka.strimzi.io/v1alpha1
2+
kind: KafkaConnector
3+
metadata:
4+
name: mq-source
5+
labels:
6+
strimzi.io/cluster: <KafkaConnect_CLUSTER_NAME>
7+
spec:
8+
class: com.ibm.eventstreams.connect.mqsource.MQSourceConnector
9+
tasksMax: 1
10+
config:
11+
topic: <TOPIC>
12+
mq.queue.manager: <QUEUE_MANAGER>
13+
mq.connection.name.list: <CONNECTION_NAME_LIST>
14+
mq.channel.name: <CHANNEL_NAME>
15+
mq.queue: <QUEUE>
16+
mq.record.builder: com.ibm.eventstreams.connect.mqsource.builders.DefaultRecordBuilder
17+
key.converter: org.apache.kafka.connect.storage.StringConverter
18+
value.converter: org.apache.kafka.connect.storage.StringConverter

pom.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
<groupId>com.ibm.eventstreams.connect</groupId>
2121
<artifactId>kafka-connect-mq-source</artifactId>
2222
<packaging>jar</packaging>
23-
<version>1.3.0-SNAPSHOT</version>
23+
<version>1.3.0</version>
2424
<name>kafka-connect-mq-source</name>
2525
<organization>
2626
<name>IBM Corporation</name>

src/main/java/com/ibm/eventstreams/connect/mqsource/MQSourceConnector.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,7 @@ public class MQSourceConnector extends SourceConnector {
135135
public static final String CONFIG_DOCUMENTATION_TOPIC = "The name of the target Kafka topic.";
136136
public static final String CONFIG_DISPLAY_TOPIC = "Target Kafka topic";
137137

138-
public static String VERSION = "1.3.0-SNAPSHOT";
138+
public static String VERSION = "1.3.0";
139139

140140
private Map<String, String> configProps;
141141

0 commit comments

Comments
 (0)