Skip to content

Commit 659cc98

Browse files
AndrewJSchofieldAndrewJSchofield
committed
Run in docker and kubernetes
1 parent 893562a commit 659cc98

File tree

3 files changed

+140
-1
lines changed

3 files changed

+140
-1
lines changed

‎Dockerfile

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
FROM ibmcom/eventstreams-kafka-ce-icp-linux-amd64:2019.2.1-3a2f93e as builder
2+
3+
4+
FROM ibmjava:8-jre
5+
6+
RUN addgroup --gid 5000 --system esgroup && \
7+
adduser --uid 5000 --ingroup esgroup --system esuser
8+
9+
COPY --chown=esuser:esgroup --from=builder /opt/kafka/bin/ /opt/kafka/bin/
10+
COPY --chown=esuser:esgroup --from=builder /opt/kafka/libs/ /opt/kafka/libs/
11+
COPY --chown=esuser:esgroup --from=builder /opt/kafka/config/ /opt/kafka/config/
12+
RUN mkdir /opt/kafka/logs && chown esuser:esgroup /opt/kafka/logs
13+
COPY --chown=esuser:esgroup target/kafka-connect-mq-sink-1.1.0-jar-with-dependencies.jar /opt/kafka/libs/
14+
15+
WORKDIR /opt/kafka
16+
17+
EXPOSE 8083
18+
19+
USER esuser
20+
21+
ENTRYPOINT ["./bin/connect-distributed.sh", "config/connect-distributed.properties"]

‎README.md

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@ The connector is supplied as source code which you can easily build into a JAR f
99

1010
- [Building the connector](#building-the-connector)
1111
- [Running the connector](#running-the-connector)
12+
- [Running the connector with Docker](#running-with-docker)
13+
- [Deploying the connector to Kubernetes](#deploying-to-kubernetes)
1214
- [Data formats](#data-formats)
1315
- [Security](#security)
1416
- [Configuration](#configuration)
@@ -65,6 +67,39 @@ To run the connector in standalone mode from the directory into which you instal
6567
bin/connect-standalone.sh connect-standalone.properties mq-sink.properties
6668
```
6769

70+
## Running with Docker
71+
72+
This repository includes a Dockerfile to run Kafka Connect in distributed mode. It also adds in the MQ Sink Connector as an available connector plugin. It uses the default connect-distributed.properties file, to provide a custom one add a `COPY` line to the Dockerfile with your customised connect-distributed.properties file.
73+
74+
1. `mvn clean package`
75+
1. `docker build -t kafkaconnect-with-mq-sink:0.0.1 .`
76+
1. `docker run -p 8083:8083 kafkaconnect-with-mq-sink:0.0.1`
77+
78+
## Deploying to Kubernetes
79+
80+
This repository includes a Kubernetes yaml file called `kafka-connect.yaml`. This will create a deployment to run Kafka Connect in distributed mode and a service to access the deployment.
81+
82+
The deployment assumes the existence of two ConfigMaps; `connect-distributed-config` and `connect-log4j-config`. These can be created using the default files in your Kafka install, however it is easier to edit them later if comment and whitespace is trimmed before creation.
83+
84+
### Creating Kafka Connect configuration ConfigMaps
85+
86+
Create ConfigMap for Kafka Connect configuration:
87+
1. `cp kafka/config/connect-distributed.properties connect-distributed.properties.orig`
88+
1. `sed '/^#/d;/^[[:space:]]*$/d' < connect-distributed.properties.orig > connect-distributed.properties`
89+
1. `kubectl -n <namespace> create configmap connect-distributed-config --from-file=connect-distributed.properties`
90+
91+
Create ConfigMap for Kafka Connect Log4j configuration:
92+
1. `cp kafka/config/connect-log4j.properties connect-log4j.properties.orig`
93+
1. `sed '/^#/d;/^[[:space:]]*$/d' < connect-log4j.properties.orig > connect-log4j.properties`
94+
1. `kubectl -n <namespace> create configmap connect-log4j-config --from-file=connect-log4j.properties`
95+
96+
### Creating Kafka Connect deployment and service in Kubernetes
97+
98+
**NOTE:** Remember to [build the Docker image](#running-with-docker) and push it to your Kubernetes image repository. You might need to update the image name in the `kafka-connect.yaml` file.
99+
100+
1. Update the namespace in `kafka-connect.yaml`
101+
1. `kubectl -n <namespace> apply -f kafka-connect.yaml`
102+
1. `curl <serviceIP>:<servicePort>/connector-plugins` to see the MQ Sink connector available to use
68103

69104
## Data formats
70105
Kafka Connect is very flexible but it's important to understand the way that it processes messages to end up with a reliable system. When the connector encounters a message that it cannot process, it stops rather than throwing the message away. Therefore, you need to make sure that the configuration you use can handle the messages the connector will process.
@@ -197,6 +232,7 @@ The configuration options for the Kafka Connect sink connector for IBM MQ are as
197232
| mq.ssl.peer.name | The distinguished name pattern of the TLS (SSL) peer | string | | Blank or DN pattern |
198233
| mq.message.builder.key.header | The JMS message header to set from the Kafka record key | string | | JMSCorrelationID |
199234
| mq.message.builder.value.converter | The class and prefix for message builder's value converter | string | | Class implementing Converter |
235+
| mq.reply.queue | The name of the reply-to queue | string | | MQ queue name or queue URI |
200236

201237

202238
### Using a CCDT file
@@ -227,6 +263,10 @@ Update the connector configuration file to reference `secret-key` in the file:
227263
mq.password=${file:mq-secret.properties:secret-key}
228264
```
229265

266+
##### Using FileConfigProvider in Kubernetes
267+
268+
To use a file for the `mq.password` in Kubernetes, you create a Secret using the file as described in [the Kubernetes docs](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod).
269+
230270
## Troubleshooting
231271

232272
### Unable to connect to Kafka
@@ -241,7 +281,7 @@ For issues relating specifically to this connector, please use the [GitHub issue
241281

242282

243283
## License
244-
Copyright 2017, 2018 IBM Corporation
284+
Copyright 2017, 2018, 2019 IBM Corporation
245285

246286
Licensed under the Apache License, Version 2.0 (the "License");
247287
you may not use this file except in compliance with the License.

‎kafka-connect.yaml

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Deployment
2+
apiVersion: apps/v1
3+
kind: Deployment
4+
metadata:
5+
name: kafkaconnect-deploy
6+
labels:
7+
app: kafkaconnect-with-mq-sink
8+
spec:
9+
replicas: 1
10+
selector:
11+
matchLabels:
12+
app: kafkaconnect-with-mq-sink
13+
template:
14+
metadata:
15+
namespace: <namespace>
16+
labels:
17+
app: kafkaconnect-with-mq-sink
18+
spec:
19+
securityContext:
20+
runAsNonRoot: true
21+
runAsUser: 5000
22+
containers:
23+
- name: kafkaconnect-container
24+
image: kafkaconnect-with-mq-sink:0.0.1
25+
readinessProbe:
26+
httpGet:
27+
path: /
28+
port: 8083
29+
httpHeaders:
30+
- name: Accept
31+
value: '*/*'
32+
failureThreshold: 2
33+
initialDelaySeconds: 5
34+
periodSeconds: 10
35+
successThreshold: 1
36+
timeoutSeconds: 2
37+
livenessProbe:
38+
httpGet:
39+
path: /
40+
port: 8083
41+
httpHeaders:
42+
- name: Accept
43+
value: '*/*'
44+
initialDelaySeconds: 20
45+
periodSeconds: 20
46+
ports:
47+
- containerPort: 8083
48+
volumeMounts:
49+
- name: connect-config
50+
mountPath: /opt/kafka/config/connect-distributed.properties
51+
subPath: connect-distributed.properties
52+
- name: connect-log4j
53+
mountPath: /opt/kafka/config/connect-log4j.properties
54+
subPath: connect-log4j.properties
55+
volumes:
56+
- name: connect-config
57+
configMap:
58+
name: connect-distributed-config
59+
- name: connect-log4j
60+
configMap:
61+
name: connect-log4j-config
62+
63+
---
64+
# Service
65+
apiVersion: v1
66+
kind: Service
67+
metadata:
68+
name: kafkaconnect-service
69+
labels:
70+
app: kafkaconnect-service
71+
spec:
72+
type: NodePort
73+
ports:
74+
- name: kafkaconnect
75+
protocol: TCP
76+
port: 8083
77+
selector:
78+
app: kafkaconnect-with-mq-sink

0 commit comments

Comments
 (0)