Skip to content

Commit bbb1102

Browse files
priyankakujoe-vullo-ibmJoel Hansonneeraj-laaddalelane
authored
MQ V2 OSS release (#136)
* Only once (#23) squash and merge of exactly once work, history is available in the onlyOnce-history branch * feat: Update mq image to latest (#45) Contributes to: event-integration/qp-planning#10883 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: Include appropriate license in packaging (#46) * fix:L Include appropriate license in packaging Include appropriate license in packaging Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad<neeraj.laad@uk.ibm.com> Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com> * fix: Move Licenses folder under META-INF Move Licenses folder under META-INF Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com> * fix: update license in README update license in README Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com> * fix: fix the no-auth MQ integration tests (#48) We use the latest tag MQ image in our integration tests. In December, MQ's container image removed the no-auth svrconn channel so this breaks the connector tests that try to make connections to MQ without credentials. This commit adds a custom MQSC script to configure the queue manager to restore the previous behaviour. The MQSourceTaskAuthIT tests still test the ability to connect to a queue manager with auth credentials, so this commit means we still test connecting with and without credentials. Contributes-to: event-integration/eventstreams-planning#12427 Signed-off-by: Dale Lane <dale.lane@uk.ibm.com> * refactor: delegate config validation to Kafka Connect (#50) * feat: pull in changes from origin Contributes to: event-integration/eventstreams-planning#12467 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Fix suggested review comments Contributes to: event-integration/eventstreams-planning#12467 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> --------- Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: avoid dead-locking the SourceTask if all commits aren't received (#51) * chore: comment updates - no functional changes Fixed some weird line-wrapping in the existing comments, edited comments that were technically inaccurate, and added some extra clarification to comments ahead of changes in the next commit. Signed-off-by: Dale Lane <dale.lane@uk.ibm.com> * test: new unit tests that illustrate desired behaviour poll() should quickly return an empty list if we are still waiting for a previous batch to complete, but throw an exception if this happens too many times. Signed-off-by: Dale Lane <dale.lane@uk.ibm.com> * fix: avoid deadlocking poll calls when commitRecord isnt called Signed-off-by: Dale Lane <dale.lane@uk.ibm.com> * chore: prepare new release I also removed an invalid config option that was included in the pom.xml at some point. Signed-off-by: Dale Lane <dale.lane@uk.ibm.com> * feat: Address review comments Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: Update kafka connect-api and connect-json version Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> --------- Signed-off-by: Dale Lane <dale.lane@uk.ibm.com> Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Co-authored-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Automate mq source release (#52) Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: travis release (#53) Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Addressing review comments Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com feat: updated versions in pom.xml Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com feat: Addressing review comments Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com feat: Addressing review comments Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com Updated BUG Report Template Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com feat: updated copyright year 2024 Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com feat: addressed the review comments and updated the version to 2.1.0 Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com feat: updated the License folder Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com feat: Updated License and Travis for MQ 2.0.2 Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com --------- Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com> Signed-off-by: Dale Lane <dale.lane@uk.ibm.com> Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com Co-authored-by: Joseph Vullo <Joseph.Vullo@ibm.com> Co-authored-by: Joel Hanson <Joel.Hanson2@ibm.com> Co-authored-by: Neeraj Laad <neeraj.laad@uk.ibm.com> Co-authored-by: Dale Lane <Dale.Lane@uk.ibm.com> Co-authored-by: Priyanka K U <Priyanka.K.U@ibm.com>
1 parent 99fc0ff commit bbb1102

File tree

57 files changed

+5628
-1247
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

57 files changed

+5628
-1247
lines changed

.github/ISSUE_TEMPLATE/BUG-REPORT.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,8 @@ body:
5757
label: Version
5858
description: What version of our software are you running?
5959
options:
60-
- 1.3.5 (Default)
60+
- 2.1.0 (Default)
61+
- 1.3.5
6162
- older (<1.3.5)
6263
validations:
6364
required: true

.gitignore

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,8 @@ target/
1515
*.iws
1616

1717
# Visual Studio Code
18-
.vscode/
18+
.vscode/
19+
20+
mqjms.log.*
21+
/src/main/resources/simplelogger.properties
22+
.envrc

Dockerfile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ FROM alpine as builder
55
RUN apk update
66
RUN apk --no-cache add curl
77

8-
RUN curl -L "https://downloads.apache.org/kafka/3.4.1/kafka_2.12-3.4.1.tgz" -o kafka.tgz
8+
RUN curl -L "https://downloads.apache.org/kafka/3.6.2/kafka_2.12-3.6.2.tgz" -o kafka.tgz
99
RUN mkdir /opt/kafka \
1010
&& tar -xf kafka.tgz -C /opt/kafka --strip-components=1
1111

@@ -27,4 +27,4 @@ EXPOSE 8083
2727

2828
USER esuser
2929

30-
ENTRYPOINT ["./bin/connect-distributed.sh", "config/connect-distributed.properties"]
30+
ENTRYPOINT ["./bin/connect-distributed.sh", "config/connect-distributed.properties"]

LICENSE

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -175,7 +175,7 @@ Apache License
175175

176176
END OF TERMS AND CONDITIONS
177177

178-
Copyright 2017 IBM Corporation
178+
Copyright 2017, 2024 IBM Corporation
179179

180180
Licensed under the Apache License, Version 2.0 (the "License");
181181
you may not use this file except in compliance with the License.

README.md

Lines changed: 86 additions & 36 deletions
Large diffs are not rendered by default.

config/mq-source-exactly-once.json

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{
2+
"name": "mq-source-exactly-once",
3+
"config":
4+
{
5+
"connector.class": "com.ibm.eventstreams.connect.mqsource.MQSourceConnector",
6+
"tasks.max": "1",
7+
"topic": "<TOPIC>",
8+
9+
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
10+
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
11+
12+
"mq.queue.manager": "<QUEUE_MANAGER>",
13+
"mq.connection.name.list": "<CONNECTION_NAME_LIST>",
14+
"mq.channel.name": "<CHANNEL_NAME>",
15+
"mq.queue": "<QUEUE>",
16+
"mq.record.builder": "com.ibm.eventstreams.connect.mqsource.builders.DefaultRecordBuilder",
17+
"mq.exactly.once.state.queue": "<EXACTLY_ONCE_STATE_QUEUE>"
18+
}
19+
}

config/mq-source.properties

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2017, 2020 IBM Corporation
1+
# Copyright 2017, 2020, 2023, 2024 IBM Corporation
22
#
33
# Licensed under the Apache License, Version 2.0 (the "License");
44
# you may not use this file except in compliance with the License.
@@ -16,6 +16,8 @@ name=mq-source
1616
connector.class=com.ibm.eventstreams.connect.mqsource.MQSourceConnector
1717

1818
# You can increase this for higher throughput, but message ordering will be lost
19+
# Exactly-once message delivery requires tasks.max to be 1
20+
# (see README.md for more details).
1921
tasks.max=1
2022

2123
# The name of the target Kafka topic - required
@@ -37,6 +39,10 @@ mq.channel.name=
3739
# The name of the source MQ queue - required
3840
mq.queue=
3941

42+
# This is required for exactly-once delivery. If not supplied, message delivery will be at-least-once.
43+
# (see README.md for more details).
44+
# mq.exactly.once.state.queue=
45+
4046
# The user name for authenticating with the queue manager - optional
4147
# mq.user.name=
4248

pom.xml

Lines changed: 51 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
* Copyright 2017, 2020, 2023 IBM Corporation
3+
* Copyright 2017, 2020, 2023, 2024 IBM Corporation
44
*
55
* Licensed under the Apache License, Version 2.0 (the "License");
66
* you may not use this file except in compliance with the License.
@@ -13,15 +13,14 @@
1313
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1414
* See the License for the specific language governing permissions and
1515
* limitations under the License.
16-
-->
17-
<project xmlns="http://maven.apache.org/POM/4.0.0"
18-
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
19-
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
16+
-->
17+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
18+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
2019
<modelVersion>4.0.0</modelVersion>
2120
<groupId>com.ibm.eventstreams.connect</groupId>
2221
<artifactId>kafka-connect-mq-source</artifactId>
2322
<packaging>jar</packaging>
24-
<version>1.3.5</version>
23+
<version>2.1.0</version>
2524
<name>kafka-connect-mq-source</name>
2625
<organization>
2726
<name>IBM Corporation</name>
@@ -46,13 +45,13 @@
4645
<dependency>
4746
<groupId>org.apache.kafka</groupId>
4847
<artifactId>connect-api</artifactId>
49-
<version>3.4.0</version>
48+
<version>3.6.2</version>
5049
<scope>provided</scope>
5150
</dependency>
5251
<dependency>
5352
<groupId>org.apache.kafka</groupId>
5453
<artifactId>connect-json</artifactId>
55-
<version>3.4.0</version>
54+
<version>3.6.2</version>
5655
<scope>provided</scope>
5756
</dependency>
5857

@@ -81,14 +80,14 @@
8180
</dependency>
8281
<dependency>
8382
<groupId>org.slf4j</groupId>
84-
<artifactId>slf4j-log4j12</artifactId>
83+
<artifactId>slf4j-api</artifactId>
8584
<version>2.0.7</version>
86-
<scope>test</scope>
8785
</dependency>
8886
<dependency>
8987
<groupId>org.slf4j</groupId>
90-
<artifactId>slf4j-api</artifactId>
88+
<artifactId>slf4j-simple</artifactId>
9189
<version>2.0.7</version>
90+
<scope>test</scope>
9291
</dependency>
9392
<!-- tests in src/integration depend on a running MQ queue manager -->
9493
<!-- in a container, configured using org.testcontainers -->
@@ -98,10 +97,34 @@
9897
<version>1.17.6</version>
9998
<scope>test</scope>
10099
</dependency>
100+
101+
<dependency>
102+
<groupId>org.assertj</groupId>
103+
<artifactId>assertj-core</artifactId>
104+
<version>3.24.2</version>
105+
<scope>test</scope>
106+
</dependency>
107+
101108
</dependencies>
102109

103110
<build>
104111
<plugins>
112+
<plugin>
113+
<artifactId>maven-resources-plugin</artifactId>
114+
<version>3.3.1</version>
115+
<configuration>
116+
<outputDirectory>${project.build.outputDirectory}/META-INF</outputDirectory>
117+
<resources>
118+
<resource>
119+
<directory>${project.basedir}</directory>
120+
<includes>
121+
<include>LICENSE</include>
122+
</includes>
123+
</resource>
124+
</resources>
125+
</configuration>
126+
</plugin>
127+
105128
<plugin>
106129
<artifactId>maven-compiler-plugin</artifactId>
107130
<version>3.1</version>
@@ -121,6 +144,9 @@
121144
<systemPropertyVariables>
122145
<connectorVersion>${project.version}</connectorVersion>
123146
</systemPropertyVariables>
147+
<additionalClasspathElements>
148+
<additionalClasspathElement>src/test/resources</additionalClasspathElement>
149+
</additionalClasspathElements>
124150
</configuration>
125151
</plugin>
126152

@@ -133,6 +159,9 @@
133159
<systemPropertyVariables>
134160
<connectorVersion>${project.version}</connectorVersion>
135161
</systemPropertyVariables>
162+
<additionalClasspathElements>
163+
<additionalClasspathElement>src/test/resources</additionalClasspathElement>
164+
</additionalClasspathElements>
136165
</configuration>
137166
<executions>
138167
<execution>
@@ -187,7 +216,6 @@
187216
</executions>
188217
</plugin>
189218

190-
191219
<!-- generate test code coverage report -->
192220
<plugin>
193221
<groupId>org.jacoco</groupId>
@@ -200,8 +228,7 @@
200228
<goal>prepare-agent</goal>
201229
</goals>
202230
<configuration>
203-
<destFile>
204-
${project.build.directory}/jacoco-output/jacoco-unit-tests.exec</destFile>
231+
<destFile>${project.build.directory}/jacoco-output/jacoco-unit-tests.exec</destFile>
205232
<propertyName>surefire.jacoco.args</propertyName>
206233
</configuration>
207234
</execution>
@@ -212,10 +239,9 @@
212239
<goal>report</goal>
213240
</goals>
214241
<configuration>
215-
<dataFile>
216-
${project.build.directory}/jacoco-output/jacoco-unit-tests.exec</dataFile>
217-
<outputDirectory>
218-
${project.reporting.outputDirectory}/jacoco-unit-test-coverage-report</outputDirectory>
242+
<dataFile>${project.build.directory}/jacoco-output/jacoco-unit-tests.exec</dataFile>
243+
<outputDirectory>${project.reporting.outputDirectory}/jacoco-unit-test-coverage-report
244+
</outputDirectory>
219245
</configuration>
220246
</execution>
221247
<execution>
@@ -225,8 +251,7 @@
225251
<goal>prepare-agent</goal>
226252
</goals>
227253
<configuration>
228-
<destFile>
229-
${project.build.directory}/jacoco-output/jacoco-integration-tests.exec</destFile>
254+
<destFile>${project.build.directory}/jacoco-output/jacoco-integration-tests.exec</destFile>
230255
<propertyName>failsafe.jacoco.args</propertyName>
231256
</configuration>
232257
</execution>
@@ -237,10 +262,10 @@
237262
<goal>report</goal>
238263
</goals>
239264
<configuration>
240-
<dataFile>
241-
${project.build.directory}/jacoco-output/jacoco-integration-tests.exec</dataFile>
265+
<dataFile>${project.build.directory}/jacoco-output/jacoco-integration-tests.exec</dataFile>
242266
<outputDirectory>
243-
${project.reporting.outputDirectory}/jacoco-integration-test-coverage-report</outputDirectory>
267+
${project.reporting.outputDirectory}/jacoco-integration-test-coverage-report
268+
</outputDirectory>
244269
</configuration>
245270
</execution>
246271
<execution>
@@ -269,18 +294,18 @@
269294
</goals>
270295
<configuration>
271296
<dataFile>${project.build.directory}/jacoco-output/merged.exec</dataFile>
272-
<outputDirectory>
273-
${project.reporting.outputDirectory}/jacoco-merged-test-coverage-report</outputDirectory>
297+
<outputDirectory>${project.reporting.outputDirectory}/jacoco-merged-test-coverage-report
298+
</outputDirectory>
274299
</configuration>
275300
</execution>
276301
</executions>
277302
</plugin>
303+
278304
<plugin>
279305
<groupId>org.apache.maven.plugins</groupId>
280306
<artifactId>maven-checkstyle-plugin</artifactId>
281307
<version>3.2.0</version>
282308
<configuration>
283-
<encoding>UTF-8</encoding>
284309
<consoleOutput>true</consoleOutput>
285310
<failsOnError>true</failsOnError>
286311
<violationSeverity>warning</violationSeverity>
@@ -301,8 +326,6 @@
301326
</executions>
302327
</plugin>
303328
</plugins>
304-
305-
<!-- some integration tests use the custom mqsc file in the resources directory -->
306329
<testResources>
307330
<testResource>
308331
<directory>src/integration/resources</directory>

src/assembly/package.xml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
* Copyright 2018 IBM Corporation
3+
* Copyright 2018, 2024 IBM Corporation
44
*
55
* Licensed under the Apache License, Version 2.0 (the "License");
66
* you may not use this file except in compliance with the License.
@@ -37,6 +37,7 @@
3737
<excludes>
3838
<exclude>**/copyright-exclude</exclude>
3939
<exclude>META-INF/maven/**</exclude>
40+
<exclude>META-INF/LICENSE*</exclude>
4041
</excludes>
4142
</unpackOptions>
4243
<useTransitiveFiltering>true</useTransitiveFiltering>

0 commit comments

Comments
 (0)