Skip to content

Commit 07d5426

Browse files
pasimoespasimoes
authored
WMS 8022 : TEQ in Oracle Database v1.1 (#464)
* Update readme * Issue #409 : Upgrade GraalVM 22.1.0 * Issue #405: Update okafka repo from local to central. * Issue #411: Upgrade Spring Boot Libraries * Issue #406: remove 8022 (WIP) * Issue #405: Remove okafka install in local mvn repo. * Issue #406: Remove 8022 code references. * WIP #416: Lab automation/scripts * WIP #416: Lab automation/scripts : Kafka topics * WIP #416: Lab automation/scripts : Kafka MS Build/Deployment * WIP #416: Lab automation/scripts : Environment * WIP #416: Lab automation/scripts : Environment * WIP #416: Lab automation/scripts : Environment * WIP #416: Lab automation/scripts : Environment * WIP #416: DB CPU Count change to 1 * WIP #416: Build/Deployment automation. * wip * WIP #416: simplify tests. * WIP #416: simplify tests. * WIP #416: simplify tests. * WIP #416: simplify teq steps. * WIP lab 1.1: setup simplification * WIP lab 1.1: change db to free_tier * WIP lab 1.1: change db to free_tier ocpu check * WIP lab 1.1: fix sh permissions * WIP lab 1.1: fix sh permissions * WIP lab 1.1: fix sh permissions * WIP lab 1.1: fix sh permissions * WIP lab 1.1: bash cleanup script * WIP lab 1.1: kafka cleanup script * WIP lab 1.1: fix issues with kafka topic conf * WIP lab 1.1: fix issues with kafka topic conf * WIP lab 1.1: fix issues with sync conf * WIP lab 1.1: fix docker registry cleanup * Upgrade Spring Boot Libraries: Issues #411 and PR #421 * Improve Lab Compartment Desc * add remove docker network * fix libs versions
1 parent a6bc9de commit 07d5426

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

59 files changed

+2446
-206
lines changed

‎workshops/oracleteq-kafka/.dcignore

Lines changed: 1547 additions & 0 deletions
Large diffs are not rendered by default.

‎workshops/oracleteq-kafka/README.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
2+
# Simplify Event-driven Apps with TEQ in Oracle Database (with Kafka interoperability) Workshop
3+
4+
## Introduction
5+
6+
### About this Workshop
7+
8+
This repository contains sample code from the workshop to help you understand Event Mesh using two message brokers and the technical capabilities inside the converged [Oracle Autonomous Database][ATP] to support a scalable event-driven microservices architecture.
9+
10+
You will create four event-driven microservice and two messaging brokers to allow communication between them. In the first lab, you will deploy an Apache Kafka broker to leverage the event-driven communication between two microservices written in Spring Boot. In the second lab, you will create an Oracle Transactional Event Queues (TEQ) and experience the Kafka APIs working in the Kafka compatibility mode. Likewise, this module has the Spring Boot producer and consumer microservices but with Kafka Java client for TEQ, using the okafka library. And finally, in the third lab, you will experiment with the concept of Event Mesh, building a bridge between Kafka and TEQ brokers, and see messages being produced on the Kafka side and consumed on the TEQ side.
11+
12+
Estimated Workshop Time: 50 minutes
13+
14+
> This workshop is part of the [Oracle LiveLabs][LiveLabs] and you can access it through the following address [TEQ in Oracle Database][TEQinOracleDatabase]
15+
16+
### About Product/Technology
17+
18+
* [Oracle Transactional Event Queues][TEQ] is a powerful messaging backbone offered by converged Oracle Database that allows you to build an enterprise-class data-centric microservices architecture.
19+
20+
* [Kafka][kafka] is an open-source distributed event streaming platform used for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
21+
22+
* [okafka][okafka] library, contains Oracle specific implementation of Kafka Client Java APIs. This implementation is built on AQ-JMS APIs.
23+
24+
* The applications will be deployed on [Oracle Cloud Infrastructure](https://www.oracle.com/cloud/) [Cloud Shell](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/cloudshellintro.htm) using pre-installed Docker Engine.
25+
26+
## Objectives
27+
28+
* The first Lab reviews the Kafka and Spring Boot Microservice built to produce and consume messages.
29+
30+
* The second Lab will use Oracle Transactional Event Queues (TEQ) and okafka library and demonstrate the Kafka compatibility of TEQ. Also, this module has the same Spring Boot producer and consumer microservices but uses okafka in place of Kafka libraries and TEQ in the database in place of Kafka broker.
31+
32+
* The third Lab will interconnect the Kafka broker and the Oracle Transactional Event Queues (TEQ), applying Kafka connector and Oracle Database Messaging libraries. This laboratory demonstrates the interoperability between the two brokers, with events flowing from the Kafka side to TEQ, finally consumed at the TEQ side.
33+
34+
## Prerequisites
35+
36+
* This workshop assumes you have an Oracle Cloud Account - Please view this workshop's LiveLabs landing page to see which environments are supported.
37+
38+
>**Note:** If you have a **Free Trial** account, when your Free Trial expires, your account will be converted to an **Always Free** account. You will not be able to conduct Free Tier workshops unless the Always Free environment is available.
39+
**[Click here for the Free Tier FAQ page.][FreeTier]**
40+
41+
## Event Mesh Architecture Overview
42+
43+
As shown in the followed diagram, we have:
44+
45+
* A Kafka Broker and a set of services consuming and producing for it.
46+
47+
* An Oracle TEQ Broker with another set of services around it.
48+
49+
* And connector between Kafka and Oracle TEQ enabling a communication path between them.
50+
51+
![Kafka and Oracle TEQ Event Mesh](images/kafka-oracle-teq-event-mesh.png " ")
52+
53+
You may now **proceed to the next lab**
54+
55+
## Want to Learn More?
56+
57+
* [Oracle Autonomous Database][ATP]
58+
* [Oracle Transactional Event Queues][TEQ]
59+
* [Microservices Architecture with the Oracle Database][MicroservicesArch]
60+
* [https://developer.oracle.com/][DRC]
61+
62+
## Acknowledgements
63+
64+
* **Authors** - Paulo Simoes, Developer Evangelist; Paul Parkinson, Developer Evangelist; Richard Exley, Consulting Member of Technical Staff, Oracle MAA and Exadata
65+
* **Contributors** - Mayank Tayal, Developer Evangelist; Sanjay Goil, VP Microservices and Oracle Database
66+
* **Last Updated By/Date** - Paulo Simoes, May 2022
67+
68+
## License
69+
70+
Copyright (c) 2022 Oracle and/or its affiliates.
71+
72+
Licensed under the Universal Permissive License v 1.0 as shown at <https://oss.oracle.com/licenses/upl>.
73+
74+
[ATP]: https://docs.oracle.com/en/cloud/paas/autonomous-database/index.html
75+
[TEQ]: https://docs.oracle.com/en/database/oracle/oracle-database/21/adque/index.html
76+
[kafka]: https://kafka.apache.org
77+
[okafka]: https://docs.oracle.com/en/database/oracle/oracle-database/21/adque/Kafka_cient_interface_TEQ.html#GUID-94589C97-F323-4607-8C3A-10A0EDF9DA0D
78+
[LiveLabs]: https://bit.ly/golivelabs
79+
[TEQinOracleDatabase]: https://bit.ly/TEQinOracleDatabase
80+
[FreeTier]: https://www.oracle.com/cloud/free/faq.html
81+
[DRC]: https://developer.oracle.com
82+
[MicroservicesArch]: https://www.oracle.com/technetwork/database/availability/trn5515-microserviceswithoracle-5187372.pdf

‎workshops/oracleteq-kafka/cloud-setup/cmd/bash-cleanup

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/bin/bash
2+
# Copyright (c) 2021 Oracle and/or its affiliates.
3+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
4+
5+
sed '/LiveLab Setup -- BEGIN/,/LiveLab Setup -- END/d' "${HOME}"/.bashrc > "${HOME}"/.bashrc.tmp
6+
mv "${HOME}"/.bashrc.tmp "${HOME}"/.bashrc

‎workshops/oracleteq-kafka/cloud-setup/cmd/container-logs

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#!/bin/bash
2+
# Copyright (c) 2021 Oracle and/or its affiliates.
3+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
4+
5+
container_name=${1}
6+
number_of_lines=${2:-10}
7+
8+
if [ -z "${container_name}" ]; then
9+
echo "ERROR: Microservice container name should not be empty!"
10+
exit 2
11+
fi
12+
13+
docker logs --tail "${number_of_lines}" "${container_name}"

‎workshops/oracleteq-kafka/cloud-setup/cmd/kafka-add-topic

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#!/bin/bash
2+
# Copyright (c) 2021 Oracle and/or its affiliates.
3+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
4+
5+
KAFKA_TOPIC=${1:-"LABTEQTOPIC"}
6+
7+
VAR=$(docker exec broker kafka-topics --bootstrap-server broker:9092 --create --topic "${KAFKA_TOPIC}")
8+
9+
if [[ "$VAR" == "Created topic"* ]]; then
10+
echo "Created topic ${KAFKA_TOPIC}"
11+
state_set KAFKA_TOPIC_NAME "${KAFKA_TOPIC}"
12+
13+
# Adjust the Kafka Producer and Consumer configuration files.
14+
echo "Configuring Kafka Producer to produce on topic ${KAFKA_TOPIC}."
15+
sed -i 's/KAFKA_TOPIC_NAME/'"${KAFKA_TOPIC}"'/g' "$LAB_HOME/springboot-kafka/kafka-producer/src/main/resources/application.yaml"
16+
17+
echo "Configuring Kafka Consumer to consume from topic ${KAFKA_TOPIC}."
18+
sed -i 's/KAFKA_TOPIC_NAME/'"${KAFKA_TOPIC}"'/g' "$LAB_HOME/springboot-kafka/kafka-consumer/src/main/resources/application.yaml"
19+
fi
20+

‎workshops/oracleteq-kafka/cloud-setup/cmd/kafka-connect-status

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/bin/bash
2+
# Copyright (c) 2021 Oracle and/or its affiliates.
3+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
4+
5+
curl -Ss http://localhost:8083/connectors/JmsConnectSync_teqlab/status | jq

‎workshops/oracleteq-kafka/cloud-setup/cmd/kafka-dequeue

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/bin/bash
2+
# Copyright (c) 2021 Oracle and/or its affiliates.
3+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
4+
5+
kafka_topic=${1:-$(state_get KAFKA_TOPIC_NAME)}
6+
7+
if ! state_get KAFKA_SETUP; then
8+
echo "ERROR: Kafka infrastructure should be setting up!"
9+
exit 2
10+
fi
11+
12+
if [ -z "${kafka_topic}" ]; then
13+
echo "ERROR: The topic name should not be empty!"
14+
exit 2
15+
fi
16+
17+
echo "Dequeuing Topic ${kafka_topic} from beginning."
18+
docker exec --interactive --tty broker \
19+
kafka-console-consumer \
20+
--bootstrap-server broker:9092 \
21+
--topic "${kafka_topic}" \
22+
--from-beginning

‎workshops/oracleteq-kafka/cloud-setup/cmd/kafka-env-cleanup

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/bin/bash
2+
# Copyright (c) 2021 Oracle and/or its affiliates.
3+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
4+
5+
rm "$LAB_HOME"/cloud-setup/state/KAFKA_SETUP
6+
rm "$LAB_HOME"/cloud-setup/state/CFLCONNECT_IMAGE
7+
rm "$LAB_HOME"/cloud-setup/state/LAB_KAFKA_NETWORK

‎workshops/oracleteq-kafka/cloud-setup/cmd/kafka-start

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/bin/bash
2+
# Copyright (c) 2021 Oracle and/or its affiliates.
3+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
4+
5+
# Fail on error
6+
set -eu
7+
8+
if ! state_get KAFKA_SETUP; then
9+
echo "ERROR: Kafka infrastructure should be setting up!"
10+
exit
11+
fi
12+
13+
# Start Kafka Infrastructure
14+
cd "$LAB_HOME"/cloud-setup/confluent-kafka
15+
./docker-compose up -d
16+
17+
# Connect Kafka Broker to Lab network
18+
echo "Connecting Kafka broker on network $(state_get LAB_KAFKA_NETWORK)"
19+
docker network connect $(state_get LAB_KAFKA_NETWORK) broker
20+
21+
state_set_done KAFKA_RUNNING

‎workshops/oracleteq-kafka/cloud-setup/cmd/kafka-status

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/bin/bash
2+
# Copyright (c) 2021 Oracle and/or its affiliates.
3+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
4+
5+
cd "${LAB_HOME}"/cloud-setup/confluent-kafka
6+
./docker-compose ps
7+
cd "${LAB_HOME}"

0 commit comments

Comments
 (0)