Skip to content

Commit 645519b

Browse files
vdesabouvdesabou
committed
wip not working
1 parent 80cacde commit 645519b

File tree

5 files changed

+267
-0
lines changed

5 files changed

+267
-0
lines changed

‎ccloud/fm-opensearch-sink/README.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Fully Managed OpenSearch Sink connector
2+
3+
4+
5+
## Objective
6+
7+
Quickly test [OpenSearch Sink](https://docs.confluent.io/cloud/current/connectors/cc-opensearch-sink.html) connector.
8+
9+
10+
11+
## Exposing docker container over internet
12+
13+
**🚨WARNING🚨** It is considered a security risk to run this example on your personal machine since you'll be exposing a TCP port over internet using [Ngrok](https://ngrok.com). It is strongly encouraged to run it on a AWS EC2 instance where you'll use [Confluent Static Egress IP Addresses](https://docs.confluent.io/cloud/current/networking/static-egress-ip-addresses.html#use-static-egress-ip-addresses-with-ccloud) (only available for public endpoints on AWS) to allow traffic from your Confluent Cloud cluster to your EC2 instance using EC2 Security Group.
14+
15+
Example in order to set EC2 Security Group with Confluent Static Egress IP Addresses and port 1521:
16+
17+
```bash
18+
group=$(aws ec2 describe-instances --instance-id <$ec2-instance-id> --output=json | jq '.Reservations[] | .Instances[] | {SecurityGroups: .SecurityGroups}' | jq -r '.SecurityGroups[] | .GroupName')
19+
aws ec2 authorize-security-group-ingress --group-name $group --protocol tcp --port 1521 --cidr 13.36.88.88/32
20+
aws ec2 authorize-security-group-ingress --group-name $group --protocol tcp --port 1521 --cidr 13.36.88.89/32
21+
etc...
22+
```
23+
24+
An [Ngrok](https://ngrok.com) auth token is necessary in order to expose the Docker Container port to internet, so that fully managed connector can reach it.
25+
26+
You can sign up at https://dashboard.ngrok.com/signup
27+
If you have already signed up, make sure your auth token is setup by exporting environment variable `NGROK_AUTH_TOKEN`
28+
29+
Your auth token is available on your dashboard: https://dashboard.ngrok.com/get-started/your-authtoken
30+
31+
Ngrok web interface available at http://localhost:4551
32+
33+
## Prerequisites
34+
35+
See [here](https://kafka-docker-playground.io/#/how-to-use?id=%f0%9f%8c%a4%ef%b8%8f-confluent-cloud-examples)
36+
37+
38+
## How to run
39+
40+
Simply run:
41+
42+
```
43+
$ just use <playground run> command and search for fully-managed-opensearch-sink.sh in this folder
44+
```

‎ccloud/fm-opensearch-sink/docker-compose.yml

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
---
2+
version: '3.5'
3+
services:
4+
opensearch-node1:
5+
image: opensearchproject/opensearch:latest
6+
hostname: opensearch-node1
7+
container_name: opensearch-node1
8+
environment:
9+
- cluster.name=opensearch-cluster
10+
- node.name=opensearch-node1
11+
- discovery.seed_hosts=opensearch-node1,opensearch-node2
12+
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
13+
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
14+
- OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
15+
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=P@szw0rd1! # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and higher
16+
ulimits:
17+
memlock:
18+
soft: -1
19+
hard: -1
20+
nofile:
21+
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
22+
hard: 65536
23+
volumes:
24+
- opensearch-data1:/usr/share/opensearch/data
25+
ports:
26+
- 9200:9200
27+
- 9600:9600 # required for Performance Analyzer
28+
# networks:
29+
# - opensearch-net
30+
opensearch-node2:
31+
image: opensearchproject/opensearch:latest
32+
hostname: opensearch-node2
33+
container_name: opensearch-node2
34+
environment:
35+
- cluster.name=opensearch-cluster
36+
- node.name=opensearch-node2
37+
- discovery.seed_hosts=opensearch-node1,opensearch-node2
38+
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
39+
- bootstrap.memory_lock=true
40+
- OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m
41+
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=P@szw0rd1!
42+
ulimits:
43+
memlock:
44+
soft: -1
45+
hard: -1
46+
nofile:
47+
soft: 65536
48+
hard: 65536
49+
volumes:
50+
- opensearch-data2:/usr/share/opensearch/data
51+
# networks:
52+
# - opensearch-net
53+
54+
55+
opensearch-dashboards:
56+
image: opensearchproject/opensearch-dashboards:latest
57+
hostname: opensearch-dashboards
58+
container_name: opensearch-dashboards
59+
ports:
60+
- 5601:5601
61+
expose:
62+
- '5601'
63+
environment:
64+
OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
65+
# networks:
66+
# - opensearch-net
67+
68+
# https://ngrok.com/docs/using-ngrok-with/docker/
69+
ngrok:
70+
image: ngrok/ngrok:latest
71+
hostname: ngrok
72+
container_name: ngrok
73+
ports:
74+
- 4040:4040
75+
restart: unless-stopped
76+
links:
77+
- opensearch-node1
78+
command:
79+
- "start"
80+
- "--all"
81+
- "--log=stdout"
82+
- "--config"
83+
- "/etc/ngrok.yml"
84+
volumes:
85+
- ../../ccloud/fm-opensearch-sink/ngrok.yml:/etc/ngrok.yml
86+
environment:
87+
NGROK_AUTHTOKEN: $NGROK_AUTH_TOKEN
88+
# networks:
89+
# - opensearch-net
90+
91+
volumes:
92+
opensearch-data1:
93+
opensearch-data2:
94+
95+
# networks:
96+
# opensearch-net:

‎ccloud/fm-opensearch-sink/fully-managed-opensearch-sink.sh

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
#!/bin/bash
2+
set -e
3+
4+
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
5+
source ${DIR}/../../scripts/utils.sh
6+
7+
NGROK_AUTH_TOKEN=${NGROK_AUTH_TOKEN:-$1}
8+
9+
display_ngrok_warning
10+
11+
bootstrap_ccloud_environment
12+
13+
14+
15+
log "Creating users topic"
16+
set +e
17+
playground topic delete --topic users
18+
sleep 3
19+
playground topic create --topic users
20+
set -e
21+
22+
docker compose build
23+
docker compose down -v --remove-orphans
24+
docker compose up -d --quiet-pull
25+
26+
sleep 5
27+
28+
log "Waiting for ngrok to start"
29+
while true
30+
do
31+
container_id=$(docker ps -q -f name=ngrok)
32+
if [ -n "$container_id" ]
33+
then
34+
status=$(docker inspect --format '{{.State.Status}}' $container_id)
35+
if [ "$status" = "running" ]; then
36+
break
37+
fi
38+
fi
39+
log "Waiting for container ngrok to start..."
40+
sleep 5
41+
done
42+
log "Getting ngrok hostname and port"
43+
NGROK_URL=$(curl --silent http://127.0.0.1:4040/api/tunnels | jq -r '.tunnels[0].public_url')
44+
NGROK_HOSTNAME=$(echo $NGROK_URL | cut -d "/" -f3 | cut -d ":" -f 1)
45+
NGROK_PORT=$(echo $NGROK_URL | cut -d "/" -f3 | cut -d ":" -f 2)
46+
47+
48+
playground --output-level WARN container logs --container opensearch-dashboards --wait-for-log "Server running at http://0.0.0.0:5601" --max-wait 300
49+
log "Navigate to http://127.0.0.1:5601 (admin/P@szw0rd1!) for OpenSearch Dashboards"
50+
51+
connector_name="OpenSearchSink_$USER"
52+
set +e
53+
playground connector delete --connector $connector_name > /dev/null 2>&1
54+
set -e
55+
56+
log "Creating fully managed connector"
57+
playground connector create-or-update --connector $connector_name << EOF
58+
{
59+
"connector.class": "OpenSearchSink",
60+
"name": "$connector_name",
61+
"kafka.auth.mode": "KAFKA_API_KEY",
62+
"kafka.api.key": "$CLOUD_KEY",
63+
"kafka.api.secret": "$CLOUD_SECRET",
64+
"input.data.format": "AVRO",
65+
"topics": "users",
66+
"instance.url": "https://$NGROK_HOSTNAME:$NGROK_PORT",
67+
"auth.type": "BASIC",
68+
"connection.user": "admin",
69+
"connection.password": "P@szw0rd1!",
70+
"indexes.num": "1",
71+
"index1.name" : "users_index",
72+
"index1.topic": "users",
73+
"request.method": "POST",
74+
"opensearch.ssl.enabled": "true",
75+
"retry.backoff.policy": "CONSTANT_VALUE",
76+
"max.retries": "1",
77+
"tasks.max" : "1"
78+
}
79+
EOF
80+
wait_for_ccloud_connector_up $connector_name 600
81+
82+
log "Sending messages to topic users"
83+
playground topic produce -t users --nb-messages 10 --forced-value '{"f1":"value%g"}' << 'EOF'
84+
{
85+
"type": "record",
86+
"name": "myrecord",
87+
"fields": [
88+
{
89+
"name": "f1",
90+
"type": "string"
91+
}
92+
]
93+
}
94+
EOF
95+
96+
sleep 10
97+
98+
# curl -XPUT --insecure -u 'admin:P@szw0rd1!' 'https://localhost:9200/users_index/_doc/1' -H 'Content-Type: application/json' -d '{"f1": "value1"}'
99+
100+
# curl -XGET --insecure -u 'admin:P@szw0rd1!' 'https://4.tcp.eu.ngrok.io:14463/users_index/_search?pretty' > /tmp/result.log 2>&1
101+
102+
log "Check that the data is available in opensearch in users_index"
103+
curl -XGET --insecure -u 'admin:P@szw0rd1!' 'https://localhost:9200/users_index/_search?pretty' > /tmp/result.log 2>&1
104+
cat /tmp/result.log
105+
grep "f1" /tmp/result.log | grep "value1"
106+
grep "f1" /tmp/result.log | grep "value10"
107+
108+
log "Do you want to delete the fully managed connector $connector_name ?"
109+
check_if_continue
110+
111+
playground connector delete --connector $connector_name

‎ccloud/fm-opensearch-sink/ngrok.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
version: 2
2+
tunnels:
3+
opensearch-node1:
4+
addr: opensearch-node1:9200
5+
proto: tcp

‎ccloud/fm-opensearch-sink/stop.sh

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash
2+
3+
4+
5+
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
6+
source ${DIR}/../../scripts/utils.sh
7+
8+
9+
docker compose down -v --remove-orphans
10+
11+
maybe_delete_ccloud_environment

0 commit comments

Comments
 (0)