Skip to content

Commit 6d445a6

Browse files
committed
Added screenshots for AWS kinesis tutorial
1 parent 2f6a325 commit 6d445a6

File tree

4 files changed

+48
-30
lines changed

4 files changed

+48
-30
lines changed
Binary file not shown.
Loading
Loading

docs/tutorials/integrate_aws_kinesis.md

Lines changed: 48 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -11,14 +11,14 @@ In this tutorial, you are going to learn, how you can use [Eclipse Vorto](https:
1111
We are going to use the GPS sensors, that we had described and connected with Vorto in the [mapping pipeline tutorial](create_mapping_pipeline.md). Make sure you had worked through that tutorial thoroughly before proceeding because it gives you a very good understanding of the essence of the Eclipse Vorto project and prepares everything you need for this tutorial.
1212
Ready? Great. Let's proceed.
1313

14-
As shown in the following illustration, we send GPS location data from two different sensors using different data formats via MQTT to Eclipse Hono. The [Eclipse Vorto normalization middleware](https://github.com/eclipse/vorto-examples/blob/master/vorto-middleware/Readme.md) consumes the sensor data and uses the [Vorto AWS Kinesis plugin](https://github.com/eclipse/vorto-examples/blob/master/vorto-middleware/middleware-ext-kinesis/Readme.md), in order to transform the data to a normalized, semantically enriched JSON before forwarding this data to an AWS Kinesis data stream. In Kinesis we will then write a small application, that analyzes the harmonized location data.
14+
As shown in the following illustration, we send GPS location data from two different sensors using different data formats via MQTT to Eclipse Hono. The [Eclipse Vorto normalization middleware](https://github.com/eclipse/vorto-examples/blob/master/vorto-middleware/Readme.md) consumes the sensor data and uses the [Vorto AWS Kinesis plugin](https://github.com/eclipse/vorto-examples/blob/master/vorto-middleware/middleware-ext-kinesis/Readme.md), in order to transform the data to a normalized, semantically enriched JSON before forwarding this data to an AWS Kinesis data stream. In Kinesis we will then write a small analytics application, that analyzes the Vorto compliant location data.
1515

1616
<img src="../images/tutorials/integrate_kinesis/overview_kinesis_vorto.png"/>
1717

1818

1919
## Prerequisites
2020

21-
* Successfully completed the [mapping pipeline tutorial](create_mapping_pipeline.md)
21+
* Successfully completed the entire [mapping pipeline tutorial](create_mapping_pipeline.md)
2222

2323
<br />
2424

@@ -28,9 +28,8 @@ Here are the steps, that we are going to take during this tutorial:
2828

2929
1. Setting up Kinesis on AWS
3030
2. Configuring and starting the Vorto middleware with your AWS Kinesis data stream settings
31-
3. Sending device data to Eclipse Hono via MQTT and monitor the incoming data in the Eclipse Vorto middleware dashboard
32-
4. Creating an AWS Kinesis analytics application
33-
5. Testing the analytics application
31+
3. Creating an AWS Kinesis analytics application
32+
4. Testing the analytics application by sending device data via MQTT
3433

3534
<br />
3635

@@ -45,24 +44,16 @@ Here are the steps, that we are going to take during this tutorial:
4544

4645
## Step 2: Configurating & Starting the Eclipse Vorto Middleware
4746

48-
1. Head over to AWS IAM, and create a technical user with AWS Kinesis full access permissions. Keep note of the access key and secret key.
49-
2. Start the Eclipse Vorto Middleware docker with the required [Kinesis environment variables](https://github.com/eclipse/vorto-examples/blob/master/vorto-middleware/middleware-ext-kinesis/Readme.md#configuration) ```docker run -it -p 8080:8080 -v //C/absolute_local_dir:/mappings -p 8080:8080 -e mapping_spec_dir=/mappings -e github.client.clientId=your_github_clientid -e github.client.clientSecret=your_github_clientsecret -e hono.tenantId=your_tenantId -e hono.password=your_hono_password -e kinesis.streamName=vortoDemo -kinesis.accessKey=mykey -e kinesis.secretKey=mysecret eclipsevorto/vorto-normalizer:nightly```
50-
3. Once the service has started successfully, open the local [Eclipse Vorto middleware dashboard](http://localhost:8080/#/plugins). You should see the AWS Kinesis plugin in active mode (green light).
47+
* Head over to AWS IAM, and create a technical user with AWS Kinesis full access permissions. Keep note of the access key and secret key.
48+
* Start the Eclipse Vorto Middleware docker with the required [Kinesis environment variables](https://github.com/eclipse/vorto-examples/blob/master/vorto-middleware/middleware-ext-kinesis/Readme.md#configuration)
5149

52-
**Congrats!** Your middleware is all set now to receive IoT device data from Eclipse Hono protocol adapters and forward it to your AWS Kinesis stream.
53-
54-
## Step 3: Sending device data
55-
56-
Send some location data to Eclipse Hono via MQTT
50+
```docker run -it -p 8080:8080 -v c:/absolute_local_dir:/mappings -e mapping_spec_dir=/mappings -e github.client.clientId=your_github_clientid -e github.client.clientSecret=your_github_clientsecret -e hono.tenantId=your_tenantId -e hono.password=your_hono_password -e kinesis.streamName=vortoDemo -kinesis.accessKey=mykey -e kinesis.secretKey=mysecret eclipsevorto/vorto-normalizer:nightly```
5751

58-
```mosquitto pub ....```
52+
* Once the service has started successfully, open the local [Eclipse Vorto middleware dashboard](http://localhost:8080/#/plugins). You should see the AWS Kinesis plugin in active mode (green light).
5953

60-
Open the [Vorto Middleware Monitoring dashboard](http://localhost:8080/#/monitoring) and observe the logs. You should see something like this:
61-
62-
//TODO: Fixme to show the monitoring logs showing the gps sensor data
63-
<img src="../images/tutorials/integrate_kinesis/kinesis_logs.png"/>
54+
**Congrats!** Your middleware is all set now to receive IoT device data from Eclipse Hono protocol adapters and forward it to your AWS Kinesis stream.
6455

65-
## Step 4: Creating an AWS Kinesis analytics application
56+
## Step 3: Creating an AWS Kinesis analytics application
6657

6758
1. Open the Kinesis service dashboard in AWS
6859
2. Click **Create Analytics Application**
@@ -72,29 +63,56 @@ Open the [Vorto Middleware Monitoring dashboard](http://localhost:8080/#/monitor
7263
6. Select **Connect streaming data** and select **Kinesis data stream** as source
7364
7. Choose the data stream, we had created in step 1, e.g. *vortoDemo*
7465
8. Leave the other selections as default.
75-
9. Click **Discover schema**. Make sure you are sending data to the Vorto middleware, so that AWS is able to discover the schema from the data it receives. You should be able to see a table, similar to this:<img src="../images/tutorials/integrate_kinesis/kinesis_discover_schema.png"/>
66+
9. Click **Discover schema**. Make sure you are sending data to the Vorto middleware, so that AWS is able to discover the schema from the data it receives. You should be able to see a table, similar to this:<img src="../images/tutorials/integrate_kinesis/schemaDiscovery.png"/>
7667
10. Select **Save and confinue**
77-
11. Choose **Go to SQL Editor** in order to process the incoming device data. Copy the following SQL snippet in the SQL editor and **save&run** the SQL:
68+
11. Choose **Go to SQL Editor** in order to process the incoming device data. Copy the following SQL snippet in the SQL editor:
7869

7970
```
80-
HERE goes the SQL analytics snippet for the gps analytics
71+
CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM"
72+
("latitude" DECIMAL(5,4),
73+
"longitude" DECIMAL(7,4),
74+
"deviceId" INTEGER);
75+
76+
CREATE OR REPLACE PUMP "STREAM_PUMP" AS
77+
INSERT INTO "DESTINATION_SQL_STREAM"
78+
SELECT STREAM "latitude", "longitude", "deviceId"
79+
FROM "SOURCE_SQL_STREAM_001"
80+
WHERE "latitude" <> 1.3475 and "longitude" <> 103.8410;
8181
```
8282

83-
TODO: Explain the SQL snippet.
83+
Firstly, we create a new destination stream, which will contain all data that we are interested in for further processing.
84+
In the second statement, we find all devices and their locations that do not match a specific reference point and insert them into our destination stream.
85+
86+
> This is very basic and simple example. Feel free to modify it to your liking, for a more complex analytics scenerio.
87+
88+
Finally, make sure, that you **save&run** the SQL. This sets up the realtime analytics application, ready to receive data from our Vorto middleware.
89+
90+
## Step 4: Testing the analytics application
91+
92+
In this step, we are going to test our analytics app by sending some test data via MQTT from both of our gps sensors:
93+
94+
* Firstly, we send data from the first GPS sensor **not meeting the location reference point condition**. This will be detected by our analytics algorithm and insert the data into our destination kinesis stream.
95+
96+
```
97+
mosquitto_pub -h mqtt.bosch-iot-hub.com -p 8883 -u 4711@tenantID -P secret -t telemetry/tenantID/4711 -m '{"longitude": "103.8410", "latitude": "1.3474"}' --cafile iothub.crt
98+
```
8499

85-
## Step 5: Testing the analytics application
100+
* Now, let's send some data from the other gps sensor that **meets the reference point condition**:
86101

87-
In this step, we are going to send some test data of our first GPS sensor with location data, **not meeting the geofence condition**.
102+
```
103+
mosquitto_pub -h mqtt.bosch-iot-hub.com -p 8883 -u 4712@tenantID -P secret -t telemetry/tenantID/4712 -m '103.8410,1.3475' --cafile iothub.crt
104+
```
88105

89-
```mosquitto pub sending JSON with location data```
90106

91-
Now, we are sending test data of the second gps sensor in CSV format that **meets the geofence condition**:
107+
* You can check if the correct location data is pushed into our kinesis destination stream:
92108

93-
```mosquitto pub sending CSV with location data```
109+
<img src="../images/tutorials/integrate_kinesis/sql.png"/>
110+
111+
112+
* Optionally: For easy debugging, you can observe the [monitoring dashboard](http://localhost:8080/#/monitoring) for any inbound/outbound device messages.
94113

95-
Now you can check our Kinesis analytics application where the destination stream now contains the data of our second sensor:
96114

97-
ADD ILLUSTRATION HERE
115+
**Way to go!** That's it. You just saw how easy it is to use the Vorto Kinesis plugin with the Eclipse Vorto normalization middleware, in order to push normalized/semantic device data to AWS Kinesis.
98116

99117
## What's next?
100118

0 commit comments

Comments
 (0)