Skip to content

Commit f8f3b2d

Browse files
davetroianobbejeck
andauthored
Flink SQL tumbling windows tutorial (#1625)
* feat: add Flink SQL tumbling windows tutorial * chore: wording tweaks and typos fixes These are largely copy/paste issues noticed during tumbling windows tutorial dev. * Update _includes/tutorials/tumbling-windows/flinksql/markup/dev/create-ratings.adoc Co-authored-by: Bill Bejeck <bbejeck@gmail.com> --------- Co-authored-by: Bill Bejeck <bbejeck@gmail.com>
1 parent 399fb8a commit f8f3b2d

File tree

81 files changed

+1213
-30
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

81 files changed

+1213
-30
lines changed

.semaphore/semaphore.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -499,6 +499,9 @@ blocks:
499499
- name: Flink SQL test for filtering
500500
commands:
501501
- make -C _includes/tutorials/filtering/flinksql/code tutorial
502+
- name: Flink SQL tumbling windows tests
503+
commands:
504+
- make -C _includes/tutorials/tumbling-windows/flinksql/code tutorial
502505
- name: Flink SQL test for deduplication
503506
commands:
504507
- make -C _includes/tutorials/finding-distinct/flinksql/code tutorial
Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
dev:
2+
steps:
3+
- title: Prerequisites
4+
content:
5+
- action: skip
6+
render:
7+
file: shared/markup/dev/docker-prerequisite.adoc
8+
9+
- title: Initialize the project
10+
content:
11+
- action: execute
12+
file: tutorial-steps/dev/init.sh
13+
render:
14+
file: tutorials/tumbling-windows/flinksql/markup/dev/init.adoc
15+
16+
- title: Get Confluent Platform
17+
content:
18+
- action: make_file
19+
file: docker-compose.yml
20+
render:
21+
file: tutorials/tumbling-windows/flinksql/markup/dev/make-docker-compose.adoc
22+
23+
- action: execute_async
24+
file: tutorial-steps/dev/docker-compose-up.sh
25+
render:
26+
file: tutorials/tumbling-windows/flinksql/markup/dev/start-compose.adoc
27+
28+
- action: execute
29+
file: tutorial-steps/dev/wait-for-containers.sh
30+
render:
31+
skip: true
32+
33+
- title: Write the program interactively using the CLI
34+
content:
35+
- action: docker_flinksql_cli_session
36+
container: flink-sql-client
37+
docker_bootup_file: tutorial-steps/dev/start-cli.sh
38+
column_width: 20
39+
render:
40+
file: tutorials/tumbling-windows/flinksql/markup/dev/start-cli.adoc
41+
stdin:
42+
- file: tutorial-steps/dev/create-ratings.sql
43+
render:
44+
file: tutorials/tumbling-windows/flinksql/markup/dev/create-ratings.adoc
45+
46+
- file: tutorial-steps/dev/populate-ratings.sql
47+
render:
48+
file: tutorials/tumbling-windows/flinksql/markup/dev/populate-ratings.adoc
49+
50+
- file: tutorial-steps/dev/transient-query-ratings-by-6hr-window.sql
51+
render:
52+
file: tutorials/tumbling-windows/flinksql/markup/dev/transient-query-ratings-by-6hr-window.adoc
53+
54+
- file: tutorial-steps/dev/create-ratings-by-6hr-window.sql
55+
render:
56+
file: tutorials/tumbling-windows/flinksql/markup/dev/create-ratings-by-6hr-window.adoc
57+
58+
stdout:
59+
directory: tutorial-steps/dev/outputs
60+
61+
- title: Validate output
62+
content:
63+
- action: execute
64+
file: tutorial-steps/dev/validate-ratings-by-6hr-window.sh
65+
stdout: tutorial-steps/dev/outputs/validate-ratings-by-6hr-window.log
66+
render:
67+
file: tutorials/tumbling-windows/flinksql/markup/dev/validate-ratings-by-6hr-window.adoc
68+
69+
test:
70+
steps:
71+
- title: Decide what testing tools to use
72+
content:
73+
- action: skip
74+
render:
75+
file: tutorials/tumbling-windows/flinksql/markup/test/test-architecture.adoc
76+
77+
- title: Create the test skeleton
78+
content:
79+
- action: execute
80+
file: tutorial-steps/test/make-test-dirs.sh
81+
render:
82+
file: tutorials/tumbling-windows/flinksql/markup/test/make-test-dirs.adoc
83+
84+
- action: make_file
85+
file: build.gradle
86+
render:
87+
file: tutorials/tumbling-windows/flinksql/markup/test/make-build-gradle.adoc
88+
89+
- action: execute
90+
file: tutorial-steps/test/gradle-wrapper.sh
91+
render:
92+
file: tutorials/tumbling-windows/flinksql/markup/test/make-gradle-wrapper.adoc
93+
94+
- title: Create SQL resources
95+
content:
96+
- action: make_file
97+
file: src/test/resources/create-ratings.sql.template
98+
render:
99+
file: tutorials/tumbling-windows/flinksql/markup/test/create-resource-create-ratings.sql.template.adoc
100+
101+
- action: make_file
102+
file: src/test/resources/populate-ratings.sql
103+
render:
104+
file: tutorials/tumbling-windows/flinksql/markup/test/create-resource-populate-ratings.sql.adoc
105+
106+
- action: make_file
107+
file: src/test/resources/create-ratings-by-6hr-window.sql.template
108+
render:
109+
file: tutorials/tumbling-windows/flinksql/markup/test/create-resource-create-ratings-by-6hr-window.sql.template.adoc
110+
111+
- action: make_file
112+
file: src/test/resources/query-ratings-by-6hr-window.sql
113+
render:
114+
file: tutorials/tumbling-windows/flinksql/markup/test/create-resource-query-ratings-by-6hr-window.sql.adoc
115+
116+
- action: make_file
117+
file: src/test/resources/expected-ratings-by-6hr-window.txt
118+
render:
119+
file: tutorials/tumbling-windows/flinksql/markup/test/create-resource-expected-ratings-by-6hr-window.txt.adoc
120+
121+
- title: Write a test
122+
content:
123+
- action: make_file
124+
file: src/test/java/io/confluent/developer/AbstractFlinkKafkaTest.java
125+
render:
126+
file: tutorials/tumbling-windows/flinksql/markup/test/make-test-base.adoc
127+
128+
- action: make_file
129+
file: src/test/java/io/confluent/developer/FlinkSqlTumblingWindowTest.java
130+
render:
131+
file: tutorials/tumbling-windows/flinksql/markup/test/make-test.adoc
132+
133+
- title: Invoke the test
134+
content:
135+
- action: execute
136+
file: tutorial-steps/test/invoke-test.sh
137+
render:
138+
file: tutorials/tumbling-windows/flinksql/markup/test/invoke-test.adoc
139+
140+
ccloud:
141+
steps:
142+
- title: Run your app to Confluent Cloud
143+
content:
144+
- action: skip
145+
render:
146+
file: shared/markup/ccloud/try-ccloud.adoc

_data/tutorials.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,7 @@ tumbling-windows:
165165
ksql: enabled
166166
kstreams: enabled
167167
confluent: enabled
168+
flinksql: enabled
168169
hopping-windows:
169170
title: How to create hopping windows
170171
meta-description: create hopping windows for time-series events

_includes/shared/markup/ccloud/try-ccloud-properties.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
Instead of running a local Kafka cluster, you may use https://www.confluent.io/confluent-cloud/tryfree/[Confluent Cloud], a fully-managed Apache Kafka service.
1+
Instead of running a local Kafka cluster, you may use https://www.confluent.io/confluent-cloud/tryfree/[Confluent Cloud], a fully managed Apache Kafka service.
22

3-
1. Sign up for https://www.confluent.io/confluent-cloud/tryfree/[Confluent Cloud], a fully-managed Apache Kafka service.
3+
1. Sign up for https://www.confluent.io/confluent-cloud/tryfree/[Confluent Cloud], a fully managed Apache Kafka service.
44
55
2. After you log in to Confluent Cloud Console, click on `Add cloud environment` and name the environment `learn-kafka`. Using a new environment keeps your learning resources separate from your other Confluent Cloud resources.
66

_includes/shared/markup/ccloud/try-ccloud.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
Instead of running a local Kafka cluster, you may use https://www.confluent.io/confluent-cloud/tryfree/[Confluent Cloud], a fully-managed Apache Kafka service.
1+
Instead of running a local Kafka cluster, you may use https://www.confluent.io/confluent-cloud/tryfree/[Confluent Cloud], a fully managed Apache Kafka service.
22

3-
1. Sign up for https://www.confluent.io/confluent-cloud/tryfree/[Confluent Cloud], a fully-managed Apache Kafka service.
3+
1. Sign up for https://www.confluent.io/confluent-cloud/tryfree/[Confluent Cloud], a fully managed Apache Kafka service.
44
55
2. After you log in to Confluent Cloud Console, click on `Add cloud environment` and name the environment `learn-kafka`. Using a new environment keeps your learning resources separate from your other Confluent Cloud resources.
66

_includes/tutorials/aggregating-count/flinksql/markup/dev/transient-query.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ This should yield the following output:
1414

1515
Enter `Q` to return to the Flink SQL prompt.
1616

17-
Note that these results were materialized in memory and printed in a human readable table representation because the default `sql-client.execution.result-mode` configuration value is `table`. You can view non-materialized streaming results as a changelog by running `SET 'sql-client.execution.result-mode' = 'changelog';`
17+
Note that these results were materialized in memory and printed in a human-readable table representation because the default `sql-client.execution.result-mode` configuration value is `'table'`. You can view non-materialized streaming results as a changelog by running `SET 'sql-client.execution.result-mode' = 'changelog';`
1818
and rerunning the same query. The results will look like this:
1919

2020
+++++

_includes/tutorials/aggregating-count/flinksql/markup/test/create-resource-create-movie-sales.sql.template.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ We could always inline the SQL statements in our Java test code, but creating se
22

33
There are a handful of resources to create for our test. These mirror the queries that we developed earlier.
44

5-
Create the following file at `src/test/resources/create-movie-sales.sql.template`. Note the `KAFKA_PORT` and `SCHEMA_REGISTRY_PORT` placeholders in this file. Out test will dynamically assign these to the ports that Testcontainers assigns.
5+
Create the following file at `src/test/resources/create-movie-sales.sql.template`. Note the `KAFKA_PORT` and `SCHEMA_REGISTRY_PORT` placeholders in this file. Our test will dynamically assign these to the ports that Testcontainers assigns.
66

77
+++++
88
<pre class="snippet"><code class="groovy">{% include_raw tutorials/aggregating-count/flinksql/code/src/test/resources/create-movie-sales.sql.template %}</code></pre>

_includes/tutorials/aggregating-count/flinksql/markup/test/test-architecture.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,11 @@ automated testing are clear, but how do we get there?
55

66
First, what do we want in an automated integration test? For starters:
77

8-
. *Real running services* that our application depends on
8+
. *Real services* (as opposed to mock) that our application depends on
99
. *Small resource footprint* so that developers can run the test locally
1010
. *Low enough latency* so that development iterations aren't hampered -- not as low latency as is required for a unit test, but test duration should be on the order of seconds
1111
. *Isolation* so that many tests can run concurrently on the same machine when this test gets run on a build automation server, e.g., no hardcoded ports
1212

1313
Luckily, tools are at our disposal to solve these problems. We'll use https://www.testcontainers.org/[Testcontainers] to run
1414
containerized Kafka and Schema Registry servers on dynamic ports, Flink's support for local execution environments so that we don't need to spin up a Flink cluster, and Flink's Table API
15-
in order to execute the Flink SQL statements that comprise our application.
15+
to execute the Flink SQL statements that comprise our application.

_includes/tutorials/aggregating-minmax/flinksql/markup/dev/transient-query.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ This should yield the following output:
1414

1515
Enter `Q` to return to the Flink SQL prompt.
1616

17-
Note that these results were materialized in memory and printed in a human readable table representation because the default `sql-client.execution.result-mode` configuration value is `table`. You can view non-materialized streaming results as a changelog by running `SET 'sql-client.execution.result-mode' = 'changelog';`
17+
Note that these results were materialized in memory and printed in a human-readable table representation because the default `sql-client.execution.result-mode` configuration value is `'table'`. You can view non-materialized streaming results as a changelog by running `SET 'sql-client.execution.result-mode' = 'changelog';`
1818
and rerunning the same query. The results will look like this:
1919

2020
+++++

_includes/tutorials/aggregating-minmax/flinksql/markup/test/create-resource-create-movie-sales.sql.template.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ We could always inline the SQL statements in our Java test code, but creating se
22

33
There are a handful of resources to create for our test. These mirror the queries that we developed earlier.
44

5-
Create the following file at `src/test/resources/create-movie-sales.sql.template`. Note the `KAFKA_PORT` and `SCHEMA_REGISTRY_PORT` placeholders in this file. Out test will dynamically assign these to the ports that Testcontainers assigns.
5+
Create the following file at `src/test/resources/create-movie-sales.sql.template`. Note the `KAFKA_PORT` and `SCHEMA_REGISTRY_PORT` placeholders in this file. Our test will dynamically assign these to the ports that Testcontainers assigns.
66

77
+++++
88
<pre class="snippet"><code class="groovy">{% include_raw tutorials/aggregating-minmax/flinksql/code/src/test/resources/create-movie-sales.sql.template %}</code></pre>

0 commit comments

Comments
 (0)