You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+64-7Lines changed: 64 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -11,11 +11,10 @@ Connector supports only STRING types.
11
11
* Maven 3
12
12
* Flink 14+
13
13
14
-
## Structure and further work
15
-
The main code can be found under `com.getindata.connectors.http.table` package plus additional classes directly from `com.getindata.connectors.http`
16
-
17
-
The `com.getindata.connectors.http.stream` package is pure PoC and currently not meant to be use. The purpose of this package was to test out the new Unified Source design for Flink Source Connector
18
-
[2]. Currently, the implementation is purely PoC, and requires further development.
14
+
## Implementation
15
+
Implementation is based on Flink's `TableFunction` and `AsyncTableFunction` classes.
16
+
To be more specific we are using a `LookupTableSource`. Unfortunately Flink's new unified source interface [2] cannot be used for this type of source.
17
+
Issue was discussed on Flink's user mailing list - https://lists.apache.org/thread/tx2w1m15zt5qnvt924mmbvr7s8rlyjmw
19
18
20
19
## Usage
21
20
Flink SQL table definition:
@@ -67,11 +66,67 @@ For example `'field.isActive.path' = '$.details.isActive'` - the value for table
67
66
| root | optional | Sets the json root node for entire table. The value should be presented as Json Path [3], for example `$.details`.|
68
67
| field.#.path | optional | The Json Path from response model that should be use for given `#` field. If `root` option was defined it will be added to field path. The value must be presented in Json Path format [3], for example `$.details.nestedDetails.balance`|
69
68
69
+
## Build and deployment
70
+
Currently, we are not publishing this artifact to any repository. The CI/CD configuration is also next thing to do.
71
+
To
72
+
73
+
To build the project locally you need to have `maven 3` and Java 11+. </br>
74
+
75
+
Project build command: `mvn package`. </br>
76
+
Detailed test report can be found under `target/site/jacoco/index.xml`.
77
+
78
+
## Demo application
79
+
You can test this connector using simple mock http server provided with this repository and Flink SQL-client.
80
+
The mock server can be started from IDE (currently only this way) by running `HttpStubApp::main` method.
81
+
It will start HTTP server listening on `http://localhost:8080/client`
82
+
83
+
Steps to follow:
84
+
- Run Mock HTTP server from `HttpStubApp::main` method.
85
+
- Start your Flink cluster, for example as described under https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/try-flink/local_installation/
SELECT o.id, o.id2, c.msg, c.uuid, c.isActive, c.balance FROM Orders AS o JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c ON o.id = c.id AND o.id2 = c.id2;
118
+
```
119
+
120
+
As a result, you should see a table with joined records like so:
121
+

122
+
123
+
The `msg` column shows parameters used with REST call for given JOIN record.
124
+
70
125
## TODO
71
126
- Implement caches.
72
127
- Add support for other Flink types. Currently, STRING type is only fully supported.
73
128
- Think about Retry Policy for Http Request
74
-
- Use Flink Format [4] to parse Json response
129
+
- Use Flink Format [5] to parse Json response
75
130
- Add Configurable Timeout value
76
131
- Check other `//TODO`'s.
77
132
@@ -82,4 +137,6 @@ For example `'field.isActive.path' = '$.details.isActive'` - the value for table
0 commit comments