Skip to content

Commit fc7fca7

Browse files
kristoffSCKrzysztof Chmielewskiswtwsk
authored
ESP-117 adding tls and mTLS support for Http Connector (#25)
* 1ESP-117_mTLS_support - Adding mTLS support for Http Source and Sink Signed-off-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com> * 1ESP-117_mTLS_support - Adding mTLS support for Http Source and Sink Signed-off-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com> * 1ESP-117_mTLS_support - Adding mTLS support for Http Source and Sink Signed-off-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com> * 1ESP-117_mTLS_support - Adding mTLS support for Http Source and Sink Signed-off-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com> * 1ESP-117_mTLS_support - Adding mTLS support for Http Source and Sink Signed-off-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com> * 1ESP-117_mTLS_support - Adding mTLS support for Http Source and Sink Signed-off-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com> * ESP-117 changes after code review. Update CHANGELOG.md Co-authored-by: Andrzej Swatowski <33041023+swtwsk@users.noreply.github.com> * 1ESP-117_mTLS_support - Changes after code review. Signed-off-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com> Signed-off-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com> Co-authored-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com> Co-authored-by: Andrzej Swatowski <33041023+swtwsk@users.noreply.github.com>
1 parent b726273 commit fc7fca7

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+1844
-85
lines changed

‎CHANGELOG.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,15 @@
22

33
## [Unreleased]
44

5+
### Added
6+
7+
- Add TLS and mTLS support for Http Sink and Lookup Source connectors.
8+
New properties are:
9+
- `gid.connector.http.security.cert.server` - path to server's certificate.
10+
- `gid.connector.http.security.cert.client` - path to connector's certificate.
11+
- `gid.connector.http.security.key.client` - path to connector's private key.
12+
- `gid.connector.http.security.cert.server.allowSelfSigned` - allowing for self signed certificates without adding them to KeyStore (not recommended for a production).
13+
514
## [0.4.0] - 2022-08-31
615

716
### Added

‎README.md

Lines changed: 49 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -211,33 +211,60 @@ An example of such a mask would be `3XX, 4XX, 5XX`. In this case, all 300s, 400s
211211
Many status codes can be defined in one value, where each code should be separated with comma, for example:
212212
`401, 402, 403`. In this example, codes 401, 402 and 403 would not be interpreted as error codes.
213213

214+
## TLS and mTLS support
215+
Both Http Sink and Lookup Source connectors supports Https communication using TLS 1.2 and mTLS.
216+
To enable Https communication simply use `https` protocol in endpoint's URL.
217+
If certificate used by HTTP server is self-signed, or it is signed byt not globally recognize CA
218+
you would have to add this certificate to connector's keystore as trusted certificate.
219+
In order to do so, use `gid.connector.http.security.cert.server` connector property,
220+
which value is a path to the certificate. You can also use your organization's CA Root certificate.
221+
You can specify many certificate, separating each path with `,`.
222+
223+
You can also configure connector to use mTLS. For this simply use `gid.connector.http.security.cert.client`
224+
and `gid.connector.http.security.key.client` connector properties to specify path to certificate and
225+
private key that should be used by connector. Key MUST be in `PCKS8` format. Both PEM and DER keys are
226+
allowed.
227+
228+
All properties can be set via Sink's builder `.setProperty(...)` method or through Sink and Source table DDL.
229+
230+
For non production environments it is sometimes necessary to use Https connection and accept all certificates.
231+
In this special case, you can configure connector to trust all certificates without adding them to keystore.
232+
To enable this option use `gid.connector.http.security.cert.server.allowSelfSigned` property setting its value to `true`.
214233

215234
## Table API Connector Options
216235
### HTTP TableLookup Source
217-
| Option | Required | Description/Value |
218-
|----------------------------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------|
219-
| connector | required | The Value should be set to _rest-lookup_ |
220-
| format | required | Flink's format name that should be used to decode REST response, Use `json` for a typical REST endpoint. |
221-
| url | required | The base URL that should be use for GET requests. For example _http://localhost:8080/client_ |
222-
| asyncPolling | optional | true/false - determines whether Async Pooling should be used. Mechanism is based on Flink's Async I/O. |
223-
| gid.connector.http.lookup.error.code | optional | List of HTTP status codes that should be treated as errors by HTTP Source, separated with comma. |
224-
| gid.connector.http.lookup.error.code.exclude | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.lookup.error.code` list, separated with comma. |
236+
| Option | Required | Description/Value |
237+
|---------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------|
238+
| connector | required | The Value should be set to _rest-lookup_ |
239+
| format | required | Flink's format name that should be used to decode REST response, Use `json` for a typical REST endpoint. |
240+
| url | required | The base URL that should be use for GET requests. For example _http://localhost:8080/client_ |
241+
| asyncPolling | optional | true/false - determines whether Async Pooling should be used. Mechanism is based on Flink's Async I/O. |
242+
| gid.connector.http.lookup.error.code | optional | List of HTTP status codes that should be treated as errors by HTTP Source, separated with comma. |
243+
| gid.connector.http.lookup.error.code.exclude | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.lookup.error.code` list, separated with comma. |
244+
| gid.connector.http.security.cert.server | optional | Path to trusted HTTP server certificate that should be add to connectors key store. More than one path can be specified using `,` as path delimiter. |
245+
| gid.connector.http.security.cert.client | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication. |
246+
| gid.connector.http.security.key.client | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication. |
247+
| gid.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |
225248

226249
### HTTP Sink
227-
| Option | Required | Description/Value |
228-
|--------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
229-
| connector | required | Specify what connector to use. For HTTP Sink it should be set to _'http-sink'_. |
230-
| url | required | The base URL that should be use for HTTP requests. For example _http://localhost:8080/client_. |
231-
| format | required | Specify what format to use. |
232-
| insert-method | optional | Specify which HTTP method to use in the request. The value should be set either to `POST` or `PUT`. |
233-
| sink.batch.max-size | optional | Maximum number of elements that may be passed in a batch to be written downstream. |
234-
| sink.requests.max-inflight | optional | The maximum number of in flight requests that may exist, if any more in flight requests need to be initiated once the maximum has been reached, then it will be blocked until some have completed. |
235-
| sink.requests.max-buffered | optional | Maximum number of buffered records before applying backpressure. |
236-
| sink.flush-buffer.size | optional | The maximum size of a batch of entries that may be sent to the HTTP endpoint measured in bytes. |
237-
| sink.flush-buffer.timeout | optional | Threshold time in milliseconds for an element to be in a buffer before being flushed. |
238-
| gid.connector.http.sink.request-callback | optional | Specify which `HttpPostRequestCallback` implementation to use. By default, it is set to `slf4j-logger` corresponding to `Slf4jHttpPostRequestCallback`. |
239-
| gid.connector.http.sink.error.code | optional | List of HTTP status codes that should be treated as errors by HTTP Sink, separated with comma. |
240-
| gid.connector.http.sink.error.code.exclude | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.sink.error.code` list, separated with comma. |
250+
| Option | Required | Description/Value |
251+
|---------------------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
252+
| connector | required | Specify what connector to use. For HTTP Sink it should be set to _'http-sink'_. |
253+
| url | required | The base URL that should be use for HTTP requests. For example _http://localhost:8080/client_. |
254+
| format | required | Specify what format to use. |
255+
| insert-method | optional | Specify which HTTP method to use in the request. The value should be set either to `POST` or `PUT`. |
256+
| sink.batch.max-size | optional | Maximum number of elements that may be passed in a batch to be written downstream. |
257+
| sink.requests.max-inflight | optional | The maximum number of in flight requests that may exist, if any more in flight requests need to be initiated once the maximum has been reached, then it will be blocked until some have completed. |
258+
| sink.requests.max-buffered | optional | Maximum number of buffered records before applying backpressure. |
259+
| sink.flush-buffer.size | optional | The maximum size of a batch of entries that may be sent to the HTTP endpoint measured in bytes. |
260+
| sink.flush-buffer.timeout | optional | Threshold time in milliseconds for an element to be in a buffer before being flushed. |
261+
| gid.connector.http.sink.request-callback | optional | Specify which `HttpPostRequestCallback` implementation to use. By default, it is set to `slf4j-logger` corresponding to `Slf4jHttpPostRequestCallback`. |
262+
| gid.connector.http.sink.error.code | optional | List of HTTP status codes that should be treated as errors by HTTP Sink, separated with comma. |
263+
| gid.connector.http.sink.error.code.exclude | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.sink.error.code` list, separated with comma. |
264+
| gid.connector.http.security.cert.server | optional | Path to trusted HTTP server certificate that should be add to connectors key store. More than one path can be specified using `,` as path delimiter. |
265+
| gid.connector.http.security.cert.client | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication. |
266+
| gid.connector.http.security.key.client | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication. |
267+
| gid.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |
241268

242269
## Build and deployment
243270
To build the project locally you need to have `maven 3` and Java 11+. </br>

‎pom.xml

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -320,12 +320,9 @@ under the License.
320320
<version>${jacoco.plugin.version}</version>
321321
<configuration>
322322
<excludes>
323-
<exclude>**/*Mock*.*</exclude>
324323
<exclude>**/HttpLookupConnectorOptions.class</exclude>
325-
<exclude>**/HttpLookupConnectorOptionsUtil.class</exclude>
326-
<exclude>**/HttpTableSourceFactoryHelper.class</exclude>
327-
<exclude>**/HttpPollTableSource.class</exclude>
328324
<exclude>**/Slf4jHttpPostRequestCallback.class</exclude>
325+
<exclude>**/SelfSignedTrustManager.class</exclude>
329326
</excludes>
330327
</configuration>
331328
<executions>

‎src/main/java/com/getindata/connectors/http/internal/SinkHttpClientResponse.java

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
import lombok.Data;
66
import lombok.NonNull;
7+
import lombok.ToString;
78

89
import com.getindata.connectors.http.internal.sink.HttpSinkRequestEntry;
910

@@ -12,6 +13,7 @@
1213
* to write, divided into two lists &mdash; successful and failed ones.
1314
*/
1415
@Data
16+
@ToString
1517
public class SinkHttpClientResponse {
1618

1719
/**

‎src/main/java/com/getindata/connectors/http/internal/config/HttpConnectorConfigConstants.java

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
@NoArgsConstructor(access = AccessLevel.NONE)
1212
public final class HttpConnectorConfigConstants {
1313

14-
public static final String ERROR_CODE_DELIM = ",";
14+
public static final String PROP_DELIM = ",";
1515

1616
/**
1717
* A property prefix for http connector.
@@ -41,4 +41,25 @@ public final class HttpConnectorConfigConstants {
4141

4242
public static final String SINK_REQUEST_CALLBACK_IDENTIFIER =
4343
GID_CONNECTOR_HTTP + "sink.request-callback";
44+
45+
// -------------- HTTPS security settings --------------
46+
public static final String ALLOW_SELF_SIGNED =
47+
GID_CONNECTOR_HTTP + "security.cert.server.allowSelfSigned";
48+
49+
public static final String SERVER_TRUSTED_CERT = GID_CONNECTOR_HTTP + "security.cert.server";
50+
51+
public static final String CLIENT_CERT = GID_CONNECTOR_HTTP + "security.cert.client";
52+
53+
public static final String CLIENT_PRIVATE_KEY = GID_CONNECTOR_HTTP + "security.key.client";
54+
55+
public static final String KEY_STORE_PATH = GID_CONNECTOR_HTTP
56+
+ "security.keystore.path";
57+
58+
public static final String KEY_STORE_PASSWORD = GID_CONNECTOR_HTTP
59+
+ "security.keystore.password";
60+
61+
public static final String KEY_STORE_TYPE = GID_CONNECTOR_HTTP
62+
+ "security.keystore.type";
63+
64+
// -----------------------------------------------------
4465
}

0 commit comments

Comments
 (0)