Skip to content

Commit 6926370

Browse files
AdrianVasiliuAdrian Vasiliu
AdrianVasiliu
and
Adrian Vasiliu
authored
HTTP-76 Allow enforcing the use of raw Authorization header (#77)
Introduce configuration parameter allowing to use the Authorization header without b64 and prefixing for Basic Authentication. Signed-off-by: Adrian Vasiliu <adrian.vasiliu3@gmail.com> Co-authored-by: Adrian Vasiliu <vasiliu@fr.ibm.com>
1 parent 4c4d312 commit 6926370

File tree

11 files changed

+176
-46
lines changed

11 files changed

+176
-46
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
/.idea/
22
.idea
33
target
4+
bin
45
/flink.http.connector.iml
56
/src/main/flink-http-connector.iml
67
/src/main/main.iml

‎CHANGELOG.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,14 @@
22

33
## [Unreleased]
44

5+
### Added
6+
7+
- Added support for passing `Authorization` headers for other purposes than Basic Authentication.
8+
New configuration parameter: `gid.connector.http.source.lookup.use-raw-authorization-header`.
9+
If set to `'true'`, the connector uses the raw value set for the `Authorization` header, without
10+
transformation for Basic Authentication (base64, addition of "Basic " prefix).
11+
If not specified, defaults to `'false'`.
12+
513
## [0.11.0] - 2023-11-20
614

715
## [0.10.0] - 2023-07-05

‎README.md

Lines changed: 21 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -385,28 +385,30 @@ In this special case, you can configure connector to trust all certificates with
385385
To enable this option use `gid.connector.http.security.cert.server.allowSelfSigned` property setting its value to `true`.
386386

387387
## Basic Authentication
388-
Connector supports Basic Authentication mechanism using HTTP `Authorization` header.
389-
The header value can set via properties same as other headers. Connector will convert passed value to Base64 and use it for request.
390-
If the used value starts from prefix `Basic `, it will be used as header value as is, without any extra modification.
388+
The connector supports Basic Authentication mechanism using HTTP `Authorization` header.
389+
The header value can be set via properties, similarly as for other headers. The connector converts the passed value to Base64 and uses it for the request.
390+
If the used value starts with the prefix `Basic `, or `gid.connector.http.source.lookup.use-raw-authorization-header`
391+
is set to `'true'`, it will be used as header value as is, without any extra modification.
391392

392393
## Table API Connector Options
393394
### HTTP TableLookup Source
394-
| Option | Required | Description/Value |
395-
|------------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
396-
| connector | required | The Value should be set to _rest-lookup_ |
397-
| format | required | Flink's format name that should be used to decode REST response, Use `json` for a typical REST endpoint. |
398-
| url | required | The base URL that should be use for GET requests. For example _http://localhost:8080/client_ |
399-
| asyncPolling | optional | true/false - determines whether Async Pooling should be used. Mechanism is based on Flink's Async I/O. |
400-
| lookup-method | optional | GET/POST/PUT (and any other) - determines what REST method should be used for lookup REST query. If not specified, `GET` method will be used. |
401-
| gid.connector.http.lookup.error.code | optional | List of HTTP status codes that should be treated as errors by HTTP Source, separated with comma. |
402-
| gid.connector.http.lookup.error.code.exclude | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.lookup.error.code` list, separated with comma. |
403-
| gid.connector.http.security.cert.server | optional | Path to trusted HTTP server certificate that should be add to connectors key store. More than one path can be specified using `,` as path delimiter. |
404-
| gid.connector.http.security.cert.client | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication. |
405-
| gid.connector.http.security.key.client | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication. |
406-
| gid.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |
407-
| gid.connector.http.source.lookup.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. |
408-
| gid.connector.http.source.lookup.request.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 8 threads will be used. |
409-
| gid.connector.http.source.lookup.response.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup response processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 4 threads will be used. |
395+
| Option | Required | Description/Value |
396+
|---------------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
397+
| connector | required | The Value should be set to _rest-lookup_ |
398+
| format | required | Flink's format name that should be used to decode REST response, Use `json` for a typical REST endpoint. |
399+
| url | required | The base URL that should be use for GET requests. For example _http://localhost:8080/client_ |
400+
| asyncPolling | optional | true/false - determines whether Async Pooling should be used. Mechanism is based on Flink's Async I/O. |
401+
| lookup-method | optional | GET/POST/PUT (and any other) - determines what REST method should be used for lookup REST query. If not specified, `GET` method will be used. |
402+
| gid.connector.http.lookup.error.code | optional | List of HTTP status codes that should be treated as errors by HTTP Source, separated with comma. |
403+
| gid.connector.http.lookup.error.code.exclude | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.lookup.error.code` list, separated with comma. |
404+
| gid.connector.http.security.cert.server | optional | Path to trusted HTTP server certificate that should be add to connectors key store. More than one path can be specified using `,` as path delimiter. |
405+
| gid.connector.http.security.cert.client | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication. |
406+
| gid.connector.http.security.key.client | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication. |
407+
| gid.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |
408+
| gid.connector.http.source.lookup.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. |
409+
| gid.connector.http.source.lookup.request.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 8 threads will be used. |
410+
| gid.connector.http.source.lookup.response.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup response processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 4 threads will be used. |
411+
| gid.connector.http.source.lookup.use-raw-authorization-header | optional | If set to `'true'`, uses the raw value set for the `Authorization` header, without transformation for Basic Authentication (base64, addition of "Basic " prefix). If not specified, defaults to `'false'`. |
410412

411413
### HTTP Sink
412414
| Option | Required | Description/Value |

‎src/main/java/com/getindata/connectors/http/internal/BasicAuthHeaderValuePreprocessor.java

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
package com.getindata.connectors.http.internal;
22

33
import java.util.Base64;
4+
import java.util.Objects;
45

56
/**
67
* Header processor for HTTP Basic Authentication mechanism.
@@ -10,20 +11,43 @@ public class BasicAuthHeaderValuePreprocessor implements HeaderValuePreprocessor
1011

1112
public static final String BASIC = "Basic ";
1213

14+
private boolean useRawAuthHeader = false;
15+
16+
/**
17+
* Creates a new instance of BasicAuthHeaderValuePreprocessor that uses
18+
* the default processing of the Authorization header.
19+
*/
20+
public BasicAuthHeaderValuePreprocessor() {
21+
this(false);
22+
}
23+
24+
/**
25+
* Creates a new instance of BasicAuthHeaderValuePreprocessor.
26+
*
27+
* @param useRawAuthHeader If set to true, the Authorization header is kept as-is,
28+
* untransformed. Otherwise, uses the default processing of the
29+
* Authorization header.
30+
*/
31+
public BasicAuthHeaderValuePreprocessor(boolean useRawAuthHeader) {
32+
this.useRawAuthHeader = useRawAuthHeader;
33+
}
34+
1335
/**
1436
* Calculates {@link Base64} value of provided header value. For Basic authentication mechanism,
1537
* the raw value is expected to match user:password pattern.
1638
* <p>
17-
* If rawValue starts with "Basic " prefix it is assumed that this value is already converted to
18-
* expected "Authorization" header value.
39+
* If rawValue starts with "Basic " prefix, or useRawAuthHeader has been set to true, it is
40+
* assumed that this value is already converted to the expected "Authorization" header value.
1941
*
2042
* @param rawValue header original value to modify.
2143
* @return value of "Authorization" header with format "Basic " + Base64 from rawValue or
22-
* rawValue without any changes if it starts with "Basic " prefix.
44+
* rawValue without any changes if it starts with "Basic " prefix or useRawAuthHeader is
45+
* set to true.
2346
*/
2447
@Override
2548
public String preprocessHeaderValue(String rawValue) {
26-
if (rawValue.startsWith(BASIC)) {
49+
Objects.requireNonNull(rawValue);
50+
if (useRawAuthHeader || rawValue.startsWith(BASIC)) {
2751
return rawValue;
2852
} else {
2953
return BASIC + Base64.getEncoder().encodeToString(rawValue.getBytes());

‎src/main/java/com/getindata/connectors/http/internal/config/HttpConnectorConfigConstants.java

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@ public final class HttpConnectorConfigConstants {
2727
public static final String LOOKUP_SOURCE_HEADER_PREFIX = GID_CONNECTOR_HTTP
2828
+ "source.lookup.header.";
2929

30+
/**
31+
* Whether to use the raw value of the Authorization header. If set, it prevents
32+
* the special treatment of the header for Basic Authentication, thus preserving the passed
33+
* raw value. Defaults to false.
34+
*/
35+
public static final String LOOKUP_SOURCE_HEADER_USE_RAW = GID_CONNECTOR_HTTP
36+
+ "source.lookup.use-raw-authorization-header";
37+
3038
// --------- Error code handling configuration ---------
3139
public static final String HTTP_ERROR_SINK_CODE_WHITE_LIST =
3240
GID_CONNECTOR_HTTP + "sink.error.code.exclude";

‎src/main/java/com/getindata/connectors/http/internal/table/lookup/HttpLookupConnectorOptions.java

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import org.apache.flink.configuration.ConfigOption;
44
import org.apache.flink.configuration.ConfigOptions;
55

6+
import static com.getindata.connectors.http.internal.config.HttpConnectorConfigConstants.LOOKUP_SOURCE_HEADER_USE_RAW;
67
import static com.getindata.connectors.http.internal.config.HttpConnectorConfigConstants.SOURCE_LOOKUP_QUERY_CREATOR_IDENTIFIER;
78

89
public class HttpLookupConnectorOptions {
@@ -40,4 +41,10 @@ public class HttpLookupConnectorOptions {
4041
ConfigOptions.key("lookup-request.format")
4142
.stringType()
4243
.defaultValue("json");
44+
45+
public static final ConfigOption<Boolean> USE_RAW_AUTH_HEADER =
46+
ConfigOptions.key(LOOKUP_SOURCE_HEADER_USE_RAW)
47+
.booleanType()
48+
.defaultValue(false)
49+
.withDescription("Whether to use the raw value of Authorization header");
4350
}

‎src/main/java/com/getindata/connectors/http/internal/table/lookup/HttpLookupTableSource.java

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,11 @@ private PollingClientFactory<RowData> createPollingClientFactory(
134134
LookupQueryCreator lookupQueryCreator,
135135
HttpLookupConfig lookupConfig) {
136136

137-
HeaderPreprocessor headerPreprocessor = HttpHeaderUtils.createDefaultHeaderPreprocessor();
137+
boolean useRawAuthHeader =
138+
lookupConfig.getReadableConfig().get(HttpLookupConnectorOptions.USE_RAW_AUTH_HEADER);
139+
140+
HeaderPreprocessor headerPreprocessor =
141+
HttpHeaderUtils.createDefaultHeaderPreprocessor(useRawAuthHeader);
138142
String lookupMethod = lookupConfig.getLookupMethod();
139143

140144
HttpRequestFactory requestFactory = (lookupMethod.equalsIgnoreCase("GET")) ?

‎src/main/java/com/getindata/connectors/http/internal/utils/HttpHeaderUtils.java

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,8 +68,13 @@ public static String[] toHeaderAndValueArray(Map<String, String> headerMap) {
6868
}
6969

7070
public static HeaderPreprocessor createDefaultHeaderPreprocessor() {
71+
return createDefaultHeaderPreprocessor(false);
72+
}
73+
74+
public static HeaderPreprocessor createDefaultHeaderPreprocessor(boolean useRawAuthHeader) {
7175
return new ComposeHeaderPreprocessor(
72-
Collections.singletonMap("Authorization", new BasicAuthHeaderValuePreprocessor())
76+
Collections.singletonMap(
77+
"Authorization", new BasicAuthHeaderValuePreprocessor(useRawAuthHeader))
7378
);
7479
}
7580
}

‎src/test/java/com/getindata/connectors/http/internal/BasicAuthHeaderValuePreprocessorTest.java

Lines changed: 10 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,26 @@
11
package com.getindata.connectors.http.internal;
22

3-
import org.junit.jupiter.api.BeforeEach;
43
import org.junit.jupiter.params.ParameterizedTest;
54
import org.junit.jupiter.params.provider.CsvSource;
65
import static org.assertj.core.api.Assertions.assertThat;
76

87
class BasicAuthHeaderValuePreprocessorTest {
98

10-
private BasicAuthHeaderValuePreprocessor preprocessor;
11-
12-
@BeforeEach
13-
public void setUp() {
14-
this.preprocessor = new BasicAuthHeaderValuePreprocessor();
15-
}
16-
179
@ParameterizedTest
1810
@CsvSource({
19-
"user:password, Basic dXNlcjpwYXNzd29yZA==",
20-
"Basic dXNlcjpwYXNzd29yZA==, Basic dXNlcjpwYXNzd29yZA=="
11+
"user:password, Basic dXNlcjpwYXNzd29yZA==, false",
12+
"Basic dXNlcjpwYXNzd29yZA==, Basic dXNlcjpwYXNzd29yZA==, false",
13+
"abc123, abc123, true",
14+
"Basic dXNlcjpwYXNzd29yZA==, Basic dXNlcjpwYXNzd29yZA==, true",
15+
"Bearer dXNlcjpwYXNzd29yZA==, Bearer dXNlcjpwYXNzd29yZA==, true"
2116
})
2217
public void testAuthorizationHeaderPreprocess(
2318
String headerRawValue,
24-
String expectedHeaderValue) {
25-
String headerValue = this.preprocessor.preprocessHeaderValue(headerRawValue);
19+
String expectedHeaderValue,
20+
boolean useRawAuthHeader) {
21+
BasicAuthHeaderValuePreprocessor preprocessor =
22+
new BasicAuthHeaderValuePreprocessor(useRawAuthHeader);
23+
String headerValue = preprocessor.preprocessHeaderValue(headerRawValue);
2624
assertThat(headerValue).isEqualTo(expectedHeaderValue);
2725
}
2826
}

0 commit comments

Comments
 (0)