TLS support for auxiliary transports

[finnegancarroll]

Description

In core auxiliary transports may implement secure variants . Secure aux transports expect to consume a SecureAuxTransportSettingsProvider as provided by the security plugin. SecureAuxTransportSettingsProvider delivers an SSLContext to the transport, enabling TLS on client/server connections.

Currently the is no way to configure TLS for an auxiliary transport in the security plugin and the provided SecureAuxTransportSettingsProvider is always empty. This PR introduces settings which allow users to configure each aux transport to provide a valid SecureAuxTransportSettingsProvider.

Configuration settings for each aux transport exist under distinct namespaces, where the identifier for the transport is the same as the key for enabling it under setting aux.transport.types.

plugins.security.ssl.aux.arrow.<setting option...>
plugins.security.ssl.aux.grpc.<setting option...>
plugins.security.ssl.aux.other.<setting option...>

Note that these changes provide no authorization component and only work to deliver an SSL configuration. Users can configure clientauth_mode to allow/disallow certificates access to a particular transport, but no permissions are enforced and no user is associated with a successfully authenticated client.

Testing

These changes add mock aux transports to unit tests for SslContextHandler.
However security plugin is agnostic as to the existence of other plugins available to OpenSearch.
Since auxiliary transports are implemented as plugins validating these changes was done manually by installing security plugin and the secure gRPC transport on a single node cluster.

Env variables for convenience:

export PLAT=darwin
export SNAP_VERSION=3.2.0
export REPOS_DIR=<path to clone OS and security feature branches>

export OS_REPO=${REPOS_DIR}OpenSearch
export SEC_REPO=${REPOS_DIR}security
export SEC_TAR=${SEC_REPO}/build/distributions/opensearch-security-${SNAP_VERSION}.0-SNAPSHOT.zip
export GRPC_TAR=${OS_REPO}/plugins/transport-grpc/build/distributions/transport-grpc-${SNAP_VERSION}-SNAPSHOT.zip
export OS_INSTALL=${OS_REPO}/distribution/archives/${PLAT}-tar/build/install/opensearch-${SNAP_VERSION}-SNAPSHOT

Build OpenSearch/security plugin/gRPC plugin:

cd ${OS_REPO} && ./gradlew :distribution:archives:${PLAT}-tar:assemble 
cd ${OS_REPO} && ./gradlew plugins:transport-grpc:assemble
cd ${OS_REPO} && ./gradlew publishToMavenLocal
cd ${SEC_REPO} && ./gradlew :assemble

Install both plugins:

${OS_INSTALL}/bin/opensearch-plugin install file://${GRPC_TAR}
${OS_INSTALL}/bin/opensearch-plugin install file://${SEC_TAR}

Create demo certs for security plugin - Respond yes yes no (keeping cluster mode disabled):

export OPENSEARCH_INITIAL_ADMIN_PASSWORD=PrTestPass369char! && \
chmod +x "${OS_INSTALL}/plugins/opensearch-security/tools/install_demo_configuration.sh" && \
"${OS_INSTALL}/plugins/opensearch-security/tools/install_demo_configuration.sh"

Enable the secure gRPC auxiliary transport:

echo "aux.transport.types: experimental-secure-transport-grpc" >> ${OS_INSTALL}/config/opensearch.yml
echo "aux.transport.experimental-secure-transport-grpc.port: '9400-9500'" >> ${OS_INSTALL}/config/opensearch.yml

Configure security settings with demo certs (.pem format):
Note: This example configures clientauth_mode: REQUIRE.

echo "plugins.security.ssl.aux.experimental-secure-transport-grpc.enabled: true" >> ${OS_INSTALL}/config/opensearch.yml
echo "plugins.security.ssl.aux.experimental-secure-transport-grpc.pemcert_filepath: esnode.pem" >> ${OS_INSTALL}/config/opensearch.yml
echo "plugins.security.ssl.aux.experimental-secure-transport-grpc.pemkey_filepath: esnode-key.pem" >> ${OS_INSTALL}/config/opensearch.yml
echo "plugins.security.ssl.aux.experimental-secure-transport-grpc.clientauth_mode: REQUIRE" >> ${OS_INSTALL}/config/opensearch.yml
echo "plugins.security.ssl.aux.experimental-secure-transport-grpc.pemtrustedcas_filepath: root-ca.pem" >> ${OS_INSTALL}/config/opensearch.yml

Start OpenSearch:

${OS_INSTALL}/bin/opensearch

Test plaintext connection:

grpcurl -plaintext localhost:9400 list

Test insecure TLS with no client certificate provided:

grpcurl -insecure localhost:9400 list

Test TLS with client certificate provided:
Note: -insecure to skip host name verification.

grpcurl -insecure -cert ${OS_INSTALL}/config/esnode.pem -key ${OS_INSTALL}/config/esnode-key.pem localhost:9400 list

Issues Resolved

#17795

Testing

Unit tests and integration tests for mock auxiliary transport CertTypes.

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 86.51163% with 29 lines in your changes missing coverage. Please review.

Project coverage is 72.82%. Comparing base (3121d88) to head (0297368).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
...nsearch/security/ssl/ExternalSecurityKeyStore.java	0.00%	10 Missing ⚠️
...rg/opensearch/security/ssl/SslSettingsManager.java	90.12%	3 Missing and 5 partials ⚠️
...a/org/opensearch/security/ssl/config/CertType.java	90.24%	2 Missing and 2 partials ⚠️
...lic/rest/api/ssl/CertificatesInfoNodesRequest.java	66.66%	1 Missing and 1 partial ⚠️
...org/opensearch/security/ssl/SslContextHandler.java	33.33%	1 Missing and 1 partial ⚠️
...ensearch/security/ssl/util/SSLConfigConstants.java	93.10%	0 Missing and 2 partials ⚠️
.../opensearch/security/ssl/config/SslParameters.java	92.30%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5375      +/-   ##
==========================================
+ Coverage   72.72%   72.82%   +0.10%     
==========================================
  Files         397      397              
  Lines       24589    24684      +95     
  Branches     3740     3744       +4     
==========================================
+ Hits        17882    17977      +95     
+ Misses       4881     4879       -2     
- Partials     1826     1828       +2     

Files with missing lines	Coverage Δ
...h/security/dlic/rest/api/ssl/CertificatesInfo.java	100.00% <100.00%> (ø)
.../api/ssl/TransportCertificatesInfoNodesAction.java	95.34% <100.00%> (-0.95%)	⬇️
...ensearch/security/ssl/DefaultSecurityKeyStore.java	30.53% <100.00%> (ø)
.../security/ssl/OpenSearchSecureSettingsFactory.java	55.38% <100.00%> (+7.17%)	⬆️
...arch/security/ssl/OpenSearchSecuritySSLPlugin.java	87.50% <100.00%> (+0.10%)	⬆️
.../opensearch/security/ssl/config/SslParameters.java	81.48% <92.30%> (+4.55%)	⬆️
...lic/rest/api/ssl/CertificatesInfoNodesRequest.java	73.68% <66.66%> (ø)
...org/opensearch/security/ssl/SslContextHandler.java	96.80% <33.33%> (-2.10%)	⬇️
...ensearch/security/ssl/util/SSLConfigConstants.java	93.93% <93.10%> (+16.16%)	⬆️
...a/org/opensearch/security/ssl/config/CertType.java	90.47% <90.24%> (-9.53%)	⬇️
... and 2 more

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[finnegancarroll]

@cwperks can you take a look when convenient?