Skip to content

docs: Document how to turn off quorum hostname verification #818

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from
Closed

docs: Document how to turn off quorum hostname verification #818

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a342b09
docs: Document how to turn off quorum hostname verification
sbernauer May 23, 2024
dbfeca6
wording
sbernauer May 23, 2024
34b83d2
typo
sbernauer May 23, 2024
bf3dd18
Update docs/modules/zookeeper/pages/usage_guide/configuration_environ…
sbernauer May 23, 2024
74b5381
Move the new Troubleshooting guide
sbernauer May 23, 2024
3406406
wording
sbernauer May 23, 2024
1cb4256
fix nav
sbernauer May 23, 2024
ac89dbf
typo
sbernauer May 23, 2024
ff33759
wording in warning
sbernauer Jun 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/modules/zookeeper/pages/troubleshooting-guide/index.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
= Troubleshooting guide

This section tries to help you in case something isn't working as expected.
29 changes: 29 additions & 0 deletions .../modules/zookeeper/pages/troubleshooting-guide/zookeeper-cluster-unhealthy.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
= Zookeeper cluster unhealthy

== Quorum hostname verification failing

In the past we have noticed problems with mutual TLS in quorums, notably with the hostname verification.
We reported the problems upstream in https://issues.apache.org/jira/browse/ZOOKEEPER-4790[ZOOKEEPER-4790].

The exception looks something like

[source]
----
2024-01-23 07:01:46,432 [myid:] - INFO [ListenerHandler-zk-server-default-0.zka1-zk-server-default.default.svc.cluster.local/100.64.9.69:3888:o.a.z.s.q.QuorumCnxManager$Listener$ListenerHandler@1076] - Received connection request from /100.64.11.99:58368
2024-01-23 07:01:46,446 [myid:] - ERROR [ListenerHandler-zk-server-default-0.zka1-zk-server-default.default.svc.cluster.local/100.64.9.69:3888:o.a.z.c.ZKTrustManager@161] - Failed to verify host address: 100.64.11.99
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <100.64.11.99> doesn't match any of the subject alternative names: [zk-server-default.default.svc.cluster.local, zk-server-default-1.zk-server-default.default.svc.cluster.local, 10.8.XXX.XXX, 10.8.XXX.XXX, 10.8.XXX.XXX, 10.XXX.XXX.XXX, 10.8.XXX.XXX, 10.8.XXX.XXX, 10.8.XXX.XXX, 10.XXX.XXX.XXX]
at org.apache.zookeeper.common.ZKHostnameVerifier.matchIPAddress(ZKHostnameVerifier.java:197)
at org.apache.zookeeper.common.ZKHostnameVerifier.verify(ZKHostnameVerifier.java:165)
----

In case you are running into issues with hostname verification, a workaround - until the problem is fixed - is to turn off hostname verification for the quorum.

[source,yaml]
----
servers:
configOverrides:
zoo.cfg:
ssl.quorum.hostnameVerification: "false"
----

WARNING: This imposes a security risk, so we don't disable the check by default.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very empty statement, what risk does it expose you to?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT of

WARNING: This imposes a security risk, so we don't disable the check by default. Any possessor of a certificate signed by the ca (even for a totally different host) can pretend to be a Zookeeper server to a Zookeeper server.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nightkr can you please have another look?

Copy link
Contributor

@nightkr nightkr Jun 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that covers the issue better, yes, and would be better if we do merge this.

Preliminarily I'd rather just solve #760 than document the workaround, unless we're running out of time for the release.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only raised this PR because it was on the tasklist. Happy to move it to Track for now.
Updated the section anyway

2 changes: 2 additions & 0 deletions docs/modules/zookeeper/partials/nav.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
*** xref:zookeeper:usage_guide/operations/pod-placement.adoc[]
*** xref:zookeeper:usage_guide/operations/pod-disruptions.adoc[]
*** xref:zookeeper:usage_guide/operations/graceful-shutdown.adoc[]
* xref:zookeeper:troubleshooting-guide/index.adoc[]
** xref:zookeeper:troubleshooting-guide/zookeeper-cluster-unhealthy.adoc[]
* xref:zookeeper:reference/index.adoc[]
** xref:zookeeper:reference/crds.adoc[]
*** {crd-docs}/zookeeper.stackable.tech/zookeepercluster/v1alpha1/[ZookeeperCluster {external-link-icon}^]
Expand Down