Skip to content

[Bug]: Most of the time certificates can't be issued: failure saving error to acme challenge: error saving acme challenge; changed since last read #2121

@LokiMidgard

Description

@LokiMidgard

Steps to Reproduce

I was able to issue a valid certificate once, but normally it fails.

It happens on a brand new docker setup. For reproduction I pushed my reproduction in this repo

deleting all containers and volums dose not fix it…

step-ca  | time="2024-12-24T14:40:45Z" level=error duration=3.770407848s duration-ns=3770407848 error="error validating challenge: failure saving error to acme challenge: error saving acme challenge: failed to commit badger transaction: Transaction Conflict. Please retry" fields.time="2024-12-24T14:40:41Z" method=POST name=ca nonce=dFlaWGtYVHRTSzJmT2FNYjNDVXBpaXE0QkJkN25zWmc path=/acme/acme/challenge/uQbnnnQDj99Epf4SOjC3h1Jhg9elqYYS/XumTdhu9LvKPIzFtmNC1oQ7oKaygZOmJ protocol=HTTP/2.0 referer= remote-address=172.18.0.3 request-id=47d800c0-e3fc-42b8-8c58-41627a615504 response="{\"type\":\"urn:ietf:params:acme:error:serverInternal\",\"detail\":\"The server experienced an internal error\"}" size=105 status=500 user-agent="nginx-proxy/acme-companion/v2.5.1-6-gea11f22 (acme.sh/3.1.0)" user-id=

this is the complete log:

step-ca  | 2024/12/24 14:39:58 Starting Smallstep CA/0.28.1 (linux/arm64)
step-ca  | 2024/12/24 14:39:58 Documentation: https://u.step.sm/docs/ca
step-ca  | 2024/12/24 14:39:58 Community Discord: https://u.step.sm/discord
step-ca  | 2024/12/24 14:39:58 Config file: /home/step/config/ca.json
step-ca  | 2024/12/24 14:39:58 The primary server URL is https://step.home:9000
step-ca  | 2024/12/24 14:39:58 Root certificates are available at https://step.home:9000/roots.pem
step-ca  | 2024/12/24 14:39:58 X.509 Root Fingerprint: 1c91fb8a845588d6fc121309467f2e6b4efed4ca420c927594d1cb95b7fdbf20
step-ca  | 2024/12/24 14:39:58 Serving HTTPS on :9000 ...
step-ca  | time="2024-12-24T14:40:05Z" level=info duration="722.498<C2><B5>s" duration-ns=722498 fields.time="2024-12-24T14:40:05Z" method=GET name=ca path=/acme/acme/directory protocol=HTTP/2.0 referer= remote-address=172.18.0.3 request-id=be49e6d1-cf61-410f-8a90-a7050797c125 response="{\"newNonce\":\"https://step.home:9000/acme/acme/new-nonce\",\"newAccount\":\"https://step.home:9000/acme/acme/new-account\",\"newOrder\":\"https://step.home:9000/acme/acme/new-order\",\"revokeCert\":\"https://step.home:9000/acme/acme/revoke-cert\",\"keyChange\":\"https://step.home:9000/acme/acme/key-change\"}" size=292 status=200 user-agent="nginx-proxy/acme-companion/v2.5.1-6-gea11f22 (acme.sh/3.1.0)" user-id=
step-ca  | time="2024-12-24T14:40:06Z" level=info duration=20.662606ms duration-ns=20662606 fields.time="2024-12-24T14:40:06Z" method=HEAD name=ca nonce=T1RpbWJSM055d1FCSFp5dXdlNVd6V1Iza1NiTGc4ZzY path=/acme/acme/new-nonce protocol=HTTP/2.0 referer= remote-address=172.18.0.3 request-id=e982e951-e8e2-4b5f-be0e-266e07bcf3d9 size=0 status=200 user-agent="nginx-proxy/acme-companion/v2.5.1-6-gea11f22 (acme.sh/3.1.0)" user-id=
step-ca  | time="2024-12-24T14:40:06Z" level=info duration=42.568646ms duration-ns=42568646 fields.time="2024-12-24T14:40:06Z" method=POST name=ca nonce=N0M3REc5VWlQTDlHZzVSTU9PZVZwRHd5enNhaURQTVE path=/acme/acme/new-account protocol=HTTP/2.0 referer= remote-address=172.18.0.3 request-id=50310159-5a12-4354-b8c0-cc8739ea9f0e response="{\"contact\":[\"mailto:maile@yourdomain.tld\"],\"status\":\"valid\",\"orders\":\"https://step.home:9000/acme/acme/account/wCm5LMhgldgqNtQDivQoiaUVHPPuhOSh/orders\"}" size=153 status=201 user-agent="nginx-proxy/acme-companion/v2.5.1-6-gea11f22 (acme.sh/3.1.0)" user-id=
step-ca  | time="2024-12-24T14:40:08Z" level=info duration="308.645<C2><B5>s" duration-ns=308645 fields.time="2024-12-24T14:40:08Z" method=GET name=ca path=/acme/acme/directory protocol=HTTP/2.0 referer= remote-address=172.18.0.3 request-id=1f49b232-2af7-4692-b9b8-9b6e0f7a33e7 response="{\"newNonce\":\"https://step.home:9000/acme/acme/new-nonce\",\"newAccount\":\"https://step.home:9000/acme/acme/new-account\",\"newOrder\":\"https://step.home:9000/acme/acme/new-order\",\"revokeCert\":\"https://step.home:9000/acme/acme/revoke-cert\",\"keyChange\":\"https://step.home:9000/acme/acme/key-change\"}" size=292 status=200 user-agent="nginx-proxy/acme-companion/v2.5.1-6-gea11f22 (acme.sh/3.1.0)" user-id=
step-ca  | time="2024-12-24T14:40:22Z" level=info duration=15.754285ms duration-ns=15754285 fields.time="2024-12-24T14:40:22Z" method=HEAD name=ca nonce=SDFLZEU2bmMyVEl0a1BEVElnVFkzT3JPZ0EyZ2ZsZ0s path=/acme/acme/new-nonce protocol=HTTP/2.0 referer= remote-address=172.18.0.3 request-id=11588208-c625-45a4-b756-c6574a3a3e71 size=0 status=200 user-agent="nginx-proxy/acme-companion/v2.5.1-6-gea11f22 (acme.sh/3.1.0)" user-id=
step-ca  | time="2024-12-24T14:40:23Z" level=info duration=86.250781ms duration-ns=86250781 fields.time="2024-12-24T14:40:23Z" method=POST name=ca nonce=MzBtQ2NYaFhSWHZlUEpXM1dvWmd0b2VuY0M1MjBrZjg path=/acme/acme/new-order protocol=HTTP/2.0 referer= remote-address=172.18.0.3 request-id=3fcf3302-9ad2-46dc-aa6a-be5bb7e2f46a response="{\"id\":\"6jwwNAaSZMk36FHGWiBH2PtgsPUdjxQ6\",\"status\":\"pending\",\"expires\":\"2024-12-25T14:40:23Z\",\"identifiers\":[{\"type\":\"dns\",\"value\":\"pi.hole\"}],\"notBefore\":\"2024-12-24T14:39:23Z\",\"notAfter\":\"2025-03-04T14:40:23Z\",\"authorizations\":[\"https://step.home:9000/acme/acme/authz/uQbnnnQDj99Epf4SOjC3h1Jhg9elqYYS\"],\"finalize\":\"https://step.home:9000/acme/acme/order/6jwwNAaSZMk36FHGWiBH2PtgsPUdjxQ6/finalize\"}" size=399 status=201 user-agent="nginx-proxy/acme-companion/v2.5.1-6-gea11f22 (acme.sh/3.1.0)" user-id=
step-ca  | time="2024-12-24T14:40:24Z" level=info duration=23.527287ms duration-ns=23527287 fields.time="2024-12-24T14:40:24Z" method=POST name=ca nonce=UDduZ3VKOUtwQlQ0WW5OUjY3bXRUTVNCZjd4cFZQRGY path=/acme/acme/authz/uQbnnnQDj99Epf4SOjC3h1Jhg9elqYYS protocol=HTTP/2.0 referer= remote-address=172.18.0.3 request-id=c21b2263-5db3-4345-9e2e-52ea093f02be response="{\"identifier\":{\"type\":\"dns\",\"value\":\"pi.hole\"},\"status\":\"pending\",\"challenges\":[{\"type\":\"dns-01\",\"status\":\"pending\",\"token\":\"gbrQ241GMPwWB1XKYELgvqoQcg8MKI7b\",\"url\":\"https://step.home:9000/acme/acme/challenge/uQbnnnQDj99Epf4SOjC3h1Jhg9elqYYS/hUqiMu5Uw83NEOobz9HlcYIXcnZYXDEJ\"},{\"type\":\"http-01\",\"status\":\"pending\",\"token\":\"gbrQ241GMPwWB1XKYELgvqoQcg8MKI7b\",\"url\":\"https://step.home:9000/acme/acme/challenge/uQbnnnQDj99Epf4SOjC3h1Jhg9elqYYS/XumTdhu9LvKPIzFtmNC1oQ7oKaygZOmJ\"},{\"type\":\"tls-alpn-01\",\"status\":\"pending\",\"token\":\"gbrQ241GMPwWB1XKYELgvqoQcg8MKI7b\",\"url\":\"https://step.home:9000/acme/acme/challenge/uQbnnnQDj99Epf4SOjC3h1Jhg9elqYYS/5kaR3qCN3xPaEcXdHIjLZZh7wKvaBDA1\"}],\"wildcard\":false,\"expires\":\"2024-12-25T14:40:23Z\"}" size=729 status=200 user-agent="nginx-proxy/acme-companion/v2.5.1-6-gea11f22 (acme.sh/3.1.0)" user-id=
step-ca  | time="2024-12-24T14:40:27Z" level=info duration="231.197<C2><B5>s" duration-ns=231197 fields.time="2024-12-24T14:40:27Z" method=GET name=ca path=/health protocol=HTTP/2.0 referer= remote-address=172.18.0.2 request-id=9e5403d1-35c1-4f55-9004-fb18e493dbbf size=16 status=200 user-agent="Smallstep CLI/0.28.2 (linux/arm64)" user-id=
step-ca  | time="2024-12-24T14:40:41Z" level=info duration=18.811413ms duration-ns=18811413 fields.time="2024-12-24T14:40:41Z" method=HEAD name=ca nonce=TjRndEpHTGg3RmFldWZBUXlPenQwMHlIb0p6S1hxQVM path=/acme/acme/new-nonce protocol=HTTP/2.0 referer= remote-address=172.18.0.3 request-id=705e1f38-30dd-4620-a6a0-30776e53de91 size=0 status=200 user-agent="nginx-proxy/acme-companion/v2.5.1-6-gea11f22 (acme.sh/3.1.0)" user-id=
step-ca  | time="2024-12-24T14:40:45Z" level=error duration=3.770407848s duration-ns=3770407848 error="error validating challenge: failure saving error to acme challenge: error saving acme challenge: failed to commit badger transaction: Transaction Conflict. Please retry" fields.time="2024-12-24T14:40:41Z" method=POST name=ca nonce=dFlaWGtYVHRTSzJmT2FNYjNDVXBpaXE0QkJkN25zWmc path=/acme/acme/challenge/uQbnnnQDj99Epf4SOjC3h1Jhg9elqYYS/XumTdhu9LvKPIzFtmNC1oQ7oKaygZOmJ protocol=HTTP/2.0 referer= remote-address=172.18.0.3 request-id=47d800c0-e3fc-42b8-8c58-41627a615504 response="{\"type\":\"urn:ietf:params:acme:error:serverInternal\",\"detail\":\"The server experienced an internal error\"}" size=105 status=500 user-agent="nginx-proxy/acme-companion/v2.5.1-6-gea11f22 (acme.sh/3.1.0)" user-id=
step-ca  | time="2024-12-24T14:40:45Z" level=info duration=20.064366776s duration-ns=20064366776 fields.time="2024-12-24T14:40:25Z" method=POST name=ca nonce=V3RZSGY2aWplQWtGOFEyQjVBS3BOd1ZiR0NGaGV4T0Y path=/acme/acme/challenge/uQbnnnQDj99Epf4SOjC3h1Jhg9elqYYS/XumTdhu9LvKPIzFtmNC1oQ7oKaygZOmJ protocol=HTTP/2.0 referer= remote-address=172.18.0.3 request-id=14f5f4f5-965f-4c89-9a11-60e4eeda5b2e response="{\"type\":\"http-01\",\"status\":\"pending\",\"token\":\"gbrQ241GMPwWB1XKYELgvqoQcg8MKI7b\",\"url\":\"https://step.home:9000/acme/acme/challenge/uQbnnnQDj99Epf4SOjC3h1Jhg9elqYYS/XumTdhu9LvKPIzFtmNC1oQ7oKaygZOmJ\",\"error\":{\"type\":\"urn:ietf:params:acme:error:connection\",\"detail\":\"The server could not connect to validation target\"}}" size=316 status=200 user-agent="nginx-proxy/acme-companion/v2.5.1-6-gea11f22 (acme.sh/3.1.0)" user-id=
step-ca  | time="2024-12-24T14:40:58Z" level=info duration="58.802<C2><B5>s" duration-ns=58802 fields.time="2024-12-24T14:40:58Z" method=GET name=ca path=/health protocol=HTTP/2.0 referer= remote-address=172.18.0.2 request-id=0efcb3f4-0a85-4fd8-b10b-44a53395ad69 size=16 status=200 user-agent="Smallstep CLI/0.28.2 (linux/arm64)" user-id=

Your Environment

  • OS : Debian GNU/Linux 12 (bookworm)
  • step-ca Version - smallstep/step-ca:latest (0.28.1 if I see that correctly)

Expected Behavior

issue certificate

Actual Behavior

no certificate issued

Additional Context

the nginx (acme-companion) outputs

[Tue Dec 24 15:02:42 UTC 2024] Using CA: https://step.home:9000/acme/acme/directory
[Tue Dec 24 15:02:43 UTC 2024] Using pre-generated key: /etc/acme.sh/maile@yourdomain.tld/pi.hole/pi.hole.key.next
[Tue Dec 24 15:02:43 UTC 2024] Generating next pre-generate key.
[Tue Dec 24 15:02:49 UTC 2024] Single domain='pi.hole'
[Tue Dec 24 15:02:52 UTC 2024] Getting webroot for domain='pi.hole'
[Tue Dec 24 15:02:52 UTC 2024] Verifying: pi.hole
[Tue Dec 24 15:03:08 UTC 2024] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 92

I set STEPDEBUG=1 in the docker compose file, but it seems not to print more infos.

the pi.hole is actually my dns server, but since it exposes port 53 directly I have no problems resolving dns names in the host machine. howerver nslookup can't resolve inside the step-ca container even when providing the ip of the resolver (host) is this expected? ping works, so the ip is reachable.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

Metadata

Metadata

Assignees

Labels

bugneeds triageWaiting for discussion / prioritization by team

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions