Nomad provisions dynamic database (PostgreSQL) credentials, immediately renews them and provides the old credentials to the container #2073
Description
Nomad version
Nomad v1.10.2
Vault v1.19.5
Operating system and Environment details
Arch Linux
Issue
The dynamically provisioned database credentials from vault for PostgreSQL are not correct (password differs). However, provisioning dynamic credentials with the Vault CLI results in working credentials that provide the correct access to PostgreSQL.
Reproduction steps
PostgreSQL container ghcr.io/immich-app/postgres:17-vectorchord0.4.3-pgvector0.8.0-pgvectors0.3.0.
Vault resources
resource "vault_mount" "postgresql" {
path = "postgresql"
type = "database"
description = "PostgreSQL database credentials backend."
}
resource "vault_database_secret_backend_connection" "postgresql" {
backend = vault_mount.postgresql.path
name = "connection-postgresql"
plugin_name = "postgresql-database-plugin"
allowed_roles = ["role-postgresql"]
postgresql {
connection_url = "postgres://{{username}}:{{password}}@postgresql.service.consul:5432/postgres"
username = var.username
password = var.password
password_authentication = "scram-sha-256"
username_template = "v_immich_{{unix_time}}"
}
}
resource "vault_database_secret_backend_role" "postgresql" {
backend = vault_mount.postgresql.path
name = "role-postgresql"
db_name = vault_database_secret_backend_connection.postgresql.name
creation_statements = [
"CREATE ROLE {{username}} WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';",
"GRANT SELECT ON ALL TABLES IN SCHEMA public TO {{username}};",
"GRANT ALL PRIVILEGES ON DATABASE immich TO {{username}};",
]
rollback_statements = [
"DROP ROLE IF EXISTS {{username}};",
]
renew_statements = [
"ALTER ROLE {{username}} WITH PASSWORD '{{password}}';",
]
revocation_statements = [
"REVOKE ALL PRIVILEGES ON DATABASE immich FROM {{username}};",
"DROP ROLE IF EXISTS {{username}};",
]
}
resource "vault_policy" "postgresql" {
name = "policy-postgresql"
policy = <<-EOT
path "${vault_mount.postgresql.path}/creds/${vault_database_secret_backend_role.postgresql.name}" {
capabilities = ["read"]
}
EOT
}When we run the photos workload (job is provided below) we can see that dynamic credentials are provisioned. However, When we look into the the lease, we can see that it has been renewed immediately.
$ curl --header "X-Vault-Token: $VAULT_TOKEN" --request LIST https://vault:8200/v1/sys/leases/lookup/postgresql/creds/role-postgresql/ | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 217 100 217 0 0 7052 0 --:--:-- --:--:-- --:--:-- 8037
{
"request_id": "719abd84-eeb2-6390-873e-61da8df70a9f",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": {
"keys": [
"u8YKcQi6KalPEA07aclHPNyK"
]
},
"wrap_info": null,
"warnings": null,
"auth": null,
"mount_type": "system"
}
$ vault lease lookup postgresql/creds/role-postgresql/u8YKcQi6KalPEA07aclHPNyK
Key Value
expire_time 2025-07-26T10:09:28.686861628Z
id postgresql/creds/role-postgresql/u8YKcQi6KalPEA07aclHPNyK
issue_time 2025-06-24T10:09:28.667139714Z
last_renewal 2025-06-24T10:09:28.686861918Z
renewable true
ttl 767h59m32s
If we manually provision dynamic database credentials from vault with vault read postgresql/creds/role-postgresql and lookup the lease we can see that the credentials have not been renewed.
Key Value
expire_time 2025-07-26T10:53:04.216999787Z
id postgresql/creds/role-postgresql/I6c3pTgTZY6BwUxSaBXmGfIW
issue_time 2025-06-24T10:53:04.216999507Z
last_renewal <nil>
renewable true
ttl 767h59m46s
Expected Result
PostgreSQL authentication success.
Actual Result
PostgreSQL authentication error.
Job file (if appropriate)
job "photos" {
datacenters = ["*"]
type = "service"
group "photos" {
count = 1
network {
mode = "host"
port "web" {
to = 2283
}
}
restart {
attempts = 10
interval = "3h"
delay = "15m"
mode = "delay"
}
task "photos" {
driver = "docker"
config {
image = "ghcr.io/immich-app/immich-server:v1.135.3"
entrypoint = ["tail", "-f", "/dev/null"]
network_mode = "nginx"
ports = []
mount {
type = "volume"
source = "photos"
target = "/usr/src/app/upload"
}
mount {
type = "bind"
source = "/etc/localtime"
target = "/etc/localtime"
readonly = true
}
}
service {
name = "photos"
port = "web"
}
template {
data = <<-EOT
{{ with secret "postgresql/creds/role-postgresql" }}
DB_USERNAME="{{ .Data.username }}"
DB_PASSWORD="{{ .Data.password }}"
{{ end }}
DB_HOSTNAME = "postgresql.service.consul"
REDIS_HOSTNAME = "redis.service.consul"
REDIS_USERNAME="immich"
REDIS_PASSWORD="..."
EOT
destination = "secrets/env"
env = true
}
vault {}
}
}
}Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
Reverse log
[INFO] client.alloc_runner.task_runner: Task event: alloc_id=e65b2dd2-3fe7-8415-c967-aa50a5c5b074 task=photos type=Started msg="Task started by cli>
[INFO] client.driver_mgr.docker: started container: driver=docker container_id=c49c19a100410307ec8218dac07675e0fac444f0bf9aa0ab6a4142375b37f959
[INFO] client.driver_mgr.docker: created container: driver=docker container_id=c49c19a100410307ec8218dac07675e0fac444f0bf9aa0ab6a4142375b37f959
[INFO] agent: (runner) rendered "(dynamic)" => "/storage/data/nomad/alloc/e65b2dd2-3fe7-8415-c967-aa50a5c5b074/photos/secrets/env"
[INFO] agent: (runner) starting
[INFO] agent: (runner) creating watcher
[INFO] agent: (runner) creating new runner (dry: false, once: false)
[INFO] client.alloc_runner.task_runner: Task event: alloc_id=e65b2dd2-3fe7-8415-c967-aa50a5c5b074 task=photos type="Task Setup" msg="Building Task >
[INFO] client.alloc_runner.task_runner: Task event: alloc_id=e65b2dd2-3fe7-8415-c967-aa50a5c5b074 task=photos type=Received msg="Task received by c>
[INFO] client.gc: marking allocation for GC: alloc_id=dd6785d4-96ea-5091-9503-82a28987b475
[INFO] agent: (runner) received finish
[INFO] agent: (runner) stopping