-
Notifications
You must be signed in to change notification settings - Fork 786
Description
I have a Docker container which, on start-up, renders a configuration file (for a datadog-agent to monitor some data stores). The file is rendered using exec
mode:
/usr/local/bin/consul-template \
-consul-addr="${REAL_CONSUL_ADDR}" \
-template="/etc/datadog-agent/datadog.yaml.ctmpl:/etc/datadog-agent/datadog.yaml" \
-template="/etc/datadog-agent/conf.d/postgres.d/postgres.yaml.ctmpl:/etc/datadog-agent/conf.d/postgres.d/conf.yaml.kms:/bin/bash -c '/usr/local/bin/gruntkms decrypt --aws-region ${REGION} < /etc/datadog-agent/conf.d/postgres.d/conf.yaml.kms > /etc/datadog-agent/conf.d/postgres.d/conf.yaml'" \
-template="/etc/datadog-agent/conf.d/redisdb.d/redisdb.yaml.ctmpl:/etc/datadog-agent/conf.d/redisdb.d/conf.yaml.kms:/bin/bash -c '/usr/local/bin/gruntkms decrypt --aws-region ${REGION} < /etc/datadog-agent/conf.d/redisdb.d/conf.yaml.kms > /etc/datadog-agent/conf.d/redisdb.d/conf.yaml'" \
-consul-retry \
-max-stale=0 \
-exec "/usr/bin/datadog-agent run"
One of the template also invokes a 400+ lines Python script which does all kinds of things (including iterating over all DBs, running CREATE OR REPLACE FUNCTION
statements for each, etc )
{{ $path := ( print "/config/infrastructure/datadog/files/postgres" ) }}
{{ key $path | plugin "updatepgconfig.py" }}
This used to work really well ; but since recently, presumably as we have more and more data-stores, this does not terminate in time and it crashes with
2025-03-05T14:58:53.760Z [ERR] (cli) /etc/datadog-agent/conf.d/postgres.d/postgres.yaml.ctmpl: execute: template: :2:15: executing "" at <plugin "/updatepgconfig.py">: error calling plugin: exec "updatepgconfig.py": did not finish in 30s
This was set to 30s following #818.
Now, I realize that this setup is perhaps abusing consul-template a little bit too much, and that this is simply not a use-case that consul-template should support, and we are considering changing that setup. Meanwhile though, I was wondering whether it would be an option to make that timeout configurable (via environment variable or something)? While no long-term fix, it would solve the immediate issue I am currently having. But as said, I would totally understand if this is declined :)
consul-template version: 0.39.1