Mounting SecretsManager secret as a file

[al-dpopowich]

I am investigating transitioning our current use of (the now deprecated) compose-ecs to copilot. One feature I'm sorely missing is the ability to declare secrets that will result in a mounted file on the running container. For example, when using compose-ecs, I could have in my docker-compose.yml file:

secrets:
   credentials:
      name: "arn:aws:secretsmanager:us-east-2:0123456789:secret:beta/credentials-YOOdYl"
      external: true
      keys: "*"

   app_config:
      name: "arn:aws:secretsmanager:us-east-2:0123456789:secret:beta/app-config-sODUXp"
      external: true

And in my running container see the following files, the content of each being the value stored in SecretsManager:

# ls -lR /run/secrets/
/run/secrets:
total 4
-rw-r--r-- 1 root root 2345 Feb 23 14:33 app_config
drwxr-xr-x 2 root root 4096 Feb 23 14:31 credentials

/run/secrets/credentials:
total 12
-rw-r--r-- 1 root root 25 Feb 23 14:31 db1
-rw-r--r-- 1 root root 25 Feb 23 14:31 db2
-rw-r--r-- 1 root root 25 Feb 23 14:31 redis

The convenience of this cannot be overstated. Note how keys: "*" exploded each key in the JSON value of credentials to its own file. And for the configuration file, app_config, containing dozens and dozens of items (many containing sensitive data), can be found by may app with one environment variable:

APP_CONFIG=/run/secrets/app_config

In compose-ecs they created a sidecar that would do this mount. I'm investigating if I can grab this image and use it as a sidecar with a copilot application, but meanwhile wondering:

1. Is this something copilot already has and I'm missing it?
2. If not, can something like this be added to copilot? (Maybe ingesting the go code from compose-ecs?)
3. Otherwise, what are best practices for loading whole configuration files (with possibly several sensitive pieces of data) without having to create a secret for each. Private s3 bucket?

[Lou1415926]

@al-dpopowich Thanks for the thoughtful request!

Is this something copilot already has and I'm missing it?

Copilot does not have the equivalent today - you do have to specify each secrets in the manifest file, individually.

However - if this helps at all - you could bulk-create secrets using copilot secret init --cli-input-yaml. Note that though secret init creates secrets in SSM as SecretStrings - these are encrypted as well, though missing some secretsmanager features such as rotation.

If not, can something like this be added to copilot? (Maybe ingesting the go code from compose-ecs?).

This is certainly a feature request for us! I see you've reacted to #3778, that's all you need to do to help us gauge the need.

Otherwise, what are best practices for loading whole configuration files (with possibly several sensitive pieces of data) without having to create a secret for each. Private s3 bucket?

Like mentioned above, copilot secret init --cli-input-yaml might be able to help alleviate some pain for you!

Private s3 bucket is an interesting idea! You can enable server-side encryption for the bucket. Then, you can give your ECS task role the permission to retrieve objects from that S3 bucket using addons. I think this is a viable option. I've also found this reddit discussion which you might be interested in as well - the permission hierarchy is an interesting point to consider!

[al-dpopowich]

Thanks for the reply!

Since I'm transitioning from compose-ecs, I have legacy dependencies and need to keep SecretsManager, so using copilot secret init ... isn't really an option. (Btw, sure would be nice to add an option to use SecretsManager instead of SSM with that command!)

In my current (compose-ecs) entrypoint, I use a passed in environment variable APP_CONFIG which points to, e.g., /run/secrets/app_config.yaml, populated by the sidecar used by compose-ecs. The rest of my application (across several containers) is built around this paradigm: APP_CONFIG points to a yaml file. So, my current solution for copilot, which seems to be working:

• Given that I can, as a minimum, pass APP_CONFIG as an environment variable with the contents of the yaml (as stored in SecretsManager), I updated my entrypoint scripts like this:

# need APP_CONFIG
[ "${APP_CONFIG:?}" ] || {
    echo "APP_CONFIG not set" >&2
    exit 1
}

# convert APP_CONFIG from holding the contents to 
# pointing to a file with the contents
app_config_file="$(mktemp -d)/app_config.yml"
echo "$APP_CONFIG" > "$app_config_file"
APP_CONFIG="$app_config_file"

With this solution, just a few lines in a bash script, I don't have to change my application or secret store.