Good way to combine optional env variables?

[Ohad31415]

I'd like to have a configuration where parameters may come from:

• file
• env
• cmd

In that order of importance.
First I thought I'd do a defaults.yaml with values like:

db:
  user: ${env:DB_USER}

But then I have a problem when merging sources how not take an env variable if not exist, doing something like this -

cfg = OmegaConf.merge(file_cfg, default_cfg, OmegaConf.from_cli())

I may end up with unresolved interpolations.

[omry]

You can provide a default value in resolver:

# corresponds to your config file
cfg1 = {
  "db": {
    "user": "${env:DB_USER, root}"
  }
}
cfg = OmegaConf.merge(cfg1, OmegaConf.from_cli())

[Ohad31415]

@Jasha10 By mix and match I meant that some parameters may have their values provided from env and those values should have precedence but some may not. The key is that env should have precedence over file but I don't want it to raise an interpolation resolution error since that value may exist in file or cmd.

🤔 Is there a way to do resolve only for "resolvable" interpolations (in this case ${env:...}) and clear unresolved ones so that I could do the merging in the right order OmegaConf.merge(file_cfg, default_cfg, OmegaConf.from_cli())?

[Jasha10]

I think that either of the approaches suggested above will work for this. Either write user: ${env:DB_USER, ohad} directly in your configuration text file (so that no merge is necessary), or use an interpolation that has a missing value as a default (default_cfg = OmegaConf.create("db: {user: '${oc.env:DB_USER, ???}'}")) and call OmegaConf.resolve before merging. The three question marks ??? are a special "missing" value that will be overwritten in a merge. This second option only works in the recent omegaconf 2.1.0.rc1, not in the stable version 2.0.

[Ohad31415]

I'll try to be more concrete and hopefully more understood:
My wish is to be able to merge config from two separate sources, yaml file and env.
Let's say I have a yaml file like this -

db:
  user: Ohad
  password: ???

Than I have another one for env -

db:
  user: ${oc.env:DB_USER, ???}
  password: ${oc.env:DB_PASSWORD, ???}

So user has a default value, but if DB_USER is not defined, I want it to be taken from the first file, but as it is I can either define the default for the env variable and get that default not the value of the first file, or ending up with MISSING, either way is a problem.
password must be given, but similar to user if I do it like this it'd have to come through env or otherwise I'd end up with MISSING even if it's provided in the first file.
I hope these was clearer!

As for now, I ended up implementing my own function to look up env variables according to the configuration field names - it gives a dictionary without _MISSING_s so I can later merge it correctly. I'd really love that I could something similar via omegaconf!

[Jasha10]

I think I understand what you're trying to achieve. The following works for me, without any MISSING in the resulting config:

# tmp.py
from omegaconf import OmegaConf
file_cfg = OmegaConf.create("db: {user: Ohad, password: secret}")
default_cfg = OmegaConf.create("db: {user: '${oc.env:DB_USER, ???}', password: '${oc.env:PASSWORD, ???}'}")
OmegaConf.resolve(default_cfg)
cfg = OmegaConf.merge(file_cfg, default_cfg, OmegaConf.from_cli())
print(OmegaConf.to_yaml(cfg.db))

Here is the result when I run this file with different environment variable settings:

$ python tmp.py
user: Ohad
password: secret

$ DB_USER=Jasha python tmp.py
user: Jasha
password: secret

Note that OmegaConf.merge will not overwrite non-missing values with missing values.
(also note that the above does require installing a version of OmegaConf greater than the stable omegaconf 2.0 branch)

[Ohad31415]

@Jasha10 Thanks a lot, yeah I tested it on the coming version and it works great. I think it makes a lot of sense the updated way to treat missings when merging.
Hope to see 2.1.0 soon!