Simplify configuration

[tatomir146]

Currently configuration definition is done in 2 sections:

1. define a config id and map json to it
2. use this id to define additional configuration parameters

Wouldn't it be easier and also more readable to have something like this:

my-cool-config:
  template: "my-cool-config.json"
  name: "Here some cool name"
  skipDeployment: true
  params: 
    - some-param: "Never enough of coolness"
    - another-param: "Yet another param"

We already have name and skipDeployment as reserved params but this is not visible in configuration, you need to take a look in documentation for this. IMO it would help us to adapt YAML structure and have these params on the first level and define all custom params within e.g. "params". This would simplify our parsing logic.

Then we could also think about defining references in more flexible way which should simplify resolving dependencies:

more-config:
  template: "template.json"
  name: "We need a name"
  params: 
    - some-reference: 
         project: "some project"
         api: "custom-service"
         configId: "custom-service-config-id"
  # or relative to project
   - another-reference:
        api: "extension"
        configId: "extension-config-id"

In case there is a reference we map all parameters including name and id and then we can use this object within json template:

{
 "name": {{ .name }},
 "id": {{ .some-reference.id }}"
 "anotherReference": {{ .some-reference.whatever }}
}

Please also notice changes in YAML data types.

[didiladi]

Related discussions and issues: #208, #25, #126

[tatomir146]

Related discussion: #222

[patrickpichler]

I really like the idea of having references explicitly declared in the config.
The nice side effect we have by doing it like this is, that we do not care
about / or \ in the config, hopefully getting rid of those nasty windows
bugs.

There are still some open question though.
First of all, I think that references should be moved into their own "section".
So instead of having them under params move them to an references
object. That way it is even more explicit, that the things defined there
are references.

Now to some question:

• Should it still be possible to use references in param definition (similiar on how we do it right now)?
• What would happen, if we define multiple parameters/references/configs with the same name?
• What data structures should we support in the params (in the end we could support pretty much anything
  go supports)?

All in all I guess by simplifying what is allowed in the config we can ease the use
of monaco. 👍

[tatomir146]

I agree, it would be cleaner to move references to it's own section. Not sure if it still should be possible to use it under params, we should maybe restrict this to only references section. IMO Duplicated names/references/params should cause validation to fail. Datastructure params: everything go templates support. It would be also nice to support some transformations in templates e.g.
{{ .some-param | default("Default value if param not set")}}

[patrickpichler]

This should actually already be possible (not the default, but using some transformations).

[kristofre]

I agree, it would be cleaner to move references to it's own section. Not sure if it still should be possible to use it under params, we should maybe restrict this to only references section. IMO Duplicated names/references/params should cause validation to fail. Datastructure params: everything go templates support. It would be also nice to support some transformations in templates e.g.
{{ .some-param | default("Default value if param not set")}}

For the default value, I had already created #224 to support this.

[patrickpichler]

I'm not sure if this is even possible with golang templates.

[patrickpichler]

After checking the code, the way we are doing it right now, it isn't really possible.
We specify explicitly that the template should fail with an error, if a key is missing.

pkg/util/template.go:39

[patrickpichler]

I actually started working on a PoC for this.
Here is a link to it.

Note: There is currently no documentation for it and one needs to build it themself (simply checkout the code
and run make given you have golang and make installed)

The PoC breaks the config syntax as we know it right now.
First major change is, that the project structure is more rigid than before.
It expects the following structure:

working-dir
    /project-a
        /dashboard
            - config1.yaml
   /project-b
        /dashboard
            - config1.yaml

It is no longer possible to have a deeper folder structure than this.

The next big change is the config file structure. Instead of defining all the templates files
in one spot, and the rest of the config in another, it got merged into one object.

Here is an example

dashboard-a:
  config:
   template: "dashboard.json"
   name: "Dashboard A"
   parameters:
    owner: han.solo
    managementZoneId: ["infrastructure", "management-zone", "zone", "id"]
    managementZoneName:
      type: "reference"
      project: "infrastructure"
      api: "management-zone"
      config: "zone"
      property: "name"
  
  groupOverrides:
    development:
      parameters:
        managementZoneName: "dev"

    production:
      parameters:
        owner: "philip.fry"

  environmentOverrides:
    environment1:
      parameters:
        owner: "bender.rodriguez"
        managementZoneId: ["management-zone", "specialZone", "id"]
        managementZoneName:
          type: "reference"
          api: "management-zone"
          config: "specialZone"
          property: "name"

dashboard-b:
  config:
   template: "dashboard-b.json"
   name: "Dashboard B"

As you can see, we define two dashboards here. All configurations under config will be available
in the json template.
References are defined either via a map with type set to reference or with the short form
of an array of strings. All values you do not provide in the reference map, will be filled with values
from the current config. Same goes for the short form.

E.g.

test:
  type: reference
  config: dashboard-b
  property: name

Here we will reference to a different dashboard config in the same project. This is equivalent to

test: ["dashboard-b", "name"]

Overrides also work different than before. They are now all defined in the config object. The priority
of overrides did not change though. So we first take the config defined under config then we apply
all values for the current group under groupOverrides then we apply all values from the corresponding
environmentsOverride object. It is also possible to redefine name and template in overrides, but I'm
not sure if this actually make sense and we might remove this later.

What also has changed is the environment definition yaml file. It now looks like this:

groups:
  development:
    environment-1:
      url: "https://my-fancy-environment"
      token:
        type: "environment"
        name: "DEV_TOKEN"

    environment-2:
      url: "https://my-fancy-environment2"
      token:
        type: "environment"
        name: "DEV_TOKEN"

  production:
    environment-3:
      url: "https://my-fancy-environment3"
      token:
        type: "environment"
        name: "PROD_TOKEN"

standalone:
  environment-4:
    url: "https://my-fancy-environment4"
    token:
      type: "environment"
      name: "SPECIAL"

Here we define four environments. Under groups there is now the possibility to define groups with
their corresponding environments. If we take for example environment-1 the old config name would have
been development.environment-1.
As of right now, there is only the possibility to have a token of type environment, which means it will be
loaded from an environment variable. Later on we could add other types of retrieving tokens, such as
executing a program or whatever.
Environments defined under standalone are not part of any group. If anyone knows a better name than standalone
I would be happy, because I'm not quite happy with it.

[didiladi]

Yeah, you are right, might be easier to understand if those are not mixed. So that would leave us with:

configs:
  - name: "Dashboard A"
    template: "dashboard.json"
    parameters:
      - owner: han.solo
    groupOverrides:
      - name: my-group
        parameters:
          - owner: "vader"
      - name: development
        parameters:
          - owner: "luke"
          - zone: "dev"
    environmentOverrides:
      - name: env-1337
        parameters:
          - owner: "leia"
  - name: "Dashboard B"
    template: "dashboard.json"
    parameters:
      - owner: chewie

(sneaking in the param list in the process) 😄

[didiladi]

Sorry, we both added messages at the same time 😄. What do you mean with "default config under its own label"?

Ad param lists: Ok, I see... Without the list, that would be a regular map, in our underlying model. Ok, fine by me, let's skip the list. I guess we should ask ourselves: "Is there a static model possible for the yaml structure?"

[patrickpichler]

default config under its own label

This would be the config sub object in the sample i posted above.

static model possible for the structure

I'm not sure if I can follow you here 😆

[patrickpichler]

Also I would separate the name we use for uploading and the name of the config.
Probably we should rename the name property in the config map to id.

[patrickpichler]

configs:
  - id: "dashboard-a"
    config:
      name: "Dashboard A"
      template: "dashboard.json"
      parameters:
        owner: han.solo

    groupOverrides:
      - group: development
        override:
          parameters:
            owner: "vader"

  - id: "dashboard-b"
    config:
      name: "Dashboard B"
      template: "dashboard.json"
      parameters:
        owner: chewie

[kristofre]

One popular use case - mainly around automated app onboarding - is to use compound parameters in combination with environment variables.

myAoo: 
    - name: "{{ .Env.Environment }} - {{ .Env.AppName }}"
    ...

Two important parts are present here:

• Loading of environment variables: can be used to make configs generic
• Creating compound parameters: using multiple parameters together

As discussed during the roadmap session, it would be great to see these capabilities with the new 2.0 config.

[tatomir146]

@patrickpichler would this be supported?

[patrickpichler]

As we discussed in our roadmap meeting last week, this is currently not supported.
Config YAMLs are currently no longer pre-processed. To actually solve this use case,
we could introduce a new parameter type (called compound or whatever, I'm terrible
at naming things), which could look something like this:

- id: test
  name: 
    type: compound
    template: "{{ .Environment }} - {{ .AppName }}"
    values:
        Environment: ["EnvironmentName"]
        AppName: ["ApplicationName"]

  parameters:
    ApplicationName:
      type: environment
      name: "APPLICATION_NAME"
    EnvironmentName:
      type: environment
      name: "EnvironmentName"
    

So we have here a config with id test where the name is defined as a compound parameter,
referencing EnvironmentName (written in the fast references style) and ApplicationName
(also written in the fast referencing style). This way one could even reference parameters
from other configurations and use them in compound templates.

[patrickpichler]

@didiladi @tatomir146 @kristofre @UnseenWizzard any thoughts on this?

[patrickpichler]

As of right now, it is possible to directly access environment variables via golang templates in the
config yamls and jsons. In the current state of the PoC, the config yamls no are no longer golang
templates, to reduce the complexity and simplify the whole system. So things like

email:
  - name: "{{ .Env.EMAIL_NAME }}"

or

{
   "name": "{{ .Env.GROUP }} - {{ .Env.MAIL }}"
}

are no longer possible. Instead one would need to create explicit
parameter that could then be used within the configuration.

So the yaml example above becomes:

- id: email
  name:
    type: environment
    name: "EMAIL_NAME"

The json example could either be solved by using an compound parameter (as described in this comment),
or it could define two environment parameters, which could then be used with golang templates
in the config json.

The main benefit I personally see with this approach is, to make every dependency of the
configuration explicit. It is then possible to just look at the configuration yaml to understand
what parameters are needed and where they come from.

[patrickpichler]

@didiladi @tatomir146 @kristofre @UnseenWizzard any input from your side?