RFC: YAML configuration

[J-N-K]

As discussed in #3636 we need configuration for user-defined tags. It was discussed that an easy to understand standard configuration format would be preferable over creating a new DSL. It was also suggested in different issues that it would be nice to be able to copy&paste from UI to files. Main UI uses a YAML representation for things and items, so this would be good choice. We already have a feature for Jackson's YAML parser, so no additional dependencies are needed for that.

However, there are a few things to discuss:

• Should different things be configured in one file (e.g. items and things) or would be prefer to have different files?
• Should there be a versioning? This would allow us to better handle future upgrades.

I would suggest to add a YamlModelRepository (similar to our XText ModelRepository) that is an OSGi service that encapsulates all YAML processing and file processing. Consumer are OSGi services that implement YamlModelListener<T> where T is a DTO that describes the format. The listeners are injected into the YamlModelRepository and whenever a matching model (identified by the root name) is found the listener is notified.

public interface YamlModelListener<T> {
    void addedModel(String modelName, List<? extends T> elements);
    void updatedModel(String modelName, List<T> elements);
    void removedModel(String modelName, List<T> elements);
    Class<T> getElementClass();
    String getRootName();
}

Example listener (not fully implemented):

@Component(service=YamlModelListener.class)
public static class TestListener implements YamlModelListener<YamlItem> {
    private final Logger logger = LoggerFactory.getLogger(TestListener.class);

    @Override
    public void addedModel(String modelName, List<? extends YamlItem> elements) {
        logger.info("Added model {} with elements {}", modelName, elements);
    }

    @Override
    public void updatedModel(String modelName, List<YamlItem> elements) {
        logger.info("Updated model {} with elements {}", modelName, elements);
    }

    @Override
    public void removedModel(String modelName, List<YamlItem> elements) {
        logger.info("Removed model {} with elements {}", modelName, elements); 
    }

    @Override
    public Class<YamlItem> getElementClass() {
        return YamlItem.class;
    }

    @Override
    public String getRootName() {
        return "items";
    }
}

Example DTO:

public class YamlItem {
    private String name = "";
    private String type = "";
    private @Nullable String label;
    private @Nullable String category;
    private Set<String> tags = Set.of();
    private List<String> groupNames = List.of();
    private List<YamlLink> links = List.of();

    public String getName() {
        return name;
    }

    public String getType() {
        return type;
    }

    public @Nullable String getLabel() {
        return label;
    }

    public @Nullable String getCategory() {
        return category;
    }

    public Set<String> getTags() {
        return tags;
    }

    public List<String> getGroupNames() {
        return groupNames;
    }

    public List<YamlLink> getLinks() {
        return links;
    }

    @Override
    public String toString() {
        return "YamlItem{" + "name='" + name + "'" + ", type='" + type + "'" + ", label='" + label + "'"
                + ", category='" + category + "'" + ", tags=" + tags + ", groupNames=" + groupNames + ", links=" + links
                + "}";
    }
}

and the file:

version: 1
items:
  - name: Outdoor_temp
      type: Number:Temperature
      label: "Outdoor [%.1f °C]"
      groupNames:               
        - gTemperatures
        - gOutdoor
        - gPersist5
      links:
        - channel: "deconz:thing1:channel1"
        - channel: "deconz:thing2:channel2"
          profile: "MyScope:MyProfile

[rkoshak]

Should different things be configured in one file (e.g. items and things) or would be prefer to have different files?

Thinking from the perspective of rule templates and sharing I think allowing it all to be configured in one file would be a nice step towards supporting a bundle of configs (e.g. a collection of Items, Rules, and Widgets). I know the marketplace can't handle that (yet?) but even with git repos and forum posts that would make sharing a little easier.

If we could support variables (like Ansible for an example) that could be very powerful from a sharing perspective. But I know, we need to walk before we run.

Should there be a versioning? This would allow us to better handle future upgrades.

I don't know the full implication from the developer perspective, but from an end user perspective it sounds like a really good idea. Upgrades are a pain point.

I would suggest to add a YamlModelRepository (similar to our XText ModelRepository) that is an OSGi service that encapsulates all YAML processing and file processing. Consumer are OSGi services that implement YamlModelListener where T is a DTO that describes the format. The listeners are injected into the YamlModelRepository and whenever a matching model (identified by the root name) is found the listener is notified.

Again, this might be jumping too far ahead, but I've always found it awkward that the only way to develop/test a rule template is to publish it in the marketplace. I would hope that this YamlModelRepository might provide a way to load a rule template locally without going through the marketplace. It would greatly improve the rule template development experience.

[lolodomo]

Ok, I understand what you want to achieve. All the stuff monitorting the folders/files and YAML files loading will then be in one unique place in the YamlModelRepository.

Should different things be configured in one file (e.g. items and things) or would be prefer to have different files?

I would suggest to keep different files and different folders, like we have today.

Should there be a versioning? This would allow us to better handle future upgrades.

That looks like a good idea. That would mean a different listener (different DTO class) for each version ?

I can try to adapt my PR to add the class YamlModelRepository and have my existing class implementing YamlModelListener.

[spacemanspiff2007]

• Should different things be configured in one file (e.g. items and things) or would be prefer to have different files?

It would be nice to be able to put multiple things into one configuration file. You already put the item configuration in items, so why not allow things as a mapping, too? I currently have a shelly_flood.things where I have the shelly thing configuration and a shelly_flood.items where I have the items. It would be nice if I could merge these into one file. If subfolders will be allowed I can still use the legacy items and things folders, so that could be a solution that fits for everyone.

But one thing that needs to be improved is how things get loaded. If I define 100 items in the yaml and only change one then the whole 100 items get recreated and loose their state. There should be a very basic check:
Does the configuration already match the existing object, if yes skip if no delete and recreate.

• Should there be a versioning? This would allow us to better handle future upgrades.

Imho this is not needed and should at least be optional defaulting to the newest version.
To support multiple versions you can try to parse the content with V3, if it fails with V2 and if that fails with V1. If that fails show the error message from parsing V3.
Also most of the time you need the new information and if it can't be derived from the default the use has to migrate the file anyway. Our data is mostly static, because things / items etc. are already defined so I don't expect much changes.

---

Concerning your example format - I think there is an easier way to model the links.
It all depends how much logic we are willing to put into the parser of the returned structure.
I think it's almost always worth it to provide a simpler way because it adds clairity

items:
  - name: Outdoor_temp
    type: Number:Temperature
    label: "Outdoor [%.1f °C]"
    groupNames:               
        - gTemperatures
        - gOutdoor
        - gPersist5

    # Option 1: allow link as a shortcut if it's only one link without a profile which makes "link" an optional entry
    # User has to either use "link" or "links" but may never use both at the same time
    link: "deconz:thing1:channel1"

    # Option 2: links takes the channel name as a key and is a mapping
    links:
        # these two are equivalent
        deconz:thing1:channel1: 
        deconz:thing1:channel1: null
        # simple profile is key: value
        deconz:thing2:channel2: MyScope:MyProfile
        # parametrized profile is key: mapping
        deconz:thing2:channel2: 
          name: MyScope:MyProfile
          lower:15

Suggested Metadata example:

Note that I reused value to allow for a more compact data structure in metadata.
I'm aware that value currently is an allowed keyword for the config but imho it makes sense
to disallow value so we don't have a structure that is too deep.

Rollershutter window_covering "Window Rollershutter" {homekit = "WindowCovering"}
Rollershutter window_covering "Window Rollershutter" {homekit = "WindowCovering"  [stop=true, stopSameDirection=true]}

items:
  - name: window_covering
    type: Rollershutter
    label: Window Rollershutter"
    metadata:
      homekit: "WindowCovering"   # first line of *.items
      homekit:                    # second line of *.items
         value: "WindowCovering"
         stop: "true"
         stopSameDirection: "true"

[rkoshak]

There should be a very basic check:
Does the configuration already match the existing object, if yes skip if no delete and recreate.

I think that's a little more challenging than it might seem because the Items get unloaded/deleted when the original .items file gets unloaded. Then when the file gets reloaded the Items get created anew.

In order to support this use case:

• OH would have to be able to recognize the difference between a config file permanently being deleted instead of having been changed.
• New checks will have to be added to support recognition that an Item was removed from the file.

I believe, with my limited knowledge that would require changes to generic openHAB core stuff far above and beyond just adding a new model listener.

That's not an argument for or against it, just pointing out it might be outside the scope of this issue as it will probably impact all the other file formats too.

Note that I reused value to allow for a more compact data structure in metadata.
I'm aware that value currently is an allowed keyword for the config but imho it makes sense
to disallow value so we don't have a structure that is too deep.

Shouldn't the config match how the metadata is stored and used and even more importantly how it's shown in the Code tab of MainUI? If the proposal is to make the formatting changed in both places I'm good with that, though it probably means also changing the underlying JSON to support it which widens the impact of the change. Otherwise I think differences between what we see in MainUI and the YAML config files would be a net negative, even if the overall change is a simplification.

[spacemanspiff2007]

I believe, with my limited knowledge that would require changes to generic openHAB core stuff far above and beyond just adding a new model listener.

Yes - it's more than just adding another listener but not as much as it seems.
We already remove the items if the file gets deleted so there is already a mapping between file and containing item names.
The other step would just e.g. to wait one or two seconds after a file event and if the file is still there reload it, otherwise remove the items. I believe denouncing of events is already happening so most of the code is already there, too.

I agree that it's easier to just attach things to the existing code but if the goal is that in the long term yaml replaces the custom dsl formats then imho it makes sense to invest a little bit more time to remove the existing quirks and limitations.

That's not an argument for or against it, just pointing out it might be outside the scope of this issue as it will probably impact all the other file formats too.

Why not keep the existing code as is, freeze it and phase it out e.g. in OH5 or OH6?

Shouldn't the config match how the metadata is stored and used and even more importantly how it's shown in the Code tab of MainUI? If the proposal is to make the formatting changed in both places I'm good with that, though it probably means also changing the underlying JSON to support it which widens the impact of the change. Otherwise I think differences between what we see in MainUI and the YAML config files would be a net negative, even if the overall change is a simplification.

Yes - showing the metadata the same way should be the goal.
My suggestion is to just modify the parser/serializer so the underlying structure is not touched which would make it a small change.

E.g. this

homekit: 
  value: "WindowCovering"
  stop: "true"
  stopSameDirection: "true"

will still be saved as

{
  "value": "WindowCovering",
  "config": {
    "stop": "true",
    "stopSameDirection": "true"
  }
}

[lolodomo]

But one thing that needs to be improved is how things get loaded. If I define 100 items in the yaml and only change one then the whole 100 items get recreated and loose their state. There should be a very basic check:
Does the configuration already match the existing object, if yes skip if no delete and recreate.

This is not yet included in the implementation I proposed in PR #3659 but it is very easy to add, I just have to compare old and new object. I am going to add this check.

[lolodomo]

I think that's a little more challenging than it might seem because the Items get unloaded/deleted when the original .items file gets unloaded. Then when the file gets reloaded the Items get created anew.

In my proposed implementation (coming from @J-N-K initial POC), this is no more a problem for the items in one file. The code detects the removed, added and updated items.
In case you remove an item from a file and then add it in another file (so move it from one file to another), in that case, the item will be first removed and then recreated.

[lolodomo]

One thing I have not yet implemented but could be done easily is to add a check (and reject) if several "things" with the same id are present either in the same file or different files.

[spacemanspiff2007]

I am going to add this check.

Awesome! 🚀 🚀 🚀

In case you remove an item from a file and then add it in another file (so move it from one file to another), in that case, the item will be first removed and then recreated.

That's how I would have expected it.

One thing I have not yet implemented but could be done easily is to add a check (and reject) if several "things" with the same id are present either in the same file or different files.

Along with a proper error message this would also be a nice feature

[rkoshak]

will still be saved as

My understanding though is that the JSON to YAML in MainUI is generic. Therefore what you see in the MainUI code tab would be

  value: "WindowCovering"
  config:
      stop: "true"
      stopSameDirection: "true"

which would be different from the YAML in the file.

So either there would have to be a special branch of code in MainUI to transform the JSON first or manipulate the YAML just for the metadata code page or the JSON would have change to become

{
  "value": "WindowCovering",
  "stop": "true",
  "stopSameDirection": "true"
}

Both are significant amounts of work and the latter would be a breaking change.

[lolodomo]

But one thing that needs to be improved is how things get loaded. If I define 100 items in the yaml and only change one then the whole 100 items get recreated and loose their state. There should be a very basic check:
Does the configuration already match the existing object, if yes skip if no delete and recreate.

This is not yet included in the implementation I proposed in PR #3659 but it is very easy to add, I just have to compare old and new object. I am going to add this check.

It is now implemented.

[lolodomo]

One thing I have not yet implemented but could be done easily is to add a check (and reject) if several "things" with the same id are present either in the same file or different files.

Along with a proper error message this would also be a nice feature

It is now implemented for objects in the same file. I kept the code simple and the warning log only mentions that "X elements elements with same ids".
Detection of objects with same ids in different files is not yet implemented.

[spacemanspiff2007]

It is now implemented for objects in the same file. I kept the code simple and the warning log only mentions that "X elements elements with same ids".

What will happen? Will the last definition overwrite all previous ones?
We could also use a mapping instead of a dedicated name attribute, then duplicate entries would be an invalid yaml:

items:
  item_name:
    type: Rollershutter
    label: Window Rollershutter"

I like this format significantly better than the obj with the name attribute.

Both are significant amounts of work and the latter would be a breaking change.

It's a one-liner in python and I suspect even with proper error handling it'll be less than 15 lines of (java) code.

But aligning the GUI with files is not the only way this can be done:
Another one could be "export to file" (which would align with the *.items import)
So we have two yamls but it's clear which is which because one is an export and the other one is code

[rkoshak]

It's a one-liner in python and I suspect even with proper error handling it'll be less than 15 lines of (java) code.

If you are talking about coding the MainUI page to show it differently, it's not Python nor Java but JavaScript and it means introducing logic to detect when the metadata is being shown and going down a different path than the standard path used everywhere else in MainUI. That's significantly more than just a one liner.

If you are talking about changing the JSON representation of metadata, it may be a one liner, but it's a one liner that impacts everything in OH that reads and uses metadata today, including potentially end user's rules.

But aligning the GUI with files is not the only way this can be done:
Another one could be "export to file" (which would align with the *.items import)
So we have two yamls but it's clear which is which because one is an export and the other one is code

And users will see a YAML posted to the forum and blindly copy and paste it into the code tab somewhere and cry when it doesn't work because it's different for some unapparent reason. And yes, shame on them. But it still takes time from those who provide support on the forum dealing with the problem.

I'm all for changing it either way or not changing it at all (it's just not that big of a deal). I am vehemently against there being two different YAML representations of metadata (or any other entity) though. Tiny little differences like these become land mines for users.

[spacemanspiff2007]

it means introducing logic to detect when the metadata is being shown and going down a different path than the standard path used everywhere else in MainUI.

No - the only place where it is shown is in the code tab, because that's the only place in the MainUI where the user sees the yaml.
All other code can stay the same.
Exactly two places have to be modified: during the creation of the yaml (in the code tab) and the parsing of the yaml (in the code tab).

If you are talking about changing the JSON representation of metadata,

This is not what I am suggesting

And users will see a YAML posted to the forum and blindly copy and paste it into the code tab somewhere and cry when it doesn't work because it's different for some unapparent reason. And yes, shame on them. But it still takes time from those who provide support on the forum dealing with the problem.

If the file format is (significantly?) different than the format code tab it'll become clear because not only the metadata representation is different but also the whole item representation.
Additionally I think it's easy to adapt the MainUI to the file format, once the file format is specified.

(it's just not that big of a deal)

As someone who works with lots of files it is a big deal.

---

I can only appeal to all the hard working maintainers to not rush the new file format and not make it blindly a yaml representation of the json. At least evaluate some structures and let's think about what is nice to edit, share and work with.
Just think about how long the *.items file has been around - I think it's worth to invest additional effort.

[lolodomo]

It is now implemented for objects in the same file. I kept the code simple and the warning log only mentions that "X elements elements with same ids".

What will happen? Will the last definition overwrite all previous ones?

Such file containing objects with same ids is considered as invalid and all the file is ignored.
Adding this check was only 3 lines of code. I can add few more lines to have a better warning log.
By the way, I realized that the number I am providing could be wrong in certain cases.

[lolodomo]

Just think about how long the *.items file has been around - I think it's worth to invest additional effort.

IMHO, what is urgent for OH4.0 is to have the core feature for the future and the YAML file syntax for the semantic tags. That is the perimeter of my PR.
Defining YAML syntax for other objects (items, things, ...) could be done in a next step and released in OH 4.1. We have time, the current XText files are there and work well.

[spacemanspiff2007]

Such file containing objects with same ids is considered as invalid and all the file is ignored.
Adding this check was only 3 lines of code. I can add few more lines to have a better warning log.

Sounds good. I think it's enough to show the duplicate ids, the number doesn't matter.

Defining YAML syntax for other objects (items, things, ...) could be done in a next step and released in OH 4.1

Sounds like a plan 👍

[rkoshak]

No - the only place where it is shown is in the code tab, because that's the only place in the MainUI where the user sees the yaml.

And that code calls a generic library to convert the JSON to YAML. That'd no longer a generic call.

As someone who works with lots of files it is a big deal.

And as someone who spends probably as much time helping users on the forum that you do writing HABApp I still maintain that I have concerns. If the YAML is different in the two places it's going to be a big impact on my time and UI users' time.

But I'll state it again, I'm not against changing the format. But I've seen no one chime in here and say "I'll make sure the YAML in these files and the YAML on the code tab match if we decide to differ." It's not even clear whether @ghys would allow such a custom set of logic in MainUI in the first place no matter how easy it is (he's rejected similar proposals for Blockly and his reasons are valid).

All I request is that we ensure it is the same in both places. How or why that is enabled is not my concern.

I think it's worth to invest additional effort.

I agree. But it's also worth the additional effort to consider the impacts to UI users, support on the forums, and maintainers. Decisions made here do have impacts on all these other people too. To reprise my statement when I've had this same discussion on openhab-js, it's not fair to impose on our least technically able users to advantage our most capable users. A balance needs to be reached and compromises on both sides need to be accommodated.

Having two different YAML representations, be they similar or very different, is an imposition on those least capable users as it adds yet another thing they need to be aware of and figure out. It's an imposition on those who want to learn how to create file based configs because they can't just take what they see in MainUI and map that to the YAML file. It's na imposition to the documentation writers to have to deal with two different YAML syntaxes and document that there is a difference in the first place. And it's an imposition on the helpers on the forum who will have to deal with problems when users just blindly assume that the formats would be the same. Why wouldn't they? It's a reasonable assumption.

[florian-h05]

All I request is that we ensure it is the same in both places. How or why that is enabled is not my concern.

Absolutely agreed! When we introduce a new YAML file format, the overall structure of the entities should match with the code tab of the UI to allow copy-and-paste.

But I'll state it again, I'm not against changing the format. But I've seen no one chime in here and say "I'll make sure the YAML in these files and the YAML on the code tab match if we decide to differ." It's not even clear whether @ghys would allow such a custom set of logic in MainUI in the first place no matter how easy it is (he's rejected similar proposals for Blockly and his reasons are valid).

I — as an UI maintainer together with Yannick — don‘t see a large problem with changing the format of the UIs code tab, even though that would invalidate examples from the forum (I have to admit that I haven‘t seen that much YAML code examples on the forum for sharing stuff with other users). However we really should keep in mind while designing a YAML file format that it also needs to be implemented in the UI that way, preferably without a ton of extra code. I also want to hear Yannick’s opinion before deciding anything on this side.
So before merging any new YAML file PR for Things, Items etc. into core, I‘d propose that we also get a UI PR in place to ensure UI and file have the same YAML structure.

Feel free to ping me and ask for help and advice for the UI side.

[spacemanspiff2007]

And that code calls a generic library to convert the JSON to YAML. That'd no longer a generic call.

That call will still be generic. The steps would be:

• get mapping for metadata (same as today)
• pull attribute one level up/down (new)
• transform to yaml (same as today)

---

All I request is that we ensure it is the same in both places. How or why that is enabled is not my concern.

I agree with you - that would be ideal and should be the goal.
But your argument is:
"We have something that is (very seldomly) used for copy paste so the file format should be the same".
And my argument is:
"We will have something that will be used by very much people very often so we should make it easy and comfortable to use."

These aren't mutually exclusive and again I agree with you and the points you make.
But we draw the different conclusion.
Yours is "The code changes are probably difficult and maybe the UI maintainers won't do it so we have to align the file format with what we have in the GUI now" (very exaggerated to make a point).
Mine is "Make the format nice and easy to use because it affects much more people and then (try to) make the UI follow that format".
And imho it's a reasonable assumption that the UI will follow: We currently have an *.items file parser in the UI which is magnitudes harder and way more complex than pulling up/down some attributes in a possible yaml file format.

---

However we really should keep in mind while designing a YAML file format that it also needs to be implemented in the UI that way, preferably without a ton of extra code.

If we decide on a new format and we want to use it in the UI why not expose it through the rest api.
That way the code tab always is in sync with the file format.
E.g. we could have a yaml/parse and a yaml/serialize or a .../yaml endpoint on the item / thing

[rkoshak]

"We have something that is (very seldomly) used for copy paste so the file format should be the same".

That is not my argument. My argument is very much that making the formats different increases complexity, adds confusion, and increases the work for lots and lots of people (users, maintainers, documenters, and helpers) outside this discussion.

And I take issue with the "very seldomly". It's only very seldomly used today because it's awkward and not well supported. If the formats are the same, I suspect it would become very frequently.

And my argument is:
"We will have something that will be used by very much people very often so we should make it easy and comfortable to use."

And your argument and mine are not against each other unless the format just changes to benefit only those who only use text files at the expense of everyone else.

Again, I don't care what the format becomes. I only care that it's the same in the files and in the UI. That's it. Having two different YAML representations for stuff in OH would be a huge mistake in my somewhat informed opinion.

And note: I'm really only talking about the stuff that you'd actually see in MainUI's code tabs. So, for example, all you see today for metadata is the "value" and the "config" so I only care that those are represented the same. The stuff that comes above those in the YAML hierarchy in the text file is not part of what I'm talking about. That stuff isn't shown in MainUI as YAML so I have no concerns about how that's formatted.

[ghys]

• You have those fromYaml () / toYaml () methods throughout the UI's code which perform the serialization/deserialization as YAML (from the JS objects that are meant to be posted as JSON to the API when clicking Save). What happens in them is open for debate as long as it makes sense and they remain reliable. Copying the server-side syntax looks sensible to me.

• Currently the YAML format is usually close to the target JS object for simplicity's sake, but can differ (e.g. omitting properties that aren't part of the object's model but are enriched, for instance statuses etc.; for scenes you also have a completely new items dictionary which doesn't exist in the rule specification and is transformed back to rule modules).

• Also note that those code editors may have CodeMirror "hints" for autocompletion (for example, thing configuration) which would have to be adjusted if the syntax changes.

• Finally, instead of (or in complement to) file watchers you might also consider the Kubernetes approach:

1. https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#organizing-resource-configurations

   Management of multiple resources can be simplified by grouping them together in the same file (separated by `---` in YAML).

2. https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#in-place-updates-of-resources (operation to "apply" the specified file(s) to the registry, or delete them in bulk)

   So you would define multiple related objects like:

kind: Thing
apiVersion: v1
UID: exec:command:MyThing1
name: MyThing1
thingTypeUID: exec:command ...
---
kind: Item
apiVersion: v1
name: MyItem1
type: Number ...
---
apiVersion: v1
kind: Link ...
---
apiVersion: v1
kind: Rule
uid: MyRule1 ...

3. Some REST endpoint accepting this kind of input would mean these could be created from a "Create from YAML input (or file") UI developer tool similar to the "create" features of the Kubernetes dashboard).

   

   https://adamtheautomator.com/kubernetes-dashboard/#Ensuring_Resources_Show_up_in_the_Dashboard

4. The same logic could be also used in the marketplace services to import these kind of multi-object YAML files.

5. This would also possibilities for new tools like an OH equivalent to Helm charts (packages of templated resources).

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/oh4-transformationservice-of-type-js-is-unavailable/148786/16

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/what-is-the-most-common-ui-you-are-using/149096/31

[lolodomo]

@rkoshak : as I understand, @J-N-K considers my PR #3659 cannot be merged due to one of your suggestion here.

In case you would like to have one unique file containing potentially any kind of OH data (is that really your request ?), that means it should contain many optional parts. When parsing the YAML file with a reference class, how do you specify in the reference class that each kind of objects is optional ? Is it technically possible with the YAML library we are relying on at your opinion @J-N-K ? I remember I tried to make the version field optional and I did not succeed.
That would require to have a reference class containing an optional list of items, an optional list of things, an optional list of sitemaps, an option list of tags, an optional list of rules, ... Then each time we would like to enhance one of these elements, we should increase the version. Then all users should update all their YAML files, even those that are not directly impacted by the change, at least to upgrade the version ?!

Today, we have files of each kind in different folders, something well structured. I understand you would like to allow a file containing items even in the folder "things" for example ?! That would require to scan all folders to find items, all folders to find sitemaps, ... etc

Would you agree to keep things simple and avoid adding a lot of complexity for a very low user profit ? Then we could maybe merge my PR.

One time again, I would also really appreciate @J-N-K if you confirm or not if it is technically doable with the YAML library we are using.

[jimtng]

@lolodomo and @J-N-K What's the intended use for these methods?

openhab-core/bundles/org.openhab.core.model.yaml/src/main/java/org/openhab/core/model/yaml/YamlModelRepository.java

Lines 28 to 32 in e9700f8

	void addElementToModel(String modelName, YamlElement element);
	void removeElementFromModel(String modelName, YamlElement element);
	void updateElementInModel(String modelName, YamlElement element);

I see that they manipulate / write into the physical YAML file. Are they going to be used to rewrite user's YAML files, or are they only meant for managing internal yaml database, similar to how it's done in jsondb, for Managed entities?

[lolodomo]

I see that they manipulate / write into the physical YAML file. Are they going to be used to rewrite user's YAML files, or are they only meant for managing internal yaml database, similar to how it's done in jsondb, for Managed entities?

Yes, @J-N-K great idea (even if I discovered that relatively recently) was to use this YAML format also for managed entities as a replacement of our JSON DB. It would allow a unique and common format for entities defined in file and those defined with UI.
I should admit this is a great idea but as you imagine there are already people against this idea.
To be honest, @J-N-K idea was also to keep a file format very close to the internal object and we change that, not too much in fact but a little, for example by using map instead of list.
The update of YAML files is even already implemented and in our code base.

We could reopen this debate for OH 5.1.

[jimtng]

My main concern is whether it will write to user's yaml, and from what I understand, it won't be.

I am working on extending our yaml structure to enable generic templating. The idea is so we can DRY repetitive elements and I'll offer several methods to achieve this. It relies however, on the fact that the user's yaml doesn't get overwritten by openhab.

However, come to think of it, one can always add readOnly: true to enforce that! So @J-N-K must've anticipated this :)

[Nadahar]

So let's go with conf/settings

It's at least better than conf/yaml, but as @rkoshak already said, "settings" isn't really a good fit (it shouldn't be used in MainUI either IMO). "Settings" should be reserved for simple settings like what color the UI should appear in or what language to display.

That said, my main point has still been missed. I assume it means that most of you don't agree with it. I for one, won't call whatever is defined in #3666 for "YAML", since that's not what it is. YAML would be a simple "flat mapping" of the internal objects, with no "rules". So, I guess I will have to continue to refer to it as either "the new™ configuration format" or "the unnamed™ configuration format".

use this YAML format also for managed entities as a replacement of our JSON DB.

The "modifications" done to this™ format makes it wholly unsuitable for such use. I like the idea, but there are several "challenges" that I can see: The fact that it doesn't map 1:1 means that you probably can't store everything that is stored internally. The fact that OH can't have more than one managed provider means that you'd have to choose between JSON DB and this.

However, come to think of it, one can always add readOnly: true to enforce that!

Not only that, it defaults to read-only if readOnly isn't specified. So, to make a file writable, you'd have to explicitly use readOnly: false. Files with this template system should probably still have some kind of "protection", like refusing to substitute placeholders (if that's the concept) in writable files.

[jimtng]

That said, my main point has still been missed. I assume it means that most of you don't agree with it. I for one, won't call whatever is defined in #3666 for "YAML", since that's not what it is. YAML would be a simple "flat mapping" of the internal objects, with no "rules". So, I guess I will have to continue to refer to it as either "the new™ configuration format" or "the unnamed™ configuration format".

I don't quite understand this, and I don't understand your obsession with "internal objects".

We shouldn't care about what the internal objects look like. We only need to worry about what the yaml configuration file looks like in order to create what we want. The aim is so the configuration file is easy to read and write by humans.

The code will abstract it and translate it to the "internal objects" as necessary. There's no rule that says it must translate 1:1 and I wonder if that's even possible. It may be nice if it happens to match 1:1 but that's most certainly not a requirement.

use this YAML format also for managed entities as a replacement of our JSON DB.

The "modifications" done to this™ format makes it wholly unsuitable for such use. I like the idea, but there are several "challenges" that I can see: The fact that it doesn't map 1:1 means that you probably can't store everything that is stored internally.

What makes you think that?

The fact that OH can't have more than one managed provider means that you'd have to choose between JSON DB and this.

No, I believe the idea is to replace the jsondb.

[Nadahar]

I don't quite understand this, and I don't understand your obsession with "internal objects".

We shouldn't care about what the internal objects look like. We only need to worry about what the yaml configuration file looks like in order to create what we want. The aim is so the configuration file is easy to read and write by humans.

The code will abstract it and translate it to the "internal objects" as necessary. There's no rule that says it must translate 1:1 and I wonder if that's even possible. It may be nice if it happens to match 1:1 but that's most certainly not a requirement.

That depends entirely on your goal. If your goal is that the files should be written/modified by humans, that makes sense. But, if it's meant to be used by code, that's not the most important property. Instead, "conversion" speed and the ability to accurately and unambiguously represent the data structure it "represents" are key. Storage efficiency (how much space is used to represent the data) is also of at least some importance.

Every (data representing) format out there is a tradeoff between the different needs. XML is superior to JSON and YAML in representing complex data types, it is simply much more capable. But, it's space inefficient and some say not very human-readable. I think XML is readable just fine, but I admit that writing XML is tedious. If you don't care about the human factor, and storage space isn't a big concern, XML is often a good choice. If you really don't care about human readability, you have "binary" formats like those made with protobuf etc. They can represent pretty much anything you want, and they are space efficient and fast, but humans can't really read or write them at all.

Both JSON and YAML are "dumbed-down" formats that can't represent complex data types, but which are more "human friendly". They only support the most basic data structures, because otherwise it would quickly be "overwhelming" for the human to read/write them. You could also say that since they (at least JSON) are often used to exchange data between systems written in different languages, they "must" restrict themselves to data structures supported by the most basic "languages".

Regardless, these all serve their purposes as compromises in some form. But, except for custom binary formats (e.g. protobuf), they all have a well-defined way to store basic data types. It is explicitly defined how a certain data type should be represented. The whole point of this is to "map them" to real data structures. Doing it this way has some huge benefits, because they can be converted and processes without "prior knowledge" of the data they contain. You can for example convert between YAML and JSON (both ways) using many different tools, and the result will be valid and able to be converted back to the original data structure.

When you "break" this "mapping", you also lose many of the benefits of using standard representation of data types. One example is that today, MainUI can "convert" between JSON and YAML itself. It doesn't need to use Core's REST API to do that, because it follows the rules. That isn't true for this™ format, which is why there's suddenly a need to go via Core's REST API to "translate": It no longer adheres to the standard rules. You meet the same limitations in different situations. You sacrifice speed and simplicity, not to forget that by having to write these "translation routines" you introduce the likelihood of bugs in the translation that wouldn't have been very likely if you used the standardized, existing rules.

So, I'm not saying that doing what this™ format is "illegal" or "bad", I'm just saying that it's not "following the rules" for normal serialization/deserialization, which makes it slower and more error-prone. The consensus is that this is worth it because it's more human friendly. That might be true (I'm not convinced that it makes much difference though), but it also makes it less suitable to when humans aren't involved.

You're right that Java data types can't be mapped 1:1 to JSON or YAML because they are so basic. Core already has DTOs for the Java classes that are used in other places in the system:

openhab-core/bundles/org.openhab.core.thing/src/main/java/org/openhab/core/thing/dto/AbstractThingDTO.java

Lines 27 to 36 in e9700f8

	public abstract class AbstractThingDTO {
	public String label;
	public String bridgeUID;
	public Map<String, Object> configuration;
	public Map<String, String> properties;
	public String UID;
	public String thingTypeUID;
	public String location;
	public String semanticEquipmentTag;

openhab-core/bundles/org.openhab.core/src/main/java/org/openhab/core/items/dto/ItemDTO.java

Lines 24 to 31 in e9700f8

	public class ItemDTO {
	public String type;
	public String name;
	public String label;
	public String category;
	public Set<String> tags;
	public List<String> groupNames;

openhab-core/bundles/org.openhab.core.automation/src/main/java/org/openhab/core/automation/dto/RuleDTO.java

Lines 27 to 39 in e9700f8

	public class RuleDTO {
	public List<TriggerDTO> triggers;
	public List<ConditionDTO> conditions;
	public List<ActionDTO> actions;
	public Map<String, Object> configuration;
	public List<ConfigDescriptionParameterDTO> configDescriptions;
	public String templateUID;
	public String uid;
	public String name;
	public Set<String> tags;
	public Visibility visibility;
	public String description;

...etc. These DTOs are mapped 1:1 where possible, and where it's not possible, they keep as close as they can.

Because this™ format has "custom rules" defined in this issue, we now have duplicate DTOs for supported "objects", e.g.

openhab-core/bundles/org.openhab.core.model.yaml/src/main/java/org/openhab/core/model/yaml/internal/things/YamlThingDTO.java

Lines 33 to 46 in e9700f8

	@YamlElementName("things")
	public class YamlThingDTO implements YamlElement, Cloneable {
	private static final Pattern THING_UID_SEGMENT_PATTERN = Pattern.compile("[a-zA-Z0-9_][a-zA-Z0-9_-]*");
	private static final Pattern CHANNEL_ID_PATTERN = Pattern
	.compile("[a-zA-Z0-9_][a-zA-Z0-9_-]*(#[a-zA-Z0-9_][a-zA-Z0-9_-]*)?");
	public String uid;
	public Boolean isBridge;
	public String bridge;
	public String label;
	public String location;
	public Map<@NonNull String, @NonNull Object> config;
	public Map<@NonNull String, @NonNull YamlChannelDTO> channels;

Maintaining a double set of DTOs (located in completely different parts of the code) in itself is bad, but that decision has already been made. But, to me, it's clear that things like persisting objects or the REST API should continue to use the "proper" DTOs, that aren't "YAML specific" (that means, designed in a way that people here think "suits YAML better"), since they most accurately represents the data, which leads to better speed and less potential for errors. In addition, as I've understood @lolodomo, this™ format doesn't even represent all the data. Things that are deemed "irrelevant to the user" are skipped for simplicity. That means that they can't be used for persistence or data exchange. In addition to this, the scope of the YamlXXXDTOs is limited to the org.openhab.core.model.yaml package, so these DTOs aren't even available to the rest of the system.

From what I have seen here, the focus has been very "tunnel visioned" on perceived user-friendliness over any other concerns. That's fine, but then it should be treated as that as well: A format especially tailored to humans, not a generic data representation format. Which again explains why I think it should have a name.

What makes you think that?

As far as I know, not everything is even represented, and data structures have been modified, multiple (independent) classes are combined into one DTO etc. To keep converting this back and forth is slow, error-prone, and generally a bad idea. I can pretty much guarantee many hard to find bugs that will "corrupt" users' data over time if this is attempted.

No, I believe the idea is to replace the jsondb.

Given what I have explained above, this certainly should be met with resistance. I certainly wouldn't want that. JSON is also superior to YAML as a storage format in every way, except that it doesn't have multi-line strings. YAML is wasteful in that the files are full of space characters. "Yeah, but it can be easily compressed" people say. Yes, but then it becomes "binary" and not human-readable at all, so all you have done then is to take a long and slow route to make a binary format.

[jimtng]

In addition, as I've understood @lolodomo, this™ format doesn't even represent all the data.

I believe/hope that the above statement is incorrect. Perhaps it's because @lolodomo is still working on it? If something is missing, that just means we need to add it.

Things that are deemed "irrelevant to the user" are skipped for simplicity.

I'm not 100% sure what you're referring to. As I understand it, some things are skipped because they can be reconstructed from default values, therefore their existence in the yaml data is unnecessary. This was primarily intended for the UI presentation purposes and afaik, it doesn't mean that the format lacks the ability to hold such data.

I understand your point about having duplicate "DTOs" and the need for a translation layer rather than direct read/write, but that's a necessary layer for us to have a nicer format for humans to read/write.

Whether we end up using this as a jsondb replacement remains to be seen maybe.

What we are working on right now is for human facing interface, first and foremost.

[Nadahar]

I'm not 100% sure what you're referring to. As I understand it, some things are skipped because they can be reconstructed from default values, therefore their existence in the yaml data is unnecessary. This was primarily intended for the UI presentation purposes and afaik, it doesn't mean that the format lacks the ability to hold such data.

Yes, and that's my point. When you "reconstruct from default values", you don't recreate identical objects. You can't do that for data storage/transfer. Also, combining different objects "into one" and then "separating them again" when deserializing, like with Thing/Channel etc. is often bug prone and slow (depending). If you have a table of metadata that is linked to the UID of some "main table" (which is a typical design) and you combine these two "into one", the deserializing code must create two separate objects and place them in different "places" (registries, whatever). That's not how you want to do persistence, you want to minimize processing and error sources, so you store them just as they are - as separate objects, linked via UID.

Don't get me wrong: I'm not saying that "translating" things this way is "bad", just that we should remember what exists for what purpose. The current YAML seen in the UI is a YAML representation of the "existing DTOs", and to be honest, I don't think the changes introduced will make much difference to users, since I think very few will "hand write" them anyway. They will probably start with some copy, either from the UI or from some other source (file, forum etc.), and modify that. Map/array, short field names etc. won't make any practical difference. But, as a consequence of this being deemed "worth it", OH will have to live with this "dual system". And that may be the right decision, I'm not saying it's not, I'm just a bit skeptical if it's worth it.

What we are working on right now is for human facing interface, first and foremost.

Exactly, so let's treat it as that, give it a name that can be referenced, and not try to use it where it's not suitable.

[rkoshak]

That depends entirely on your goal. If your goal is that the files should be written/modified by humans

Everything in this issue is for this. We are creating a file that will be read, written and edited by humans as the most important use. All the computer stuff can and should be using the raw JSON.

MainUI can "convert" between JSON and YAML itself. It doesn't need to use Core's REST API to do that, because it follows the rules

But it can't for Rules DSL, so the REST API is still required. If it's going to be there anyway, and the most important part of what we are doing here is to create a format for humans to work with. MainUI doesn't need to be in the business of converting between any formats and the YAML humans have to deal with can be changed to be more human friendly.

That isn't true for this™ format, which is why there's suddenly a need to go via Core's REST API to "translate"

This puts the cart before the horse. The REST API was created to see Rules DSL in MainUI. That's not something that MainUI was ever going to be able to support. This new YAML format did not drive that, but it certainly can take advantage of that to give a better experience to the end users. But again, this PR is not driving the creation of that REST API.

since I think very few will "hand write" them anyway.

But they will have to read and understand them.

Exactly, so let's treat it as that, give it a name that can be referenced, and not try to use it where it's not suitable.

We use "thing", "item", "channel", "rule", "scene", "widget" et al as technical terms of art in OH. We use "script" as a term of art in no fewer than three different ways with orthogonal definitions. This is the ditch we are going to die in?

Right now, when this work is done, there will be exactly one YAML format in use in OH that end users will ever see. This one.

I think any discussion of replacing or adding an alternative to JSONDB is very premature at this point.

[Nadahar]

Everything in this issue is for this. We are creating a file that will be read, written and edited by humans as the most important use. All the computer stuff can and should be using the raw JSON.

Yes, that has become my understanding too. My comment was in the context of this™ format being suggested as a replacement for JSON for persistence or REST API communication, what you call "the computer stuff". All I was trying to say was since this™ format has been "dedicated to" being convenient for humans, it should not be attempted used for the things it's not suitable for.

MainUI doesn't need to be in the business of converting between any formats and the YAML humans have to deal with can be changed to be more human friendly.

MainUI should not be in the business of "translating" formats, I very much agree with that. If something has to be mapped in some specific way, the logic for that should be in Core, not in a UI. But, this is what I seem to struggle to express, YAML <-> JSON is well-defined outside of OH. So, any library can do it, you can use a web page converted, you can use your "tool X" installed on your computer to convert between the two. In such a case, MainUI can do it, because it requires no logic except what is already built into the formats.

I can't stand YAML, so when working with Widgets, I've sometimes used a converter to transform it to JSON to work with it without having to indent/unindent huge sections (which is a pain, and you newer know if it went as planned). When I've done the things I want to do, I convert it back to YAML and paste it into the editor. So, to try to sum up: Conversion between YAML and JSON can be done in a standardized way that is "known to the world", and as far as I know they "map perfectly" except that JSON will "lose" comments. With such a conversion, there is no need to go via Core, because there is no "special logic" to apply.

When you're talking about something like this™ format, that isn't a standardized conversion/mapping of the internal objects, it should be done in Core, because otherwise you'd have to replicate the logic in each and every UI (or other component) that would want to use it.

Just to be clear: I'm not arguing that this™ format should be ditched and the "standard DTOs" used instead. I have accepted that this decision has been made. My argument was merely that this has a cost (in duplication of DTOs, more complicated logic to maintain, more network traffic and more load on the OH server hardware), and that I'm not convinced that the benefits outweigh the costs.

But they will have to read and understand them.

Yes, but I don't think config is easier or better to read than configuration. The only "benefit" is that it's shorter to type. Regarding presenting things as maps when they are really arrays, I really don't think that improves readability in any meaningful way either. It makes it a bit more "logical" to write, maybe. But, I feel this is a digression, that "ship has sailed", too much effort has been put into making it a separate format/structure that it's much point in revisiting this now.

We use "thing", "item", "channel", "rule", "scene", "widget" et al as technical terms of art in OH. We use "script" as a term of art in no fewer than three different ways with orthogonal definitions. This is the ditch we are going to die in?

I'm not sure that I understand your point here. It's not about "correctness of terms" to me, it's about separating different entities. As I've said before, as far as I'm concerned, it could have an arbitrary name with no inherent meaning. What I consider important is to have a name for whatever this™ is. So that it can be referenced and documented.

By the time discussions have been had and decisions have been made over most things that can be configured using this™ format, don't you agree that these rules should be documented somewhere? That you have to translate certain field names and certain values, and that you have to make a structure that adheres to certain rules?

I think a document that describes this should exist, so that people can know how to use it. This document also needs a name for what it is it describes.

I hope nobody is going to die in a ditch, and I'm not sure why you say this. To me, the need is just so clear, that I honestly don't understand why the rest of you don't see the need. It makes me "feel" that I can't have explained it well enough, otherwise others would see it too. I guess this is what you're observing, that I can't quite fathom that the need only is obvious to me. I'll try to shut up about it, as I'm pretty sure that I've given everybody ample opportunity to see my point if they want to.

[rkoshak]

What I consider important is to have a name for whatever this™ is. So that it can be referenced and documented.

But you also insist that name cannot be "openHAB's YAML" or "YAML" for short.

don't you agree that these rules should be documented somewhere?

Absolutely it needs to be documented. I expect there to be either a new reference page created that covers everything on one page, or (more likely) the existing pages to be updated with new sections describing how to use this new format. In a lot of cases we can use tabbed code blocks to switch between DSL and the new YAML in the examples in these pages similar to how examples are shown for the various rules languages.

I don't think the format has stabilized enough yet to start tackling the docs though.

That you have to translate certain field names and certain values, and that you have to make a structure that adheres to certain rules?

The end users won't have to translate anything though. All they will know is this format or the DSL format. If they choose they can use MainUI or the REST API to translate whatever they have to Rules DSL or translate Rules DSL to YAML. Maybe another format will be supported in the future. I think that REST API endpoint can also translate it to the internal JSON, but I'm not positive about that.

I beleive the end goal is that if a user wants to go from file based configs to managed configs, they can import what they already have unchanged no matter the format. If a managed configs user wants to move to files, they can export the managed configs (which are JSON) to what ever file format they want. When they are working in the UI and click on the code tab, they will see DSL or the YAML file format so that no matter where they look, they are seeing the same format they would see when looking in a file.

One of the main goals here is to avoid translation like that. What you see works in the UI as well as in files. No longer will OH have this two-track system where everything looks different when you are using files or when you are using the UI.

To me, the need is just so clear, that I honestly don't understand why the rest of you don't see the need. It makes me "feel" that I can't have explained it well enough, otherwise others would see it too.

Which is why I've really tried to limit how much I participate in these specific discussions and have abandoned others. You seem to think "if only they understood me they would agree" and never consider "they understand me but still disagree." It is ok for us all to disagree though. We just have to drop it at some point and move on. I think we've reached that point here.,

[jimtng]

@lolodomo

I will create a separate issue / topic about yaml enhancements I'm working on, but before that's ready, I would like to request (I'll make a PR to do this) to reserve the following:

• secret.yaml as a special filename to store passwords / secrets / keys, to be referred to from the main yaml files
• *.inc.yaml as a reserved filename for yaml include files

The alternative is that I'll use secret.yml and anything.yml as the include files.

Which will be preferable?

[jimtng]

What I consider important is to have a name for whatever this™ is. So that it can be referenced and documented.

But you also insist that name cannot be "openHAB's YAML" or "YAML" for short.

From my pov, I just need a decision for the directory name: is it yaml or settings, or conf.d, or something else, so we can move on to more exciting stuff. @rkoshak which one is your vote?

[jimtng]

See:

• YAML File "templating" #4812

[Nadahar]

But you also insist that name cannot be "openHAB's YAML" or "YAML" for short.

Yes, and I've explained why in every way I can.

The end users won't have to translate anything though. All they will know is this format or the DSL format.

It depends on the end user. All the existing YAML in MainUI today is based on the "regular" DTOs, that map 1:1 as much as possible. I'd find it strange if nobody has any "relation" to that. Then there's the JSON DB, I certainly do stuff there as an "end user" (to clean up things that have gone wrong in the UI, or to simply study what the UI has done). Then there's the source code, the JavaDocs and what you get from the REST API. I'd say that "most sources" for this information needs to be translated. Not everybody will use any of these sources, and they probably won't have the need to "translate" - but I'm certain there's also quite a few that do. And, let's not forget developers. They can't all possibly remember all these rules, they will also need a place to "look it up".

I beleive the end goal is that if a user wants to go from file based configs to managed configs, they can import what they already have unchanged no matter the format. If a managed configs user wants to move to files, they can export the managed configs (which are JSON) to what ever file format they want.

I think "the equality" between file based and managed is great 👍 I haven't been a fan of the "push" to move everything to managed.

You seem to think "if only they understood me they would agree" and never consider "they understand me but still disagree."

That is true, I tend to fall in this "trap". But, I'd like to say that I understand that people simply disagree on some things and not on others, and I think a part of the reason is the way the response is formed. If the points I'm trying to make are addressed or at least acknowledged, it will usually get through to me (albeit a bit late at times). But, when my arguments are simply ignored, or the reply indicates that they are misunderstood, I will very likely try again (and again..).

I think we've reached that point here.

I agree.

• secret.yaml as a special filename to store passwords / secrets / keys, to be referred to from the main yaml files
• *.inc.yaml as a reserved filename for yaml include files

The alternative is that I'll use secret.yml and anything.yml as the include files.

Which will be preferable?

I don't have any particular "preference", but I'd like to caution that parsers usually just look at the extension. *.inc.yaml will be seen as just another YAML file, the extension is usually consider the part after the last period. So, a different extension might be the "cleanest" way to differentiate the files. That will lead to the same old syntax highlighting problem of course, but if the file is a "partial YAML" or has some other syntax that "doesn't fit" standard YAML, many editors might have linters that go haywire, so it might be preferrable if it's not treated as YAML?

If you only need to exclude these files from this™ particular parser, anything will work, as it's just about hardcoding the exceptions in the parser.

[jimtng]

I don't have any particular "preference", but I'd like to caution that parsers usually just look at the extension. *.inc.yaml will be seen as just another YAML file, the extension is usually consider the part after the last period. So, a different extension might be the "cleanest" way to differentiate the files. That will lead to the same old syntax highlighting problem of course, but if the file is a "partial YAML" or has some other syntax that "doesn't fit" standard YAML, many editors might have linters that go haywire, so it might be preferrable if it's not treated as YAML?

Yes, I understand how file extensions work, and I'm not saying that you shouldn't try to explain it. I'm just saying that I am aware of what you described. However:

• I'd like to keep it as YAML so editors can recognize them instantly.
• The include files are, and must also be, 100% valid standard YAML just like the main files.

I'm thinking of also reserving *.package.yaml to differentiate packages vs simple includes, but for now *.inc.yaml is fine. One can do *.package.inc.yaml if we were limited to just *.inc.yaml, leaving it to personal preference.

[rkoshak]

secret.yaml as a special filename to store passwords / secrets / keys, to be referred to from the main yaml files

There is an open issue that I thought had some recent activity on it for secrets handling in OH core but I think I have it confused with the RBAC PR. #3289 appears to not have had any activity for a year so I guess it's dead. But there might be something that can be salvaged from there, and it would be really great if a whole OH solution that could be used by add-ons and rules were implemented.

From my pov, I just need a decision for the directory name: is it yaml or settings, or conf.d, or something else, so we can move on to more exciting stuff. @rkoshak which one is your vote?

This folder won't really behave like a typical conf.d you'd see in a Linux /etc folder, but I don't know if that's a problem. I like it better than settings except for the fact that MainUI calls it settings and consistency is something where OH could do a better job. So I'll vote for "settings" even though I kind of like "conf.d" better on its own.

[Nadahar]

This folder won't really behave like a typical conf.d you'd see in a Linux /etc folder, but I don't know if that's a problem. I like it better than settings except for the fact that MainUI calls it settings and consistency is something where OH could do a better job. So I'll vote for "settings" even though I kind of like "conf.d" better on its own.

I'd vote for conf.d of the two because, while I also like consistency, it shouldn't be replicated when the "source is wrong". Maybe this should be renamed in MainUI instead?

[rkoshak]

Maybe this should be renamed in MainUI instead?

It certainly shouldn't be named "conf.d" in MainUI.

[Nadahar]

It certainly shouldn't be named "conf.d" in MainUI.

Hehe, no, I was thinking more of "Configuration" 😉 As far as I know, conf.d is short for "configuration directory", so I'd still say that is consistent.

[JustinGeorgi]

I prefer settings to conf.d just because even though they are a minority, there are users who run oh on something other than Linux and they would find conf.d completely opaque. Add in users where OH is their first encounter with linux and that leaves a lot of users behind. settings, on the other hand, is understandable to everyone.

[jimtng]

@florian-h05 wdyt about renaming main UI Settings to Configurations?

[jimtng]

secret.yaml as a special filename to store passwords / secrets / keys, to be referred to from the main yaml files\n\nThere is an open issue that I thought had some recent activity on it for secrets handling in OH core but I think I have it confused with the RBAC PR. #3289 appears to not have had any activity for a year so I guess it's dead. But there might be something that can be salvaged from there, and it would be really great if a whole OH solution that could be used by add-ons and rules were implemented.

Wow thanks for this. It's much bigger but nice.

In that case I might leave secrets out for now and implement that later. I'm just wondering though that the linked topic whilst sounding awesome, would also be far more complicated for the average user to manage.

My current plan was to copy the idea from esphome in how the secrets are managed (simple yaml files in the same directory) but that would be very file/yaml specific.

Maybe we need a special conf/secrets and managed structurally with formal subdirectory structure, e.g secrets/bindingid.secrets.yaml etc.

and for the config file secrets, conf/secrets/settings/ (or whatever the name we'll end up deciding)

[Nadahar]

wdyt about renaming main UI Settings to Configurations?

I think "Configuration" would be better.

[rkoshak]

but that would be very file/yaml specific.

That's my main concern. I'm not against it by any means, but it sure would be nice if everyone could use it everywhere, not just users of this new file format. But something is better than nothing.

And since the linked issue appears dead, waiting forever is basically accepting nothing. I hate when the perfect becomes the enemy of the good enough.

[jimtng]

I might as well create a secrets registry, with a yaml secrets provider. Since that then becomes a bit more involved, I'll split it into a separate PR. That would simplify the rest (for now).

[jimtng]

PR created:

• Yaml config enhancements to support variable substitutions, includes and packages #4818

[Nadahar]

Regarding the naming issue, although I know that everybody is fed up with it, I just discovered that the plan to remove the parser for the "old" YAML format for rule templates can't be done at this time without breaking the parsing of the rule templates currently on the marketplace. Since this "new format" isn't yet implemented for the marketplace, this would break YAML based rule templates on the marketplace completely.

There are also lots of UI widgets on the marketplace in the "old" YAML format. So, I don't think it's true that OH will have "just one YAML format" in the foreseeable future.

[rkoshak]

There are also lots of UI widgets on the marketplace in the "old" YAML format. So, I don't think it's true that OH will have "just one YAML format" in the foreseeable future.

It's not unreasonable to require a translation for OH 5 marketplace rule templates. Like I said above or somewhere, rule templates should probably be reassessed for major OH version changes anyway. But if that doesn't happen right now, 🤷‍♂ it's going to have to happen eventually.