concatinating json arrays

[damo2929]

NetBox version

V3.51

Feature type

Change to existing functionality

Proposed functionality

would it be possible to combine values in a json array like it is possible to combine json object attributes.

example

have a config context with

{ 
  "sudo: [
       'group1' 
  ]
}

and a second one with

{ 
  "sudo: [
       'group2' 
  ]
}

desired outcome is

{ 
  "sudo: [
       'group1' ,  
       'group2'
   ]
}

Use case

this would allow for adding rules based on properties in which both the MSP and the Tenant could need sudo access for instance

Database changes

No response

External dependencies

No response

[candlerb]

I think that would require something like JSONNET instead of JSON - or to be able to provide custom code for the merging function.

[kkthxbye-code]

There is currently no code for merging, it's all done in postgres with the jsonb_agg function, which is where the limitations of deep merging comes from. To achieve any changes in that regards would likely be a larger undertaking, and if the merging has to be done in code, it might be hard to keep performance in check.

[candlerb]

I suspect any performance difference would be immeasurably small, as it would be when rendering individual devices. At worst, you could use a postgres stored function, but I can't see it would be needed.

[kkthxbye-code]

Config contexts are included by default in the device list API.

Several ideas have been proposed, but if using a different dialect of json, I assume the column type would have to be changed from jsonb to text or similar and all processing would have to be done in python code.

A fully functional PoC would probably be the first step to get a FR accepted, as to me it sounds like a lot of work to implement.

[andershagman]

I have a similar problem. I use a Netbox to Zabbix integration and it uses config context to list what management templates to add to an object in Zabbix. Normally, for devices, we use to factors to decide what templates to use and that is device type and role.
If you have 4 different device types and 4 different roles, you end up with 16 different config context objects.
It would be simpler to do 4 device type and 4 role config context objects and combine them.
Here is a module with these properties: https://pypi.org/project/jsonmerge/

Device type 1

{
  "zabbix" {
      "templates": [
           "Template dev-type 1"
          ]
     }
}

Role 1

{
  "zabbix" {
      "templates": [
           "Template role 1"
          ]
     }
}

Devices of type 1 with role 1

{
  "zabbix" {
      "templates": [
           "Template dev-type 1",
           "Template role 1"
          ]
     }
}

[candlerb]

You could do this by having "device_role_templates" and "device_type_templates" and combining them externally in your Zabbix integration.

[damo2929]

There are multiple use cases for the so what your saying is have config context sprawl, or do the config management outside netbox. Kind of defeats the point of the config context feature. Sent from Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Brian Candler ***@***.***> Sent: Saturday, June 3, 2023 5:36:07 PM To: netbox-community/netbox ***@***.***> Cc: Damo Dye ***@***.***>; Author ***@***.***> Subject: Re: [netbox-community/netbox] concatinating json arrays (Discussion #12719) You could do this by having "device_role_templates" and "device_type_templates" and combining them externally in your Zabbix integration. — Reply to this email directly, view it on GitHub<#12719 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKPM7ZA2EIOMROT5TFJ74O3XJNRXPANCNFSM6AAAAAAYZH3YZQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[andershagman]

I could but the integration is also a python package from Internet and changing the code either in Netbox or in the Zabbix-Netbox isn’t desirable. I rather discuss a change in Netbox to become even better. MHV /Anders Hagman

…

3 juni 2023 kl. 18:36 skrev Brian Candler ***@***.***>:  You could do this by having "device_role_templates" and "device_type_templates" and combining them externally in your Zabbix integration. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[candlerb]

I rather discuss a change in Netbox to become even better.

The trouble is, I don't think there's a new behaviour which is unambiguously "better" in all cases for merging raw JSON objects.

Say for example, at one level I have "dns_servers": ["8.8.8.8","8.8.4.4"]. At another level, I want to override this with "dns_servers": ["1.1.1.1","1.0.0.1"]. I don't want them to be merged into a list of 4 dns servers.

Therefore, I think the starting point would be to dinfe exactly what semantics you want 'object merge' to have, and exactly what extra knobs or configuration settings you want to have, so that it doesn't break existing users of config contexts.

This is a problem which other projects have found complex to solve. For example in kubernetes / kustomize there is inline patch, strategic merge patch, and JSON patch (RFC6902). Another solution would be to use JSONNET instead of JSON.

[damo2929]

This is definitely in netbox, when you do to the device context tab on a device or VM. The issue is netbox replaces the config in that context rather than combine as such this is a netbox defect because it correctly combines keys on 1st level objects. So why have to do the rest outside by having to make multiple API call to get round the a incorrect calculation on the main object is a netbox issue. Either that or the documentation needs to reflect that this feature is limited only works on 1 level object keys, not with arrays or sub objects as they are taken as a key and replaced as a redeclaration rather than follow the same behaviour of 1st level keys which only replaces values but combines the object. Sent from Outlook for Android<https://aka.ms/AAb9ysg>

________________________________ From: andershagman ***@***.***> Sent: Saturday, June 3, 2023 7:13:55 PM To: netbox-community/netbox ***@***.***> Cc: Damo Dye ***@***.***>; Author ***@***.***> Subject: Re: [netbox-community/netbox] concatinating json arrays (Discussion #12719) I could but the integration is also a python package from Internet and changing the code either in Netbox or in the Zabbix-Netbox isn’t desirable. I rather discuss a change in Netbox to become even better. MHV /Anders Hagman

3 juni 2023 kl. 18:36 skrev Brian Candler ***@***.***>:  You could do this by having "device_role_templates" and "device_type_templates" and combining them externally in your Zabbix integration. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

— Reply to this email directly, view it on GitHub<#12719 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKPM7ZCQXLAUGGRZE5RFYY3XJN5GHANCNFSM6AAAAAAYZH3YZQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[andershagman]

You are right. There are conflicts but we have weight. Example: Same weight - append Different weight - overwrite Device local - overwrite MHV /Anders Hagman

…

3 juni 2023 kl. 21:12 skrev Brian Candler ***@***.***>:  I rather discuss a change in Netbox to become even better. The trouble is, I don't think there's a new behaviour which is unambiguously "better" in all cases for merging raw JSON objects. Say for example, at one level I have "dns_servers": ["8.8.8.8","8.8.4.4"]. At another level, I want to override this with "dns_servers": ["1.1.1.1","1.0.0.1"]. I don't want them to be merged into a list of 4 dns servers. Therefore, I think the starting point would be to dinfe exactly what semantics you want 'object merge' to have, and exactly what extra knobs or configuration settings you want to have, so that it doesn't break existing users of config contexts. This is a problem which other projects have found complex to solve. For example in kubernetes / kustomize there is inline patch, strategic merge patch, and JSON patch (RFC6902). Another solution would be to use JSONNET instead of JSON. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[candlerb]

That's still not a precise definition of the semantics. For example, if you have two items of the same weight, then what order will you append the lists?

If we accept a limitation that JSON keys cannot end with "+" then we can borrow rules from JSONNET.

• "foo": ...: replace "foo" entirely
• "foo+": ["xxx", "yyy"]: append to end of "foo" (if existing "foo" is an array)
• "foo+": {"bar": "baz"}: merge sub-object into "foo" (if existing "foo" is an object)

[andershagman]

The rendering order today seems to be some sort of sorting by the letter.

Here are two objects with the same weight different names. The result is dns-2.
If I change the name of dns-1 to dns-3 it will win. So in practice we already have an order.

dns-1

{
    "DNS-servers": [
        "10.1.1.1",
        "10.1.1.2"
    ]
}

dns-2

{
    "DNS-servers": [
        "10.1.1.3",
        "10.1.1.4"
    ]
}

Rendered Context

{
    "DNS-servers": [
        "10.1.1.3",
        "10.1.1.4"
    ]
}

[damo2929]

Am happy with that approach as logically that works fine to resolve the conflicts Sent from Outlook for Android<https://aka.ms/AAb9ysg>

________________________________ From: andershagman ***@***.***> Sent: Saturday, June 3, 2023 9:14:25 PM To: netbox-community/netbox ***@***.***> Cc: Damo Dye ***@***.***>; Author ***@***.***> Subject: Re: [netbox-community/netbox] concatinating json arrays (Discussion #12719) You are right. There are conflicts but we have weight. Example: Same weight - append Different weight - overwrite Device local - overwrite MHV /Anders Hagman

3 juni 2023 kl. 21:12 skrev Brian Candler ***@***.***>:  I rather discuss a change in Netbox to become even better. The trouble is, I don't think there's a new behaviour which is unambiguously "better" in all cases for merging raw JSON objects. Say for example, at one level I have "dns_servers": ["8.8.8.8","8.8.4.4"]. At another level, I want to override this with "dns_servers": ["1.1.1.1","1.0.0.1"]. I don't want them to be merged into a list of 4 dns servers. Therefore, I think the starting point would be to dinfe exactly what semantics you want 'object merge' to have, and exactly what extra knobs or configuration settings you want to have, so that it doesn't break existing users of config contexts. This is a problem which other projects have found complex to solve. For example in kubernetes / kustomize there is inline patch, strategic merge patch, and JSON patch (RFC6902). Another solution would be to use JSONNET instead of JSON. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

— Reply to this email directly, view it on GitHub<#12719 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKPM7ZHHIHUSL64KWZNRF2LXJOLKDANCNFSM6AAAAAAYZH3YZQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>