Upper level abstraction for environments

[andy-shady-org]

Hi,

It would be nice to have another level of abstraction that handles 'environments'.

For example, many companies now have multiple overlapping DNS environments, such as Cloud and On-Premise, where views are not sufficient for zone identification.

An example might be for 'example.com', where a sub-zone 'cloud.example.com' is delegated from on-premise DNS to a cloud DNS provider. In this case, the zone 'cloud.example.com' would have to exist as a 'delegated zone' inside the on-premise DNS servers, and also within the cloud provider DNS.

This could be separated with a top level of abstraction to define different environments:

• Environment/View/Zone

In this case, it would be possible to have 'cloud.example.com' in 2 different environments.

Is this a feature that could be added? Im more than happy to contribute if you think this is something that is acceptable.

Thanks

Andy

[peteeckel]

Hi @andy-shady-org, thanks for raising this idea. But I must admit that I don't see the proposed functionality yet.

Views are not suitable for this use case, as they would completely isolate the zones, that's obvious. Delegation from zones in NetBox DNS to externally managed zones can be done right now. So that's not the issue as well.

So I assume that you want to have two different 'cloud.example.com' zones with different records, but in the same view, correct? How would you like PTR records to be handled in this case? The logical functionality would be to have separate sets of reverse zones for the two forward zones as well, correct? How would the delegation to these zones look? What are the NS records in example.com for the two 'cloud.example.com' zones?

Automatically setting up delegations is one thing that is planned for the forseeable future anyway, and this seems to be closely related to it - so yes, integrating that kind of functionality would be a good idea IMHO.

[andy-shady-org]

We have several environments right now, and we use Infoblox. So, lets say we have an external facing grid, internal grid, and cloud grids. IP space, and therefore zones can and do overlap within the internal on-prem grids and cloud grids.

So, if we have say within the on-prem grid, 10/8 in multiple views, and then we have 10/8 also within the cloud grid, even in a single view, and these have different records, we would need to identify by environment. We can route these using NAT, if required.

Sometimes things need to be resolvable in both envs, sometimes differently, sometimes delegation is enough however, where on-prem can delegate the zone to the cloud grid. Not in all cases though :)

So, having a 'top level' environment abstraction, would allow us to have zones in netbox per environment, from which we can do provisioning. The environment would define which grid the record should go onto, or come from, and then its just the normal view, zone type affair.

Hope this makes more sense now.

[peteeckel]

Hi @andy-shady-org, thanks for the attempt to explain the issue, but to be honest I still don't get the proposed change in functionality for NetBox DNS.

The question is: What consequences would being in different environments have for the records in a zone (with or without a view)? For your n instances of 10/8, what's the difference between being in a different view and being associated to a different environment?

Or do you only need another field in which you can specifiy the environment for a zone/view combination? In that case - what's the downside of just defining a custom field for each zone that specifies the environment (or grid, or whatever) so you can use it when you are provisioning?

[andy-shady-org]

Hi @peteeckel, thank you for your response.

I'll try and give some more context.

Different views are within the same instance, different environments contain different instances but may contain the same zone names within the same views.

My specific 'cloud' example, is down to an overlap of IP space used between cloud hosted services which use 10/8 and on-prem hosted services which also use 10/8. Today within the plugin, I can only manage one reverse zone 'default/0.10.in-addr.arpa' zone, which isn't allowing me to manage these independently, as this zone in reality exists on different instances, and in multiple views within the on-prem instance.

This leads to the requirement whereby I need to have 2 copies of this zone, identified as 'default/0.10.in-addr.arpa'.

So, an on-prem host which may be called server1.example.com and have 10.0.1.2, and a cloud host which maybe called server1.cloud.example.com and have 10.0.1.2, can only have a single entry today in 'default/0.10.in-addr.arpa', but really, it needs 2 unique entries thereby driving the requirement to have 2 unique reverse zones.

The reverse is only relevant to the environment but is relevant nonetheless. Creating a new view for the cloud instance means that code handling the provisioning would have to adjust the view for that instance internally, which is possible, but not ideal (mapping known view name of 'cloud_default' -> 'default' for example).

Allowing for uniqueness of a zone to be environment/view/name would allow for the above scenario.

Finally, I do not think this cannot be handled by a custom field I think, as the unique zone is based on name and view, I would not be able to define multiple instances of 'default/0.10.in-addr.arpa', with different parameters, and this is the restriction. Defining a custom field will not overcome this limitation. However, if there is a way to do this, by simply specifying a custom field, and judging uniqueness on view/name/custom field then it would work, but I am not aware of this being possible.

I hope this makes it clearer.

Another example, maybe when a Company A manages the reverse zones for a Company B, C, D and E, and these reverse zones overlap as above. Company B hosts its DNS in a default view, as does Company C, D and E. All are using 10/8 space, and have the reverse zone requirement. Company A can manage each of the zones, without having to handle mappings internally, through a higher level 'environment', for example Company Name/default/zone name.

Thanks

Andy

[tobus3000]

Hi guys,
I really hope you can come to an agreement on this. :-)
The mentioned feature would indeed be a massive improvement.
We run a complete internal DNS root infrastructure where we have break-out points to 3rd parties in each relevant country.
Because of the internal root, we do have zone X as a delegation to the Extranet environment. On the Extranet env in the DMZ, we basically do what we call "DNS routing" by configuring forward zones who then forward to 3rd parties..
Meaning, we do have a separate Extranet environment with one view per country. Inside these regional views we have the zones with forwarders set per region. In some cases we expect a different DNS response for the same name, depending on the region (i.e. market feeds)..
To further complicate things, we do have cloud providers where we run conditional forwarders to forward our namespace back to our on-prem root DNS.

So, the setup could look like this (simplified):

Env	View	Zone	Zone Type	Notes
Internal Root	default	thirdpartyx.com	Delegation	Internal root delegation under 'com.'
Extranet DMZ	china	thirdpartyx.com	Forward	Forwarding to 3rd party DNS through VPN in china.
Extranet DMZ	usa	thirdpartyx.com	Forward	Forwarding to 3rd party DNS through VPN in usa.
Extranet DMZ	default	thirdpartyx.com	Forward	Catch all view. If nothing in there, return NXDOMAIN.
Cloud Provider A	default	thirdpartyx.com	Forward	Conditional forwarding back to on-prem DNS root.
Cloud Provider B	default	thirdpartyx.com	Forward	Conditional forwarding back to on-prem DNS root.
Random Company just bought and needs to be integrated	default	thirdpartyx.com	Forward	Conditional forwarding to our on-prem DNS root.

The example above also holds true for the various reverse zones in each respecitve environment.
Not to mention the duplication in RFC1918 reverse zones when we integrate the "Random company just bought"..

If I'm not understanding Andy's proposal wrong, it would make it possible for us to properly document our setup in netbox.

I hope this helps and the feature finds it's way into the plugin! :-)

Thanks!
tobi

[peteeckel]

@andy-shady-org: Thanks. If I get your point correctly now, the environment would be something like a namespace for views, functionally equivalent to some mapping like:

Environment	View	Mapped View
env1	view1	env1:view1
env1	view2	env1:view2
env2	view1	env2:view1
...

The advantage would then lie in the option to have identical view names in different environments, which is currently not possible. The workaround is to modify the view names in the provisioning process. Correct?

[andy-shady-org]

@peteeckel this would be correct, but mapping is less than ideal, as then the netbox data then does not reflect what is actually out there.

Our setup is akin to the setup described by @tobus3000 above, whereby we have multiple environments that we manage, that require views/zone names to overlap. Having this data represented correctly in Netbox simplifies the management of the zone for us, without having to adjust provisioning code mappings to handle new zone onboarding.

[peteeckel]

this would be correct, but mapping is less than ideal, as then the netbox data then does not reflect what is actually out there.

Yes, I see. It's not 'shut up and use the workaround', it's 'this would work exactly like that', which helps understanding how it should be implemented :-)

[andy-shady-org]

I'll let @tobus3000 run with this, as it seems a very similar requirement to me, if not the same, so I think that both of our requirements would be met should you choose to implement the ability to set environments on a per zone basis.

[peteeckel]

@tobus3000: Again, if i got Andys point correctly now, yours is about something else but also interesting (and in some way related). You are introducing the 'zone type', which can be used to specify if you have, for instance, a forward or dynamic zone (a delegation zone would be a 'normal' zone that happens to have delegation records, so it's not so special after all). This is something that isn't covered in NetBox DNS at all at the moment.

Can we move that to another discussion to keep things separate? Currently I'm solving this with custom fields and some special handling in the provisioning code, but it would make a good enhancement to have something like a zone type field and some specific functionalities for, for instance, forward zones (they would have relaxed requirements for SOA fields), while delegated zones would create delegation records in the parent zones.

[tobus3000]

G'day!

Apols for opening a can of worms here... :-)
While I've only provided the zone type in the above table for illustration purposes, you're right! It would indeed make a good addition to netbox as well..
But my primary focus is on the environment bit that Andy has described.
This becomes more and more important with all the cloud providers today.
Some of them are really not up to speed when it comes to DNS (eg Azure private resolver is pretty useless so far)... This leads to companies spinning up their own caching forwarders in these clouds to cover for the missing functionalities.
It'd be great to be able to also properly document the conditional forwarding zones in these environments.

[peteeckel]

Hi @tobus3000, no need to apologise - this is what the discussions should be about! We just need to keep track of the ideas, and when there are too many in one thread the discussion tends to diverge in too many directions and some things will eventually get lost.

[peteeckel]

I mulled a bit about this, and I don't think it will make it very high on the priority list.

1. Implementing the Environment hierarchy in NetBox DNS code will require a substantial amount of new and changed code (I just started implementing it, just to get a feeling for it). A new model (with views, forms, tables etc.) is required, the existing models View, Zone and Record all need to be adjusted to have optional references to an Environment instance, all import and API views and forms need to be extended, and everything needs to be covered with tests (that also get a lot more complicated, as the number of test cases needs to be multiplied by the number of combinations for the environment. We're talking about hundreds of test cases, just in case you're asking yourself ...).
2. While I see the advantage of having another abstraction layer for the use cases you described, it's a very special case that won't affect the majority of users, so I'm not convinced it's worth the effort. There are other things (e.g. RFC2317, which has been dormant for far too long now) that are more pressing, even more so as there is a simple and efficient workaround for the missing environment layer (see below).
3. Having the optional environment layer in the hierarchy will make the forms for creating, editing and bulk editing zones and records more complicated and increase the amount of validation checks that need to be done when entering and especially importing and bulk editing zones and records - which negatively impacts performance, and it does so for everyone. Improving performance especially for bulk operations is much higher on my list of priorities than this.

Regarding the workaround for not having Environment I'd suggest the following setup:

• Create custom fields environment and local_name or similar for views
• For each view view that exists in a specific environment env, use env:view as the view's name, which makes it unique
• Adjust the provisioning to use local_name instead of name
• Optionally just enter the name of a view as env:view and let a custom script/webhook take care of filling the custom fields automatically

You can then search or filter for (local) names and environments, have multiple instances of identically named zones (with the same local_name, but distinct environment) and views for different environments and the disadvantages outlined above do not apply.

And the best of it is that you can have it today, as it takes only a couple of minutes to set up.

[tobus3000]

No worries! You set the priorities. No pressure. :-)
I'll see what I can do with the proposal you have laid out above.
Thanks for the analysis of what would have to be done to add the feature.

[peteeckel]

Thanks for your patience and understanding! If you get stuck with my suggestion feel free to come back to it - maybe we can find some ideas together that may also help others with the same challenge.

[andy-shady-org]

@peteeckel I would also like to thank you for taking the time to evaluate this request. Its very much appreciated.