sync in a keepalived cluster

[tessus]

I have 2 piholes in a keepalived cluster. Thus the primary might switch, but it seems that nebula-sync requires a fixed primary. I could use the virtual IP, but then I would have to use both nodes in the replicas config, which would mean that it syncs with itself. Thus this idea is out.

Having used gravity-sync before, nebula-sync's architecture is not yet clear to me. It seems to me that nebula-sync does not perform a sync, but rather a push to all replicas. Is this correct?
Do I install and run it only on the primary?

In a 2 way setup, this does not really matter that much, since a failover happens when the FTL process is dead, thus syncing (or pushing) won't work. Although it will be a problem, when the original primary comes back online.

But I was thinking of adding a 3rd one to the mix, in which case it starts to make more sense. However, if it is push-only architecture, I will run into issues (no matter how many nodes).

Scenario: primary goes offline, one of the 2 other members takes the primary role. First problem: the new primary does not know that it is the primary. I could script something and start nebula-sync on the new primary.
After a few days the original primary comes online. Second problem: nebula-sync is still running on one of the replicas. Third problem: nebula-sync is started on the original primary and pushes a days-old config to the replicas and wipes out all changes made during the time the original primary was offline.

As you can see, I am unsure how to run nebula-sync in a keepalived cluster.
My current setup syncs between the 2 nodes, no matter who the primary is. The latest changes are always applied to the other node.

I really would like to use nebula-sync in my 2 node setup, since gravity-sync throws a few errors now and then with pihole 6.

Any pointers, ideas, dark magic?

[tessus]

@lovelaze should I have opened an issue instead?

[lovelaze]

Hi @tessus, I think this is a good place to discuss this.

Having used gravity-sync before, nebula-sync's architecture is not yet clear to me. It seems to me that nebula-sync does not perform a sync, but rather a push to all replicas. Is this correct?

Correct, you can call it a unidirectional "sync"/"push" from primary to replica(s)

Do I install and run it only on the primary?

Yes and no. You only need one instance of nebula sync, but it does not necessarily need to be installed on a Pi-hole host since it uses the API to communicate. It can be deployed separately.

Any pointers, ideas, dark magic?

I don't personally run keepalived for Pi-hole, but it sounds like you are over complicating it a bit.

I would set it up similar to this:

• Setup a virtual IP for all your Pi-hole hosts using keepalived
• "Pick" one of these to be the primary and the rest your replicas
• Configure nebula-sync with the correct hosts

This will make your DNS HA in the case of a Pi-hole failure with the caveat that you will need to only make configuration changes to the master Pi-hole host that nebula-sync uses to sync to the replicas.

[tessus]

Thanks a bunch for the reply.

I think this is a good place to discuss this.

Thanks. I have noticed this depends a lot on the project, as some like to monitor just issues and convert issues with questions to discussions.

Correct, you can call it a unidirectional "sync"/"push" from primary to replica(s)

Thanks. gravity-sync checked the changes and applied (synced, not pushed) them accordingly, thus I didn't have to worry about losing any data (changes).

Yes and no. You only need one instance of nebula sync, but it does not necessarily need to be installed on a Pi-hole host since it uses the API to communicate. It can be deployed separately.

Interesting, so it retrieves the config via the API from the primary and pushes it to the replicas. This could make the keepalived cluster setup easier.

I would set it up similar to this

Thanks, I read this article about the same time I created this discussion. This setup has exactly the issue I was describing - which you also mentioned as a caveat just now.

I access the pi-hole's web interface for configuration via the virtual IP, so unless I check the hostname I don't know whether it is the primary or not. And if I have to change something, I might have to do it right away. e.g. yesterday the upstream servers (1.1.1.1) were down and my network couldn't resolve anything. I couldn't have waited for the primary to be back up, if one of the replicas were actually the active node.

While I understand the "you will need to only make configuration changes to the master Pi-hole" restriction, it is too error prone, especially if a keepalived cluster automates failover and was primarily developed to not require/make any manual changes.

I have to think about it a bit more, but I am afraid using nebula-sync in a fully automated way is impossible in a cluster setup. While I could run nebula-sync on a separate VM or LXC, change the config file when a node becomes the primary, and restart the service automatically, there is still an issue that cannot be resolved. nebula-sync is not triggered when a change is applied to pi-hole, but runs manually or on a schedule. This means that data (configuration settings) can be lost. Scenario: nebula-sync is scheduled to run at 18:00 every day. I make a change at 16:00 and the primary goes offline at 17:00. My automation had made one of the replicas the new primary, so at 18:00, the changes I made at 16:00 are reverted. The same would be true, if I made any changes after 18:00 and the node failed over again before 18:00 the next day.

Does the API tell you when the last change was? In that case, nebula-sync could block a push to a replica that has a more recent change timestamp (and use a different return code and allow that code to be sent via a webhook). Such a behavior could be enabled/disabled via a config parameter.

[lovelaze]

I access the pi-hole's web interface for configuration via the virtual IP, so unless I check the hostname I don't know whether it is the primary or not. And if I have to change something, I might have to do it right away. e.g. yesterday the upstream servers (1.1.1.1) were down and my network couldn't resolve anything. I couldn't have waited for the primary to be back up, if one of the replicas were actually the active node.

While I understand the "you will need to only make configuration changes to the master Pi-hole" restriction, it is too error prone, especially if a keepalived cluster automates failover and was primarily developed to not require/make any manual changes.

Please correct me if I'm wrong - you want the virtual IP to load balance all requests and to have the Pi-hole nodes serving requests be in a high-availability or fail-over cluster. While that makes a lot of sense I believe the latter part is something that would not make sense to have nebula-sync responsible for.

Really what gravity/orbital/nebula-sync achieves is to make up for the fact that Pi-hole any form of HA natively - but it only does so partially - like you have discovered. My guess would be that for the absolute majority of nebula-sync users it is sufficient to have a primary and secondary dns distributed by dhcp, only configure changes on the primary and pretty much forget about the secondary.

nebula-sync is not triggered when a change is applied to pi-hole, but runs manually or on a schedule.

This would be real nice to have and I wanted to implement this, but AFAIK the Pi-hole API does not provide functionality to support this.

Does the API tell you when the last change was?

No, it does not.

I have to think about it a bit more, but I am afraid using nebula-sync in a fully automated way is impossible in a cluster setup.

Yeah, I agree. Wouldn't mind adding support to nebula-sync to help you achieve this, but I don't see a clear path forward and it feels like what you're looking for probably should not be in a third party app for Pi-hole. Perhaps you could look at something similar to BIND9 with a master/slave setup.

[tessus]

you want the virtual IP to load balance all requests and to have the Pi-hole nodes serving requests be in a high-availability or fail-over cluster.

Yes and no. The virtual IP does not do load balancing. Load balamcing would mean the LB is a single point of failure, unless you build another failover cluster for the LB. Additionally, UDP is a bit tricky when it comes to load balancing. HAproxy supports UDP for DNS only in their enterprise version. I rather use Traefik in Nomad envs. Never used LVS. DNSdist would probably be the best option.
But I am not adding another component (which would require another failover cluster)...

keepalived does the failover of the virtual IP when triggered by events. In my case, the failover happens, if the FTL service is down or dies. I have usually less than 25,000 DNS queries a day, thus I certainly do not need an active-active solution.

have a primary and secondary dns distributed by dhcp

Yes, I am not so happy with this approach. Depending on the local DNS resolver process, a switch to the next DNS IP (e.g. let's say your DHCP provided 2 DNS IPs) might take a while, during which time name resolution does not work. Also, it usually does not switch back, if the first one becomes available again.

Wouldn't mind adding support to nebula-sync to help you achieve this, but I don't see a clear path forward

Thanks, but I certainly do understand that you rely on the API. Since the API does not provide what is needed, there is nothing any of us can do.

Automation is out when it comes to pihole sync in my cluster env, so I have decided to do the following:

• whenever I make a change on pihole, I check the hostname/ip of the active node
• then I run nebula-sync manually with the config (env) file that uses the hostname/ip of the active node as primary

[HanSooloo]

Did you ever consider running nebula-sync on a schedule on your "main" node? I have the exact same setup as yours and wanted to have a periodic sync from "main" to "backup", regardless of the failed state, i.e., pi-hole-01 is always "main" and pi-hole-02 is always "backup".
However, I couldn't find any examples of a systemd service for nebula-sync. Any thoughts?

[tessus]

Creating a systemd service is not very complicated. But the question is the following: Do you want nebula-sync to run with the CRON env var set, or do you want to run it via your crontab?

Here is an example of a unit file that should work with nebula-sync:

[Unit]
Description=nebula-sync
After=network.target

[Service]
Type=simple
User=nebula   # remove, if you don't use a separate user
Group=nebula  # remove, if you don't use a separate user
EnvironmentFile=/etc/nebula-sync/nebula-sync.cfg
ExecStart=/usr/bin/nebula-sync    # adjust to where your binary is located
StandardOutput=append:/var/log/nebula-sync/nebula-sync.log
StandardError=append:/var/log/nebula-sync/nebula-sync.log
Restart=on-failure
RestartSec=3
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

If you don't use the CRON env var, nebula-sync terminates after each run, so you would rather have to create systemd timer unit file, instead of a service unit.

Did you ever consider running nebula-sync on a schedule on your "main" node? I have the exact same setup as yours and wanted to have a periodic sync from "main" to "backup", regardless of the failed state, i.e., pi-hole-01 is always "main" and pi-hole-02 is always "backup".

I see what you are saying. If the primary is down, no sync will be happening anyway. In theory I agree with you. However, in my very long time in IT I have encountered edge cases and unexpected situations more often than I like to admit.
This is too dangerous. To play the devil's advocate here: imagine a failover to the secondary even though the FTL service on the primary is perfectly fine. And we are right at the issue I described earlier.
So that's a no-no for me.

[JamesTX10]

Are you having failover that often? I have Keepalived configured as well but I have my primary PiHole configured in Nebula rather than the virtual IP and would check which PiHole I am making changes to but it pretty much only fails over when I am doing updates on the primary.

[tessus]

No, my nodes do not failover that often. Pretty similar to what you are saying: they failover due to an update and/or reboot, both of which are rather transitory events.

However, I am speaking in terms of cluster automation. While I agree that the cognitive load is minimal to remember on which node I made the config changes and when the last sync was, it still leaves room for mistakes. Mistakes on my part, which I would like to minimize and even eliminate with automation.
This cluster is not the only infra I have and take care off. All these little things add up.

I love fault tolerant systems and automation, including the challenges that come with them.

[JamesTX10]

I run mine the same way. I only publish the VIP as my DNS record rather than the 2 piholes. I do that because there are sometimes when we need to disable blocking for 30 sec to test or access a site that has not been whitelisted yet. My wife is not going to want to log into multiple interfaces to do that. Having the VIP means there is always just one interface to manage at a time.

Another option that could work here would be to have a 3rd pihole setup that is NOT part of the KeepAliveD configuration and is not used for DNS. Use that one as your configuration master in Nebula-Sync and only make changes on that one. Seems like overkill to me but it would solve your concern.

[tessus]

I run mine the same way. I only publish the VIP as my DNS record rather than the 2 piholes. I do that because there are sometimes when we need to disable blocking for 30 sec to test or access a site that has not been whitelisted yet. My wife is not going to want to log into multiple interfaces to do that. Having the VIP means there is always just one interface to manage at a time.

Makes sense.

Another option that could work here would be to have a 3rd pihole setup that is NOT part of the KeepAliveD configuration and is not used for DNS. Use that one as your configuration master in Nebula-Sync and only make changes on that one. Seems like overkill to me but it would solve your concern.

Yep, this could work, but is also a manual process, since I would have to manually sync/push as soon as I make config changes on that 3rd pihole.

I have decided to do the following: see bottom of this comment.