Webhooks No Longer Triggering after v3.7.0 Upgrade/Event Rule Migration

[eramnes]

We have been using Netbox for a while now and had set up some webhooks to call some APIs to perform actions on object updates. These had been working great.

We recently updated to v3.7.0 and with the migration to Event Rules, it seems the webhooks no longer trigger on object update. The auto-creation of the Event Rules from the conditions on the old webhooks was successful, and the conditions are still there, but changing an object so that it would trigger the Event Rule does nothing.

I've checked the request queue after making a change that should match an Event Rule, and there's no request that shows up there, in any state (active, finished, failed, scheduled). I've verified that netbox-rq is running, and the PIDs for the workers show in the request queue interface. I also tried creating a brand new Event Rule with the same conditions and pointing it to the old webhook but that didn't seem to make a difference. We are using custom Status values as the condition for the Event Rule, but I tried using a provided value instead and that didn't seem to help.

Given that I haven't seen anyone else report this as an issue, I think I must be doing something wrong. Can anyone point me in the correct direction? I've tried turning logging on for Netbox and Django but I don't see anything that stands out as having to do with webhooks or Event Rules. If you need any further info I would be happy to provide that.

Thanks.

[DanSheps]

Could you provide the rule and webhook (sanitize as required obvivously)?

I do have a webhook (that does nothing) and that appears to be working as intended

[markkuleinio]

Works here as well: tcpdump shows all attempts to connect, and Failed Jobs show all my attempts: create, update, delete.

[eramnes]

Here is the event rule:

Here is the webhook that the event rule should be calling:

If there's a better way to provide the settings than screenshots please let me know. Thanks!

[candlerb]

any ideas on what else to dig for?

Nope: if it was affecting me I would be sprinkling log or print statements around the code to try to understand where the webhook gets dropped.

[adparis99]

Do you mean that they all originally had a condition configured?

Yes, that is correct.

Do you have any webhooks without conditions? (If yes, are they working?)

No we do not. I did try removing conditions from various webhooks in the troubleshooting process, but we didn't have any webhooks pre-upgrade without conditions.

Any idea what was the change after which it started to work?

At the time I discovered there was an issue, I had all four webhooks enabled, each targeting dcim.interfaces. At some point I thought that maybe having all four was causing some sort of conflict, so I disabled all but the webhook that is currently working (that is, the one triggered for interfaces updated with an 802.1x type of access). That webhook continues to work now, even with the others enabled. So it's possible that disabling and re-enabling those had some kind of effect. It's also possible that the webhook that's working now was always working. I feel like I tested it and didn't see results, but I've done so many variations that it's possible I overlooked it. In any case, now I can shift my attention to figuring out why only that 802.1x type is triggering and not the others.

[DanSheps]

Is "802.1x type" a custom field? 802.1x is access control (nothing to do with access, tagged)

[adparis99]

@DanSheps Sorry, definitely meant 802.1Q mode.

[DanSheps]

Ah yeah, that makes sense!

[markkuleinio]

@eramnes @adparis99 What was the earlier NetBox version you had before upgrading to 3.7.0?

[eramnes]

We were on 3.6.5 before upgrading. I did go ahead and roll back our Netbox instance to the 3.6.5 backup, and I can confirm that reverting fixed the issue I was seeing with the webhooks. It's Saturday here, but I am planning to try the 3.7.0 upgrade again this coming week, however I'm going to delete all webhooks before the upgrade and then recreate them once on 3.7.0 to see if that makes a difference.

If doing an interim upgrade first before moving to 3.7.0 might help (e.g. 3.6.5 -> 3.6.9 or something) I can try that as well. Also if anyone has any thoughts on where to put in some logging statements I would be willing to do that as well.

I have been wondering if somehow the Webhook model changes didn't quite complete successfully during the upgrade, and there are still pieces of the old model associated with the webhooks that don't work. Just a guess as I'm not familiar with how the code really functions. In any case, if removing the webhooks first and recreating them after works, that will be a solution for us.

[markkuleinio]

Ok, I just did a quick 3.6.5 install (using https://github.com/markkuleinio/netbox-freezes/blob/main/netbox-3.6.5.txt), added a working webhook (with a condition on prefix status "deprecated"), then upgraded to 3.7.0, still works. So there is something more complicated behind this.

[adparis99]

Honestly - I can't remember. It was either 3.6.7 or 3.6.8.

[adparis99]

@candlerb @markkuleinio I believe I've found a way to replicate the behavior I and @eramnes have been seeing in the NetBox demo instance. To put it succinctly, I suspect that when multiple event rules with different conditions are applied to a single model, only the first event rule is being evaluated (and executed, should the condition match). In my example, I'm using dcim.interface as the model, and the 802.1x mode as the conditions (access, tagged, and tagged-all).
Would you mind trying the steps below and note if the behavior matches my description?

In my testing, event rules do not trigger (or more specifically, a webhook is not generated) for tagged interfaces after following these steps, but they do trigger for access interfaces:

1. Create two webhooks pointed toward a webhook receiver
2. Create two event rules
   • Configure the first rule to be enabled, associated with first of the two new webhooks, triggered by updates to dcim.interfaces, and with the condition below:
     • { "and": [ { "attr": "mode.value", "value": "access" } ] }
   • Configure the second rule the same, but associated with the second of the two new webhooks, and with the condition below:
     • { "and": [ { "attr": "mode.value", "value": "tagged" } ] }
3. Update an interface with an 802.1x mode of access, and then update an interface with an 802.1x mode of tagged
4. Check webhook receiver. There is a webhook based on the interface with mode access, but not with mode tagged

If I swap the conditions between the two rules - that is, I put the tagged condition on the first event rule that I created, and access on the second rule - the behavior swaps as well. I can then generate webhooks for tagged interfaces, but not for access.

[markkuleinio]

Good catch.

Issue #14827 opened, please comment there as well if you have something to add.

[kuhball]

We have encountered a maybe related issue. Recently upgraded our netbox prod instance from 3.6.6 to 3.7.1. Doing a bulk edit on 2 interfaces resulted in only one webhook. Expected are 2 webhooks, one for each interface.

Unfortunately I wasn't able to reproduce this with the following steps:

• netbox 3.6.6 with imported demo data
• create a webhook for interfaces pointing to http://127.0.0.1:9000
• try a bulk change -> 2 webhooks
• upgrade to 3.7.1
• try a bulk change also 2 webhooks

Importing our database into a fresh 3.7.1 docker netbox instance recreates this problem. Was also thinking this could be a problem with having multiple webhooks for the same object types (we currently have 3 webhooks for interface object types). But this was also not reproducible.

The solution for us was to delete all the event rules and recreate them. Afterwards all bulk edits produced the expected webhooks for all edited objects.

Posting this mostly if other users happen to have a similar problem.