API returns JSON custom field as string on certian ipam.prefix entries

[FliesLikeABrick]

Deployment Type

Self-hosted

NetBox Version

3.7.7

Python Version

3.10

Steps to Reproduce

I do not know what preconditions led to this.

We recently upgraded from NetBox 3.6.3 to 3.7.7, and now some of our tooling is receiving unexpected string responses for fields that should be JSON. If there were prior actions taken in our data to create a precondition for this, we do not know what it was.

Expected Behavior

For a JSON-typed custom field, we expected the API to return this custom field as JSON payload instead of a string

Observed Behavior

>>> for prefix in pynetbox_instance.ipam.prefixes.filter('10.30.15'):
...   print(f"{prefix} custom_fields['references'] type: {type(prefix.custom_fields['references'])}")
... 
10.30.150.0/24 custom_fields['references'] type: <class 'dict'>
10.30.151.0/24 custom_fields['references'] type: <class '**str**'>
2606:ae80:1415:150::/64 custom_fields['references'] type: <class 'dict'>
2606:ae80:1415:151::/64 custom_fields['references'] type: <class 'dict'>

[FliesLikeABrick]

The database values unsurprisingly appear to be handling this differently

These are the same prefixes as listed above:

>>> for prefix in pynetbox_instance.ipam.prefixes.filter('10.30.15'):
...   print(f"{prefix.id} custom_fields['references'] type: {type(prefix.custom_fields['references'])}")
... 
8814 custom_fields['references'] type: <class 'dict'>
8815 custom_fields['references'] type: <class 'str'>
20521 custom_fields['references'] type: <class 'dict'>
8816 custom_fields['references'] type: <class 'dict'>
>>> 

Here are the relevant prefix rows' custom_field_data, the broken prefix of 8815 has the 'references' custom field value as an escaped string instead of continuing the native JSON representation.

This data has not been updated in some time (as evidenced by our 'last updated' date embedded in the data'), I believe the handling of this data changed from 3.6.3 to 3.7.7 (or a database migration was applied?)

netbox_prod=> select custom_field_data from ipam_prefix where id=8814
                                                                                                           
                                    custom_field_data                                                                                                                                             
                                                                                          
-------------------------------------------------------------------------------------------------------------------------
 {"vlan_id": 150, "last_audit": "2024-05-28", "references": {"sw-vm01a.sjc.cnvr.net": {"updated_date": "2023-02-06", "interface_name": "Vlan150", "interface_description": "FutureVMHost:10.30.150
.0/24"}, "sw-vm01b.sjc.cnvr.net": {"updated_date": "2022-05-12", "interface_name": "Vlan150", "interface_description": "FutureVMHost:10.30.150.0/24"}}, "data_source": "sw-vm01a.sjc.cnvr.net inte
rface via cnvripam", "equivalent_mgmt_prefix": null, "associated_ipv6_prefixes": [20521]}
(1 row)

netbox_prod=> select custom_field_data from ipam_prefix where id=8815;
                                                                                                                                                                                                  
                                                custom_field_data                                                                                                                                 
                                                                                                                 
-------------------------------------------------------------------------------------------------------------------------
 {"vlan_id": 151, "last_audit": "2024-05-06", "references": "{\"sw-vm01a.sjc.cnvr.net\": {\"updated_date\": \"2023-02-06\", \"interface_name\": \"Vlan151\", \"interface_description\": \"WindowsV
M:10.30.151.0/24\"}, \"sw-vm01b.sjc.cnvr.net\": {\"updated_date\": \"2022-05-12\", \"interface_name\": \"Vlan151\", \"interface_description\": \"WindowsVM:10.30.151.0/24\"}}", "data_source": "sw
-vm01a.sjc.cnvr.net interface via cnvripam", "equivalent_mgmt_prefix": null, "associated_ipv6_prefixes": [8816]}
(1 row)

netbox_prod=> 

[FliesLikeABrick]

A new bug issue has been opened after we investigated and found the actions causing this corruption
#16640