Bulk-importing only certain columns? / Why are a lot of fields required when patching interfaces?

[pv2b]

Hi!

I'm running into an issue in general where bulk-updating specific data seems to either be needlessly hard, or I'm missing something.

The use case is that I want to import some interface descriptions that I have in a CSV file. The interfaces already have a lot of information associated with them, such as cable terminations, already set media types, etc, which makes deleting an re-creating the interfaces impractical. Because I thought ahead, I made sure that I have the ID number of the interface in question in the CSV file, so my file might look something like:

id,description
123,awesome interface
124,more awesome interface

The intent is then to change the descriptions of interface 123 and 124 to "awesome interface" and "more awesome interface" respectively.

My first thought is, no problem, I'll just use the Interface Bulk Import. Unfortunately, that doesn't seem to be very great, because there are a lot of required columns such as device, name and type. It's not clear to me why these fields should be required when performing an update of an existing interface with a known netbox interface ID, but anyway, it does seem to complain if those headers are missing.

Undeterred, I decide to try to explore the underlying API, and I find the API endpoints /api/dcim/interfaces/ and /api/dcim/interfaces/{id} with the PATCH method, but, to make a long story short, it would seem that even those underlying API methods seem to require a lot of information that I'd rather not touch. There doesn't seem to be a way to say "just change the description, leave whatever's already there alone".

So in the end, what I ended up having to do was to write a script that went through my CSV file, did a GET on the interface in question, extracted the media type (that I didn't need to change but needed to be there in the bulk import) and then CSV object including the (spuriously) mandatory parameters in it, and then bulk imported that same into Netbox. This isn't great because you end up telling Netbox what it already knows (and all that implies). It also ended up being very chatty because I couldn't figure out a way to do a bulk-query of objects given an array of IDs (even if the documentation suggests that should be possible).

That seems like a lot of work for what should be something simple, but before I make any suggestions about improvement to functionality or the API, am I missing some already existing functionality that makes these types of operations easy?

[kkthxbye-code]

My first thought is, no problem, I'll just use the Interface Bulk Import. Unfortunately, that doesn't seem to be very great, because there are a lot of required columns such as device, name and type. It's not clear to me why these fields should be required when performing an update of an existing interface with a known netbox interface ID, but anyway, it does seem to complain if those headers are missing.

Just tested this and it worked fine with no issues. I copy/pasted your csv, just changing the id to an exisiting interface.

Undeterred, I decide to try to explore the underlying API, and I find the API endpoints /api/dcim/interfaces/ and /api/dcim/interfaces/{id} with the PATCH method, but, to make a long story short

Just tried this as well and it works with no issues. Example curl command (you need to add the authorization stuff):

curl -X 'PATCH' \
  'https://demo.netbox.dev/api/dcim/interfaces/1/' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "description": "test description"
}'

So the two things you tried and claim do not work - works just fine. You didn't say what version of netbox you are using, but at least the API part should work in all versions.

[pv2b]

Thank you for your example, looking at it I was able to figure out a couple of things that sent me down the wrong rabbit hole with the JSON:

1. I was using "ID" as my key, not "id". Seems the key names are case sensitive. Oops.
2. I was passing the ID number as a string, not as an integer.

Both of these lead to pretty cryptic error messages, so I was probably changing the wrong part of the JSON to try to fix this.

For example, putting the id number as a string will provide this error message which isn't exactly clear:

{
    "non_field_errors": [
        "No data provided"
    ]
}

I interpreted this to mean that the "media" property was missing.

Also when using "ID" as the key, this error message is given:

[
    {
        "id": [
            "This field is required."
        ]
    }
]

In my confusion I didn't notice it said "id" over there, I was still in my wrong mental model that the "media" property is requied.

Anyway, the same thing regarding case sensitivity was causing the CSV import to fail. Turns out my test case wasn't exactly as written, but this test case will fail for example:

ID,description
1393,test description99

This will produce error messages implying that the "media" type is required, and this is how I was confused.

Possibly some better error messages could have helped but it was still my fault.

[candlerb]

My first thought is, no problem, I'll just use the Interface Bulk Import. Unfortunately, that doesn't seem to be very great, because there are a lot of required columns such as device, name and type

AFAIK, those columns are not required when you're updating an existing object (by providing its id). Only when creating new ones.

[mtinberg]

Welcome to computing, its always been like this 😉 — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: pv2b ***@***.***> Sent: Wednesday, August 9, 2023 10:51 AM To: netbox-community/netbox ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [netbox-community/netbox] Bulk-importing only certain columns? / Why are a lot of fields required when patching interfaces? (Discussion #13421) Thank you for your example, looking at it I was able to figure out a couple of things that sent me down the wrong rabbit hole with the JSON: 1. I was using "ID" as my key, not "id". Seems the key names are case sensitive. Oops. 2. I was passing the ID number as a string, not as an integer. Both of these lead to pretty cryptic error messages, so I was probably changing the wrong part of the JSON to try to fix this. — Reply to this email directly, view it on GitHub<#13421 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UMY4F3CDISEOHSP5ZRTXUOWWPANCNFSM6AAAAAA3J52KTI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>