Database Migration Failure During Upgrade (custom fields)

[parker-portlock]

NetBox version

2.11.2

Python version

3.6

Steps to Reproduce

This is an upgrade from version 2.9.9 to 2.11.2 via git. I followed these instructions:

cd /opt/netbox
sudo git checkout master
sudo git pull origin master

It looks to have errored out on one of my custom fields. They are configured as follows:

Host OS: Ubuntu 18.04
Postgres version: psql (PostgreSQL) 10.16 (Ubuntu 10.16-0ubuntu0.18.04.1)
Python version: Python 3.6.9

Expected Behavior

Expected behavior: upgrade completed without issues.

Observed Behavior

ValueError occurred during the database migration:

Running migrations:
  Applying virtualization.0018_custom_field_data... OK
  Applying tenancy.0010_custom_field_data... OK
  Applying secrets.0010_custom_field_data... OK
  Applying ipam.0038_custom_field_data... OK
  Applying extras.0048_exporttemplate_remove_template_language... OK
  Applying extras.0049_remove_graph... OK
  Applying extras.0050_customfield_changes... OK
  Applying dcim.0117_custom_field_data... OK
  Applying circuits.0020_custom_field_data... OK
  Applying extras.0051_migrate_customfields...AWS Region (select): us-east-1 (35)
Traceback (most recent call last):
  File "netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 246, in handle
    fake_initial=fake_initial,
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/opt/netbox/venv/lib/python3.6/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox/netbox/extras/migrations/0051_migrate_customfields.py", line 72, in migrate_customfieldvalues
    raise e
  File "/opt/netbox/netbox/extras/migrations/0051_migrate_customfields.py", line 69, in migrate_customfieldvalues
    cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value)
  File "/opt/netbox/netbox/extras/migrations/0051_migrate_customfields.py", line 15, in deserialize_value
    return field._choices.get(pk=int(value)).value
ValueError: invalid literal for int() with base 10: 'us-east-1'

[pspdbx]

Similar problem here. I think the upgrade is risky when you have custom fields. Its not being handled carefully. OS = Ubuntu 20 , Python version = 3.8 , Netbox version = 2.11.2 (Trying to upgrade from 2.8.8 to 2.11.2)

Applying dcim.0117_custom_field_data... OK
Applying circuits.0019_nullbooleanfield_to_booleanfield... OK
Applying circuits.0020_custom_field_data... OK
Applying extras.0051_migrate_customfields...

Stuck here forever ^^

So I dug deeper and observed strange behavior with .exclude... it doesn't seem to work. In my case, the custom field called 'Role' has a default value of ''.
But the exclude filter still returned the object anyway:
$ foo = CustomField.objects.exclude(default='')[0]
$ foo
<CustomField: Role>
$ foo.default
''
$

!?

I tried to execute https://github.com/netbox-community/netbox/blob/v2.11.2/netbox/extras/migrations/0051_migrate_customfields.py, and it fails here: https://github.com/netbox-community/netbox/blob/v2.11.2/netbox/extras/migrations/0051_migrate_customfields.py#L38

`
$ CustomField.objects.filter(pk=1).update(default2='none')
Traceback (most recent call last):
File "/home/foo/netbox_dev/netbox-server.runfiles/main/thirdparty/netbox/pip/django/django-cpython-38/lib/django/db/models/options.py", line
608, in get_field
return self.fields_map[field_name]
KeyError: 'default2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/home/foo/netbox_dev/netbox-server.runfiles/main/thirdparty/netbox/pip/django/django-cpython-38/lib/django/db/models/query.py", line 7
79, in update
query.add_update_values(kwargs)
File "/home/foo/netbox_dev/netbox-server.runfiles/main/thirdparty/netbox/pip/django/django-cpython-38/lib/django/db/models/sql/subqueries.py
", line 85, in add_update_values
field = self.get_meta().get_field(name)
File "/home/foo/netbox_dev/netbox-server.runfiles/main/thirdparty/netbox/pip/django/django-cpython-38/lib/django/db/models/options.py", line
610, in get_field
raise FieldDoesNotExist("%s has no field named '%s'" % (self.object_name, field_name))
django.core.exceptions.FieldDoesNotExist: CustomField has no field named 'default2'
$`

CustomField has no field name default2 , even though this migration seemingly got applied successfully:

Applying extras.0050_customfield_changes... OK. < which is supposed to create the field called 'default2` ..

😞

[parker-portlock]

I've tried this again with a fresh installation of the latest netbox version (2.11.12) and importing a copy of our existing database. Same result

It is extremely disappointing that something like a custom field can cause this problem. Moreover, It's incredibly frustrating that I submitted this as a bug, and it was ignored.

[sdktr]

Can you link the bug issue? If you can reproduce it with a clean install of 2.9 (or whichever version came last before 0051 migration) there must be something that can be done.