Upgrade to v3.6.0 No space left on device: devicetype_component_counters

[LoSaMe]

When upgrading from v3.5.9 to 3.6.0, migrating devicetype_component_counters will result in running out of disk space. The process consumes 50 GB, takes a long time and did not lead to success. Any suggestions?

Running migrations:
  Applying dcim.0177_devicetype_component_counters...Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/netbox/venv/lib/python3.8/site-packages/psycopg/cursor.py", line 737, in execute
    raise ex.with_traceback(None)
psycopg.errors.DiskFull: could not write to file "base/pgsql_tmp/pgsql_tmp731207.13": No space left on device

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox/netbox/dcim/migrations/0177_devicetype_component_counters.py", line 9, in recalculate_devicetype_template_counts
    device_types = list(DeviceType.objects.all().annotate(
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/query.py", line 380, in __len__
    self._fetch_all()
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1881, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/netbox/venv/lib/python3.8/site-packages/psycopg/cursor.py", line 737, in execute
    raise ex.with_traceback(None)
django.db.utils.OperationalError: could not write to file "base/pgsql_tmp/pgsql_tmp731207.13": No space left on device

[madecto]

I might also add that, after the failed migration attempt, trying to revert to 3.5.9 fails in the same way.

[candlerb]

How much free space was available on the system before attempting this migration?

How much disk space is being used now by postgres? (du -sch /var/lib/postgresql if it's Ubuntu or similar)

I suspect you might need to blow away postgres and restore from a backup.

[candlerb]

Once the disk has actually become full, I'm not sure the database is recoverable.

"The process consumes 50 GB" - do you mean RAM then?

When you say "trying to revert to 3.5.9 fails in the same way", what exactly do you mean? Are you dropping the database, recreating it, and then using psql to restore a backup made with 3.5.9 as documented here? That's the only safe way I know to revert, and if it's failing, you should show the error.

[madecto]

Update:
I tried giving the PostgreSQL filesystem 500GB in order for the migration to run unhindered, but it kept filling up the disk and failing.
I have no backup of the VM (no comment please: I've already made sure to repeatedly bang my head against a wall).

pgdump works and I fired up a v3.5.9 instance with it, but obviously something in the DB has already been migrated so - depending on the accessed object - things stop working:

<class 'django.db.utils.ProgrammingError'>

column dcim_platform.napalm_driver does not exist
LINE 1: ...cturer_id", "dcim_platform"."config_template_id", "dcim_plat...
                                                             ^

Question:
is there anybody out there offering some kind of paid service to reconstruct data which has been mangled by the migration?

[jeremystretch]

@madecto if you can give me access to a copy of your database, I'll take a look and see what I can do with it. I'm not sure why disk usage would be so excessive. You can reach me at jstretch at netboxlabs.com.

[candlerb]

Just to be clear then:

1. You were doing a migration in-place?
2. You don't have a pgdump from before the migration started, nor a VM snapshot?
3. When you run ./upgrade.sh it fills all available disk space?

When the migration crashes, does the disk return to mostly empty, or does it remain full? (This is the difference between an infinite transaction which gets rolled back, versus an infinite series of inserts)

[madecto]

1. yes
2. correct
3. yes

it returns to mostly empty

[jeremystretch]

This looks like bug #13605, which was addressed during the v3.6 beta for devices but apparently not for device types. It should be simple enough for us to fix in the next release, but to get you back up and running for now, we can disable the problematic logic within the migration file.

Open the file netbox/dcim/migrations/0177_devicetype_component_counters.py and comment out the following lines toward the bottom:

        migrations.RunPython(
            recalculate_devicetype_template_counts,
            reverse_code=migrations.RunPython.noop
        ),

It should look like this after editing:

#        migrations.RunPython(
#            recalculate_devicetype_template_counts,
#            reverse_code=migrations.RunPython.noop
#        ),

The, run the upgrade script again (or run manage.py migrate directly, within NetBox Python virtual environment). This will apply the database schema changes without calculating the component counts for each device type.

At this point all migrations should apply successfully, however your device types will all report having zero component templates. We can fix this by running the calculate_cached_counts management command. To do this, enable the Python virtual environment and run the command using manage.py (adjust the first command below for your actual installation path if necessary):

$ cd /opt/netbox
$ source venv/bin/activate
$ /netbox/manage.py calculate_cached_counts

You should see this output:

Updating Device console_port_count...
Updating Device console_server_port_count...
Updating Device power_port_count...
Updating Device power_outlet_count...
Updating Device interface_count...
Updating Device front_port_count...
Updating Device rear_port_count...
Updating Device device_bay_count...
Updating Device module_bay_count...
Updating Device inventory_item_count...
Updating DeviceType console_port_template_count...
Updating DeviceType console_server_port_template_count...
Updating DeviceType power_port_template_count...
Updating DeviceType power_outlet_template_count...
Updating DeviceType interface_template_count...
Updating DeviceType front_port_template_count...
Updating DeviceType rear_port_template_count...
Updating DeviceType device_bay_template_count...
Updating DeviceType module_bay_template_count...
Updating DeviceType inventory_item_template_count...
Updating VirtualChassis member_count...
Updating VirtualMachine interface_count...
Finished.

Finally, to restore a pristine environment, revert the changes you made to 0177_devicetype_component_counters.py above by removing the hash marks.

Let me know how this goes! 🤞

[madecto]

As much as I hate the ubiquitous "lifesaver" comments, @jeremystretch you definitely have been one!
Everything is back online as you anticipated and I've already got a daily pg_dump and backup in place ;-)

This is what I deserve for living on the bleeding edge and automatically upgrading every night.

[jeremystretch]

Nice! Thanks for helping us track this down. I've already submitted bug #13745 and this should be resolved in the next release (v3.6.2).