restore dump not works from other running netbox into docker instance

[jancovic]

Hi

I was trying to replicate already running netbox app to my docker Lab env.
I use Docker Desktop for Windows.
I successfully run new netbox container app in docker via https://github.com/netbox-community/netbox-docker/wiki/Getting-Started
Then I was trying to restore data according this below steps, but I got some errors, and data was not restored.
https://github.com/netbox-community/netbox-docker/wiki/Troubleshooting

Restore that database:

Stop all NetBox instances that access the db
docker compose stop netbox netbox-worker netbox-housekeeping

Restore the DB dump
gunzip -c db_dump.sql.gz | docker-compose exec -T postgres sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'

Here are my errors:

SET
ERROR: relation "auth_group" already exists
ALTER TABLE
ERROR: relation "auth_group_id_seq" already exists
ALTER TABLE
ALTER SEQUENCE
ERROR: relation "auth_group_permissions" already exists
ALTER TABLE
ERROR: relation "auth_group_permissions_id_seq" already exists
ALTER TABLE
ALTER SEQUENCE
ERROR: relation "auth_permission" already exists
ALTER TABLE
ERROR: relation "auth_permission_id_seq" already exists
ALTER TABLE
ALTER SEQUENCE
ERROR: relation "auth_user" already exists
ALTER TABLE
ERROR: relation "auth_user_groups" already exists
...
COPY 640
ERROR: duplicate key value violates unique constraint "auth_permission_pkey"
DETAIL: Key (id)=(1) already exists.
CONTEXT: COPY auth_permission, line 1
ERROR: duplicate key value violates unique constraint "auth_user_pkey"
DETAIL: Key (id)=(1) already exists.
CONTEXT: COPY auth_user, line 4
ERROR: insert or update on table "auth_user_groups" violates foreign key constraint "auth_user_groups_user_id_6a12ed8b_fk_auth_user_id"
DETAIL: Key (user_id)=(10) is not present in table "auth_user".
COPY 0
COPY 0
ERROR: column "connected_endpoint_id" of relation "circuits_circuittermination" does not exist
invalid command .
COPY 0
COPY 0
ERROR: column "termination_a_id" of relation "dcim_cable" does not exist
invalid command .
ERROR: syntax error at or near "1"
LINE 1: 1 2020-07-01 2020-07-01 07:15:40.693781-05 277 229 mmf-om1 c...
invalid command \N
invalid command \N
invalid command \N

[jancovic]

OK, I fixed 1st issue with making replication on standard new netbox app without docker,
restore was completed OK with these commands:
psql -c 'drop database netbox'
psql -c 'create database netbox'
psql netbox < netbox.sql

However when I run the Upgrade Script, I got another error and env was not created.

:/opt/netbox/netbox$ sudo PYTHON=/usr/bin/python3.8 /opt/netbox/upgrade.sh
Using Python 3.8.10
Removing old virtual environment...
Creating a new virtual environment at /opt/netbox/venv...
Updating pip (pip install --upgrade pip)...
Collecting pip
Using cached pip-22.2.2-py3-none-any.whl (2.0 MB)
....

Applying database migrations (python3 netbox/manage.py migrate)...
Traceback (most recent call last):
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 276, in build_graph
self.graph.validate_consistency()
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 198, in validate_consistency
[n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 198, in
[n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 60, in raise_error
raise NodeNotFoundError(self.error_message, self.key, origin=self.origin)
django.db.migrations.exceptions.NodeNotFoundError: Migration dcim.0131_consoleport_speed dependencies reference nonexistent parent node ('dcim', '0130_sitegroup')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "netbox/manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/init.py", line 446, in execute_from_command_line
utility.execute()
File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/init.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/base.py", line 414, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/base.py", line 460, in execute
output = self.handle(*args, **options)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/base.py", line 98, in wrapped
res = handle_func(*args, **kwargs)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 108, in handle
executor = MigrationExecutor(connection, self.migration_progress_callback)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 18, in init
self.loader = MigrationLoader(self.connection)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 58, in init
self.build_graph()
File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 295, in build_graph
raise NodeNotFoundError(
django.db.migrations.exceptions.NodeNotFoundError: Migration dcim.0131_consoleport_speed depends on nonexistent node ('dcim', '0130_sitegroup'). Django tried to replace migration dcim.0130_sitegroup with any of [dcim.0003_squashed_0130] but wasn't able to because some of the replaced migrations are already applied.

Any idea what is wrong ?

[jancovic]

FYI: I am migrating netbox from v2.8.6 to uptodate 3.3.5

[kkthxbye-code]

You have to upgrade to v2.11 first, then to v3.3.5.

https://github.com/netbox-community/netbox/releases/tag/v3.0.0

WARNING: Upgrading an existing NetBox deployment to version 3.0 must be done from version 2.11.0 or later. If attempting to upgrade a deployment of NetBox v2.10 or earlier, first upgrade to a NetBox v2.11 release, and then upgrade from v2.11 to v3.0. This will avoid any problems with the database migration optimizations implemented in version 3.0. (This is not necessary for new installations.)

We recommend that you at least read all major and minor release notes before upgrading, as these can include backwards incompatible changes.

[jancovic]

OK, so as I have running in production this old ver. 2.8.6,
Where and how can I download from github this old version 2.8.6 ?
Reason : Replicate first this old version in dev. env. if restore is working and then I can upgrade version in production.
When I tried to search 2.8.6 in github branches, I could find anything.

[jancovic]

OK, I found old release here https://github.com/netbox-community/netbox/releases/tag/v2.8.6
I tried to install the database schema according below old Installation steps, but I got errors.
Any idea what is wrong ?

_## Run Database Migrations

Before NetBox can run, we need to install the database schema. This is done by running python3 manage.py migrate from the netbox directory (/opt/netbox/netbox/ in our example):_

(venv) # cd /opt/netbox/netbox/
(venv) # python3 manage.py migrate

(venv) root@:/opt/netbox/netbox# python3 manage.py migrate
Traceback (most recent call last):
File "manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/init.py", line 401, in execute_from_command_line
utility.execute()
File "/opt/netbox/venv/lib/python3.8/site-packages/django/core/management/init.py", line 377, in execute
django.setup()
File "/opt/netbox/venv/lib/python3.8/site-packages/django/init.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/opt/netbox/venv/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
app_config.import_models()
File "/opt/netbox/venv/lib/python3.8/site-packages/django/apps/config.py", line 211, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 848, in exec_module
File "", line 219, in _call_with_frames_removed
File "/opt/netbox/netbox/circuits/models.py", line 8, in
from dcim.models import CableTermination
File "/opt/netbox/netbox/dcim/models/init.py", line 23, in
from dcim.elevations import RackElevationSVG
File "/opt/netbox/netbox/dcim/elevations.py", line 7, in
from utilities.utils import foreground_color
File "/opt/netbox/netbox/utilities/utils.py", line 8, in
from jinja2 import Environment
File "/opt/netbox/venv/lib/python3.8/site-packages/jinja2/init.py", line 12, in
from .environment import Environment
File "/opt/netbox/venv/lib/python3.8/site-packages/jinja2/environment.py", line 25, in
from .defaults import BLOCK_END_STRING
File "/opt/netbox/venv/lib/python3.8/site-packages/jinja2/defaults.py", line 3, in
from .filters import FILTERS as DEFAULT_FILTERS # noqa: F401
File "/opt/netbox/venv/lib/python3.8/site-packages/jinja2/filters.py", line 13, in
from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/opt/netbox/venv/lib/python3.8/site-packages/markupsafe/init.py)
(venv) root@:/opt/netbox/netbox#

[jancovic]

ok, I fixed this with

pip install markupsafe==2.0.1

and migrate script finished, also app is running now

(venv) root@:/opt/netbox/netbox# python3 manage.py migrate
/opt/netbox/netbox/utilities/api.py:103: SyntaxWarning: "is" with a literal. Did you mean "=="?
if obj is '':
/opt/netbox/netbox/utilities/api.py:118: SyntaxWarning: "is" with a literal. Did you mean "=="?
if data is '':
Operations to perform:
Apply all migrations: admin, auth, circuits, contenttypes, dcim, extras, ipam, secrets, sessions, taggit, tenancy, users, virtualization
Running migrations:
No migrations to apply.
(venv) root@:/opt/netbox/netbox#

so Final step is to upgrade to to uptodate version