Can't migrate docker instance from 3.77 to 4.0.1

[EHRETic]

Hi there,

Don't know if Portainer makes it more difficult, but whenever I try to migrate my docker instance (don't worry, I tried first on a test instance), it starts and then crash

Those are the logs of netbox server container (worker & housekeeping will not start) when I start the migration process:

2024-05-13T15:02:21.987600186Z ⚙️ Applying database migrations
2024-05-13T15:02:32.161406009Z 🧬 loaded config '/etc/netbox/config/configuration.py'
2024-05-13T15:02:32.161444788Z 🧬 loaded config '/etc/netbox/config/authentik.py'
2024-05-13T15:02:32.161450677Z 🧬 loaded config '/etc/netbox/config/extra.py'
2024-05-13T15:02:32.161454892Z 🧬 loaded config '/etc/netbox/config/logging.py'
2024-05-13T15:02:32.161459015Z 🧬 loaded config '/etc/netbox/config/plugins.py'
2024-05-13T15:02:32.161463630Z Operations to perform:
2024-05-13T15:02:32.161467924Z   Apply all migrations: account, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, vpn, wireless
2024-05-13T15:02:32.161473139Z Running migrations:
2024-05-13T15:02:32.328320086Z   Applying dcim.0186_location_facility... OK
2024-05-13T15:02:32.825488653Z   Applying dcim.0187_alter_device_vc_position... OK
2024-05-13T15:02:33.048554513Z   Applying extras.0108_convert_reports_to_scripts... OK
2024-05-13T15:02:33.690684047Z   Applying extras.0109_script_model... OK
2024-05-13T15:02:33.866802292Z   Applying extras.0110_remove_eventrule_action_parameters... OK
2024-05-13T15:02:36.439299141Z   Applying extras.0111_rename_content_types... OK
2024-05-13T15:02:36.659687401Z   Applying extras.0112_tag_update_object_types... OK
2024-05-13T15:02:36.744616037Z   Applying extras.0113_customfield_rename_object_type... OK
2024-05-13T15:02:36.808889065Z   Applying extras.0114_customfield_add_comments... OK
2024-05-13T15:02:37.015120978Z   Applying extras.0115_convert_dashboard_widgets... OK
2024-05-13T15:02:38.214899068Z   Applying tenancy.0015_contactassignment_rename_content_type... OK
2024-05-13T15:02:38.876423000Z   Applying users.0005_alter_user_table... OK
2024-05-13T15:02:39.808513461Z   Applying users.0006_custom_group_model... OK
2024-05-13T15:02:40.019548670Z   Applying users.0007_objectpermission_update_object_types... OK
2024-05-13T15:02:41.188572864Z   Applying users.0008_flip_objectpermission_assignments... OK
2024-05-13T15:02:41.699445052Z   Applying vpn.0005_rename_indexes... OK
2024-05-13T15:02:42.973469189Z ⚙️ Running trace_paths
2024-05-13T15:02:52.451739345Z 🧬 loaded config '/etc/netbox/config/configuration.py'
2024-05-13T15:02:52.451784675Z 🧬 loaded config '/etc/netbox/config/authentik.py'
2024-05-13T15:02:52.451794413Z 🧬 loaded config '/etc/netbox/config/extra.py'
2024-05-13T15:02:52.451801725Z 🧬 loaded config '/etc/netbox/config/logging.py'
2024-05-13T15:02:52.451809288Z 🧬 loaded config '/etc/netbox/config/plugins.py'
2024-05-13T15:02:52.451816321Z Found no missing console port paths; skipping
2024-05-13T15:02:52.451823285Z Found no missing console server port paths; skipping
2024-05-13T15:02:52.451830335Z Found no missing interface paths; skipping
2024-05-13T15:02:52.451855978Z Found no missing power feed paths; skipping
2024-05-13T15:02:52.451863758Z Found no missing power outlet paths; skipping
2024-05-13T15:02:52.451870973Z Found no missing power port paths; skipping
2024-05-13T15:02:52.451878103Z Finished.
2024-05-13T15:02:53.476511809Z ⚙️ Removing stale content types
2024-05-13T15:03:04.197941157Z 🧬 loaded config '/etc/netbox/config/configuration.py'
2024-05-13T15:03:04.197973977Z 🧬 loaded config '/etc/netbox/config/authentik.py'
2024-05-13T15:03:04.197978572Z 🧬 loaded config '/etc/netbox/config/extra.py'
2024-05-13T15:03:04.197982492Z 🧬 loaded config '/etc/netbox/config/logging.py'
2024-05-13T15:03:04.197986097Z 🧬 loaded config '/etc/netbox/config/plugins.py'
2024-05-13T15:03:05.195063491Z ⚙️ Removing expired user sessions
2024-05-13T15:03:15.039052099Z 🧬 loaded config '/etc/netbox/config/configuration.py'
2024-05-13T15:03:15.039089265Z 🧬 loaded config '/etc/netbox/config/authentik.py'
2024-05-13T15:03:15.039093495Z 🧬 loaded config '/etc/netbox/config/extra.py'
2024-05-13T15:03:15.039096808Z 🧬 loaded config '/etc/netbox/config/logging.py'
2024-05-13T15:03:15.039100019Z 🧬 loaded config '/etc/netbox/config/plugins.py'
2024-05-13T15:03:16.097496583Z ⚙️ Building search index (lazy)
2024-05-13T15:03:25.497692066Z 🧬 loaded config '/etc/netbox/config/configuration.py'
2024-05-13T15:03:25.497823505Z 🧬 loaded config '/etc/netbox/config/authentik.py'
2024-05-13T15:03:25.497838599Z 🧬 loaded config '/etc/netbox/config/extra.py'
2024-05-13T15:03:25.497849330Z 🧬 loaded config '/etc/netbox/config/logging.py'
2024-05-13T15:03:25.497859781Z 🧬 loaded config '/etc/netbox/config/plugins.py'
2024-05-13T15:03:25.497870255Z Reindexing 75 models.
2024-05-13T15:03:25.498116754Z Indexing models
2024-05-13T15:03:25.514581797Z   core.datasource... No objects found.
2024-05-13T15:03:25.531500566Z   core.datafile... No objects found.
2024-05-13T15:03:25.554469681Z   circuits.circuit... No objects found.
2024-05-13T15:03:25.565566312Z   circuits.circuittermination... No objects found.
2024-05-13T15:03:25.574294944Z   circuits.circuittype... No objects found.
2024-05-13T15:03:25.579369287Z   circuits.provider... No objects found.
2024-05-13T15:03:25.586290030Z   circuits.providernetwork... No objects found.
2024-05-13T15:03:25.591460850Z   dcim.cable... No objects found.
2024-05-13T15:03:25.601733470Z   dcim.consoleport... No objects found.
2024-05-13T15:03:25.606023626Z   dcim.consoleserverport... No objects found.
2024-05-13T15:03:25.607072603Z   dcim.device... Skipping (found 10 existing).
2024-05-13T15:03:25.615847131Z   dcim.devicebay... No objects found.
2024-05-13T15:03:25.617635500Z   dcim.devicerole... Skipping (found 2 existing).
2024-05-13T15:03:25.618547025Z   dcim.devicetype... Skipping (found 2 existing).
2024-05-13T15:03:25.631325701Z   dcim.frontport... No objects found.
2024-05-13T15:03:25.645083091Z   dcim.interface... No objects found.
2024-05-13T15:03:25.651847857Z   dcim.inventoryitem... No objects found.
2024-05-13T15:03:25.666104234Z   dcim.location... No objects found.
2024-05-13T15:03:25.668599200Z   dcim.manufacturer... Skipping (found 2 existing).
2024-05-13T15:03:25.682908839Z   dcim.module... No objects found.
2024-05-13T15:03:25.687278067Z   dcim.modulebay... No objects found.
2024-05-13T15:03:25.697522968Z   dcim.moduletype... No objects found.
2024-05-13T15:03:25.706893924Z   dcim.platform... No objects found.
2024-05-13T15:03:25.725903469Z   dcim.powerfeed... No objects found.
2024-05-13T15:03:25.735337856Z   dcim.poweroutlet... No objects found.
2024-05-13T15:03:25.739632598Z   dcim.powerpanel... No objects found.
2024-05-13T15:03:25.744981021Z   dcim.powerport... No objects found.
2024-05-13T15:03:25.745991654Z   dcim.rack... Skipping (found 6 existing).
2024-05-13T15:03:25.750964555Z   dcim.rackreservation... No objects found.
2024-05-13T15:03:25.758768905Z   dcim.rackrole... No objects found.
2024-05-13T15:03:25.770661299Z   dcim.rearport... No objects found.
2024-05-13T15:03:25.783224931Z   dcim.region... No objects found.
2024-05-13T15:03:25.787912658Z   dcim.site... Skipping (found 2 existing).
2024-05-13T15:03:25.798515612Z   dcim.sitegroup... No objects found.
2024-05-13T15:03:25.802759855Z   dcim.virtualchassis... No objects found.
2024-05-13T15:03:25.810245952Z   dcim.virtualdevicecontext... No objects found.
2024-05-13T15:03:25.816627647Z   ipam.aggregate... No objects found.
2024-05-13T15:03:25.824321314Z   ipam.asn... No objects found.
2024-05-13T15:03:25.837925998Z   ipam.asnrange... No objects found.
2024-05-13T15:03:25.846328183Z   ipam.fhrpgroup... No objects found.
2024-05-13T15:03:25.856660350Z   ipam.ipaddress... No objects found.
2024-05-13T15:03:25.864781510Z   ipam.iprange... No objects found.
2024-05-13T15:03:25.882274970Z   ipam.prefix... No objects found.
2024-05-13T15:03:25.891536146Z   ipam.rir... No objects found.
2024-05-13T15:03:25.899625604Z   ipam.role... No objects found.
2024-05-13T15:03:25.905724765Z   ipam.routetarget... No objects found.
2024-05-13T15:03:25.912225741Z   ipam.service... No objects found.
2024-05-13T15:03:25.917426248Z   ipam.servicetemplate... No objects found.
2024-05-13T15:03:25.935857660Z   ipam.vlan... No objects found.
2024-05-13T15:03:25.941442948Z   ipam.vlangroup... No objects found.
2024-05-13T15:03:25.949617133Z   ipam.vrf... No objects found.
2024-05-13T15:03:25.959596593Z   extras.customfield... No objects found.
2024-05-13T15:03:25.966555653Z   extras.journalentry... No objects found.
2024-05-13T15:03:25.973919949Z   extras.webhook... No objects found.
2024-05-13T15:03:25.979912455Z   tenancy.contact... No objects found.
2024-05-13T15:03:25.988640396Z   tenancy.contactgroup... No objects found.
2024-05-13T15:03:25.992953546Z   tenancy.contactrole... No objects found.
2024-05-13T15:03:26.007209301Z   tenancy.tenant... No objects found.
2024-05-13T15:03:26.017173174Z   tenancy.tenantgroup... No objects found.
2024-05-13T15:03:26.026127931Z   virtualization.cluster... No objects found.
2024-05-13T15:03:26.034437903Z   virtualization.clustergroup... No objects found.
2024-05-13T15:03:26.047348767Z   virtualization.clustertype... No objects found.
2024-05-13T15:03:26.061270476Z   virtualization.virtualmachine... No objects found.
2024-05-13T15:03:26.071421379Z   virtualization.vminterface... No objects found.
2024-05-13T15:03:26.078524629Z   virtualization.virtualdisk... No objects found.
2024-05-13T15:03:26.083494038Z   vpn.tunnel... No objects found.
2024-05-13T15:03:26.088741367Z   vpn.ikeproposal... No objects found.
2024-05-13T15:03:26.094492677Z   vpn.ikepolicy... No objects found.
2024-05-13T15:03:26.099774906Z   vpn.ipsecproposal... No objects found.
2024-05-13T15:03:26.104781610Z   vpn.ipsecpolicy... No objects found.
2024-05-13T15:03:26.114370590Z   vpn.ipsecprofile... No objects found.
2024-05-13T15:03:26.121488192Z   vpn.l2vpn... No objects found.
2024-05-13T15:03:26.127380829Z   wireless.wirelesslan... No objects found.
2024-05-13T15:03:26.135773289Z   wireless.wirelesslangroup... No objects found.
2024-05-13T15:03:26.145673225Z   wireless.wirelesslink... No objects found.
2024-05-13T15:03:26.145690741Z Completed. Total entries: 24

This is my docker compose file:

services:
  netbox: &netbox-test
    container_name: netbox-test
    image: docker.io/netboxcommunity/netbox:v4.0.1
    hostname: netbox-test
        
    restart: unless-stopped
    depends_on:
      - netbox-test-postgres

    user: 'unit:root'

    healthcheck:
      start_period: 60s
      timeout: 3s
      interval: 15s
      test: "curl -f http://localhost:8080/api/ || exit 1"

    volumes:
      - configuration:/etc/netbox/config
      - media-files:/opt/netbox/netbox/media
      - reports-files:/opt/netbox/netbox/reports
      - scripts-files:/opt/netbox/netbox/scripts

    networks:
      default:
        ipv4_address: 192.168.X.X

    environment:
      DB_HOST: '${DB_HOST}'
      DB_NAME: '${DB_NAME}'
      DB_USER: '${DB_USER}'
      DB_PASSWORD: '${DB_PASSWORD}'
      SECRET_KEY: '${SECRET_KEY}'
      REDIS_HOST: '${REDIS_HOST}'
      REDIS_DATABASE: '${REDIS_DATABASE}'
      REDIS_PASSWORD: '${REDIS_PASSWORD}'
      REDIS_INSECURE_SKIP_TLS_VERIFY: '${REDIS_INSECURE_SKIP_TLS_VERIFY}'
      REDIS_SSL: '${REDIS_SSL}'
      REDIS_CACHE_HOST: '${REDIS_CACHE_HOST}'
      REDIS_CACHE_DATABASE: '${REDIS_CACHE_DATABASE}'
      REDIS_CACHE_PASSWORD: '${REDIS_CACHE_PASSWORD}'
      REDIS_CACHE_INSECURE_SKIP_TLS_VERIFY: '${REDIS_CACHE_INSECURE_SKIP_TLS_VERIFY}'
      REDIS_CACHE_SSL: '${REDIS_CACHE_SSL}'
      TIME_ZONE: '${TIME_ZONE}'
      RELEASE_CHECK_URL: '${RELEASE_CHECK_URL}'
      LOGIN_REQUIRED: '${LOGIN_REQUIRED}'
      LOGIN_TIMEOUT: '${LOGIN_TIMEOUT}'
      CORS_ORIGIN_ALLOW_ALL: '${CORS_ORIGIN_ALLOW_ALL}'
      REMOTE_AUTH_ENABLED: '${REMOTE_AUTH_ENABLED}'
      REMOTE_AUTH_BACKEND: '${REMOTE_AUTH_BACKEND}'
      SOCIAL_AUTH_OIDC_ENDPOINT: '${SOCIAL_AUTH_OIDC_ENDPOINT}'
      SOCIAL_AUTH_OIDC_KEY: '${SOCIAL_AUTH_OIDC_KEY}'
      SOCIAL_AUTH_OIDC_SECRET: '${SOCIAL_AUTH_OIDC_SECRET}'
      LOGOUT_REDIRECT_URL: '${LOGOUT_REDIRECT_URL}'

  netbox-worker:
    <<: *netbox-test
    container_name: netbox-test-worker
    depends_on:
      netbox:
        condition: service_healthy
    
    command:
    - /opt/netbox/venv/bin/python
    - /opt/netbox/netbox/manage.py
    - rqworker
    
    healthcheck:
      start_period: 20s
      timeout: 3s
      interval: 15s
      test: "ps -aux | grep -v grep | grep -q rqworker || exit 1"

    networks:
      default:
        ipv4_address: 192.168.X.X

  netbox-housekeeping:
    <<: *netbox-test
    container_name: netbox-test-housekeeping
    depends_on:
      netbox:
        condition: service_healthy
    command:
    - /opt/netbox/housekeeping.sh
    healthcheck:
      start_period: 20s
      timeout: 3s
      interval: 15s
      test: "ps -aux | grep -v grep | grep -q housekeeping || exit 1"
    networks:
      default:
        ipv4_address: 192.168.X.X

  netbox-test-postgres:
    container_name: netbox-test-postgres
    image: postgres:16.2-alpine
    
    restart: unless-stopped

    volumes: 
      - postgres:/var/lib/postgresql/data

    networks:
      default:
        ipv4_address: 192.168.X.X

    environment:
      POSTGRES_USER: '${POSTGRES_USER}'
      POSTGRES_PASSWORD: '${POSTGRES_PASSWORD}'

volumes:
  configuration:
  media-files:
  reports-files:
  scripts-files:
  postgres:

networks:
  default:
    external: true
    name: docker_test

The only "not standard" thing is I use Authentik as OIDC SSO, so there is some configuration/scrip for it.
I also use macvlan containers (every single container has it's own IP), but that should not prevent migration, no?

authentik.py:

from os import environ

#############
# Docker
#############

# python-social-auth configuration
SOCIAL_AUTH_OIDC_ENDPOINT = environ.get('SOCIAL_AUTH_OIDC_ENDPOINT')
SOCIAL_AUTH_OIDC_KEY = environ.get('SOCIAL_AUTH_OIDC_KEY')
SOCIAL_AUTH_OIDC_SECRET = environ.get('SOCIAL_AUTH_OIDC_SECRET')
LOGOUT_REDIRECT_URL = environ.get('LOGOUT_REDIRECT_URL')

Where do we start troubleshooting? Thanks in advance for you help

K.R.
Franck

[EHRETic]

Anyone for help? :-)

[candlerb]

worker & housekeeping will not start

A good starting point would be looking at logs for those containers.

[EHRETic]

They will not generate any logs because netbox server container crashes (and therefore stops the starting chain)

[squintfox]

It looks like your docker-compose file isn't based on the latest release of netbox-docker. The healthcheck test was changed from /api to /login for 4.0. You should take a look at the newest version and integrate any changes.

[EHRETic]

I'll have a look at this, thanks for pointing that out! ;-)

[EHRETic]

The healthcheck test was changed from /api to /login for 4.0. You should take a look at the newest version and integrate any changes.

The container crashed/stopped the same after DB migration, but second start was successfull.
Thanks a lot for spoting that little detail! :-)