Netbox v3.6.x does not work with pgpool #13994

rc-networks · 2023-10-06T11:56:44Z

rc-networks
Oct 6, 2023

NetBox version

v3.6.3

Python version

3.11

Steps to Reproduce

Install bitnami/postgresql-ha

PSQL_IMAGE_VERSION="15.4.0-debian-11-r43"
PGPOOL_IMAGE_VERSION="4.4.4-debian-11-r32"

helm upgrade -i $POSTGRES_RELEASE \
    --create-namespace \
    --namespace $NAMESPACE \
    --set postgresql.password="$PSQL_PASSWORD" \
    --set postgresql.repmgrUsername="$PSQL_REPMGR" \
    --set postgresql.repmgrPassword="$PSQL_REPMGR_PASS" \
    --set postgresql.repmgrDatabase="$PSQL_REPMGR" \
    --set postgresql.image.tag="$PSQL_IMAGE_VERSION" \
    --set postgresql.initdbScriptsCM=init-db-script \
    --set pgpool.adminUsername="$PGPOOL_ADMIN" \
    --set pgpool.adminPassword="$PGPOOL_ADPASS" \
    --set pgpool.customUsers.usernames="$PSQL_USERNAME" \
    --set pgpool.customUsers.passwords="$PSQL_PASSWORD" \
    --set pgpool.image.tag="$PGPOOL_IMAGE_VERSION" \
    --set persistence.storageClass="$STG_CLASS" \
    --set persistence.size="2Gi" \
    oci://registry-1.docker.io/bitnamicharts/postgresql-ha

Go to bootc/netbox-chart, copy the values.yaml file and point netbox to use the bitnamicharts/postgresql-ha's SVC. Then Install netbox via the given helm-chart.

helm upgrade -i $NETBOX_RELEASE \
    --create-namespace \
    --namespace $NAMESPACE \
    --set superuser.name="$NETBOX_USERNAME" \
    --set superuser.password="$NETBOX_PASS" \
    --set superuser.apiToken="$NETBOX_TOKEN" \
    --set napalm.username="$NAPALM_USR" \
    --set napalm.password="$NAPALM_PAS" \
    --set externalDatabase.database="$NETBOX_PSQL_DB" \
    --set externalDatabase.username="$PSQL_USERNAME" \
    --set externalDatabase.password="$PSQL_PASSWORD" \
    --set image.tag="$NETBOX_VERSION" \
    --set persistence.storageClass="$STG_CLASS" \
    -f config/netbox-values-dev.yaml \
    bootc/netbox

Expected Behavior

Same as with the Netbox v3.5x and below, installed without issues:

NOTE: This is my prod environment running Netbox v3.5.9 with exactly the same config and variables as with the deployment on dev namespace for testing newer versions.
Dev namespace will be an entirely fresh deployment without any data.

Observed Behavior

Netbox pods and or Worker Pod erroring out:

% kubectl logs netbox-dev-58ffdb69fb-ddb7f -n netbox-dev
Defaulted container "netbox" out of: netbox, init-dirs (init)
⚙️ Applying database migrations
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
Operations to perform:
  Apply all migrations: account, admin, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, wireless
Running migrations:
Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/server_cursor.py", line 294, in execute
    raise ex.with_traceback(None)
psycopg.OperationalError: consuming input failed: unable to execute Describe
DETAIL:  unable to get the bind message
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 114, in debug_sql
    yield
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 102, in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/server_cursor.py", line 294, in execute
    raise ex.with_traceback(None)
django.db.utils.OperationalError: consuming input failed: unable to execute Describe
DETAIL:  unable to get the bind message
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
<omitted output too long>

huxcrux · 2023-10-06T13:08:52Z

huxcrux
Oct 6, 2023

I also came across this, I moved to a single database for now. An alternative would also be to test using pgbouncer(I have not yet tested this) however it would be nice to have a more official discussion on what netbox supports to avoid the need of replacing parts if the database setup in the near future

3 replies

rc-networks Oct 7, 2023
Author

I originally opened this as a BUG since it was working with the older versions.
This being moved here means, they intentionally removed the support for pgpool?

jschewebbn Oct 24, 2023

@rc-networks @huxcrux what version of pgpool are you using that caused this issue? I'm due for an upgrade and wondering if perhaps that will help.
Did either of you find a way around this and still use pgpool?

jschewebbn Oct 24, 2023

I think I have a fix, set DISABLE_SERVER_SIDE_CURSORS to true in your database configuration. That's what fixed it for goauthentik.

jschewebbn · 2023-10-24T21:02:19Z

jschewebbn
Oct 24, 2023

I'm trying to upgrade to 3.6.4 from 3.5.2 and didn't have issues on my development system backed in pgpool, however my production system is running into this issue.

I see that another project has a patch for perhaps a similar issue goauthentik/authentik#6949

Operations to perform:                                                                                                                                                    
  Apply all migrations: account, admin, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization,
 wireless                                                                                                                                                                 
Running migrations:                          
  Applying social_django.0013_migrate_extra_data...Traceback (most recent call last):                                                                                     
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)                                                                                                                               
  File "/srv/netbox/venv/lib64/python3.8/site-packages/psycopg/server_cursor.py", line 294, in execute
    raise ex.with_traceback(None)                                                                                                                                         
psycopg.OperationalError: consuming input failed: unable to execute Describe
DETAIL:  unable to get the bind message                                                                                                                                   
SSL SYSCALL error: EOF detected
                                                                                                                                                                          
The above exception was the direct cause of the following exception:
                                                                                                                                                                          
Traceback (most recent call last):             
  File "netbox/manage.py", line 10, in <module>                                                                                                                           
    execute_from_command_line(sys.argv)    
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line                                        
    utility.execute()                                          
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/core/management/__init__.py", line 436, in execute                                                          
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/core/management/base.py", line 412, in run_from_argv                                                        
    self.execute(*args, **cmd_options)
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(  
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/srv/netbox/venv/lib64/python3.8/site-packages/social_django/migrations/0013_migrate_extra_data.py", line 13, in migrate_json_field
    for auth in (           
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/models/query.py", line 516, in _iterator
    yield from iterable
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)                                                                                                                                           
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute                                                                  
    return self._execute_with_wrappers(                                                                                                                                   
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/srv/netbox/venv/lib64/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/srv/netbox/venv/lib64/python3.8/site-packages/psycopg/server_cursor.py", line 294, in execute
    raise ex.with_traceback(None)
django.db.utils.OperationalError: consuming input failed: unable to execute Describe
DETAIL:  unable to get the bind message
SSL SYSCALL error: EOF detected

0 replies

eguarc · 2023-10-27T20:00:14Z

eguarc
Oct 27, 2023

I'm also running with this issue at moment of implementation (not upgrading).

⚙️ Applying database migrations
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
Operations to perform:
  Apply all migrations: account, admin, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, wireless
Running migrations:
Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/server_cursor.py", line 294, in execute
    raise ex.with_traceback(None)
psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox/netbox/./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox/netbox/core/migrations/0004_replicate_jobresults.py", line 12, in replicate_jobresults
    for job_result in JobResult.objects.order_by('pk').iterator(chunk_size=100):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py", line 516, in _iterator
    yield from iterable
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/server_cursor.py", line 294, in execute
    raise ex.with_traceback(None)
django.db.utils.OperationalError: consuming input failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
  Applying core.0004_replicate_jobresults...

I try setting:

  - name: CONN_MAX_AGE
    value: "60"
  - name: DISABLE_SERVER_SIDE_CURSORS
    value: "True"

but no success at all.
Solution was to pointing DB_HOST to the backend master pg server.
Then:

⚙️ Applying database migrations
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
Operations to perform:
  Apply all migrations: account, admin, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, wireless
Running migrations:
  Applying core.0004_replicate_jobresults... OK
  Applying core.0005_job_created_auto_now... OK
  Applying tenancy.0010_tenant_relax_uniqueness... OK
  Applying ipam.0064_clear_search_cache... OK
  Applying extras.0085_synced_data... OK
  Applying extras.0086_configtemplate... OK
  Applying extras.0087_dashboard... OK
  Applying ipam.0065_asnrange... OK
  Applying ipam.0066_iprange_mark_utilized... OK
  Applying dcim.0167_module_status... OK
  Applying dcim.0168_interface_template_enabled... OK
  Applying dcim.0169_devicetype_default_platform... OK
  Applying dcim.0170_configtemplate... OK
  Applying dcim.0171_cabletermination_change_logging... OK

Netbox docker image: netboxcommunity/netbox:v3.6.4-2.7.0
Postgres server version: 14.7

0 replies

anubisg1 · 2023-11-20T11:04:23Z

anubisg1
Nov 20, 2023

ok, so this worked for me...

in the netbox helm cart set the variable

externalDatabase.disableServerSideCursors

to true.

0 replies

jschewebbn · 2023-11-20T15:18:56Z

jschewebbn
Nov 20, 2023

Here us the section from my configuration.py file. I have deployed netbox directly on a host, not using docker or other such tools.

DATABASE = {
    'NAME': database_name,         # Database name
    'USER': secrets['DATABASE_USER'],               # PostgreSQL username
    'PASSWORD': secrets['DATABASE_PASSWORD'],           # PostgreSQL password
    'HOST': 'server-name',      # Database server
    'PORT': '9898',               # Database port (leave blank for default)
    'CONN_MAX_AGE': 300,      # Max database connection age
    'CONN_HEALTH_CHECKS': True,
    'DISABLE_SERVER_SIDE_CURSORS': True, # for compatibility with pgpool
    'POOL_OPTIONS': {
        'POOL_SIZE': 10,
        'MAX_OVERFLOW': 10,
        'RECYCLE': 24 * 60 * 60
    }
}

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Netbox v3.6.x does not work with pgpool #13994

Uh oh!

{{title}}

Uh oh!

Replies: 5 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Netbox v3.6.x does not work with pgpool #13994

Uh oh!

NetBox version

Python version

Steps to Reproduce

Expected Behavior

Observed Behavior

Replies: 5 comments · 3 replies

Uh oh!

Uh oh!

rc-networks Oct 7, 2023 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 5 comments 3 replies

rc-networks Oct 7, 2023
Author