Re-relating mattermost-k8s to postgresql-k8s causes strange behaviour

[adhityaravi]

Bug Description

After successfully deploying the Mattermost stack on a K8s cloud with juju following this tutorial, when the relation between the mattermost-k8s and postgresql-k8s is removed and then re-integrated, something strange happens.

1. when the relation is removed - the mattermost-k8s application continues to work with the status active. I would have expected that it would be waiting for a db relation similar to when the postgresql-k8s application did not exist.
2. when the relation is integrated again - there seems to be a second mattermost-k8s unit with a completely different IP which becomes the leader, then old unit is removed. This seems to be the behavior only with the mattermost-k8s charm (atleast not reproducible with a couple of other charms).

To Reproduce

Deploy the stack

juju deploy mattermost-k8s
juju deploy postgresql-k8s --channel 14/stable --trust
juju deploy tls-certificates-operator
juju config tls-certificates-operator generate-self-signed-certificates="true" ca-common-name="Test CA"
juju integrate postgresql-k8s tls-certificates-operator
juju integrate mattermost-k8s postgresql-k8s:db

Re-integrate mattermost-k8s and postgresql-k8s

juju remove-relation mattermost-k8s postgresql-k8s:db
juju integrate mattermost-k8s postgresql-k8s:db

Environment

As per the tutorial the operations were performed on a multipass charm-dev VM in a microK8s cloud which uses

• juju v3.6.5
• mattermost-k8s latest/stable
• postgresql-k8s 14/stable

Relevant log output

unit-postgresql-k8s-0: 10:35:00 INFO juju.worker.uniter.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:35:01 INFO juju.worker.uniter.operation ran "db-relation-joined" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:35:02 INFO unit.mattermost-k8s/0.juju-log db:5: emitting master_available event for relation 5
application-mattermost-k8s: 10:35:02 INFO unit.mattermost-k8s/0.juju-log db:5: emitting master_changed event for relation 5
unit-postgresql-k8s-0: 10:35:02 WARNING unit.postgresql-k8s/0.juju-log db:5: DEPRECATION WARNING - `db` is a legacy interface
unit-postgresql-k8s-0: 10:35:03 INFO juju.worker.uniter.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:35:03 INFO unit.mattermost-k8s/0.juju-log db:5: emitting standby_available event for relation 5
application-mattermost-k8s: 10:35:03 INFO unit.mattermost-k8s/0.juju-log db:5: emitting standby_changed event for relation 5
application-mattermost-k8s: 10:35:03 INFO unit.mattermost-k8s/0.juju-log db:5: emitting database_available event for relation 5
application-mattermost-k8s: 10:35:03 INFO unit.mattermost-k8s/0.juju-log db:5: emitting database_changed event for relation 5
unit-postgresql-k8s-0: 10:35:03 INFO juju.worker.uniter.operation ran "restart-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:35:04 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:35:06 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:35:06 INFO juju.worker.caasoperator started pod init on "mattermost-k8s/0"
application-mattermost-k8s: 10:35:08 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:35:09 WARNING unit.postgresql-k8s/0.juju-log db:5: DEPRECATION WARNING - `db` is a legacy interface
application-mattermost-k8s: 10:35:10 INFO unit.mattermost-k8s/0.juju-log db:5: emitting master_changed event for relation 5
unit-postgresql-k8s-0: 10:35:10 INFO juju.worker.uniter.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:35:10 INFO unit.mattermost-k8s/0.juju-log db:5: emitting master_gone event for relation 5
application-mattermost-k8s: 10:35:11 INFO unit.mattermost-k8s/0.juju-log db:5: emitting standby_changed event for relation 5
application-mattermost-k8s: 10:35:11 INFO unit.mattermost-k8s/0.juju-log db:5: emitting standby_gone event for relation 5
application-mattermost-k8s: 10:35:11 INFO unit.mattermost-k8s/0.juju-log db:5: emitting database_changed event for relation 5
application-mattermost-k8s: 10:35:11 INFO unit.mattermost-k8s/0.juju-log db:5: emitting database_gone event for relation 5
application-mattermost-k8s: 10:35:12 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:35:13 INFO unit.mattermost-k8s/0.juju-log db:5: emitting master_available event for relation 5
application-mattermost-k8s: 10:35:13 INFO unit.mattermost-k8s/0.juju-log db:5: emitting master_changed event for relation 5
application-mattermost-k8s: 10:35:15 INFO unit.mattermost-k8s/0.juju-log db:5: emitting standby_available event for relation 5
application-mattermost-k8s: 10:35:15 INFO unit.mattermost-k8s/0.juju-log db:5: emitting standby_changed event for relation 5
application-mattermost-k8s: 10:35:15 INFO unit.mattermost-k8s/0.juju-log db:5: emitting database_available event for relation 5
application-mattermost-k8s: 10:35:15 INFO unit.mattermost-k8s/0.juju-log db:5: emitting database_changed event for relation 5
application-mattermost-k8s: 10:35:16 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:35:17 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:36:36 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:38:37 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:39:01 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-departed" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:39:01 INFO juju.worker.uniter.operation ran "db-relation-departed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:39:02 INFO unit.mattermost-k8s/0.juju-log db:5: emitting master_changed event for relation 5
unit-postgresql-k8s-0: 10:39:02 INFO juju.worker.uniter.operation ran "db-relation-broken" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:39:02 INFO juju.worker.uniter.relation unknown relation 5 resolving next op
application-mattermost-k8s: 10:39:03 INFO unit.mattermost-k8s/0.juju-log db:5: emitting master_gone event for relation 5
application-mattermost-k8s: 10:39:03 INFO unit.mattermost-k8s/0.juju-log db:5: emitting standby_changed event for relation 5
application-mattermost-k8s: 10:39:03 INFO unit.mattermost-k8s/0.juju-log db:5: emitting standby_gone event for relation 5
application-mattermost-k8s: 10:39:03 INFO unit.mattermost-k8s/0.juju-log db:5: emitting database_changed event for relation 5
application-mattermost-k8s: 10:39:03 INFO unit.mattermost-k8s/0.juju-log db:5: emitting database_gone event for relation 5
application-mattermost-k8s: 10:39:03 INFO unit.mattermost-k8s/0.juju-log db:5: emitting database_relation_broken event for relation 5
application-mattermost-k8s: 10:39:03 INFO unit.mattermost-k8s/0.juju-log db:5: cleaning up broken relation 5
application-mattermost-k8s: 10:39:04 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-broken" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:39:04 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.relation unknown relation 5 resolving next op
unit-postgresql-k8s-0: 10:39:04 INFO juju.worker.uniter.relation unknown relation 5 resolving next op
unit-tls-certificates-operator-0: 10:39:37 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:40:51 INFO juju.worker.uniter.operation ran "db-relation-created" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:40:51 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-created" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:40:52 INFO unit.mattermost-k8s/0.juju-log db:6: emitting database_relation_joined event for relation 6
unit-postgresql-k8s-0: 10:40:52 INFO juju.worker.uniter.operation ran "db-relation-joined" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:40:53 WARNING unit.postgresql-k8s/0.juju-log db:6: DEPRECATION WARNING - `db` is a legacy interface
unit-postgresql-k8s-0: 10:40:53 WARNING unit.postgresql-k8s/0.juju-log db:6: Early exit on_relation_changed: No database name provided
application-mattermost-k8s: 10:40:53 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-joined" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:40:53 INFO juju.worker.uniter.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:40:54 WARNING unit.postgresql-k8s/0.juju-log db:6: DEPRECATION WARNING - `db` is a legacy interface
unit-postgresql-k8s-0: 10:40:54 INFO juju.worker.uniter.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:40:54 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:40:55 WARNING unit.postgresql-k8s/0.juju-log db:6: DEPRECATION WARNING - `db` is a legacy interface
application-mattermost-k8s: 10:40:55 INFO unit.mattermost-k8s/0.juju-log db:6: emitting master_available event for relation 6
application-mattermost-k8s: 10:40:55 INFO unit.mattermost-k8s/0.juju-log db:6: emitting master_changed event for relation 6
unit-postgresql-k8s-0: 10:40:56 INFO juju.worker.uniter.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:40:56 INFO unit.mattermost-k8s/0.juju-log db:6: emitting standby_available event for relation 6
application-mattermost-k8s: 10:40:56 INFO unit.mattermost-k8s/0.juju-log db:6: emitting standby_changed event for relation 6
application-mattermost-k8s: 10:40:57 INFO unit.mattermost-k8s/0.juju-log db:6: emitting database_available event for relation 6
application-mattermost-k8s: 10:40:57 INFO unit.mattermost-k8s/0.juju-log db:6: emitting database_changed event for relation 6
application-mattermost-k8s: 10:40:58 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:40:59 INFO juju.worker.caasoperator.runner start "mattermost-k8s/1"
application-mattermost-k8s: 10:40:59 INFO juju.worker.leadership mattermost-k8s leadership for mattermost-k8s/1 denied
application-mattermost-k8s: 10:40:59 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-mattermost-k8s-1
application-mattermost-k8s: 10:40:59 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1 unit "mattermost-k8s/1" started
application-mattermost-k8s: 10:40:59 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1 resuming charm install
application-mattermost-k8s: 10:40:59 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1.charm downloading ch:amd64/mattermost-k8s-27 from API server
application-mattermost-k8s: 10:40:59 INFO juju.downloader downloading from ch:amd64/mattermost-k8s-27
application-mattermost-k8s: 10:40:59 INFO juju.downloader download complete ("ch:amd64/mattermost-k8s-27")
application-mattermost-k8s: 10:40:59 INFO juju.downloader download verified ("ch:amd64/mattermost-k8s-27")
application-mattermost-k8s: 10:40:59 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:01 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1 hooks are retried true
application-mattermost-k8s: 10:41:01 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1 found queued "install" hook
application-mattermost-k8s: 10:41:02 INFO unit.mattermost-k8s/1.juju-log Running legacy hooks/install.
application-mattermost-k8s: 10:41:03 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1.operation ran "install" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:41:04 INFO juju.worker.uniter.operation ran "db-relation-joined" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:04 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1.operation ran "db-relation-created" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:41:05 WARNING unit.postgresql-k8s/0.juju-log db:6: DEPRECATION WARNING - `db` is a legacy interface
unit-postgresql-k8s-0: 10:41:05 INFO juju.worker.uniter.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:05 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:05 INFO juju.worker.caasoperator started pod init on "mattermost-k8s/1"
unit-postgresql-k8s-0: 10:41:06 WARNING unit.postgresql-k8s/0.juju-log db:6: DEPRECATION WARNING - `db` is a legacy interface
unit-postgresql-k8s-0: 10:41:07 INFO juju.worker.uniter.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:07 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:08 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1.operation ran "config-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:08 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1 found queued "start" hook
application-mattermost-k8s: 10:41:11 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:11 INFO unit.mattermost-k8s/1.juju-log Running legacy hooks/start.
application-mattermost-k8s: 10:41:12 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1.operation ran "start" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:13 ERROR unit.mattermost-k8s/1.juju-log db:6: db-relation-changed hook run before db-joined! Deferring event.
application-mattermost-k8s: 10:41:14 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:14 ERROR unit.mattermost-k8s/1.juju-log db:6: db-relation-changed hook run before db-joined! Deferring event.
application-mattermost-k8s: 10:41:15 INFO unit.mattermost-k8s/1.juju-log db:6: emitting database_relation_joined event for relation 6
application-mattermost-k8s: 10:41:16 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1.operation ran "db-relation-joined" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:17 INFO unit.mattermost-k8s/1.juju-log db:6: emitting master_available event for relation 6
application-mattermost-k8s: 10:41:17 INFO unit.mattermost-k8s/1.juju-log db:6: emitting master_changed event for relation 6
application-mattermost-k8s: 10:41:18 INFO unit.mattermost-k8s/1.juju-log db:6: emitting standby_available event for relation 6
application-mattermost-k8s: 10:41:18 INFO unit.mattermost-k8s/1.juju-log db:6: emitting standby_changed event for relation 6
application-mattermost-k8s: 10:41:18 INFO unit.mattermost-k8s/1.juju-log db:6: emitting database_available event for relation 6
application-mattermost-k8s: 10:41:18 INFO unit.mattermost-k8s/1.juju-log db:6: emitting database_changed event for relation 6
application-mattermost-k8s: 10:41:19 WARNING juju.worker.caasoperator.uniter.mattermost-k8s/0.operation we should run a leader-deposed hook here, but we can't yet
application-mattermost-k8s: 10:41:20 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1.operation ran "db-relation-changed" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:41:20 INFO juju.worker.uniter.operation ran "db-relation-departed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:21 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-departed" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:21 INFO unit.mattermost-k8s/0.juju-log db:6: emitting master_changed event for relation 6
application-mattermost-k8s: 10:41:22 INFO unit.mattermost-k8s/0.juju-log db:6: emitting master_gone event for relation 6
application-mattermost-k8s: 10:41:22 INFO unit.mattermost-k8s/0.juju-log db:6: emitting standby_changed event for relation 6
application-mattermost-k8s: 10:41:23 INFO unit.mattermost-k8s/0.juju-log db:6: emitting standby_gone event for relation 6
application-mattermost-k8s: 10:41:23 INFO unit.mattermost-k8s/0.juju-log db:6: emitting database_changed event for relation 6
application-mattermost-k8s: 10:41:23 INFO unit.mattermost-k8s/0.juju-log db:6: emitting database_gone event for relation 6
application-mattermost-k8s: 10:41:23 INFO unit.mattermost-k8s/0.juju-log db:6: emitting database_relation_broken event for relation 6
application-mattermost-k8s: 10:41:23 INFO unit.mattermost-k8s/0.juju-log db:6: cleaning up broken relation 6
application-mattermost-k8s: 10:41:23 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "db-relation-broken" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:23 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.relation unknown relation 6 resolving next op
application-mattermost-k8s: 10:41:24 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "stop" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:24 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.relation unknown relation 6 resolving next op
application-mattermost-k8s: 10:41:25 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0.operation ran "remove" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:41:25 INFO juju.worker.caasoperator.uniter.mattermost-k8s/0 unit "mattermost-k8s/0" shutting down: unit dead
application-mattermost-k8s: 10:41:25 INFO juju.worker.leadership mattermost-k8s/1 promoted to leadership of mattermost-k8s
application-mattermost-k8s: 10:41:25 INFO juju.worker.caasoperator.runner stopped "mattermost-k8s/0", err: <nil>
application-mattermost-k8s: 10:41:25 WARNING juju.worker.caasoperator stopping uniter for dead unit "mattermost-k8s/0": worker "mattermost-k8s/0" not found
application-mattermost-k8s: 10:41:25 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1 found queued "leader-elected" hook
application-mattermost-k8s: 10:41:27 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-postgresql-k8s-0: 10:43:07 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
application-mattermost-k8s: 10:45:08 INFO juju.worker.caasoperator.uniter.mattermost-k8s/1.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-tls-certificates-operator-0: 10:45:35 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)

Additional context

No response