Pods are not getting into ready state with Error in process <0.275.0> on node

[viveksinghggits]

Description

I installed a couchdb instance on my Kubernetes cluster and used this platform to backup the volumes and the workload manifests that well.
Now when I try to restore the the database using those volumes and manifests the pods are not getting into ready state and this is the error that I can see in the pod logs

[error] 2020-10-27T05:03:22.610384Z couchdb@my-release-couchdb-0.my-release-couchdb.justcouch-restored.svc.cluster.local emulator -------- Error in process <0.275.0> on node 'couchdb@my-release-couchdb-0.my-release-couchdb.justcouch-restored.svc.cluster.local' with exit value:
{{rexi_DOWN,{'couchdb@my-release-couchdb-1.my-release-couchdb.justcouch-ns.svc.cluster.local',noconnect}},[{mem3_rpc,rexi_call,3,[{file,"src/mem3_rpc.erl"},{line,394}]},{mem3_seeds,'-start_replication/1-fun-0-',1,[{file,"src/mem3_seeds.erl"},{line,99}]}]}

[notice] 2020-10-27T05:03:42.400712Z couchdb@my-release-couchdb-0.my-release-couchdb.justcouch-restored.svc.cluster.local <0.529.0> fb5c28d307 10.244.0.133:5984 10.244.0.242 undefined GET /_up 404 ok 0
[info] 2020-10-27T05:03:42.416955Z couchdb@my-release-couchdb-0.my-release-couchdb.justcouch-restored.svc.cluster.local <0.32.0> -------- SIGTERM received - shutting down

[error] 2020-10-27T05:03:42.421902Z couchdb@my-release-couchdb-0.my-release-couchdb.justcouch-restored.svc.cluster.local <0.486.0> -------- gen_server <0.486.0> terminated with reason: killed
  last msg: {'EXIT',<0.373.0>,killed}
     state: {state,#Ref<0.61916457.2739535873.205975>,couch_replicator_doc_processor,nil,<<"_replicator">>,#Ref<0.61916457.2739404801.205976>,nil,[],true}
    extra: []

Complete pod logs can be found here
I was not able to figure out what exactly the reason could be.

Steps to Reproduce

Expected Behaviour

I should get all the couchdb pods up and running.

Your Environment

this is the helm release that we installed

my-release	justcouch-ns	1       	2020-10-26 19:10:40.322422397 +0530 IST	deployed	couchdb-3.3.4	3.1.0

Additional Context

» kubectl version 
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.11", GitCommit:"ea5f00d93211b7c80247bf607cfa422ad6fb5347", GitTreeState:"clean", BuildDate:"2020-08-13T15:11:47Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

[wohali]

Have you looked at the CouchDB Helm Chart? https://github.com/apache/couchdb-helm

[viveksinghggits]

Hey @wohali,
yeah so I installed the DB using that helm chart itself, and it gets installed as expected. The problem is when I restore after taking backup of the volumes and workload manifests.

[willholley]

The restored node names suggest a different FQDN to the original install (couchdb@my-release-couchdb-0.my-release-couchdb.justcouch-restored.cluster.local vs couchdb@my-release-couchdb-0.my-release-couchdb.justcouch.svc.cluster.local). From the logs, that looks to be breaking the seedlist (see /opt/couchdb/etc/default.d/seedlist.ini as the node names that the cluster expects are not reachable).

If you perform a full volume restore of a Helm-chart deployed cluster like this you'll need to make sure the restored cluster has the same name and namespace as the original.

[viveksinghggits]

ok, so we can restore that into a new namespace because the seed list is persisted somehow, would this sentence be correct?

[willholley]

if you restore to a new namespace you would need to update the seedlist (defined in a configmap at https://github.com/apache/couchdb-helm/blob/56dc7ea595ae64fbda49a9b921c0d610bc6f76e2/couchdb/templates/configmap.yaml#L21). You're effectively changing the identify of the couchdb nodes, which will also mean you need to update the shard maps of all your databases and possibly other places as well - it's a non-trivial task.

[viveksinghggits]

ok, so I am gonna try to change that configmap before restore so that it would have correct seeds. Do you think changing the configmap with seed list is going to resolve the issue..

[willholley]

You'll also need to update the shard maps of all the databases you restore to reference the new node names (see https://docs.couchdb.org/en/master/cluster/sharding.html#examining-database-shards).

[viveksinghggits]

Hey Thank you. let me have a look at how we can update the shards, but updating the configmap actually at least got the pods into running state.