Elasticsearch 7 upgrade help and support

[tommyli]

Describe the bug

This is not a bug report but more a support/help request. We at MCRI recently merged latest upstream (this repo) into our forked repo (up to inclusive commit d86fb91).
Last time we did this was around 4 months ago. A major change within these 4 months is the upgrade from Elasticsearch 6 to 7 as described here. So far our merging and testing has been pretty good. Thanks very much for all your hard work in the ongoing development and improvement to Seqr and making it all freely available!

We had to make a couple of changes in the seqr.utils.elasticsearch modules to ensure backwards compatibility with our ES 6 instance. We do have plans to upgrade to ES 7 as we feel matching upstream as close as possible is important for maintainability. However, we may do that separately. We have a few questions regarding using latest Seqr codebase with ES 6 and upgrading to ES 7 in general.

• Can you think of any major problems with running latest Seqr codebase on ES 6 cluster? I've read through the changes in breaking_70_search_changes
  and cannot identify anything major. More important is that I've done some manual testing to ensure latest codebase can query our MCRI Elasticsearch 6 cluster fine.
• The couple of changes we made makes Seqr backwards compatible on ES 6, would you like a PR for this?
• We plan to follow rolling upgrade instructions
  to upgrade our ES 6 (6.3) cluster to match upstream (7.8). Were there anything to watch for when upgrading?

[hanars]

• Can you think of any major problems with running latest Seqr codebase on ES 6 cluster? I've read through the changes in breaking_70_search_changes
  and cannot identify anything major. More important is that I've done some manual testing to ensure latest codebase can query our MCRI Elasticsearch 6 cluster fine.

As long as you are able to query and add new datasets, I don't see any issues with this. Indices generated for ES6 can be read by ES7 with no issues, so theres no risk that you are going to lose data by using ES6 for longer. I would expect a few places in the code to need some tweaking to maintain ES6 compatibility, the biggest place we needed to make changes was with mappings, as the mapping endpoints changed from returning {"mapping": {"variant": <mapping>}} to {"mapping": <mapping>}. But if search is working that means you are probably handling that okay

• The couple of changes we made makes Seqr backwards compatible on ES 6, would you like a PR for this?

We don't generally support backwards compatibility as it increases code complexity and makes maintenance harder, so I don't expect we would want to merge that PR. However, if you want to put up a PR that we will then close so other users can use that as a reference, please feel free!

• We plan to follow rolling upgrade instructions
  to upgrade our ES 6 (6.3) cluster to match upstream (7.8). Were there anything to watch for when upgrading?

We did not do rolling updates. We deploy elasticsearch in GCP and have our data on persistent disks, so we snapshotted those disks and then deployed a new ES7 instance backed by those disks. We did this because it allowed us to have ES6 and ES7 running in parallel for testing purposes, and it meant that the switchover from 6 to 7 was very fast with minimal down time.
I don't think there is any real reason to do it the way we did, and I expect rolling updates should work fine. I just don't have any insight I can share, as I never tried it

[tommyli]

We managed to create a new ES7 cluster using persistent volumes backed by existing persistent disks (as suggested above). These disks are snapshots of disks from our old ES6 instance. The new ES7 cluster is up and running but returns no indices when I run curl -u "elastic:$PASSWORD" -k "http://localhost:9200/_cat/indices?v". Were any steps/tasks required to "upgrade" the old indices to be queryable on the new ES7 cluster?

[hanars]

You need to make sure the data is in the correct location. Is you are using kubernetes or if you use the docker-compose file we provide, your data has to be mounted to /usr/share/elasticsearch/data. Is that where the data on the disks is?

As a warning, when we were spinning up the new clusters off of the persistent disks we needed to go into the data folders on the disks and delete the _state data and also sometimes the node.lock files. This was only the first time, I think it was a version mismtach issue, but might be worth noting for you guys

[tommyli]

Thanks for your help @hanars and pointing me in the right direction. Here are my findings:

• The problem was to do with mounts and incorrect path locations. The pods/containers' mountPath was correctly set
  to /usr/share/elasticsearch/data. However, the data/files on the disk was at /data as opposed to /. This meant
  the data in the pods were sitting at /usr/share/elasticsearch/data/data instead of /usr/share/elasticsearch/data.
  I'm not 100% sure why the data is sitting at /data and not /, I suspect it's how it was
  configured
  in our current ES6 instance.
• One approach to fix this is to add a symlink: ln -sf /usr/share/elasticsearch/data/data/nodes /usr/share/elasticsearch/data/nodes. This needs to be done for each old disk that needs to be brought over/migrated,
  in our case we had 3 disks for data nodes and 2 disks master nodes. This only needs to be done once as the symlink
  is persisted as part of the disk.
• Another approach is to use K8s Volume subPath.
  For our data and master node configs, we explicitly added below in our elasticsearch.gcloud.yaml (also see
  here):

  containers:
  - name: elasticsearch
  ...
    volumeMounts:
    - mountPath: /usr/share/elasticsearch/data
      name: elasticsearch-data
      subPath: data
  ...

The subPath: data config allowed this mount to use a subdirectory of the referenced volume instead of its root.
We opted for this option as we wanted such tweaks/configurations to be more explicit.

• Another approach I suppose is to manually copy all the data from /data back to / on each disk. I didn't try this
  option but I can confirm both approaches above work.

[hanars]

Oh sorry for not thinking of this, we actually also had this problem. We solved it by mounting 2 persistent disks to each data node, the one with the old data and a blank one. Then we used rsync to copy the data to the new disks at the right path and then we redeployed using those disks and not the old disks. I think your solutions are just as good so I don't recommend you do this, just wanted to share

[tommyli]

Another question we had is how Seqr should communicate with this Kubernetes cluster. I can see an Internal Load Balancer Service (elasticsearch-es-data-nodes) but is this how Seqr should access Elasticsearch? If so, then why only data-nodes are selected? I thought the selector should match all nodes, i.e. similar to the Cluster IP Service (elasticsearch-es-http) that kubernetes-elasticsearch-all-in-one.yaml already creates for you.

We created another Internal Load Balancer but selecting all nodes and it seems to work so want some clarification on how the Elasticsearch cluster was intended to be used.

[hanars]

we created that load balancer for data loading, so we would have a stable IP address to use for out loading jobs that wouldn't need to be reset whenever elasticsearch redeployed, as we run the loading pipeline in a separate kubernetes cluster which therefore can't directly access the pods.

ECk will automatically create a kubernetes service named elasticsearch-es-http that includes all the nodes, so we use that for seqr to connect. We just set the ELASTICSEARCH_SERVICE_HOSTNAME variable to elasticsearch-es-http and it works without needing to configure anything special

[tommyli]

I see. Our setup is a bit different, we use Kubernetes for ES and the rest uses docker-compose. Both Seqr (Django) and our loading pipeline will access Elasticsearch cluster the same way (i.e. via docker-compose) so we'll just create one load balancer for both Seqr and the loading pipeline.

Thanks for your help and clarifications.