Variant search page won't load

[tomiles]

We managed to stand up a new Seqr instance from scratch using the Hail backend. The loading pipeline ran successfully, the Hail table was reloaded, but when we go to the variant search page for the loaded family, we just get a blank page with the following errors in the browser console. None of the backend services show any errors in their logs, so we don't have much to go on to debug. Can you give any pointers on how we can try and figure this out? We tested both the gcloud-dev and gcloud-prod version of the seqr container and both show this issue.

[Info] Blocked connection to known tracker https://www.googletagmanager.com/gtag/js in frame displaying https://seqr.prod.cmgg.be/variant_search/family/F000001_censored (F000001_censored, line 1)
[Info] Blocked connection to known tracker https://www.googletagmanager.com/gtag/js (F000001_censored, line 1)
[Error] Refused to apply a stylesheet because its hash, its nonce, or 'unsafe-inline' does not appear in the style-src directive of the Content Security Policy. (F000001_censored, line 1)
[Error] TypeError: null is not an object (evaluating 'styleElement.sheet.insertRule') — hidden:151
	handleMessage (advanced-script.js:262)
[Info] Blocked connection to known tracker https://www.googletagmanager.com/gtag/js in frame displaying https://seqr.prod.cmgg.be/variant_search/family/F000001_censored
[Info] Blocked connection to known tracker https://www.googletagmanager.com/gtag/js
[Log] initialJSON:  – {meta: Object, user: Object} (F000001_censored, line 1)
{meta: Object, user: Object}Objectmeta: {version: "v1.0-c8341b45", hijakEnabled: false, oauthLoginProvider: "", elasticsearchEnabled: false, warningMessages: [], …}Objectuser: {username: "censored", email: "censored", firstName: "", lastName: "", lastLogin: "2025-02-07T09:17:10.437Z", …}ObjectObject Prototype
[Log] Creating store with initial state: – {meta: Object, user: Object, projectsTableState: {filter: "SHOW_ALL"}, …} (app-c8341b45.js, line 1)
{meta: Object, user: Object, projectsTableState: {filter: "SHOW_ALL"}, familyTableState: {familiesSearch: "", familiesSortOrder: "FAMILY_NAME", familiesSortDirection: 1}, savedVariantTableState: Object, …}ObjectfamilyTableFilterState: {}ObjectfamilyTableState: {familiesSearch: "", familiesSortOrder: "FAMILY_NAME", familiesSortDirection: 1}Objectmeta: {version: "v1.0-c8341b45", hijakEnabled: false, oauthLoginProvider: "", elasticsearchEnabled: false, warningMessages: [], …}ObjectprojectsTableState: {filter: "SHOW_ALL"}ObjectsavedVariantTableState: {hideExcluded: false, hideReviewOnly: false, categoryFilter: "ALL", sort: "FAMILY_GUID", page: 1, …}ObjectsearchFamiliesByHash: {}ObjectsearchesByHash: {}Objectuser: {username: "root", email: "censored", firstName: "", lastName: "", lastLogin: "2025-02-07T09:17:10.437Z", …}ObjectvariantSearchDisplay: {sort: "XPOS", page: 1, recordsPerPage: 100}Objectpage: 1recordsPerPage: 100sort: "XPOS"Object PrototypeObject Prototype
[Error] TypeError: Array(o).keys().map is not a function. (In 'Array(o).keys().map((function(e){return t.slice(e*i,(e+1)*i)}))', 'Array(o).keys().map' is undefined) — app-c8341b45.js:1:791973
	fl (app-c8341b45.js:1:341488)
	(anonymous function) (app-c8341b45.js:1:343603)
	hi (app-c8341b45.js:1:307741)
	gl (app-c8341b45.js:1:345295)
	Nu (app-c8341b45.js:1:361683)
	(anonymous function) (app-c8341b45.js:1:381102)
	Du (app-c8341b45.js:1:358429)
	Eu (app-c8341b45.js:1:354564)
	hu (app-c8341b45.js:1:351729)
	ec (app-c8341b45.js:1:367545)
	(anonymous function) (app-c8341b45.js:1:368931)
	wu (app-c8341b45.js:1:354716)
	ac (app-c8341b45.js:1:368916)
	(anonymous function) (app-c8341b45.js:1:1586035)
	(anonymous function) (app-c8341b45.js:1:1587759)
	Global Code (app-c8341b45.js:1:1587762)
[Error] TypeError: Array(o).keys().map is not a function. (In 'Array(o).keys().map((function(e){return t.slice(e*i,(e+1)*i)}))', 'Array(o).keys().map' is undefined) — app-c8341b45.js:1:791973
	fl (app-c8341b45.js:1:341488)
	(anonymous function) (app-c8341b45.js:1:343603)
	hi (app-c8341b45.js:1:307741)
	gl (app-c8341b45.js:1:345295)
	Nu (app-c8341b45.js:1:361683)
	(anonymous function) (app-c8341b45.js:1:381102)
	Du (app-c8341b45.js:1:358429)
	Eu (app-c8341b45.js:1:354564)
	hu (app-c8341b45.js:1:351729)
	ec (app-c8341b45.js:1:367545)
	(anonymous function) (app-c8341b45.js:1:368931)
	wu (app-c8341b45.js:1:354716)
	ac (app-c8341b45.js:1:368916)
	(anonymous function) (app-c8341b45.js:1:1586035)
	(anonymous function) (app-c8341b45.js:1:1587759)
	Global Code (app-c8341b45.js:1:1587762)
[Error] TypeError: Array(o).keys().map is not a function. (In 'Array(o).keys().map((function(e){return t.slice(e*i,(e+1)*i)}))', 'Array(o).keys().map' is undefined) — app-c8341b45.js:1:791973
	fl (app-c8341b45.js:1:341488)
	(anonymous function) (app-c8341b45.js:1:343603)
	hi (app-c8341b45.js:1:307741)
	gl (app-c8341b45.js:1:345295)
	Nu (app-c8341b45.js:1:361683)
	(anonymous function) (app-c8341b45.js:1:381102)
	Du (app-c8341b45.js:1:358429)
	Eu (app-c8341b45.js:1:354564)
	hu (app-c8341b45.js:1:351729)
	ec (app-c8341b45.js:1:367545)
	(anonymous function) (app-c8341b45.js:1:368931)
	wu (app-c8341b45.js:1:354716)
	ac (app-c8341b45.js:1:368916)
	(anonymous function) (app-c8341b45.js:1:1586035)
	(anonymous function) (app-c8341b45.js:1:1587759)
	Global Code (app-c8341b45.js:1:1587762)
[Error] TypeError: Array(o).keys().map is not a function. (In 'Array(o).keys().map((function(e){return t.slice(e*i,(e+1)*i)}))', 'Array(o).keys().map' is undefined)
	Nu (app-c8341b45.js:1:362335)
	(anonymous function) (app-c8341b45.js:1:381102)
	Du (app-c8341b45.js:1:358429)
	Eu (app-c8341b45.js:1:354564)
	hu (app-c8341b45.js:1:351729)
	ec (app-c8341b45.js:1:367545)
	(anonymous function) (app-c8341b45.js:1:368931)
	wu (app-c8341b45.js:1:354716)
	ac (app-c8341b45.js:1:368916)
	(anonymous function) (app-c8341b45.js:1:1586035)
	(anonymous function) (app-c8341b45.js:1:1587759)
	Global Code (app-c8341b45.js:1:1587762)

[hanars]

I am not sure what you mean when you say you tried the "gcloud-prod" version of the container. For the hail backend, you need to be deploying using helm, and you should not be specifying container tags, only chart versions. Can you confirm that you are running seqr using the helm charts as described in the documentation:

• https://github.com/broadinstitute/seqr/blob/master/deploy/LOCAL_INSTALL_HELM.md
• https://github.com/broadinstitute/seqr-helm/blob/main/README.md

[tomiles]

That's what we did, setup is done with the helm charts. I just meant to say is at the low level I tested the latest seqr (helm) release (where appVersion corresponds to the container tag in gcr.io/seqr-project/seqr) as well as the most recent dev version tagged as gcloud-dev (by manually editing the seqr appversion) just to verify that my issue wasn't solved already in an as of yet unreleased version.

[hanars]

Thanks for clarifying. Are you seeing this JS error on every single page, or only some? Are you for example still able to view the project page for the corresponding project, or no? Can you provide the URL path for the page(s) where this error is occurring?

[tomiles]

I’m only seeing the error on the variant search page. URL https://seqr.prod.cmgg.be/variant_search/family/F000001_censored

Tracing the error a bit in-browser console seems to indicate it happens in the JS/React code that builds up the filter form. But not easy to trace exactly with the minified js code.

[hanars]

Thanks. Can you also confirm that you are accessing seqr in a recent version of the Chrome browser?

[tomiles]

Indeed, it seems it works in Chrome. I didn't realize this was the only browser that is supported, since we previously used it without issue in other browsers. I will make note of that when we do internal user training.

Although now we are fighting hail-search to give us timely results, because we keep hitting time-outs. Any tips on how best to scale up/optimize the resources (CPU/memory/storage type) for the Hail-Search task?

[hanars]

Yes, seqr is only supported in chrome. This is documented in the seqr FAQ but I can see how that might be easy to miss when setting up your own installation

Regarding optimization, as we communicated in our backend announcement this backend is substantially slower than the elasticsearch backend., and this is not something that can be solved through resource optimization - this backend will have certain searches that will be slow no matter what you do.
That said, you can potentially improve performance by setting memory limits that are better tuned to the available memory in your deployment. Add the following variables to your custom values.yaml

hail-search:
    environment:
      MACHINE_MEM: "<YOUR_MACHINE_MEM>"
      JAVA_OPTS_XSS: "<YOUR_JAVA_STACK_SIZE>"

You want to set MACHINE_MEM to close to the amount of available memory on the machine you are running (in GB). So for example if you have 100GB available you might want to use "96". When initializing, seqr will run its won adjustment of this value down to prevent hail from using 100% of available memory, so you can set this to pretty close to the available memory on the machine.
You want to set JAVA_OPTS_XSS to as higher value without going so high that java crashes by running out of memory. The default is "4M", we use "16M"

In general, the backed performs better with lots of memory available to it, although at some point this starts to have diminishing returns and more memory does not help anymore. How much memory is optimal varies depending on your total data size, so we can't give specific recommendations for an individual deployment. While having more CPUs also improves performance, we found memory was a bigger factor than number of CPUs. In terms of storage type, the more local the better, so a mounted local SSD will be better than non-local. If you have the ability to use an in-memory file system that could potentially be even faster but it would be difficult to get that to set up properly so I do not recommend that

[tomiles]

Thanks, for the tips. I'm testing out some permutations of those variables.

One last question: Do you know if it is an option to use s3://... or gs://... paths for
HAIL_SEARCH_DATA_DIR, HAIL_TMP_DIR, LOADING_DATASETS_DIR, REFERENCE_DATASETS_DIR, ... like it is mentioned on the LOCAL_INSTALL_HELM.md docs but without running the pipeline on DATAPROC ? This would make managing our data collection easier, and we could apply some data policies instead of shuffling matrix tables between SSD/hot and non-SSD/cold/archive tiers on traditional network filesystems. In theory, it seems Hail should be able to handle these paths.

[bpblanken]

Hi @tomiles,

The pipeline will work with gs:// for some of those directories (HAIL_TMP_DIR and LOADING_DATASETS_DIR ) but will not work with s3://. There are several places where we're relying on luigi's GCS integration and have not yet built out support for s3 itself.

There's a special piece of logic to handle HAIL_SEARCH_DATA_DIR and REFERENCE_DATASETS_DIR being gs://-prefixed though, that I think will not work correctly right now. Those directories are, by default, shared between the pipeline that writes to them and the "hail-search" app that reads from them on local disk. The pipeline has a step that will sync between gcs buckets and the local disks needed for "hail-search", but that step is currently tied to the RUN_PIPELINE_ON_DATAPROC environment variable. I can make a ticket for us to decouple this behavior.