Announcing a New seqr Search Backend #4531
Replies: 4 comments 14 replies
-
|
Hi @lynnpais . Thanks for this update! :). I was curious about the new search backend. Have you all made any choices regarding how you will move away from Hail as a backend variant store? |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for the heads up @hanars. Do you have any information about the database or format or service you will be using for the variant data store? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Robert, our team does not have the bandwidth to document our internal development for external consumption. Is there a reason you are so interested in getting such documentation early? We will provide meaningful documentation once we are ready to release this code to the community |
Beta Was this translation helpful? Give feedback.
-
|
Hi @hanars. I think we're getting our wires crossed. I understand there's a lot going on, so I was looking for something more general and not something super detailed like documentation. If "it's complicated" or you don't know or you can't say... well that's completely fine. We were going to take a look at Clickhouse anyways to see how it works since you dropped that in the comment section already. But... if you can supply any general information regarding the database/service/format, that would be helpful for me and my team as we run our own instance of seqr. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Robert, we will not be releasing any detailed information about the new backend until it is ready to be installed and run. We do not see utility in your group attempting to preempt the release by trying to create your own preliminary implementation in your local instance as we would be unable to support that going forward. At this time we ask for everyone's patience until we have a maintainable and supportable backend available |
Beta Was this translation helpful? Give feedback.
-
|
"no means no" 😆. Got it. |
Beta Was this translation helpful? Give feedback.
-
|
Since the loading pipeline for full genome sample data seems pretty slow in our "local install" setup we where wondering if there is an option to run the loading pipeline locally (with |
Beta Was this translation helpful? Give feedback.
-
|
Hi @tomiles, Unfortunately I'm not sure a standalone spark cluster will work without a bit of configuration work. One potential approach would be to override the docker image and kubernetes |
Beta Was this translation helpful? Give feedback.
-
|
Hello, hope you're doing well and thanks for this update! I have question regarding the calculation for Global Allele Frequencies: The scenario that I am thinking of is, if I had 3 singleton exomes (A,B,C) across three different project in seqr. They're not joint called as they belong to different projects and if I did a lookup for a hypothetical variant, say chr-101-A-C, that is present in exome A in a heterozygous state but not called in exome B and C. Would the reported global allele frequency be 1/2 (0.5) or 1/6 (0.166) (i.e. assuming ref at the site of a missing call)? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Anjali, This is actually an issue we have struggled with a bit with this new data model. In the example you provide, the AF would be 1/2 because we do not have enough information to accurately extrapolate about the call in samples B and C. For example, what if project A used a different sequencing product than B and C and had slightly different capture regions - the absence of the variant in B and C might therefore be because it was ref for all samples and should therefore count toward the AN but it might also be truly absent from the calls and should not count toward the AN. Since we have no way to know why a call is absent from a joint called VCF we do not consider it at all. However, this also creates its own problem - an AF of 1/2 is very high given that, in fact, this is potentially a totally de-novo variant never seen in any other project and it could be very rare. Over the coming months, we are planning to fully remove the global AN and AF from seqr and only expose the global AC and Hom count. We will be adjusting our standard searches to instead use a hard AC cutoff instead of an AF cutoff for the seqr frequency in searches. We are still in the design phase for this update so we have not finished deciding on the final UI for this change, but will probably be including some additional details about the total available sample count in seqr on hover to give users a sense of the rough AN. In the meantime, the ability to use an AC filter for the global seqr frequency is already available, and if the scenario you describe is one that is common in your seqr installation (i.e. loading data from VCFs with very few samples going into seqr) it may be a good idea for your team to adjust your searches to use that AC filter instead of the AF filter for now. I hope this helps, let me know if you have any further questions! |
Beta Was this translation helpful? Give feedback.
-
|
Hi Hana, thank you very much for explaining this, its very helpful! |
Beta Was this translation helpful? Give feedback.
-
|
Is there any docs/guidance on how to remove family data from a project (using the hail backend)? so we later run it through the loading pipeline again after an full reanalysis. In the past we could simple remove the ES index for that family ( we load data per family) and reimport. I tried running the |
Beta Was this translation helpful? Give feedback.
-
|
In addition to deleting data from the backend you need to disable search on the family in the seqr app itself. This is well-supported at the whole project level but not at the family level. To disable search in a family, you have to go into the |
Beta Was this translation helpful? Give feedback.
-
|
Also, we encourage you to open new discussion posts with these sorts of questions to enable other users to be able to find similar questions and answers on their own. Ideally, we would like to keep the discussion on thes post directly related to questions around how and whether to switch backends |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
The seqr team is excited to announce support for a new backend variant search infrastructure powered by Hail. We will continue to support the existing Elasticsearch backend as a core part of seqr. Please review the new changes and considerations to determine if and when to upgrade your instance of seqr from the old Elasticsearch backend to the new backend.
New Features
Self Service Data Loading Interface: Loading data to seqr can be done directly from the user interface by a non-technical user, removing the need for complicated infrastructure setup and manually running scripts. Additionally, a lightweight user interface for the loading pipeline is available in seqr to simplify troubleshooting issues with loading. A walkthrough of how to use the loading interface is found in the local install documentation within the seqr repo.
Updated VEP and Plugin Annotations: We’ve updated the data loading pipeline to run VEP 110 and include support for the AlphaMissense, UTRAnnotator, and SpliceRegion plugins.
Global Allele Frequencies: Allele frequencies are computed internally within the seqr backend across all loaded callsets, and users see this global frequency regardless of their access to the underlying callsets.
Variant Lookup Tool: Users can now look into any single variant in seqr, including variants outside of your own projects, to see affected/unaffected status and some phenotype data. The information shown will depend on the level of access. Project-level opt out for this feature is an option.
Allele Registration w/ Clingen Allele Registry: New variants loaded into seqr may be optionally registered within the Clingen Allele Registry. Information on how to enable this feature is found below in the local install documentation in the seqr repo.
Automatic Reference Data Updates: seqr will automatically source variant reference data updates from both the codebase and external sources (e.g., Clinvar). These updates will propagate to all variants in seqr without needing to reload data or any other manual intervention.
Data Size and Cost Efficiency: The new variant store representation is ~20x more efficient in comparison to the elasticsearch structure.
Modernized Deployment: The minimal docker-compose deployment of seqr has been replaced by a Helm Chart, enabling scalable cloud deployments on Kubernetes. Technical documentation for how to set up the new infrastructure and how to migrate existing application data is available in the open source seqr-helm repository.
Performance Considerations
While the new search backend is more efficient and scalable, most searches, especially recessive searches and searches across multiple projects, are experiencing substantially slower performance. See the below “Future Compatibility” section for more details on how possible future performance improvements will be made available.
We strongly recommend testing out the performance of seqr on the main Broad instance before deciding whether or not the performance will be acceptable for your team. You can run test searches here to get a better understanding of the new search features and performance (you will first need to register your email for a free account in AnVIL).
Future Compatibility
For the time being, all seqr development will be compatible with either the existing Elasticsearch backend or the new Hail backend. However, all new search-specific functionality (i.e., eventual support for structural variants or long read data) will only be made available in the new Hail backend.
Due to ongoing performance concerns, it is likely that seqr will transition to a different search backend in 2025. Should this happen, tooling will be made available to easily migrate from the Hail backend to whatever comes next.
Deprecated Features
This release is cloud-agnostic. It does not yet support running the loading pipeline on Google Dataproc clusters, but we anticipate that this feature will be available in 2025.
Data Migration
In order to migrate your search backend, you will need to re-load each VCF into seqr via the loading interface. There is no functionality to export directly from Elasticsearch. Our prospective plan is that this new Hail backend will support direct data migrations to future versions.
There are instructions in the helm chart docs for migrating all other application data and static files. This means that all user-provided data including case metadata and tagged variants will be preserved.
Beta Was this translation helpful? Give feedback.
All reactions