Dev #1050
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| import hail as hl | ||
|
|
||
| from v03_pipeline.lib.annotations import sv | ||
| from v03_pipeline.lib.migration.base_migration import BaseMigration | ||
| from v03_pipeline.lib.model import DatasetType, ReferenceGenome | ||
| from v03_pipeline.lib.reference_datasets.reference_dataset import ReferenceDataset | ||
|
|
||
| # This vcf was generated with the gatk command: | ||
| # | ||
| # gatk SVConcordance --verbosity DEBUG --evaluation /var/seqr/phase4.seqr.gnomad_v4_tmp.vcf.gz | ||
| # --truth /var/seqr/gnomad.v4.1.sv.sites.modified.vcf.bgz | ||
| # --sequence-dictionary gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.dict | ||
| # | ||
| # Followed by: | ||
| # bcftools annotate --rename-annots /var/seqr/remap /var/seqr/phase4.seqr.gnomad_v4_tmp.vcf.gz | bgzip > /var/seqr/phase4.seqr.gnomad_v4.vcf.gz | ||
| # | ||
| # where remap contains "INFO/TRUTH_VID GNOMAD_V4.1_TRUTH_VID" | ||
| PHASE_4_CALLSET_WITH_GNOMAD_V4 = 'gs://seqr-loading-temp/phase4.seqr.gnomad_v4.vcf.gz' | ||
|
|
||
|
|
||
| class AddGnomadSVs(BaseMigration): | ||
| reference_genome_dataset_types: frozenset[ | ||
| tuple[ReferenceGenome, DatasetType] | ||
| ] = frozenset( | ||
| ((ReferenceGenome.GRCh38, DatasetType.SV),), | ||
| ) | ||
|
|
||
| @staticmethod | ||
| def migrate(ht: hl.Table, **_) -> hl.Table: | ||
| mapping_ht = ( | ||
| hl.import_vcf( | ||
| PHASE_4_CALLSET_WITH_GNOMAD_V4, | ||
| reference_genome=ReferenceGenome.GRCh38.value, | ||
| force_bgz=True, | ||
| ) | ||
| .key_rows_by('rsid') | ||
| .rows() | ||
| ) | ||
| ht = ht.annotate( | ||
| **{ | ||
| 'info.GNOMAD_V4.1_TRUTH_VID': mapping_ht[ht.key].info[ | ||
| 'GNOMAD_V4.1_TRUTH_VID' | ||
| ], | ||
| }, | ||
| ) | ||
| gnomad_svs_ht = ReferenceDataset.gnomad_svs.get_ht(ReferenceGenome.GRCh38) | ||
| ht = ht.annotate(gnomad_svs=sv.gnomad_svs(ht, gnomad_svs_ht)) | ||
| ht = ht.drop('info.GNOMAD_V4.1_TRUTH_VID') | ||
| return ht.annotate_globals( | ||
| versions=ht.globals.versions.annotate(gnomad_svs='1.0'), | ||
| enums=ht.globals.enums.annotate(gnomad_svs=hl.Struct()), | ||
|
There was a problem hiding this comment. this feels unnecessary, as this would already be in the existing table globals and does not seem to be changing There was a problem hiding this comment. |
||
| ) | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wouldn't this already be the version- shouldn't we bump it?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this migration will create the
gnomad_svstable, but you're right, the globals annotation happens when we callReferenceDataset.gnomad.get_htso we don't need to set the version/enums here.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oooh, there's some confusing bits here.
ReferenceDataset.gnomad_svs.get_htsets the globals on thegnomad_svsreference dataset (gs://seqr-reference-data/v3.1/GRCh38/gnomad_svs/1.0.htif persisted). This code was setting the globals on the SV annotations table as they're not currently there. It's not overwhelmingly important, as we exclude gnomad_svs from consideration when computing the reference datasets to update, and it would also be annotated by the normal process because I kept separatefor_reference_genome_dataset_type_annotations(which includesgnomad_svs) andfor_reference_genome_dataset_type_annotations_updatesmethods (which does not). I felt the migration should include the correct globals on annotations table for completeness though!Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
right I got jumbled, this is an
annotationtable migration.This is good reason to keep the globals annotation in this migration. @hanars, I'm going to ad the globals
versionsandenumsannotations back and re-request review.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, it would be awesome if these had tests, but not entirely sure it would have caught this 😬.