20230424 developer call notes

[hammer]

Pre-notes

• Notes from previous meeting
• CSIRO meeting
• Potential new users

PRs

• sgkit#1021 Generate VCF header when writing, if no header is explicitly supplied
• sgkit#1056 Add variant id to simulate dataset
• sgkit#1072 [WIP] Add tutorial on relationship matrices
• sgkit#1073 Make zarr_array_sizes run in parallel

Issues

• sgkit#1059 sgkit.io.vcf.vcf_to_zarr() fails to convert VCFs with INFO/CSQ and other unbounded annotations
• sgkit#1061 Update pedigree_kinship API to match hybrid_relationship
• sgkit#1062 Endelman-Jannik GRM
• sgkit#1066 Release 0.7.0
• sgkit#1067 Division by zero error with clinvar vcf

Notes

Attendees

1. Jeff Hammerbacher
2. Tom White
3. Jerome Kelleher
4. Ben Jeffery
5. Tim Millar

Discussion

• Use case discussion
  • CSIRO: would like to replace Hail with sgkit for QC prior to VariantSpark for GWAS. Some interest in merging and munging multiple VCFs, not entirely clear how they’re doing this right now.
  • Genomics England: 30M variants, 80k samples VCF converted to sgkit.
  • BDI: UKB loading chr19 but hitting memory issues for bigger chromosomes. MalariaGen having some issues with duplicated rows.
  • NZ PFR: Relationship matrices work motivated by doing work in front of downstream R analysis. Tim has sgkit installed on an internal cluster, using SLURM but needs JIT caching disabled.
  • Common theme: sgkit useful in data preparation prior to downstream analysis in R. Should we add docs on R interop? Add anything to BiocPy or reticulate?
• sgkit#1021
  • Committed! Don’t need to manually specify VCF headers when writing.
• sgkit#1056
  • Almost ready, need to fix coverage failure.
• sgkit#1072
  • Tim could use feedback on level of detail
  • Would like a few more estimators some day
• sgkit#1059
  • Stymied on ragged arrays
  • VEP can export as JSON file
  • Common use case: filter variants using VEP annotations. Could we query the JSON with an external tool like DuckDB?
  • What other data should live outside of Zarr? Alternate alleles, for example?
  • Big Xarray PR from Tom Nicholas to allow Cubed to be use with Xarray: Generalize handling of chunked array types pydata/xarray#7019
• sgkit#1061
  • Small update to API to be consistent
• sgkit#1062
  • Promising new estimator
• sgkit#1066
  • Tim and Ben have some PRs to get in!
• sgkit#1067
  • From CZI person
• Paper?
  • JK emerging from sleep deprivation, has another preprint in front, will hit sgkit after that.

[tomwhite]

Also https://github.com/xarray-contrib/cubed-xarray is the other piece for xarray-cubed integration