20200709 developer call notes

[hammer]

Pre-notes

Process and tools

• CONTRIBUTING: sgkit#8
  • Need to formalize when and how to block on Alistair + Jerome input
• Docs: sgkit#19
• Code coverage: sgkit#20

IO

• Okay to close data ingest sgkit#2?
• BGEN: not enthusiastic about 3rd party libraries
  • May need someone who's good at C++ for BGEN
  • More generally, when to use Numba → Cython → C/C++?
  • Quansight? Gavin Band?
• VCF: does Alistair want to lead, or okay for Tom to pick up after BGEN?

Data structures

• Initial Xarray call data representation sgkit#10 is in!
• Follow up
  • Constants for dimensions and variables sgkit#17
  • Document use of dtype "S1"?
  • Document how we handle missing values?
• What else is unresolved?

Dispatching

• Guidelines sgkit#11 to fall out of count alleles sgkit#3 and linear regression sgkit#16 implementation. How should we document them?
• Time to get configuration set up?

Core operations

• Count alleles sgkit#3: does Alistair want to lead, or okay for Tom to pick up?
• Plan for PCA, LD pruning, PC-Relate
• Synthetic data: should we start work on it?

Notes

Attendees

• Jeff Hammerbacher
• Jerome Kelleher
• Alistair Miles
• Eric Czech
• Tom White
• Rafal Wojdyla

Discussion

• Alistair: Quansight engaged! Dale, Jillian, Kim, Dharhas have a reading list.
  • Dale Tovar (@daletovar) will be main developer
  • Dharhas Pothina (@dharhas) is a senior architect
  • Kim Pevey (@kcpevey) is a senior developer w/ experience in project infrastructure
  • Jillian Rowe (@jerowe) has life sciences experience
  • Ashley Baal is the program manager
• How Quansight might contribute
  • Jeff: testing, benchmarking
  • Alistair: dispatch design
• CONTRIBUTING: sgkit#8
  • Tom will write up
  • Jerome wants to avoid being overly prescriptive about it
  • 3 +1s for large changes
  • 1 +1 is fine for most PRs
    • It’s considerate to wait for other time zone to +1
  • RTC preferred to CTR
  • Use of mergify will bring a set of rules
  • Jerome notes Ben Jeffery at Oxford is the mergify guru
  • Eric notes: contributors should suggest a reviewer
  • Jerome: use GitHub review proces
• Docs: sgkit#19
  • MyST is nice but will still need ReST for docstrings
  • Alistar + Tom fine to use ReST for now
  • Alistair notes that most API docs just import docstrings
  • Alistair notes tutorials may be in notebooks anyways
  • ReST wins over MyST!
  • Now need to figure out docstring style and Sphinx extensions
  • Alistair likes the numpy docstring style
  • Alistair notes PyCharm can parse certain styles of docstring
• Code coverage: sgkit#20
  • Tom will set up Codecov
  • Eric Q: testing for JIT-compiled functions?
  • Alistair A: environment variable to disable JITing, then run both ways
  • Alistair also notes there’s a CUDA emulator
• Git merge strategy: sgkit#26
  • Rebase vs. merge
    • Rebase book
  • Rebase pro: commit graph is neater, git bisect is easier to use
  • Squash + merge is kind of like a rebase but we lose commit granularity
  • Mergify has a setting, so hopefully won’t require anyone to rebase manually
  • Jerome: you can force linear merges and git bisect works well
  • Eric: whatever is the least amount of manual work!
• Okay to close data ingest sgkit#2?
  • Eric: what about methods/data used for validation?
    • Example: C code for linear regression reference implementation lives where?
    • The environment for this companion code is the hard part. Probably need a Docker image to go with the code.
    • Start w/ folder in sgkit, spin it out if it becomes unwieldy
• pystatgen/sgkit-bgen is alive!
  • Using bgen-reader for now
  • Blocked on representation for dosages
  • Wait to reach out to Gavin until we have more experience
  • Jerome has written his own Python bgen reader: simplebgenmodule.c
• VCF
  • Let’s try to use Alistair’s code for now
  • Ultimately will need to rely on htslib
• Data structures for results of methods
  • Discussion started on linear regression PR sgkit#16
  • Accumulating results back into the same data set is a common pattern
  • It’s nice when functions are pure
  • Only return a Dataset if you’re returning two or more things?
  • You can use merge if you want to lump new results into old data
  • Set common names as default keyword arguments and allow for override?
• Dispatch
  • Can’t just look at types to decide backends
  • Default backend: Dask on CPU
  • How do you switch to Dask on GPU?
  • 2 separate PRs: xarray part, dispatch part; or, mix them?
  • Eric: maybe leave up to user for simple functions
  • Variant summary statistics issue includes our approach to masks
• Simulated genotypes sgkit#23
  • Too early to consolidate how we simulate data for tests
  • Eric needs to learn msprime to do more sophisticated stuff