20200507 developer call notes

[hammer]

@hammer's notes

• Alistair's pre-notes
• IO
  • Alistair question about chunking in multiple dimensions during IO
  • Eric assumes we'll write to Zarr first and do everything else later
• Data structures
  • Eric C likes that Hail hides information from users
  • Alistair M curious how xarray is working out
  • Eric C
    • Option 1: make xarray the main API through which users do everything
    • Option 2: ignore xarray methods, just use data container
    • Option 3: use xarray extensions, e.g. HWE
  • Xarray pro: makes ad hoc analyses easier
  • Xarray con: evaluation is eager; Dask would optimize whole graph
  • Xarray con: materializes coordinates into memory
  • Alistair: currently organize arrays in a Zarr hierarchy but don't use Xarray now
    • If we use Xarray, do we need to fix organization and names of arrays?
    • PLINK vs. VCF are different, for example
  • Definitely need a top-level organization of arrays; if not xarray, what? Zarr?
    • Use case: subset data but carry metadata along
    • Alistair tried to defer evaluation in his API
    • ExpressionSet/AnnData/loom
      • Changing the backend to xarray anndata#32
      • ExpressionSet/SummarizedExperiment
      • AnnData
• Indexing
  • Pandas requires indexes to be in memory
  • Alistair's issue: Low memory/out-of-core index? xarray#1650
  • Solved by Explicit indexes in xarray's data-model (Future of MultiIndex) #1603?
  • Project board for explicit indexes
  • Uses of indexes: identifying uniqueness, joins, pull genes/regions
  • Workload: liftover variants between references
  • Workload: harmonization
• Dispatching
  • Matt R concerned about automatic dispatch
  • Eric C trying to hide conditions from user
  • Alistair M wants to push burden onto users as maintenance is hard
  • Eric C: set backend once
  • Alistair M found uarray hard to grok and didn't like coercion
  • Users should be aware that data may move from CPU to GPU
  • Eric C: methods that use multiple arrays that could be on multiple backends should have whitelist of combinations, not the uarray approach of automatic conversion
• What next
  • Quansight engagement: starting end of May through early 2021
    • Alistair thinking 10% - 20% of his time; provide requirements mostly
  • Short analysis workflow: "analysis stories"
  • Design: how to organize into repos and packages
    • End user and library developer UX
  • Paper comes after that