20210412 benchmarking call notes

[hammer]

• People
  • @hammer
  • @ravwojdyla
  • @eric-czech
  • @tomwhite
  • @aktech
  • @arunkk09
  • @shy218
• LD pruning
  • Benefit from exponential decay in LD
  • SY: does diversity computation require LD pruning? EC: maybe.
  • ArK: Dask does not provide good multi-node operators
  • EC: map_blocks a workhorse for us
  • ArK: worth hiding multiple physical operators behind a single logical operator
  • ArK: SystemML as baseline
  • ArK: pbdr (on ScaLAPACK) as a baseline
  • JH: revisit scale out later
  • ArK: low arithmetic intensity so wouldn't expect GPU to be best
  • ArK: Krypton project uses CUDA code to do parallel memory copies across the GPU memory hierarchy
• Relatedness pruning
  • Quadratic in number of people.
  • 500k people is the target; sgkit implementation will not work at this scale.
  • RW: Implementation is mostly CPU, but could work on GPUs.
  • RW: PC-Relate implemented to compare to Hail. Tested 2k people.
  • RW: AmK's generic pairwise work could be useful
• Population structure
  • ArK: SVD is hard; SLAB didn't find a good one anywhere
  • EC: Dask SVD is multi-node scalable
• Association testing
  • SLAB had a heteroskedasticity test that was even harder than OLS
  • EC: per-variant OLS using lstsq
  • ArK: Mike Mahoney and Michael Jordan have QR decomposition
• ArK path to publication
  • Extend SLAB benchmark suite
  • Run benchmark suite for Plink, Hail, sgkit
  • SY can design an experiment suite that varies data and machine parameters
• ArK: what data and machines do we have?
  • SLAB had synthetic and Criteo
  • JH: rice, canine, mosquito data
  • ArK: when is the data sensitive to distributions?
  • EC: LD pruning, data distribution is going to impact the runtime
  • RW: relatedness could be done in two passes to limit computation to only suspected relations
  • JH: previous discussion on synthetic data discourse#48
• VLDB paper
  • Send paper to VLDB experiments, analysis, and benchmarking track
  • Try to make it more general, not too application-specific discourse#32
  • Cerebro: "integrationist", "modular" philosophy

[arunkk09]

Wow, thank you for the detailed yet crisp set of notes, Jeff! Shaoqing plans to start a draft in VLDB template to read/think/write about the following 4 things before our next joint call on Thu (Apr 22). (This Thu is not feasible for us.)

1. Read about, look through the docs, and summarize all the compared tools as background material. Try their APIs out on toy data on CloudLab.
2. Write down the 6 algorithms/workloads precisely in math notation. Cull out and explain common access patterns. Potentially group them into two--simple and complex--for starters.
3. Flesh out the key scale factors: data shape parameters and machine parameters. At least 3 levels seems needed on the latter: single-node multi-core, single-node GPU(s), and multi-node.
4. Flesh out an initial empirical comparison plan in terms of machine setup, dataset setup, algorithm, comparators, x-axes, y-axes, and hypotheses. We can walk through this and discuss what to add/drop and how to prioritize.