Hydroflow 1.0 Roadmap

[MingweiSamuel]

P0

1. Singleton syntax/usability

P1

1. State externalization (dataflow) (State externalization meta issue #1059)
2. Hydroflow+ polish (documentation)
   • Prune Hydroflow operator set
3. Standard library for Hydroflow+
   • distributed async/await (futures/promises/etc.)
   • actors
   • distributed protocols, CRDTs/[semi]rings/groups, KVS, transaction manager, BFT, etc
4. Deployment (hydro-deploy)
5. Performant KVS
6. Debugging/diagnostics/telemetry
7. Decent Ops support
   • cleaner k8s integration?
   • Version management
8. Lattices/properties (semantics)
   • Singletons, flows,
   • ticks/deltas
   • David Chu optimization preconditions
   • Groups, rings?
9. Dynamic/Auto-Ops
   • Live reconfiguration
   • Auto-elasticity

P2

1. Choose benchmarks (maybe a subset of the below)
   • against Timely / Rust Iterators
   • Java functional stream processing benchmarks
     • StreamBench
   • Scalability at what COST examples
   • Classical application level benchmarks of systems community:
     • SPLASH-2: https://www.capsl.udel.edu/splash/
     • Phoenix v2.0.0: http://csl.stanford.edu/~christos/publications/2007.cmp_mapreduce.hpca.pdf
   • Against hand-written code / actors
2. Networking?
   • Backpressure?
   • Reconnection?
   • Shared memory performance
     • No serialization overhead
3. Integrations
   • Apache Arrow in-memory format
   • Various serialization/IO formats
   • Polars "dataframes for rust"

P3

1. Fault tolerance specs (on Hydroflow functions?)
2. State externalization (for replication)
   1. checkpointing Checkpoint/migration handling #1049
3. Sequence operators (windowing)
   • (check out stream-it)
   • Caleb Stanford ordered streams
4. Extended Performance
   • Cache locality
   • Vectorization / Columnar join
   • Instruction-level benchmarking (Vtune, etc)
5. Dataflow Algebra Optimizations

[MingweiSamuel]

Moved here from #930

Performance

• How low is our latency for a simple flow? Identity? Anna? Can we get to "line rate"? How does the KVS bench compare?
• Can a fast box with many Hydroflow transducers handle a high-speed (40Gb?) NIC?
• Similar questions for bandwidth -- seems easy, but have we done something wrong?
• What else should we benchmark?
• We get lots of goodness from "shared nothing". What's the downside of explicit comm vs shared memory? Let's do some adversarial analysis of our weaknesses.
• Revisit early timely comparison?

Expressivity

• Is map and fold enough generality for non-dataflow user code?
• If you're justifying that you're "a general language", what constructs do you need to demonstrate?
• What about linear algebra/ML?