Public Beta 0.11 Release Planning

[jhellerstein]

Deliverables

• Hydroflow+ as primary DSL (@shadaj)
  • [ x ] 1.0 of language/API
  • docs
  • examples
• Initial release of stdlib (@jhellerstein)
  • move HF+ examples
  • flesh out initial list of targets
    • Anna KVS
      • Gossip
    • 2PC
    • Voting
    • Paxos
    • PBFT
    • Collaborative editor
    • Leader election?
    • Membership?
    • Load Balancer?
• Lattices/Algebra crate (@conor-23, @jhellerstein)
  • CRDTs for consistency
  • Move/benchmark RGA (collab editing tree)
  • Z-sets
• Hydroflow cleanup (@jhellerstein)
  • operator pruning and naming consistency
  • finish docs
  • re-add poll_futures() feat(hydroflow): Added poll_futures and poll_futures_async operators. #1143 revert: "feat(hydroflow): Added poll_futures and poll_futures_async operators. (#1143)", fix #1183 #1216 (@MingweiSamuel)
• Flo syntax/optimizations (@MingweiSamuel)
  • 1. Flo semantics/syntax (nested iteration) feat(hydroflow): implement flo loop syntax parsing #1507 feat(hydroflow_lang): flo basic graph structure checks #1513
  • 2. Implement this in the current scheduler
  • 3. Pipeline types, operator type negotiation (Implement this in the codegen/operatorinfo types)
  • 4+. Runtime optimizations beyond in/out trees
    • Graph is DAG within an iteration
    • Later: eliminate operator resumability (in many cases). All operators become fully “stateless”
    • Laziness short-circuiting
    • Borrowing tee
• KVS service (@rohitkulshreshtha)
  • benchmarking replicated HF Anna 1.0
  • scale-out/elasticity support (Anna 2.0)
    • metrics in HF Anna
    • read replicas
      • serializing HF data structures (collections, lattices, etc) to NW and disk
        • HF+ currently uses bincode
    • (re-)partitioning in HF??

Post 1.0

• HydroDeploy 2.0 / k8s integration
• unordered semantics support for HF+
  • unordered collections for HF+
  • unordered networking for HF+ clusters (union)
  • FT model for clusters in HF+
  • CALM-style consistency guaranteed by HF+ types

[ydewit]

First of all, thanks for this very interesting work!

Are there any plans to support running multiple Process<..> instances within a single OS process? From a developer perspective, having the option to run all flows initially in one OS process would greatly simplify debugging and iterative development. Additionally, this could offer valuable flexibility for scaling from small deployments to larger distributed setups.

[shadaj]

Great question! We are indeed planning this in a "simulation" mode (where all processes / clusters run in a single OS process) to allow developers to simulate network delays / concurrency scenarios. It's a bit farther out on our roadmap, but definitely something we want to do.

Are you noticing high latency with compile times or launching the processes? Wondering if there are things we can do in the meantime to improve iterative development.

[ydewit]

Sorry, I’m still too new to Hydro to provide meaningful feedback based on actual usage, but I’ll reach out when I have something concrete.

Beyond direct developer convenience, it would be nice if Hydro could consistently scale from a single thread to the cloud—similar to how Timely describes itself: “a distributed data-parallel compute engine, which scales the same program up from a single thread on your laptop to distributed execution across a cluster of computers.”

Could you clarify how Hydro schedules flows within a single Process? Does it use multiple threads, or is it single-threaded? ... Here’s my current understanding: A Process corresponds to a single OS process and is single-threaded, with concurrency achieved by running multiple Process instances across different machines. However, scheduling within a Process doesn’t operate on the entire dataflow graph at once, but rather on subgraphs. The dataflow graph is partitioned, and “…the operators within a subgraph are monomorphized (i.e., inlined and optimized) together”. Each subgraph consumes data from input queues and produces data into output queues (handoff queues). These subgraphs are scheduled for execution according to a well-defined hierarchy based on Ticks and strata.