Post-Offsite Planning, Summer 24

[jhellerstein]

Themes

1. Push out Anna as a flagship app and dogfooded substrate: Lead: @rohitkulshreshtha
   • and then pick another app to stretch us
2. Make Hydroflow+ the default interface, well-documented and complete. Lead: @MingweiSamuel with @shadaj
   • review old design, propose new design (language design and implementation)
   • documentation and examples move over
   • How does Shadaj's tick thinking fit into this?
   • Standard Interfaces support - GRPC/REST
   • Disk I/O
   • Deployment (?)
3. Enable research from students/faculty: Oversight: @jhellerstein
   • Algebra library integration. Lead: @conor-23
   • Cost model via performance monitoring (for query optimization, benchmarking). Lead: @davidchuyaya
   • Query Optimization Framework/Rules (Shadaj/David/Conor). Lead: @jhellerstein

Offsite Feature List

1. Language Design: @MingweiSamuel

• Hydroflow+ usability/review
• Distributed Systems idioms (e.g. search martinfowler.com)

2. Query Optimization: @jhellerstein

• Implement an optimizer framework
• Controlling which optimizations get applied

3 Single Machine Testing & Perf: @davidchuyaya needs some of this over the summer

• Local testing (simulate failures, etc)
• Visual Performance Profiling
• Resource Utilization + machine counters like CPU idleness, queue lengths
• Benchmarking Harness (e.g. criterion)
• Expose low-level optimization (eg core pinning)
• Good multicore support (shared memory not TCP loopback)

Additional Features

• K8s integration, HydroDeploy review
• Ways to manage security (crypto, etc)
• GRPC integration

Distributed Concerns Testing/Observability

• Lineage debugging across services (a la Dapper)
• Observability in Production: Logging, Metrics, Perf (a la DataDog)