ODC SC Meeting 2019 06 27

Attendees

• Andrew Cherry (Chair)
• Kirill Kouzoubov (GA)
• Alex Leith (GA)
• Federica Moscato (Catapult)
• Peter Wang (Data61)
• Rob Woodcock (CSIRO)
• Syed Rizvi (AMA)
• Tony Butzer (USGS)
• George Dyke (Symbios)

Apologies

• Alex Leith (GA)

Minutes

See Google Drive document.

Agenda

• Welcome
• Previous Actions
  • See below
• ODC Project Code of Conduct (Alex L)
• ODC AWS EKS k8s cluster release v0.5 milestone
  • SC confirm ODC supported feature set
  • Project board for 0.5
• Enhancement Proposals
  • Clean up:
    • Duplicate process - delete one or both of:
      • enhancement-proposal repo
      • datacube-core Wiki Enhancement Proposals
      • and alternate just use Issues flagged as enhancement proposal against each repository (this can also be supported by automation to remove stale issues not seeing any action).
  • New Proposals
    • ODC-EP01 Add Support for 3D Datasets
    • ODC-EP02 Support for multiple databases
    • ODC datacube-core v2.0
  • Committed Proposals
• State of the Cube Highlights
  • CSIRO
    • Working with GA and FrontierSI on the Jupyterhub/K8s/EKS implementation on github and CSIRO deployment.
      • Now available for authorised users at https://hub.easi-eo.solutions.
      • Dask is supported. Hit a few issues with parallel IO at scale swamping coredns. Manifests as connection resets in the Dask worker logs. dns-autoscaler resolves the issue by increasing the number of coredns replicas in proportion to worker node/core count (not CPU load Since that isn’t the issue). A PR is being developed to add this option to the opendatacube/datacube-k8s-eks repository.
      • Borrowed quite a few ideas from Pangeo. Not surprisingly the components add together well though we don’t have support for Pangeo binder (yet).
    • Next steps: integrate with CSIRO Live Ingester and ODC/GA indexing components, add datacube-ows.
  • Catapult
    • Working on the JupyterHub setup with Kubernetes for on-premise, single server (microk8s/minikube), and public cloud deployments
      • Goals:
        • Clear provenance of the setup material
        • Computing environment and storage agnostic: on-premise, dev laptop, or public cloud
        • Simple setup: all Helm/kubectl YAML blueprints in one place on GitHub
      • Setup material available here
      • Deployment instructions here
      • Internal environment setup blueprints shared here
  • GA
    • DE Africa
      • Working with NASA and AMA to acquire initial data for DE Africa, including Landsat and ALOS
      • Developing a Technical Roadmap for internal purposes at first, hopefully can share
    • DE Australia
      • Working on collection upgrade, which will involve freshly indexing
      • Collection upgrade will also involve re-running downstream products (WOfS and FC etc.), which will be a datacube-driven process
      • Work underway to define what an ODC 2.0 might look like...
  • USGS
    • Working with the USGS Pangeo Team to stand up a JupyterHub/K8s/EKS implementation in the USGS Secure AWS Space
      • pangeo.chs.usgs.gov
      • https://pangeo.io/architecture.html#where-we-began
      • Richard Signell - https://www.usgs.gov/staff-profiles/richard-p-signell?qt-staff_profile_science_products=0#qt-staff_profile_science_products
      • Scott Henderson - https://github.com/scottyhq/esip-tech-dive/blob/master/notebooks/0-demo-aws.ipynb
        • geophysicist and python enthusiast University of Washington, @uwescience
      • Ryan Abernathey (Columbia University) - The Future of Data-driven Discovery in the Cloud
    • Planning on attending the ARD19 - Planet Radiant.Earth Second ARD Workshop on Remote Sensing Data Interoperability - AUGUST 5-7th
      • Labahn, Kline, Renee, Covington, etc to attend also from USGS as well as others from all over the USA
    • Digital Earth Africa
      • We should decide on some target test sites for C2L2 Data - for example: South Africa
      • Developing a conops with Randy Sunne and Ron Dilley at EROS
      • L0Ra and L0Rp - will be no problem - all the way to L2 - still some schedule risk for August - mitigations?
  • AMA
  • FrontierSI
• Other Business
  • ODC Conference 2020
  • Mailing lists
• Next Meeting and Close
  • Date

Actions

See the action tracking project.

Outstanding Actions

• Close out 1.7 Release
• Next ODC Conference structure
• Architecture document
• Licenses in ODC organisation on GitHub
• PR template text in datacube-core
• Copyright licensing of code contributions
• CSIRO to merge develop branch in to their code -> datacube-core 0.7

New Actions