Skip to content

Conversation

sitaowang1998
Copy link

@sitaowang1998 sitaowang1998 commented Oct 9, 2025

Note

This PR depends on y-scope#1178.

Description

This PR adds spider scheduler to the docker compose packaging. User can choose to use spider for compression orchestration by setting CompressionScheduler.type in configuration file.

This PR:

  • Adds spider database and scheduler configuration.
  • Adds credential validation for spider.
  • Adds a script to setup spider database and the use of it in create database service.
  • Adds a new docker compose file for spider services.
  • Update docker controller to setup environment variables for spider services.
  • Adds docker service to start spider scheduler.
  • Adds a script to start multiple spider workers and the use of it in docker service.
  • Updates compression scheduler to determine which task manager backend to use.

Checklist

  • The PR satisfies the contribution guidelines.
  • This is a breaking change and that has been indicated in the PR title, OR this isn't a
    breaking change.
  • Necessary docs have been updated, OR no docs need to be updated.

Validation performed

  • Compression with celery works.
  • Compression with spider works.
  • GitHub workflows pass.

sitaowang1998 and others added 30 commits September 25, 2025 11:29
Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>
…ost type and removing duplicated validators.
sitaowang1998 and others added 30 commits October 10, 2025 14:16
…r` and `spider_worker`) in the `clp-package` image; Set `mariadb-connector-cpp` build type to `Release` (fixes y-scope#1410).
…notated serializers. (y-scope#1417)

Co-authored-by: Junhao Liao <junhao@junhao.ca>
Co-authored-by: Junhao Liao <junhao.liao@yscope.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants