Improve schedule engine performance

[ericallam]

Fixed the calculation for distributing schedules across the "calculation window" timeframe to include the schedule instanceId in the random seed and not just the schedule time, which was causing all schedules to be "distributed" into the exact same timestamp (not good).

Also fixed an issue where the schedule engine workers and tasksPerWorker options were not being passed down to the internal RedisWorker, causing the defaults of 1 and 10 to be used.

I also added some additional logging into RedisWorker and also spaced-out the starting times of each run worker loop.

[changeset-bot]

⚠️ No Changeset found

Latest commit: fcc78b3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Walkthrough

This change introduces new log level configuration options for three worker categories—legacy run engine worker, common worker, and alerts worker—by extending the environment schema to accept log level values from environment variables, defaulting to "info". The logging level for each worker is now dynamically set from these environment variables instead of being hardcoded. The default concurrency settings for the schedule worker are increased, with the number of workers raised from 1 to 2 and tasks per worker from 1 to 10. Additional worker concurrency parameters are added to the schedule engine, and distributed execution time calculation now supports an optional instance identifier and uses the FNV-1a 32-bit hash algorithm. Worker loops are staggered with delays based on their index and enhanced with detailed logging for improved observability.

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ec2175f and fcc78b3.

📒 Files selected for processing (10)

• apps/webapp/app/env.server.ts (4 hunks)
• apps/webapp/app/v3/alertsWorker.server.ts (1 hunks)
• apps/webapp/app/v3/commonWorker.server.ts (1 hunks)
• apps/webapp/app/v3/legacyRunEngineWorker.server.ts (1 hunks)
• apps/webapp/app/v3/scheduleEngine.server.ts (1 hunks)
• internal-packages/run-engine/src/engine/index.ts (1 hunks)
• internal-packages/schedule-engine/src/engine/distributedScheduling.ts (1 hunks)
• internal-packages/schedule-engine/src/engine/index.ts (2 hunks)
• internal-packages/schedule-engine/src/engine/types.ts (1 hunks)
• packages/redis-worker/src/worker.ts (4 hunks)

⏰ Context from checks skipped due to timeout of 90000ms (1)

• GitHub Check: Analyze (javascript-typescript)

🔇 Additional comments (18)

apps/webapp/app/env.server.ts (4)

621-623: LGTM! Consistent log level configuration added.

The new LEGACY_RUN_ENGINE_WORKER_LOG_LEVEL environment variable follows the established pattern with the same enum values and default as other worker log levels.

---

667-667: LGTM! Consistent log level configuration added.

The new COMMON_WORKER_LOG_LEVEL environment variable maintains consistency with other worker log level configurations.

---

706-706: LGTM! Consistent log level configuration added.

The new ALERTS_WORKER_LOG_LEVEL environment variable follows the same pattern as other worker log level configurations.

---

740-741: Improved schedule worker concurrency defaults.

The increased defaults (workers: 1→2, tasks per worker: 1→10) should significantly improve schedule engine performance by enabling better parallelism and throughput.

apps/webapp/app/v3/scheduleEngine.server.ts (1)

64-65: LGTM! Worker concurrency configuration properly integrated.

The addition of workers and tasksPerWorker properties enables the schedule engine to use the configurable concurrency settings from environment variables, supporting the performance improvements outlined in the PR objectives.

apps/webapp/app/v3/commonWorker.server.ts (1)

199-199: LGTM! Dynamic log level configuration implemented.

Replacing the hardcoded "debug" level with env.COMMON_WORKER_LOG_LEVEL enables runtime configuration of logging verbosity, improving operational flexibility and observability control.

apps/webapp/app/v3/legacyRunEngineWorker.server.ts (1)

71-71: LGTM! Dynamic log level configuration implemented.

Replacing the hardcoded "debug" level with env.LEGACY_RUN_ENGINE_WORKER_LOG_LEVEL enables runtime configuration of logging verbosity, consistent with the pattern applied across all worker components.

apps/webapp/app/v3/alertsWorker.server.ts (1)

64-64: LGTM! Dynamic log level configuration implemented.

Replacing the hardcoded "debug" level with env.ALERTS_WORKER_LOG_LEVEL completes the consistent pattern of enabling runtime log level configuration across all worker components, improving operational control and observability.

internal-packages/schedule-engine/src/engine/types.ts (1)

38-39: LGTM! Clean interface extension for worker concurrency options.

The addition of optional workers and tasksPerWorker properties properly extends the worker configuration to support the enhanced concurrency settings mentioned in the PR objectives.

internal-packages/run-engine/src/engine/index.ts (1)

155-155: LGTM! Improved logging configurability.

Changing from hardcoded "debug" to configurable options.logLevel ?? "info" provides better control over logging verbosity and aligns with the standardization effort across worker components.

internal-packages/schedule-engine/src/engine/index.ts (2)

95-96: LGTM! Critical fix for worker concurrency configuration.

These additions ensure that the workers and tasksPerWorker configuration options are properly passed to the RedisWorker component, resolving the issue where it was defaulting to 1 worker and 10 tasks per worker regardless of intended settings.

---

595-596: LGTM! Essential fix for schedule distribution.

Adding the instanceId parameter to calculateDistributedExecutionTime is crucial for fixing the core issue where all schedules were getting the same timestamp. This ensures proper distribution across the calculation window by providing instance-specific seeds for the random distribution.

packages/redis-worker/src/worker.ts (3)

209-209: LGTM! Proper parameter passing for staggered worker starts.

The addition of workerIndex and totalWorkers parameters enables the staggered start functionality to prevent simultaneous execution spikes across worker loops.

---

393-417: LGTM! Well-implemented staggered start mechanism.

The staggered delay calculation and comprehensive logging provide excellent operational improvements:

• Prevents thundering herd issues by spacing out worker loop starts
• Detailed logging enhances observability of worker behavior
• Delay calculation using (totalWorkers - workerIndex) ensures proper distribution

---

421-461: LGTM! Enhanced debugging capabilities.

The detailed logging at capacity checks and during dequeue operations provides valuable insights into worker states, concurrency limits, and processing activity. This will significantly improve troubleshooting and performance monitoring.

internal-packages/schedule-engine/src/engine/distributedScheduling.ts (3)

8-9: LGTM! Critical parameter addition for instance-specific distribution.

The optional instanceId parameter is essential for fixing the core issue where all schedules were assigned the same timestamp due to identical random seeds.

---

11-24: LGTM! Excellent hash function implementation.

The FNV-1a hash algorithm implementation is technically sound:

• Correct FNV offset basis (2166136261) and prime (16777619)
• Proper XOR-then-multiply order for FNV-1a variant
• Appropriate 32-bit unsigned integer handling with >>> 0
• Smart seed construction combining timestamp with instanceId for unique distribution per instance

---

26-28: LGTM! Improved normalization for better distribution.

Using the full 32-bit range (0xffffffff) for normalization provides more uniform distribution compared to previous approaches, enhancing the quality of the distributed scheduling algorithm.

✨ Finishing Touches

• 📝 Generate Docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.