From a7cc30e84d0a15ea16cfc07ff65e93a285292a30 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Fri, 11 Jul 2025 23:22:18 +0200 Subject: [PATCH 01/20] qg/queues --- queues.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/queues.md b/queues.md index 7bc48c8d67..9e77ee129b 100644 --- a/queues.md +++ b/queues.md @@ -577,6 +577,8 @@ public function middleware(): array } ``` +Releasing an overlapping job back onto the queue will still increment the job's total number of attempts. You may wish to tune your `tries` and `maxExceptions` properties on your job class accordingly. For exemple, leaving the `tries` property to 1 as it is by default would prevent any overlapping job to be retried later. + Any overlapping jobs of the same type will be released back to the queue. You may also specify the number of seconds that must elapse before the released job will be attempted again: ```php @@ -1213,7 +1215,21 @@ class ProcessPodcast implements ShouldQueue #### Max Attempts -If one of your queued jobs is encountering an error, you likely do not want it to keep retrying indefinitely. Therefore, Laravel provides various ways to specify how many times or for how long a job may be attempted. +Job attempts are a core concept of Laravel's queue system and power many advanced features. While they may seem confusing at first, it's important to understand how they work before modifying the default configuration. + +When a job is dispatched, it is pushed onto the queue. A worker then picks it up and attempts to execute it, this is known as a job attempt. + +However, an attempt does not necessarily mean the job’s handle method was executed. Attempts can be consumed in several ways: + +- The job's `handle` method runs and completes without throwing an exception. +- The job encounters an unhandled exception during execution. +- The job is manually released back to the queue using `$this->release()`. +- Middleware such as `WithoutOverlapping` or `RateLimited` fails to acquire a lock and releases the job. + +You likely do not want it to keep retrying indefinitely a job. Therefore, Laravel provides various ways to specify how many times or for how long a job may be attempted. + +> [!NOTE] +> By default, Laravel will only attempt a job once. If your job uses middleware like `WithoutOverlapping` or `RateLimited`, or if you're manually releasing jobs, you’ll likely need to increase the number of allowed attempts. One approach to specifying the maximum number of times a job may be attempted is via the `--tries` switch on the Artisan command line. This will apply to all jobs processed by the worker unless the job being processed specifies the number of times it may be attempted: @@ -1417,6 +1433,9 @@ public function handle(): void } ``` +> [!NOTE] +> A job marked failed will never be retried. + If you would like to mark your job as failed because of an exception that you have caught, you may pass the exception to the `fail` method. Or, for convenience, you may pass a string error message which will be converted to an exception for you: ```php @@ -2249,6 +2268,13 @@ class ProcessPodcast implements ShouldQueue > [!WARNING] > A new instance of the job is instantiated before invoking the `failed` method; therefore, any class property modifications that may have occurred within the `handle` method will be lost. +A failed job is not necessarily one that encountered an unhandled exception. A job is considered failed when it has exhausted all of its allowed attempts. These attempts can be consumed in several ways: + +- The job encounters an unhandled exception during execution. +- The job is released back to the queue either manually or by a middleware. + +If the final attempt fails due to an exception, the exception will be passed to the failed method. However, if the job fails simply because it reached the maximum number of attempts, the `$exception` will be an instance of `Illuminate\Queue\MaxAttemptsExceededException`. + ### Retrying Failed Jobs From 52a1620c7c4b56ceb31631dab5a109e494280a4f Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Fri, 11 Jul 2025 23:22:26 +0200 Subject: [PATCH 02/20] qg/horizon --- horizon.md | 131 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 128 insertions(+), 3 deletions(-) diff --git a/horizon.md b/horizon.md index b104d2b807..7aa61891e7 100644 --- a/horizon.md +++ b/horizon.md @@ -3,7 +3,11 @@ - [Introduction](#introduction) - [Installation](#installation) - [Configuration](#configuration) + - [Queue Priorities](#queue-priorities) - [Balancing Strategies](#balancing-strategies) + - [Max Job Attempts](#max-job-attempts) + - [Job Timeout](#job-timeout) + - [Job Backoff](#job-backoff) - [Dashboard Authorization](#dashboard-authorization) - [Silenced Jobs](#silenced-jobs) - [Upgrading Horizon](#upgrading-horizon) @@ -92,6 +96,23 @@ You may also define a wildcard environment (`*`) which will be used when no othe When you start Horizon, it will use the worker process configuration options for the environment that your application is running on. Typically, the environment is determined by the value of the `APP_ENV` [environment variable](/docs/{{version}}/configuration#determining-the-current-environment). For example, the default `local` Horizon environment is configured to start three worker processes and automatically balance the number of worker processes assigned to each queue. The default `production` environment is configured to start a maximum of 10 worker processes and automatically balance the number of worker processes assigned to each queue. +You may also define Horizon's environment independently of the application's `APP_ENV` by adding an `env` key to your Horizon configuration file: + +```php +/* +| ------------------------------------------------------------------ +| Horizon Environement +| ------------------------------------------------------------------ +| +| This value determine Horizon's environment! +*/ +'env' => env('HORIZON_ENV', env('APP_ENV')), +``` + +This can be particularly useful when running Horizon across multiple servers. For example, you might have one server processing general-purpose jobs using the default queue configuration, while a second server, using in a different environment key, is dedicated to handling resource-intensive jobs, such as video processing or large data imports. + +Using two different environments makes it easy to configure Horizon on a per-server basis rather than a per-environment basis, giving you more granular control over worker settings and job handling. + > [!WARNING] > You should ensure that the `environments` portion of your `horizon` configuration file contains an entry for each [environment](/docs/{{version}}/configuration#environment-configuration) on which you plan to run Horizon. @@ -123,6 +144,24 @@ While your application is in [maintenance mode](/docs/{{version}}/configuration# Within Horizon's default configuration file, you will notice a `defaults` configuration option. This configuration option specifies the default values for your application's [supervisors](#supervisors). The supervisor's default configuration values will be merged into the supervisor's configuration for each environment, allowing you to avoid unnecessary repetition when defining your supervisors. + +### Queue Priorities + +Each supervisor can process one or more queues. Just like Laravel's default queue system, queues are processed in the order they are listed in the queue configuration option, giving higher priority to queues defined first: + +```php +'environments' => [ + 'production' => [ + 'supervisor-1' => [ + 'connection' => 'redis', + 'queue' => ['high', 'default', 'low'], + ], + ], +], +``` + +In this example, jobs in the `high` queue will be prioritized over those in the `default` or `low` queues. + ### Balancing Strategies @@ -132,7 +171,12 @@ Unlike Laravel's default queue system, Horizon allows you to choose from three w The `auto` strategy, which is the configuration file's default, adjusts the number of worker processes per queue based on the current workload of the queue. For example, if your `notifications` queue has 1,000 pending jobs while your `render` queue is empty, Horizon will allocate more workers to your `notifications` queue until the queue is empty. -When using the `auto` strategy, you may define the `minProcesses` and `maxProcesses` configuration options to control the minimum number of processes per queue and the maximum number of worker processes in total Horizon should scale up and down to: +When using the auto scaling strategy, you may define the `minProcesses` and `maxProcesses` configuration options: + +- `minProcesses` defines the minimum number of worker processes per queue. This value must be greater than or equal to 1. +- `maxProcesses` defines the maximum total number of worker processes Horizon may scale up to across all queues. This value should typically be greater than the number of queues multiplied by the `minProcesses` value. To prevent the supervisor from spawning any processes, you may set this value to 0. + +For example, you may configure Horizon to maintain at least one process per queue and scale up to a total of 10 worker processes: ```php 'environments' => [ @@ -146,18 +190,99 @@ When using the `auto` strategy, you may define the `minProcesses` and `maxProces 'maxProcesses' => 10, 'balanceMaxShift' => 1, 'balanceCooldown' => 3, - 'tries' => 3, ], ], ], ``` -The `autoScalingStrategy` configuration value determines if Horizon will assign more worker processes to queues based on the total amount of time it will take to clear the queue (`time` strategy) or by the total number of jobs on the queue (`size` strategy). +The `autoScalingStrategy` configuration option determines how Horizon will assign more worker processes to queues. You can choose between two strategies: +- The `time` strategy will assign workers based on the total estimated amount of time it will take to clear the queue. +- The `size` strategy will assign workers based on the total number of jobs on the queue. The `balanceMaxShift` and `balanceCooldown` configuration values determine how quickly Horizon will scale to meet worker demand. In the example above, a maximum of one new process will be created or destroyed every three seconds. You are free to tweak these values as necessary based on your application's needs. When the `balance` option is set to `false`, the default Laravel behavior will be used, wherein queues are processed in the order they are listed in your configuration. + +### Max Job Attempts + +> [!NOTE] +> Before refining these options, make sure you are familiar with Laravel's base [queue services](/docs/{{version}}/queues#max-job-attempts-and-timeout) and the concept of 'attempts'. + +You can define the maximum number of attempts a job can consume directly at the supervisor level: + +```php +'environments' => [ + 'production' => [ + 'supervisor-1' => [ + // ... + 'tries' => 10, + ], + ], +], +``` + +> [!NOTE] +> This option is similar to the `--tries` option when using the Artisan command to process queues. + +Fine-tuning the `tries` option is essential when using middlewares such as `WithoutOverlapping` or `RateLimited` because they consume attempts, however depending on your case, it may be more appropriate to set the job class' `$tries` property. + +If you do not specify a value for the `tries` option, jobs will only be attempted once or as many times as specified by the job class `$tries` property. + + +### Job Timeout + +Similarly, you can define a `timeout` value at the supervisor level. This value determines how many seconds a worker process is allowed to run a job before it is forcefully terminated. If a job exceeds the specified time limit, the worker process will be terminated, and the job will either be retried or marked as failed based on your queue configuration. + +```php +'environments' => [ + 'production' => [ + 'supervisor-1' => [ + // ... + /** + * The number of seconds the job can run before timing out. + */ + 'timeout' => 60, + ], + ], +], +``` + +>[!WARNING] +> The `timeout` value should always be at least several seconds shorter than the `retry_after` value define in your `config/queue.php` configuratin file. Otherwise your jobs may be processed twice. + + +### Job Backoff + +You can define the `backoff` value at the supervisor level to specify how long Horizon should wait before retrying a job that encounters an unhandled exception: + +```php +'environments' => [ + 'production' => [ + 'supervisor-1' => [ + // ... + /** + * The number of seconds to wait + */ + 'backoff' => 10, + ], + ], +], +``` + +You may also configure "exponential" backoffs by using an array for the `backoff` value. In this example, the retry delay will be 1 second for the first retry, 5 seconds for the second retry, 10 seconds for the third retry, and 10 seconds for every subsequent retry if there are more attempts remaining: + +```php +'environments' => [ + 'production' => [ + 'supervisor-1' => [ + // ... + 'backoff' => [1, 5, 10], + ], + ], +], +``` + ### Dashboard Authorization From 4c4341745b7413b57ced38cea997756ef2113c88 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Fri, 11 Jul 2025 23:26:15 +0200 Subject: [PATCH 03/20] format --- horizon.md | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/horizon.md b/horizon.md index 7aa61891e7..716ff7819b 100644 --- a/horizon.md +++ b/horizon.md @@ -99,14 +99,19 @@ When you start Horizon, it will use the worker process configuration options for You may also define Horizon's environment independently of the application's `APP_ENV` by adding an `env` key to your Horizon configuration file: ```php -/* -| ------------------------------------------------------------------ -| Horizon Environement -| ------------------------------------------------------------------ -| -| This value determine Horizon's environment! -*/ -'env' => env('HORIZON_ENV', env('APP_ENV')), + +return [ + /* + | ------------------------------------------------------------------ + | Horizon Environement + | ------------------------------------------------------------------ + | + | This value determine Horizon's environment! + */ + 'env' => env('HORIZON_ENV', env('APP_ENV')), + + // ... +]; ``` This can be particularly useful when running Horizon across multiple servers. For example, you might have one server processing general-purpose jobs using the default queue configuration, while a second server, using in a different environment key, is dedicated to handling resource-intensive jobs, such as video processing or large data imports. From 96a4084a1cfe97d4757a609c72df604486703024 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Sat, 12 Jul 2025 13:59:07 +0200 Subject: [PATCH 04/20] add documentation about TimeoutExceededException --- queues.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/queues.md b/queues.md index 9e77ee129b..5682682ecc 100644 --- a/queues.md +++ b/queues.md @@ -2273,7 +2273,11 @@ A failed job is not necessarily one that encountered an unhandled exception. A j - The job encounters an unhandled exception during execution. - The job is released back to the queue either manually or by a middleware. -If the final attempt fails due to an exception, the exception will be passed to the failed method. However, if the job fails simply because it reached the maximum number of attempts, the `$exception` will be an instance of `Illuminate\Queue\MaxAttemptsExceededException`. +If the final attempt fails due to an exception thrown during job execution, that exception will be passed to the job’s failed method. + +However, if the job fails because it has reached the maximum number of allowed attempts, the `$exception` will be an instance of `Illuminate\Queue\MaxAttemptsExceededException`. + +Similarly, if the job fails due to exceeding the configured timeout, the `$exception` will be an instance of `Illuminate\Queue\TimeoutExceededException`. ### Retrying Failed Jobs From a57d763fe830604658e7ca46b46f1fe84e06a996 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Sat, 12 Jul 2025 14:05:26 +0200 Subject: [PATCH 05/20] Clarify timeout doc --- queues.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/queues.md b/queues.md index 5682682ecc..3605e0ef3e 100644 --- a/queues.md +++ b/queues.md @@ -1223,6 +1223,7 @@ However, an attempt does not necessarily mean the job’s handle method was exec - The job's `handle` method runs and completes without throwing an exception. - The job encounters an unhandled exception during execution. +- The job timed out. - The job is manually released back to the queue using `$this->release()`. - Middleware such as `WithoutOverlapping` or `RateLimited` fails to acquire a lock and releases the job. @@ -1386,6 +1387,10 @@ If you would like to indicate that a job should be marked as [failed](#dealing-w public $failOnTimeout = true; ``` +> [!NOTE] +> By default, when a job times out, it consumes one attempt and is released back to the queue (if retries are allowed). +> However, if you configure the job to fail on timeout, it will not be retried, regardless of the value set for tries. + ### Error Handling @@ -2271,6 +2276,7 @@ class ProcessPodcast implements ShouldQueue A failed job is not necessarily one that encountered an unhandled exception. A job is considered failed when it has exhausted all of its allowed attempts. These attempts can be consumed in several ways: - The job encounters an unhandled exception during execution. +- The job timed out. - The job is released back to the queue either manually or by a middleware. If the final attempt fails due to an exception thrown during job execution, that exception will be passed to the job’s failed method. From db11c73b3241faa555361024c567d84b6a1b7f55 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Sat, 12 Jul 2025 17:32:48 +0200 Subject: [PATCH 06/20] rewrite strategies --- horizon.md | 109 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 85 insertions(+), 24 deletions(-) diff --git a/horizon.md b/horizon.md index 716ff7819b..771d6b0370 100644 --- a/horizon.md +++ b/horizon.md @@ -3,7 +3,6 @@ - [Introduction](#introduction) - [Installation](#installation) - [Configuration](#configuration) - - [Queue Priorities](#queue-priorities) - [Balancing Strategies](#balancing-strategies) - [Max Job Attempts](#max-job-attempts) - [Job Timeout](#job-timeout) @@ -149,32 +148,15 @@ While your application is in [maintenance mode](/docs/{{version}}/configuration# Within Horizon's default configuration file, you will notice a `defaults` configuration option. This configuration option specifies the default values for your application's [supervisors](#supervisors). The supervisor's default configuration values will be merged into the supervisor's configuration for each environment, allowing you to avoid unnecessary repetition when defining your supervisors. - -### Queue Priorities - -Each supervisor can process one or more queues. Just like Laravel's default queue system, queues are processed in the order they are listed in the queue configuration option, giving higher priority to queues defined first: - -```php -'environments' => [ - 'production' => [ - 'supervisor-1' => [ - 'connection' => 'redis', - 'queue' => ['high', 'default', 'low'], - ], - ], -], -``` - -In this example, jobs in the `high` queue will be prioritized over those in the `default` or `low` queues. - ### Balancing Strategies -Unlike Laravel's default queue system, Horizon allows you to choose from three worker balancing strategies: `simple`, `auto`, and `false`. The `simple` strategy splits incoming jobs evenly between worker processes: +Each supervisor can process one or more queues but unlike Laravel's default queue system, Horizon allows you to choose from three worker balancing strategies: `auto`, `simple`, and `false`. - 'balance' => 'simple', + +#### Auto Strategy -The `auto` strategy, which is the configuration file's default, adjusts the number of worker processes per queue based on the current workload of the queue. For example, if your `notifications` queue has 1,000 pending jobs while your `render` queue is empty, Horizon will allocate more workers to your `notifications` queue until the queue is empty. +The `auto` strategy, which is the default value, adjusts the number of worker processes per queue based on the current workload of the queue. For example, if your `notifications` queue has 1,000 pending jobs while your `default` queue is empty, Horizon will allocate more workers to your `notifications` queue until the queue is empty. When using the auto scaling strategy, you may define the `minProcesses` and `maxProcesses` configuration options: @@ -188,7 +170,7 @@ For example, you may configure Horizon to maintain at least one process per queu 'production' => [ 'supervisor-1' => [ 'connection' => 'redis', - 'queue' => ['default'], + 'queue' => ['default', 'notifications'], 'balance' => 'auto', 'autoScalingStrategy' => 'time', 'minProcesses' => 1, @@ -206,7 +188,86 @@ The `autoScalingStrategy` configuration option determines how Horizon will assig The `balanceMaxShift` and `balanceCooldown` configuration values determine how quickly Horizon will scale to meet worker demand. In the example above, a maximum of one new process will be created or destroyed every three seconds. You are free to tweak these values as necessary based on your application's needs. -When the `balance` option is set to `false`, the default Laravel behavior will be used, wherein queues are processed in the order they are listed in your configuration. + +#### Queue Priorities With Auto + +When using the `auto` balancing strategy, Horizon does not enforce strict priority between queues. The order in which queues are listed within a supervisor's configuration does not influence how jobs are assigned or processed. Instead, Horizon uses the configured balance strategy to dynamically allocate worker processes. + +For example, int the following configuration, the high queue is not prioritized over the default queue, despite appearing first in the list: + +```php +'environments' => [ + 'production' => [ + // ... + 'supervisor-1' => [ + 'queue' => ['high', 'default'], + 'minProcesses' => 1, + 'maxProcesses' => 10, + ], + ], +], +``` + +If you need to enforce a strict priority between queues, you may define multiple supervisors and explicitly allocate processing resources: +```php +'environments' => [ + 'production' => [ + // ... + 'supervisor-1' => [ + 'queue' => ['default'], + 'minProcesses' => 1, + 'maxProcesses' => 10, + ], + 'supervisor-2' => [ + 'queue' => ['images'], + 'minProcesses' => 1, + 'maxProcesses' => 1, + ], + ], +], +``` + +In this example, one process is always assigned to each queue, ensuring that all jobs are processed regardless of how many are present in each queue. However, the `images` queue will never have more than one process, even if a large number of jobs accumulate there. This guarantees that the `default` queue can scale independently of the `images` queue and that the `images` queue cannot spawn too many processes. + + +#### Simple Strategy + +The `simple` strategy splits incoming jobs evenly between queues. + +```php +'environments' => [ + 'production' => [ + 'supervisor-1' => [ + // ... + 'queue' => ['default', 'notifications'], + 'balance' => 'simple', + 'processes' => 5, + ], + ], +], +``` + +In the exemple, both 'default' and 'notifications' will have 5 processes assigned to them. + + +#### False Strategy + +When the `balance` option is set to `false`, the default Laravel behavior will be used, wherein queues are processed in the order they are listed in your configuration: + +```php +'environments' => [ + 'production' => [ + 'supervisor-1' => [ + // ... + 'queue' => ['default', 'notifications'], + 'balance' => false, + 'processes' => 5, + ], + ], +], +``` + +In this example, jobs in the `default` queue will be prioritized over those in the `notifications` queue. ### Max Job Attempts From 5f323686fda65d802692816c2afd15c2543b9592 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Sat, 12 Jul 2025 17:39:07 +0200 Subject: [PATCH 07/20] typo --- horizon.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/horizon.md b/horizon.md index 771d6b0370..4232f3d53d 100644 --- a/horizon.md +++ b/horizon.md @@ -193,7 +193,7 @@ The `balanceMaxShift` and `balanceCooldown` configuration values determine how q When using the `auto` balancing strategy, Horizon does not enforce strict priority between queues. The order in which queues are listed within a supervisor's configuration does not influence how jobs are assigned or processed. Instead, Horizon uses the configured balance strategy to dynamically allocate worker processes. -For example, int the following configuration, the high queue is not prioritized over the default queue, despite appearing first in the list: +For example, in the following configuration, the high queue is not prioritized over the default queue, despite appearing first in the list: ```php 'environments' => [ From 0471e2cf7eed56da2ce59be8cb0583fb09b5b9ed Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Sat, 12 Jul 2025 17:39:28 +0200 Subject: [PATCH 08/20] format --- horizon.md | 1 + 1 file changed, 1 insertion(+) diff --git a/horizon.md b/horizon.md index 4232f3d53d..0b82fc74fc 100644 --- a/horizon.md +++ b/horizon.md @@ -209,6 +209,7 @@ For example, in the following configuration, the high queue is not prioritized o ``` If you need to enforce a strict priority between queues, you may define multiple supervisors and explicitly allocate processing resources: + ```php 'environments' => [ 'production' => [ From cd779ccdacd94e57f7833d7c869f73aec91f0d54 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Sat, 12 Jul 2025 17:40:04 +0200 Subject: [PATCH 09/20] format --- horizon.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/horizon.md b/horizon.md index 0b82fc74fc..8498083b8b 100644 --- a/horizon.md +++ b/horizon.md @@ -198,8 +198,8 @@ For example, in the following configuration, the high queue is not prioritized o ```php 'environments' => [ 'production' => [ - // ... 'supervisor-1' => [ + // ... 'queue' => ['high', 'default'], 'minProcesses' => 1, 'maxProcesses' => 10, @@ -213,13 +213,14 @@ If you need to enforce a strict priority between queues, you may define multiple ```php 'environments' => [ 'production' => [ - // ... 'supervisor-1' => [ + // ... 'queue' => ['default'], 'minProcesses' => 1, 'maxProcesses' => 10, ], 'supervisor-2' => [ + // ... 'queue' => ['images'], 'minProcesses' => 1, 'maxProcesses' => 1, From 17caa35c3763e97d219b65677bbd027a959a16a2 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Sat, 12 Jul 2025 17:45:25 +0200 Subject: [PATCH 10/20] simplify --- horizon.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/horizon.md b/horizon.md index 8498083b8b..9b74d5f416 100644 --- a/horizon.md +++ b/horizon.md @@ -229,7 +229,7 @@ If you need to enforce a strict priority between queues, you may define multiple ], ``` -In this example, one process is always assigned to each queue, ensuring that all jobs are processed regardless of how many are present in each queue. However, the `images` queue will never have more than one process, even if a large number of jobs accumulate there. This guarantees that the `default` queue can scale independently of the `images` queue and that the `images` queue cannot spawn too many processes. +In this example, the default `queue` can scale up to 10 processes, while the `images` queue is limited to one process. This configuration ensures that your queues can scale independently. #### Simple Strategy From 982ad732caf64bf52322404879b12df4ab59a9f9 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Sat, 12 Jul 2025 17:45:57 +0200 Subject: [PATCH 11/20] simplify --- horizon.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/horizon.md b/horizon.md index 9b74d5f416..4e61263ddc 100644 --- a/horizon.md +++ b/horizon.md @@ -208,7 +208,7 @@ For example, in the following configuration, the high queue is not prioritized o ], ``` -If you need to enforce a strict priority between queues, you may define multiple supervisors and explicitly allocate processing resources: +If you need to enforce a relative priority between queues, you may define multiple supervisors and explicitly allocate processing resources: ```php 'environments' => [ From a9aeeeab2e6c939e2bb5e02e41f7b23b7d55aaee Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Sun, 13 Jul 2025 22:25:22 +0200 Subject: [PATCH 12/20] Correct False Strategy --- horizon.md | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/horizon.md b/horizon.md index 4e61263ddc..28beda1d36 100644 --- a/horizon.md +++ b/horizon.md @@ -156,9 +156,9 @@ Each supervisor can process one or more queues but unlike Laravel's default queu #### Auto Strategy -The `auto` strategy, which is the default value, adjusts the number of worker processes per queue based on the current workload of the queue. For example, if your `notifications` queue has 1,000 pending jobs while your `default` queue is empty, Horizon will allocate more workers to your `notifications` queue until the queue is empty. +The `auto` strategy, which is the default strategy, adjusts the number of worker processes per queue based on the current workload of the queue. For example, if your `notifications` queue has 1,000 pending jobs while your `default` queue is empty, Horizon will allocate more workers to your `notifications` queue until the queue is empty. -When using the auto scaling strategy, you may define the `minProcesses` and `maxProcesses` configuration options: +You may define the `minProcesses` and `maxProcesses` configuration options: - `minProcesses` defines the minimum number of worker processes per queue. This value must be greater than or equal to 1. - `maxProcesses` defines the maximum total number of worker processes Horizon may scale up to across all queues. This value should typically be greater than the number of queues multiplied by the `minProcesses` value. To prevent the supervisor from spawning any processes, you may set this value to 0. @@ -191,7 +191,7 @@ The `balanceMaxShift` and `balanceCooldown` configuration values determine how q #### Queue Priorities With Auto -When using the `auto` balancing strategy, Horizon does not enforce strict priority between queues. The order in which queues are listed within a supervisor's configuration does not influence how jobs are assigned or processed. Instead, Horizon uses the configured balance strategy to dynamically allocate worker processes. +When using the `auto` balancing strategy, Horizon does not enforce strict priority between queues. The order of queues in a supervisor's configuration does not affect how worker processes are assigned. Instead, Horizon relies on the selected `autoScalingStrategy` to dynamically allocate worker processes based on queue load For example, in the following configuration, the high queue is not prioritized over the default queue, despite appearing first in the list: @@ -231,6 +231,9 @@ If you need to enforce a relative priority between queues, you may define multip In this example, the default `queue` can scale up to 10 processes, while the `images` queue is limited to one process. This configuration ensures that your queues can scale independently. +>[!NOTE] +> When dispatching resource-intensive jobs, it's best to assign them to a dedicated queue with a limited `maxProcesses` value. Otherwise, these jobs could consume excessive CPU resources and overload your system. + #### Simple Strategy @@ -248,13 +251,12 @@ The `simple` strategy splits incoming jobs evenly between queues. ], ], ``` - In the exemple, both 'default' and 'notifications' will have 5 processes assigned to them. #### False Strategy -When the `balance` option is set to `false`, the default Laravel behavior will be used, wherein queues are processed in the order they are listed in your configuration: +Setting the `balance` option to `false` means Horizon will not distribute worker processes dynamically between queues. Instead, queues are processed strictly in the order they are listed. Horizon will still scale up the number of processes if jobs begin to accumulate. ```php 'environments' => [ @@ -263,13 +265,19 @@ When the `balance` option is set to `false`, the default Laravel behavior will b // ... 'queue' => ['default', 'notifications'], 'balance' => false, - 'processes' => 5, + 'minProcesses' => 1, + 'maxProcesses' => 10, ], ], ], ``` -In this example, jobs in the `default` queue will be prioritized over those in the `notifications` queue. +In this example, jobs in the default queue are always prioritized over those in the notifications queue. For example, if there are 1,000 jobs in the default queue and only 10 in the notifications queue, Horizon will process all default jobs before beginning any from the notifications queue. + +You can also configure the minProcesses and maxProcesses options: + +- `minProcesses` defines the minimum number of worker processes in total. This value must be greater than or equal to 1. +- `maxProcesses` defines the maximum total number of worker processes Horizon may scale up to. ### Max Job Attempts From f2eb5ff90ec1c32c55a71d5770bdc5eaef4231ae Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Mon, 14 Jul 2025 20:37:27 +0200 Subject: [PATCH 13/20] use min/maxProcesses instead of processes --- horizon.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/horizon.md b/horizon.md index 28beda1d36..bc47d3fa4f 100644 --- a/horizon.md +++ b/horizon.md @@ -246,7 +246,8 @@ The `simple` strategy splits incoming jobs evenly between queues. // ... 'queue' => ['default', 'notifications'], 'balance' => 'simple', - 'processes' => 5, + 'minProcesses' => 1, + 'maxProcesses' => 10, ], ], ], From 433ec8b9d914872f13fc0a9117c9ee115e0e4cc8 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Mon, 14 Jul 2025 21:08:47 +0200 Subject: [PATCH 14/20] rewrite simple strategy --- horizon.md | 36 +++++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/horizon.md b/horizon.md index bc47d3fa4f..b13fb3aa5f 100644 --- a/horizon.md +++ b/horizon.md @@ -237,7 +237,7 @@ In this example, the default `queue` can scale up to 10 processes, while the `im #### Simple Strategy -The `simple` strategy splits incoming jobs evenly between queues. +The `simple` strategy distributes worker processes evenly across the specified queues. With this strategy, Horizon does not automatically scale the number of worker processes, it uses a fixed number of processes. ```php 'environments' => [ @@ -246,18 +246,40 @@ The `simple` strategy splits incoming jobs evenly between queues. // ... 'queue' => ['default', 'notifications'], 'balance' => 'simple', - 'minProcesses' => 1, - 'maxProcesses' => 10, + 'processes' => 10, ], ], ], ``` -In the exemple, both 'default' and 'notifications' will have 5 processes assigned to them. +In the example above, Horizon will assign 5 processes to each queue, splitting the total of 10 evenly. + +If you'd like to control the number of worker processes assigned to each queue individually, you can define multiple supervisors: + +```php +'environments' => [ + 'production' => [ + 'supervisor-1' => [ + // ... + 'queue' => ['default'], + 'balance' => 'simple', + 'processes' => 10, + ], + 'supervisor-notifications' => [ + // ... + 'queue' => ['notifications'], + 'balance' => 'simple', + 'processes' => 2, + ], + ], +], +``` + +With this configuration, Horizon will assign 10 processes to the `default@ queue and 2 processes to the `notifications` queue. #### False Strategy -Setting the `balance` option to `false` means Horizon will not distribute worker processes dynamically between queues. Instead, queues are processed strictly in the order they are listed. Horizon will still scale up the number of processes if jobs begin to accumulate. +When the `balance` value is set to `false` Horizon will not distribute worker processes dynamically between queues. Instead, queues are processed strictly in the order they are listed. Horizon will still scale up the number of processes if jobs begin to accumulate. ```php 'environments' => [ @@ -273,9 +295,9 @@ Setting the `balance` option to `false` means Horizon will not distribute worker ], ``` -In this example, jobs in the default queue are always prioritized over those in the notifications queue. For example, if there are 1,000 jobs in the default queue and only 10 in the notifications queue, Horizon will process all default jobs before beginning any from the notifications queue. +In the example above, jobs in the `default` queue are always prioritized over jobs in the `notifications` queue. For instance, if there are 1,000 jobs in `default` and only 10 in `notifications`, Horizon will fully process all `default` jobs before handling any from `notifications`. -You can also configure the minProcesses and maxProcesses options: +You can control Horizon's ability to scale worker processes using the `minProcesses` and `maxProcesses` options: - `minProcesses` defines the minimum number of worker processes in total. This value must be greater than or equal to 1. - `maxProcesses` defines the maximum total number of worker processes Horizon may scale up to. From 61476ee87fdf4804503c37f0415ca8b05f153df3 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Mon, 14 Jul 2025 21:12:59 +0200 Subject: [PATCH 15/20] typo --- horizon.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/horizon.md b/horizon.md index b13fb3aa5f..13f338c842 100644 --- a/horizon.md +++ b/horizon.md @@ -274,7 +274,7 @@ If you'd like to control the number of worker processes assigned to each queue i ], ``` -With this configuration, Horizon will assign 10 processes to the `default@ queue and 2 processes to the `notifications` queue. +With this configuration, Horizon will assign 10 processes to the `default` queue and 2 processes to the `notifications` queue. #### False Strategy From 4784ecc0807905e735840e92cd4bb46d78ba93cc Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Mon, 14 Jul 2025 21:17:01 +0200 Subject: [PATCH 16/20] simplify false --- horizon.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/horizon.md b/horizon.md index 13f338c842..c9ee091bc5 100644 --- a/horizon.md +++ b/horizon.md @@ -279,7 +279,7 @@ With this configuration, Horizon will assign 10 processes to the `default` queue #### False Strategy -When the `balance` value is set to `false` Horizon will not distribute worker processes dynamically between queues. Instead, queues are processed strictly in the order they are listed. Horizon will still scale up the number of processes if jobs begin to accumulate. +When the `balance` option is set to `false`, Horizon processes queues strictly in the order they are listed. Horizon will still scale up the number of worker processes if jobs begin to accumulate. This strategy behaves similarly to Laravel’s default queue system. ```php 'environments' => [ From 2a739d129ee87bcb40e27bc77176514bd0a02bfc Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Mon, 14 Jul 2025 21:18:23 +0200 Subject: [PATCH 17/20] reformulate --- horizon.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/horizon.md b/horizon.md index c9ee091bc5..034092b518 100644 --- a/horizon.md +++ b/horizon.md @@ -279,7 +279,7 @@ With this configuration, Horizon will assign 10 processes to the `default` queue #### False Strategy -When the `balance` option is set to `false`, Horizon processes queues strictly in the order they are listed. Horizon will still scale up the number of worker processes if jobs begin to accumulate. This strategy behaves similarly to Laravel’s default queue system. +When the `balance` option is set to `false`, Horizon processes queues strictly in the order they're listed, similar to Laravel’s default queue system. However, it will still scale the number of worker processes if jobs begin to accumulate. ```php 'environments' => [ From 5f34adb1cef2a690ff1ff9d79f4fabead165bb95 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Mon, 14 Jul 2025 21:22:17 +0200 Subject: [PATCH 18/20] reformulate --- horizon.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/horizon.md b/horizon.md index 034092b518..c5f5b1d520 100644 --- a/horizon.md +++ b/horizon.md @@ -331,7 +331,7 @@ If you do not specify a value for the `tries` option, jobs will only be attempte ### Job Timeout -Similarly, you can define a `timeout` value at the supervisor level. This value determines how many seconds a worker process is allowed to run a job before it is forcefully terminated. If a job exceeds the specified time limit, the worker process will be terminated, and the job will either be retried or marked as failed based on your queue configuration. +Similarly, you can set a `timeout` value at the supervisor level, which specifies how many seconds a worker process can run a job before it’s forcefully terminated. Once terminated, the job will either be retried or marked as failed, depending on your queue configuration. ```php 'environments' => [ From 8ce5cda2e8aa4a7a7a95a9fd6a7c89d88936b9d4 Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Mon, 14 Jul 2025 21:23:11 +0200 Subject: [PATCH 19/20] typo --- horizon.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/horizon.md b/horizon.md index c5f5b1d520..ec861d7d7e 100644 --- a/horizon.md +++ b/horizon.md @@ -348,7 +348,7 @@ Similarly, you can set a `timeout` value at the supervisor level, which specifie ``` >[!WARNING] -> The `timeout` value should always be at least several seconds shorter than the `retry_after` value define in your `config/queue.php` configuratin file. Otherwise your jobs may be processed twice. +> The `timeout` value should always be at least a few seconds shorter than the `retry_after` value defined in your `config/queue.php` configuration file. Otherwise, your jobs may be processed twice. ### Job Backoff From 488f88f96ee2180a7249b00e299ce0fa1ccbb06f Mon Sep 17 00:00:00 2001 From: Quentin Gabriele Date: Mon, 14 Jul 2025 21:24:04 +0200 Subject: [PATCH 20/20] remove useless comments --- horizon.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/horizon.md b/horizon.md index ec861d7d7e..1981d4f7ad 100644 --- a/horizon.md +++ b/horizon.md @@ -337,10 +337,7 @@ Similarly, you can set a `timeout` value at the supervisor level, which specifie 'environments' => [ 'production' => [ 'supervisor-1' => [ - // ... - /** - * The number of seconds the job can run before timing out. - */ + // ...¨ 'timeout' => 60, ], ], @@ -360,9 +357,6 @@ You can define the `backoff` value at the supervisor level to specify how long H 'production' => [ 'supervisor-1' => [ // ... - /** - * The number of seconds to wait - */ 'backoff' => 10, ], ],