Add HTTPError to retryable errors and increase max retries for network failures

[cgoldshtein]

Description:

Currently, XLoader only retries jobs for database-related errors (DeadlockDetected, LockNotAvailable, ObjectInUse) but doesn't retry for network-related failures during resource download.

Problem:

The _download_resource_data function can raise HTTPError for temporary network issues

These network errors are not included in RETRYABLE_ERRORS, so jobs fail permanently

MAX_RETRIES is set to 1, which may not be sufficient for transient network issues

Proposed Solution:

Add HTTPError to RETRYABLE_ERRORS tuple

Increase MAX_RETRIES from 1 to 3 for better resilience

Current Code (lines 42-47):

MAX_RETRIES = 1
RETRYABLE_ERRORS = (
    errors.DeadlockDetected,
    errors.LockNotAvailable,
    errors.ObjectInUse,
)

Proposed Change:

MAX_RETRIES = 3
RETRYABLE_ERRORS = (
    errors.DeadlockDetected,
    errors.LockNotAvailable,
    errors.ObjectInUse,
    HTTPError,
)

Benefits:

Improved reliability for resources hosted on unreliable servers

Reduces false negatives from temporary network issues

Maintains existing retry logic for database errors

Environment:

CKAN version: 2.10.8

XLoader version: 2.0.1

[ThrawnCA]

I partially like the idea, but HTTPError covers a wide range of errors, of which many are not retryable, especially 4xx errors.

The intent of this feature was to handle transaction locking timeouts, which almost universally can be retried successfully.

[duttonw]

Maybe only should run on some status codes, default list but allows override

say only these at this time:
408 Request Timeout
429 Too Many Requests
500 Internal Server Error
503 Service Unavailable
504 Gateway Timeout
507 Insufficient Storage