Skip to content

fix(aci): Retry process_delayed_workflows timeouts #95379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

kcons
Copy link
Member

@kcons kcons commented Jul 11, 2025

Timeouts are expected here and assumed transient, and the existing retry decorator and config don't support retrying ProcessingDeadlineExceeded.
So, we add a new task decorator to trigger retries on timeouts. This avoids opening the door to other BaseExceptions and keeps it opt-in as designed.

@kcons kcons requested a review from a team as a code owner July 11, 2025 21:19
@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 11, 2025
cursor[bot]

This comment was marked as outdated.

Copy link

codecov bot commented Jul 11, 2025

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
27216 1 27215 240
View the top 1 failed test(s) by shortest run time
tests.sentry.tasks.test_base::test_retry_on
Stack Traces | 0.048s run time
#x1B[1m#x1B[.../sentry/tasks/base.py#x1B[0m:248: in wrapped
    return func(*args, **kwargs)
#x1B[1m#x1B[.../sentry/silo/base.py#x1B[0m:165: in override
    return handler(*args, **kwargs)
#x1B[1m#x1B[.../sentry/tasks/base.py#x1B[0m:51: in handle
    raise self.AvailabilityError(message)
#x1B[1m#x1B[31mE   sentry.silo.base.SiloLimit.AvailabilityError: Cannot call or spawn retry_on_task in REGION,#x1B[0m

#x1B[33mDuring handling of the above exception, another exception occurred:#x1B[0m
#x1B[1m#x1B[.../sentry/tasks/test_base.py#x1B[0m:168: in test_retry_on
    retry_on_task("bruh")
#x1B[1m#x1B[.../sentry/tasks/base.py#x1B[0m:262: in wrapped
    retry_task(exc)
#x1B[1m#x1B[.../sentry/taskworker/retry.py#x1B[0m:59: in retry_task
    assert False, "unreachable"
#x1B[1m#x1B[31mE   AssertionError: unreachable#x1B[0m

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@kcons kcons marked this pull request as draft July 11, 2025 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant