Skip to content

[Hotfix] Change Promise.all to for await in project status handler to catch UnauthorizedException #3213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 16, 2024

Conversation

andrewmurraydavid
Copy link
Contributor

Cord API memory leak causing AWS crash

Description

This PR changes how the engagements are updated in the project handler.
There seems to be an issue, either with the the way Promise.all handles the UnauthorizedException for the edgecase where a project has 3 engagements, one (or more) of them being in a step previous (for example 2 in Active change plan and 1 in Discussing change plan), the engagement update would fail with the UnauthorizedException.

It is assumed that because Promis.all not handling the exception properly, would cause a neo4j transaction creation to retry in an infinite loop.

This infinite loop caused the API service to crash completely being OOM (out of memory).

Steps to reproduce the bug

In the description

Expected behavior

Not crash, lol

Ready for review checklist

Use [N/A] if the item is not applicable to this PR or remove the item

  • Change the task url above to the actual Monday task
  • Add reviewers to this PR
  • THIS IS A HOTFIX

@bryanjnelson bryanjnelson merged commit 7bffdaa into master May 16, 2024
21 checks passed
@bryanjnelson bryanjnelson deleted the hotfix/api-crashes-promise-all branch May 16, 2024 21:32
@CarsonF
Copy link
Member

CarsonF commented May 16, 2024

UnauthorizedExceptions should not be treated as retryable. But maybe it was the error from the other queries?
I don't think this is the root solution, other queries in parallel could trigger this symptom too.
But I'm pretty sure EdgeDB doesn't even let you run queries in parallel while in a transaction so there's that too. Which warrants this change.

Good job @andrewmurraydavid! Race conditions are hard to track down 👍

adam-soltech pushed a commit that referenced this pull request May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants