fix: Retry on NeptuneRetryableError. Call on_async_lag_callback #99
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
NeptuneAsyncLagThresholdExceeded is a NeptuneScaleError, so when it was after NeptuneScaleError in the except block, the more general on_error_callback was called to handle this exception and on_async_lag_callback was a dead code
NeptuneRetryableError currently has 3 subclasses (NeptuneConnectionLostError, NeptuneUnexpectedResponseError, NeptuneInternalServerError)
NeptuneConnectionLostError is retried for 60 seconds with an exponential backoff, the other two subtypes simply triggered on_error_callback, which by default kills the sender process. It was possible to provide a custom on_error_callback to make them actually retriable, but with the default config they were simply fatal. I added them to the backoff instruction, so they are all handled in the same way, ie. retried for 60s and only afterwards the decision is delegated to the on_error_callback