Skip to content

Do not crash the client if ingestion server is down #164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

vizath
Copy link

@vizath vizath commented May 27, 2025

Recently, we had received a few 503 from axiom logging.
When the Client receives a 503 after few retries, it raise an Exception.
That's problematic because it also crashes the whole app.

How to reproduce in a production python app:

  • update your logger with a 503 error: axiom_py.Client(token=token, url_base="https://httpstat.us/503")
  • Have a FastAPI endpoint creating a log
  • See the logging crashing the route

Another solution would be to move the try/except in the flush function of logging.py.

I'm not super fluent in python, feel free to propose changes.

Some related stack trace:

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.12/threading.py", line 1433, in run
    self.function(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.12/site-packages/axiom_py/logging.py", line 67, in flush
    self.client.ingest_events(self.dataset, local_buffer)
  File "/usr/local/lib/python3.12/site-packages/axiom_py/client.py", line 262, in ingest_events
    return self.ingest(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/axiom_py/client.py", line 239, in ingest
    res = self.session.post(
          ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests_toolbelt/sessions.py", line 76, in request
    return super(BaseUrlSession, self).request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 710, in send
    r = dispatch_hook("response", hooks, r, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/hooks.py", line 30, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/axiom_py/client.py", line 182, in <lambda>
    "response": lambda r, *args, **kwargs: raise_response_error(r)
                                           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/axiom_py/client.py", line 145, in raise_response_error
    raise AxiomError(res.status_code, error_res)
axiom_py.client.AxiomError: API error 404: Not Found

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant