ValueError when null byte in URL

[stefan-hoelzl]

Hello,

today someone tried a bunch of URLs on my online service (using starlette).
On one URL (/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/etc/passwd%00) starlette (specific the StaticFiles Middleware) raised an ValueError: embedded null byte.

This is the full StackTrace

2024-04-03T12:22:52+02:00	ValueError: embedded null byte
2024-04-03T12:22:52+02:00	    st = os.lstat(newpath)
2024-04-03T12:22:52+02:00	  File "/usr/local/lib/python3.10/posixpath.py", line 431, in _joinrealpath
2024-04-03T12:22:52+02:00	    path, ok = _joinrealpath(filename[:0], filename, strict, {})
2024-04-03T12:22:52+02:00	  File "/usr/local/lib/python3.10/posixpath.py", line 396, in realpath
2024-04-03T12:22:52+02:00	    full_path = os.path.realpath(joined_path)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/staticfiles.py", line 164, in lookup_path
2024-04-03T12:22:52+02:00	    result = context.run(func, *args)
2024-04-03T12:22:52+02:00	  File "/packages/anyio/_backends/_asyncio.py", line 851, in run
2024-04-03T12:22:52+02:00	    return await future
2024-04-03T12:22:52+02:00	  File "/packages/anyio/_backends/_asyncio.py", line 2134, in run_sync_in_worker_thread
2024-04-03T12:22:52+02:00	    return await get_async_backend().run_sync_in_worker_thread(
2024-04-03T12:22:52+02:00	  File "/packages/anyio/to_thread.py", line 56, in run_sync
2024-04-03T12:22:52+02:00	    full_path, stat_result = await anyio.to_thread.run_sync(
2024-04-03T12:22:52+02:00	  File "/packages/starlette/staticfiles.py", line 122, in get_response
2024-04-03T12:22:52+02:00	    response = await super().get_response(path, scope)
2024-04-03T12:22:52+02:00	  File "/packages/api/app.py", line 122, in get_response
2024-04-03T12:22:52+02:00	    response = await self.get_response(path, scope)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/staticfiles.py", line 103, in __call__
2024-04-03T12:22:52+02:00	    await self.app(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/routing.py", line 487, in handle
2024-04-03T12:22:52+02:00	    await route.handle(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/routing.py", line 778, in app
2024-04-03T12:22:52+02:00	    await self.middleware_stack(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/routing.py", line 758, in __call__
2024-04-03T12:22:52+02:00	    await app(scope, receive, sender)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-04-03T12:22:52+02:00	    raise exc
2024-04-03T12:22:52+02:00	  File "/packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-04-03T12:22:52+02:00	    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/exceptions.py", line 62, in __call__
2024-04-03T12:22:52+02:00	    return await old_call(app, scope, new_receive, new_send, **kwargs)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
2024-04-03T12:22:52+02:00	    await old_call(self, scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/starlette.py", line 256, in _sentry_exceptionmiddleware_call
2024-04-03T12:22:52+02:00	    await self.app(scope, receive_or_disconnect, send_no_error)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/base.py", line 151, in coro
2024-04-03T12:22:52+02:00	    raise app_exc
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/base.py", line 165, in call_next
2024-04-03T12:22:52+02:00	    return await call_next(request)
2024-04-03T12:22:52+02:00	  File "/packages/api/app.py", line 64, in log_request_duration
2024-04-03T12:22:52+02:00	    response = await self.dispatch_func(request, call_next)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/base.py", line 191, in __call__
2024-04-03T12:22:52+02:00	    raise exc
2024-04-03T12:22:52+02:00	  File "/packages/starlette/_utils.py", line 93, in collapse_excgroups
2024-04-03T12:22:52+02:00	    self.gen.throw(typ, value, traceback)
2024-04-03T12:22:52+02:00	  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
2024-04-03T12:22:52+02:00	    with collapse_excgroups():
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/base.py", line 189, in __call__
2024-04-03T12:22:52+02:00	    return await old_call(app, scope, new_receive, new_send, **kwargs)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
2024-04-03T12:22:52+02:00	    await self.app(scope, receive, _send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/errors.py", line 164, in __call__
2024-04-03T12:22:52+02:00	    raise exc
2024-04-03T12:22:52+02:00	  File "/packages/starlette/middleware/errors.py", line 186, in __call__
2024-04-03T12:22:52+02:00	    return await old_call(app, scope, new_receive, new_send, **kwargs)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
2024-04-03T12:22:52+02:00	    await self.middleware_stack(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/starlette/applications.py", line 123, in __call__
2024-04-03T12:22:52+02:00	    return await self.app(
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/asgi.py", line 234, in _run_app
2024-04-03T12:22:52+02:00	    raise exc from None
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/asgi.py", line 241, in _run_app
2024-04-03T12:22:52+02:00	    return await self._run_app(scope, receive, send, asgi_version=3)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/asgi.py", line 146, in _run_asgi3
2024-04-03T12:22:52+02:00	    return await middleware(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/sentry_sdk/integrations/starlette.py", line 364, in _sentry_patched_asgi_app
2024-04-03T12:22:52+02:00	    await super().__call__(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/fastapi/applications.py", line 1054, in __call__
2024-04-03T12:22:52+02:00	    return await self.app(scope, receive, send)
2024-04-03T12:22:52+02:00	  File "/packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
2024-04-03T12:22:52+02:00	    result = await app(  # type: ignore[func-returns-value]
2024-04-03T12:22:52+02:00	  File "/packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
2024-04-03T12:22:52+02:00	Traceback (most recent call last):
2024-04-03T12:22:52+02:00	2024-04-03 10:22:52,088 ERROR    uvicorn.error        Exception in ASGI application
2024-04-03T12:22:52+02:00	2024-04-03 10:22:51,990 INFO     uvicorn.access       GET /%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD/etc/passwd%00 HTTP/1.1 500 Internal Server Error

I would expect starlette to handle this in someway instead of raising an exception.

Important

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[Zaczero]

For some more context, this happens when filename contains a null byte (such filenames are invalid). I suppose a fix would be to sanitize filenames or to improve exception handling. I prefer the latter since it will not impact the performance of normal requests. It will also be much simpler in code.

[stefanhoelzl]

This seems to be a cpython issue.
os.path.realpath raises an ValueError when a path contains a null byte.
Unlike other path operations e.g. os.path.exists, os.path.islink, os.fspath.

I created a PR to fix this issue python/cpython#117573

[nggit]

That "someone" is trying to exploit a vulnerability in your service.

Allowing null bytes in paths can have bad implications,
e.g. if an application naively thinks /etc/passwd%00.jpg is a jpg, but file system functions like fopen() may treat it as a C string.
It ignores everything after null resulting in reading the wrong file.

Luckily, that ugly error came to your rescue.

I personally think this should be anticipated at the ASGI app (or middleware), e.g. rejecting filenames with null.
Rather than relying on CPython behavior, whether it is handled correctly or not on the CPython side.