[Bug]: Custom root path (still) does not work

[ak11234]

What happened?

Based on #10761, we would expect for the custom root path (as referred to in docs to work. Meaning we set SERVER_ROOT_PATH as an environment variable when starting the litellm proxy.

However, it does not. For example, in setting SERVER_ROOT_PATH=/proxy, we see the following:

• Upon visiting the Swagger UI, the main html loads, and attempts to load its dependent scripts from the bare URL, not using the custom path.
• Upon visiting the management UI, the main HTML loads, but again the dependent scripts attempt to load from the bare URL, not using the custom path.
  • This problem may not be immediately obvious if you are directly accessing the container without a proxy in front that will ONLY allow traffic with the custom base path. If litellm receives the request for the js/css at the non-custom base path, it will redirect those requests to the correct custom path, so it appears to work in bench testing.

If we add in the old setting PROXY_BASE_URL that is no longer listed on that docs page:

• The Swagger UI now loads correctly
• The management UI's html still refers to the wrong paths
• The /litellm/.well-known/litellm-ui-config API does show the correct base urls

I have tried a number of versions, up to and including the official docker images from v1.72.1-nightly to no avail.

I see that @krrishdholakia closed the bug report as completed/fixed back in v1.71.2, but I am curious to here if they actually tested to make sure it worked...

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.72.1-nightly

Twitter / LinkedIn details

No response

[krrishdholakia]

Hi @ak11234 - can you share a video of what you're seeing?

[krrishdholakia]

we were able to confirm this is working for another user as well

[krrishdholakia]

The key thing is to reserve /litellm route on the base url

so ${BASE_URL}/litellm -> point to litellm proxy

Not ${BASE_URL}/my-custom-path/litellm

[ak11234]

The key thing is to reserve /litellm route on the base url

so ${BASE_URL}/litellm -> point to litellm proxy

Not ${BASE_URL}/my-custom-path/litellm

Ahh. That expected behavior does align with what I am seeing (photo below), thank you for clarifying.

However, I must say that this really isn't a full custom path implementation... You seem to be suggesting that we can use a custom path in the URL bar, but that we still need to have the exact standard-issue paths (/litellm/*) available anyways. That makes this feel more like a vanity feature than a true base path modification. In my corporate setting, I need to have everything running under a certain path, where this application is confined to.

I understand that making the frontend refer to paths with a different base is hard to do dynamically. I am happy to custom-compile the JS app, and the old instructions used to have a sample dockerfile to do so (albeit a dockerfile that no longer worked given your now alpine-based images). But that has since been removed.

At the end of the day, all I need is to be able to confine litellm to run under mydomain.com/custom-path/... so that litellm can share network/domain infrastructure with other applications without requiring that mydomain.com/litellm/... is specifically allocated to it.

 

[krrishdholakia]

The /litellm route is also where we serve our .well-known to know how to construct the custom url.

This allows the UI to know how to call the backend.

How have you seen other products handle this?

[ak11234]

The /litellm route is also where we serve our .well-known to know how to construct the custom url.

This allows the UI to know how to call the backend.

How have you seen other products handle this?

In general, when other products accept a custom base URL, it lists everything under that URL. So anywhere a route is defined, whether it be a backend API route handler, or a frontend reference/redirect/etc, that base URL is used as a prefix, rather than just using / as a prefix.

It seems you have mostly achieved this already. Your backend API already supports nesting under some custom base url, and the frontend seems to almost be there, other than the UI needing that .well-known in order to find the custom url.

Currently (based on what I can tell), you have code that responds to requests for /ui and is serving the initial UI html file, which then loads up the js and css for the web ui. The core problem is that the html page refers to those hardcoded url in the src attributes that load those resources. Then, once the bundles are loaded, your JS tries to use that .well-known API to figure out how to call the backend.

If I may suggest, you should find a way to have the frontend "know" the custom base path without needing that hardcoded /litellm route to be there to 1) serve the static assets (CSS and JS) and 2) have the reliable .well-known api to then locate the backend. One thing you could do is however you are serving that HTML and static assets, bake in support for subpaths. The code that serves the frontend could dynamically create the html, prefacing the urls in those src attributes to have whatever the base url is. Which by default can be / or /litellm or whatever you want, but is customizable. If you backend API is serving that webapp bundle, it knows about the base url, and could assemble the correct links just-in-time when it serves the html/js/css.

There are of course other ways to serve static web apps that have built-in support for this, if you feel the above is too complicated.

And if worst comes to worst, you can go back to making us use a dockerfile to recompile the JS bundle given the expected base url, rather than trying to figure out how to do this all at runtime. Would be fine with me, although I respect that you have already put in the effort to try and get this to work at runtime, and I would not be surprised if you want to get it to work!

[ak11234]

Yeah, looks like based on this line of code, you are bundling some static web app and then serving it directly from FastAPI. Pretty standard thing to do, no issues, but sort of confirms (in my opinion) that you have 2 choices. Either provide instructions and a stable way to provide the custom URL and recompile the UI, or have FastAPI inject the correct base URL into the bundle when it serves it. The latter is obviously a bit more complex...

[krrishdholakia]

have FastAPI inject the correct base URL into the bundle when it serves it

Oh, how could we do this?

Ideally we don't require users to maintain a custom docker image / build process

[ak11234]

Hey @krrishdholakia the PR looks super promising! I just deployed that version and it seems better, but not perfect yet.
With SERVER_ROOT_PATH and PROXY_BASE_URL set, I can see that the /custom/ui page correctly loads the .well-known API from the custom location (big win), but then it redirects to /sso/key/generate without the custom base URL. So close!

From breakpointing and correlating with your code, I think the culprit is this useEffect which relies on the proxyBaseUrl. When I break here, I see that the proxyBaseUrl that is being used here is still null, therefore the redirect url is wrong.

I can see that the .well-known is being fetched from the correct place, however, it is still "pending" when this redirect happens!

I think that we have a race condition on our hands, where your useEffect hook in page.tsx is attempting to do the login redirect before the networking.tsx code has had a chance to resolve the correct url. Since page.tsx is reading the variable directly, it is able to read the value before it is resolved.

There are a few ways you could try to resolve this. One way could be that instead of directly exporting the proxyBaseUrl variable, you export a getProxyBaseUrl function that returns the proxyBaseUrl value, but if the value has not yet been resolved and stored into a private variable, it fetches it, stores it, then returns it.