[Bug]: /model/delete API breaks when using multiple workers

[kota-iizuka]

What happened?

When I set NUM_WORKERS to 8 (presumably any number greater than 1), model information remains in /v1/models and /v1/model/info even though I deleted the model via the /model/delete API.

Here is some sample code that reproduces this issue, along with the output:

(The environment was created according to https://docs.litellm.ai/docs/proxy/deploy, except that NUM_WORKERS: 8 was added to the environment variable at the following location.)

litellm/docker-compose.yml

Line 18 in 049e65a

environment:

import requests

LITELLM_API_URL = "http://localhost:4000"
LITELLM_KEY = "sk-1234"

def get_litellm_models():
    response = requests.get(
        f"{LITELLM_API_URL}/v1/models",
        headers={
            "accept": "application/json",
            "Authorization": f"Bearer {LITELLM_KEY}",
        },
    )
    return response.json()


def get_litellm_model_info():
    response = requests.get(
        f"{LITELLM_API_URL}/v1/model/info",
        headers={
            "accept": "application/json",
            "Authorization": f"Bearer {LITELLM_KEY}",
        },
    )
    models = []
    for model in response.json()["data"]:
        models.append(
            {
                "model_name": model["model_name"],
                "model_litellm_id": model["model_info"]["id"],
            }
        )
    return models


def add_litellm_model(model_name: str):
    response = requests.post(
        f"{LITELLM_API_URL}/model/new",
        headers={
            "accept": "application/json",
            "Content-Type": "application/json",
            "Authorization": f"Bearer {LITELLM_KEY}",
        },
        json={
            "model_name": model_name,
            "litellm_params": {
                "model": f"openai/{model_name}",
            },
            "model_info": {"version": 2},
        },
    )
    return response.json()


def delete_litellm_model(model_name: str):
    litellm_models = get_litellm_model_info()
    ret = []
    for model in litellm_models:
        if model["model_name"] == model_name:
            # todo: error handling
            response = requests.post(
                f"{LITELLM_API_URL}/model/delete",
                headers={
                    "accept": "application/json",
                    "Content-Type": "application/json",
                    "Authorization": f"Bearer {LITELLM_KEY}",
                },
                json={"id": model["model_litellm_id"]},
            )
            if response.status_code != 200:
                raise RuntimeError(f"Failed to delete model {model_name}: {response.text}")
            ret.append(response.json())
    return ret


def main():
    for i in range(10):
        print(i, get_litellm_models())
    print(add_litellm_model("model1"))
    for i in range(10):
        print(i, get_litellm_models())
    print(delete_litellm_model("model1"))
    for i in range(10):
        print(i, get_litellm_models())


if __name__ == "__main__":
    main()

Relevant log output

0 {'data': [], 'object': 'list'}
1 {'data': [], 'object': 'list'}
2 {'data': [], 'object': 'list'}
3 {'data': [], 'object': 'list'}
4 {'data': [], 'object': 'list'}
5 {'data': [], 'object': 'list'}
6 {'data': [], 'object': 'list'}
7 {'data': [], 'object': 'list'}
8 {'data': [], 'object': 'list'}
9 {'data': [], 'object': 'list'}
{'model_id': '32ba4821-fb74-41e1-b965-cadb53888736', 'model_name': 'model1', 'litellm_params': {'model': 'QvI3p+IdHJmyvcXw4FznKzwjJXyhPzdwcmyeKNxjH7H54vW22+lDffNmHJgeE4qu3R93hJQ=', 'use_litellm_proxy': False, 'use_in_pass_through': False, 'merge_reasoning_content_in_choices': False}, 'model_info': {'id': '32ba4821-fb74-41e1-b965-cadb53888736', 'version': 2, 'db_model': False}, 'created_at': '2025-06-05T01:56:24.965000Z', 'created_by': 'default_user_id', 'updated_at': '2025-06-05T01:56:24.965000Z', 'updated_by': 'default_user_id'}
0 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
1 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
2 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
3 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
4 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
5 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
6 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
7 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
8 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
9 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
[{'message': 'Model: 5fc30770-ae54-4517-9af9-008466b0395f deleted successfully'}]
0 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
1 {'data': [], 'object': 'list'}
2 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
3 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
4 {'data': [], 'object': 'list'}
5 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
6 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
7 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}
8 {'data': [], 'object': 'list'}
9 {'data': [{'id': 'model1', 'object': 'model', 'created': 1677610602, 'owned_by': 'openai'}], 'object': 'list'}

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.71.1-stable.patch1

Twitter / LinkedIn details

No response

[kota-iizuka]

• Data inconsistency seems to occur only when the model list becomes empty after executing /model/delete. The results of /model/new model1, /model/new model2, /model/delete model1 were accurate (except for the behavior that asynchronous database updates take time).
• As a workaround, you can use docker compose restart litellm to reflect database updates.