How to use two models in same inference code

[siddhantwaghjale]

I'm trying with two model inference in the same code using vLLM. But while trying to load the 2nd model it fails with error
AssertionError: tensor model parallel group is already initialized.

Any help will be appreciated

[thiner]

I think it's not feasible with vllm currently(please correct me if I was wrong). But you can try to search "LLM gateway" in github.