RuntimeError when setting up self hosted model + langchain integration

[dcavadia]

Im having this bug when trying to setup a model within a lambda cloud running SelfHostedHuggingFaceLLM() after the rh.cluster() function.

`
from langchain.llms import SelfHostedPipeline, SelfHostedHuggingFaceLLM
from langchain import PromptTemplate, LLMChain
import runhouse as rh
gpu = rh.cluster(name="rh-a10", instance_type="A10:1").save()
template = """Question: {question}

Answer: Let's think step by step."""

prompt = PromptTemplate(template=template, input_variables=["question"])
llm = SelfHostedHuggingFaceLLM(model_id="gpt2", hardware=gpu, model_reqs=["pip:./", "transformers", "torch"])
`

I made sure with sky check that the lambda credentials are set, but the error i get within the log is this, which i havent been able to solve.

If i can get any help solving this i would appreciate it.

[dongreenberg]

Hi! Thanks for raising this. It looks like the GPU type you're specifying is "A10", which is not a valid GPU type. Can you try "A100:1"? To see all the GPU types available, you can run sky show-gpus or sky show-gpus --cloud lambda.

[dongreenberg]

Cc @concretevitamin - it looks to me like the accelerator validation for lambda is not catching properly?

[concretevitamin]

Hey, thanks for the report. This bug showed when Lambda console contained existing instances in addition to using SkyPilot to launch new instances.

This has been fixed in SkyPilot main branch.

[dongreenberg]

Ya, I spun up an A10 shortly after I wrote the above and realized it works and just wasn't in the catalogue 😄. Excellent, glad to hear it's fixed. @dcavadia I can help you get set up on the SkyPilot main branch if that's helpful, or use an existing Lambda instance you have up if you'd prefer to do that instead.

[concretevitamin]

@dongreenberg It's a quirk on our end: sky show-gpus --cloud lambda shows NVIDIA_GPU which really means an opinionated list of "common NVIDIA GPUs". If you pass --all / -a to the above, all supported GPUs in the catalog will be shown including A10, A6000, RTX6000, etc. Let us know if showing all GPUs by default or at least A10 is a good idea.

[dongreenberg]

Based on my intuition to run the show-gpus command to see if a particular variant exists in the catalogue, my bias would be to either show all by default or print a warning that this is only common GPUs, and to run -a to see the full list. Maybe a middle ground would be that if I just run sky show-gpus it shows only the common hardware variants so it's not a mess of cloud-specific hardware (with a warning about running -a), but if I run with --cloud it show the full catalogue for the given cloud.

[dcavadia]

Ya, I spun up an A10 shortly after I wrote the above and realized it works and just wasn't in the catalogue 😄. Excellent, glad to hear it's fixed. @dcavadia I can help you get set up on the SkyPilot main branch if that's helpful, or use an existing Lambda instance you have up if you'd prefer to do that instead.

Hi! im glad you guys find the issue thanks a lot. I just set up the SkyPilot main branch with pip install git+https://github.com/skypilot-org/skypilot and that solved the problem i had before... it now set up the instance in lambda i can launch it but i get a new error while still running the function, looks like a InactiveRpcError. Have any idea on this?

[dongreenberg]

Great! Glad this worked.

It's because your working directory (referenced in reqs by "./") is being detected as "ubuntu" (I assume your home directory), and the pip modifier in front of it is telling the grpc server to try to pip install it. Try changing "pip:./" to just "./" or "local:./" to avoid pip installing it. Sorry, the notebook you're using was inside the langchain directory (meaning langchain was the working directory) so it needed to be pip installed. You'll probably need to add "langchain" into the reqs too if you haven't already to make sure to install it on the server.

If for some reason you're getting an error about gRPC not finding methods, the gRPC server on your instance went down from this error. You can restart it by running gpu.restart_grpc_server().

[dcavadia]

Great! Glad this worked.

It's because your working directory (referenced in reqs by "./") is being detected as "ubuntu" (I assume your home directory), and the pip modifier in front of it is telling the grpc server to try to pip install it. Try changing "pip:./" to just "./" or "local:./" to avoid pip installing it. Sorry, the notebook you're using was inside the langchain directory (meaning langchain was the working directory) so it needed to be pip installed. You'll probably need to add "langchain" into the reqs too if you haven't already to make sure to install it on the server.

If for some reason you're getting an error about gRPC not finding methods, the gRPC server on your instance went down from this error. You can restart it by running gpu.restart_grpc_server().

Thanks for the quick reply. I created a new instance and set it all up again with the ./ and langchain parameter as SelfHostedHuggingFaceLLM(model_id="gpt2", hardware=gpu, model_reqs=["./", "transformers", "torch","langchain"]) and it seems that finally the setup has been successful.

I dont get an error about the gRPC but it get hang at the Running _generate_text via gRPC. Im not sure if its normal for the model integration to take >20mins with a A10 lambda instance.

[dongreenberg]

Great! But no, it shouldn't take nearly that long to download the model with a small model like gpt2. One way to see what's happening on the server is to call the RPC with stream_logs=True (though it's not integrated in a user-facing way into langchain). Can you halt that and try running the following:

llm.client(pipeline=llm.pipeline_ref, prompt="My prompt...", stream_logs=True)

If that doesn't work, there's a way to inspect the server logs directly that I can point you to. Thank you for bearing with us!

[dcavadia]

Yes that would be great if you can point me where to get the server logs directly. Thanks!

[dongreenberg]

If you ssh into the cluster (you can just run ssh rh-a10 from your command line) and then type screen -r, you can view the screen in which the server is running. Just be careful not the ctrl-C to exit or you'll kill the server (not a big deal, you can just restart it with the restart_grpc_server call I mentioned above). CMD-A-D exits screen without killing the server. Happy to live debug too, I'm free pretty much the rest of the day.

[dcavadia]

Great i now can look at the logs of the server, i noticed something about no available nodes can fulfill resource request.

And this is the ray status within the server instance.

[dongreenberg]

That would indeed cause the thread to hang. Confusing why Ray would be halting that when the resources are clearly available in ray status. Could you try running gpu.restart_grpc_server(restart_ray=True)?