Run with model running on colab

[briancunning]

I'm trying to get this to work with a model running on a colab as running models locally on my laptop is incredibly slow.

https://colab.research.google.com/github/camenduru/text-generation-webui-colab/blob/main/llama-2-7b-chat.ipynb

The above colab would need to be modified to create a local tunnel to expose the URL to model.

Something along the lines of

# Install LocalTunnel
!npm install -g localtunnel

# Start your local server here (Replace with your server command)
# Example: !python3 -m http.server 8080
%cd /content/text-generation-webui
!python server.py --share --settings /content/settings.yaml --model /content/text-generation-webui/models/Llama-2-7b-chat-hf

# Start LocalTunnel to expose the server
get_ipython().system_raw('lt --port 8080 > url.txt 2>&1 &')

# Retrieve the public URL
!cat url.txt

[rahulnyk]

Thanks for sharing
I am pretty new to Colab.

In my opinion, the slow part is the LLM. And I don't know if Colab will help with that. You need an Ollama container hosted on the cloud or a HuggingFace Inference API for the model. If you can get the model working with Colab, please share the knowhow.