Is it possible to change the AI LLM.

[Zirgite]

I would like to change the AI LLM.
According to the manual these two models are known to work well:

https://huggingface.co/TheBloke/Llama-2-7B-chat-GGUF
https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF (recommended)

But how to switch between them? On huggingface I have the option to download less or more powerful version to adapt to my needs.

Also if I want to deploy a more powerful version how to do it. In the previous version it was easy just changing the .env file. But how that works in the present version.

[Zirgite]

I tried to work it on my own with my limited knowledge. So it is transformed now into Guide to Configuring and Deploying a Local Language Model. What I did test is the following. Changing the default mistral-7b-instruct-v0.1.Q4_K_M.gguf with the slightly more powerfull mistral-7b-instruct-v0.1.Q5_K_S.gguf

1. Back up and Clearing data and models
In order to do that I made a local copy of my working installation.
-I deleted the local files local_data/private_gpt (we do not delete .gitignore)
-I delete under /models the installed model
-I delete the embedding, by deleting the content of the folder /model/embedding (not necessary if we do not change them)

2. Model Configuration
Update the settings file to specify the correct model repository ID and file name.
This is contained in the settings.yaml file. The logic is the same as the .env change under the legacy privateGPT

local:
  llm_hf_repo_id: <Your-Model-Repo-ID>
  llm_hf_model_file: <Your-Model-File>
  embedding_hf_model_name: BAAI/bge-base-en-v1.5

In my case i made the following changes, not just the model but also the embeddings passing from small to the base model.
Passing from Q4 to Q5 is a great deal for me, as the difference between totally illogical results and spot on reasoning of the same model for my personal needs.

local:
  llm_hf_repo_id: TheBloke/Mistral-7B-Instruct-v0.1-GGUF
  llm_hf_model_file: mistral-7b-instruct-v0.1.Q5_K_S.gguf
  embedding_hf_model_name: BAAI/bge-base-en-v1.5

The possibilities are in this page: https://huggingface.co/BAAI/bge-small-en-v1.5

3.Running Setup Scripts
This is where it was all the difference. I ran into multitude of different errors the whole afternoon. The solution was to run all the install scripts all over again.

poetry install --with ui,local

poetry install --with local

poetry run python scripts/setup

Here the script will read the new model and new embeddings (if you choose to change them) and should download them for you into --> privateGPT/models

4. Environment Variables

Setting Local Profile: Set the environment variable to tell the application to use the local configuration.

$env:PGPT_PROFILES = "local"  # For Windows
export PGPT_PROFILES="local"  # For Unix/Linux

5. Launching the Application
Starting the Server: Use the make command to start the application.

make run
5.Accessing the UI: Open your web browser and navigate to the local server.

http://localhost:8001/

[NadaComa]

You only need to edit the settings.yaml

Go to hugging face, and select whatever model you want any copy the path here.
If it is a GGUF model, copy the specific file name as well.

I update the settings, download the appropriate file, store it in ./models and that is it

[Zirgite]

I did that in the beginning but I ran in all sorts of dependency issues. They disappeared when I run the install scripts all over again.