Running in docker with custom model

[d1g33k]

My local installation on WSL2 stopped working all of a sudden yesterday. It was working fine and without any changes, it suddenly started throwing StopAsyncIteration exceptions.

Since setting everything up from scratch, including WSL2 made no difference, I realised my only option is to run it in Docker.

Being new to Docker, I installed Docker Desktop and was expecting it to be a seamless process using the provided Dockerfile. However, executing the Dockerfile directly results in missing model error.

So I tried to set it up using "docker compose up" instead, hoping it will pick up the model from the local folder but it doesn't. In addition, I get the error below.

private-gpt-1  | [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default', 'docker']
private-gpt-1  | Traceback (most recent call last):
private-gpt-1  |   File "<frozen runpy>", line 198, in _run_module_as_main
private-gpt-1  |   File "<frozen runpy>", line 88, in _run_code
private-gpt-1  |   File "/home/worker/app/private_gpt/__main__.py", line 5, in <module>
private-gpt-1  |     from private_gpt.main import app
private-gpt-1  |   File "/home/worker/app/private_gpt/main.py", line 3, in <module>
private-gpt-1  |     from private_gpt.di import global_injector
private-gpt-1  |   File "/home/worker/app/private_gpt/di.py", line 3, in <module>
private-gpt-1  |     from private_gpt.settings.settings import Settings, unsafe_typed_settings
private-gpt-1  |   File "/home/worker/app/private_gpt/settings/settings.py", line 324, in <module>
private-gpt-1  |     unsafe_typed_settings = Settings(**unsafe_settings)
private-gpt-1  |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/pydantic/main.py", line 164, in __init__
private-gpt-1  |     __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
private-gpt-1  | pydantic_core._pydantic_core.ValidationError: 2 validation errors for Settings
private-gpt-1  | llm.mode
private-gpt-1  |   Input should be 'llamacpp', 'openai', 'openailike', 'sagemaker', 'mock' or 'ollama' [type=literal_error, input_value='local', input_type=str]
private-gpt-1  |     For further information visit https://errors.pydantic.dev/2.5/v/literal_error
private-gpt-1  | embedding.mode
private-gpt-1  |   Input should be 'huggingface', 'openai', 'sagemaker', 'ollama' or 'mock' [type=literal_error, input_value='local', input_type=str]
private-gpt-1  |     For further information visit https://errors.pydantic.dev/2.5/v/literal_error
private-gpt-1 exited with code 1

Can anyone provide a simple guide as to how to get privateGPT working as a docker image?

[estkae]

same error in Windows
settings-vllm file
llm:
mode: openailike

embedding:
mode: local
ingest_mode: simple

local:
embedding_hf_model_name: BAAI/bge-small-en-v1.5

openai:
api_base: http://localhost:1234/v1 # LM Studio base URL
api_key: EMPTY
model: mixtral # this can be whatever you want

[hamvi-ai]

Same issue any guidance is appreciated

[estkae]

This is for Windows 11

conda create -n privategpt python=3.11
conda activate privategpt
pip install poetry
$env:PGPT_PROFILES='ollama' #or Dialog Windows Systemvariable
rm poetry.lock #will be new create
poetry install --no-root #Now it Take the file pyproject.toml
pip install build
poetry install --extras "ui llms-openai-like llms-ollama embeddings-ollama vector-stores-qdrant embeddings-huggingface"

Install Ollama on windows. After installation stop Ollama server
Ollama pull nomic-embed-text
Ollama pull mistral
Ollama serve

in Folder privateGPT and Env privategpt
make run

Here the file settings-ollama.yaml:
server:
env_name: ${APP_ENV:Ollama}

llm:
mode: ollama
max_new_tokens: 512
context_window: 3900
temperature: 0.1 #The temperature of the model. Increasing the temperature will make the model answer more creatively. A value of 0.1 would be more factual. (Default: 0.1)

embedding:
mode: ollama

ollama:
llm_model: mistral
embedding_model: nomic-embed-text
api_base: http://localhost:11434
tfs_z: 1.0 # Tail free sampling is used to reduce the impact of less probable tokens from the output. A higher value (e.g., 2.0) will reduce the impact more, while a value of 1.0 disables this setting.
top_k: 40 # Reduces the probability of generating nonsense. A higher value (e.g. 100) will give more diverse answers, while a lower value (e.g. 10) will be more conservative. (Default: 40)
top_p: 0.9 # Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text. (Default: 0.9)
repeat_last_n: 64 # Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx)
repeat_penalty: 1.2 # Sets how strongly to penalize repetitions. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. (Default: 1.1)

vectorstore:
database: qdrant

qdrant:
path: local_data/private_gpt/qdrant

This should similar run for Docker.
--wtth is replace --extra