flags: response token, temperatur, p and k

[kalle07]

where i can add those parameters and how ?
(i know where i can chage the model in settings.yaml)
thx a lot !

[piotrmasior]

private_gpt\components\llm\llm_component.py
https://github.com/imartinez/privateGPT/blob/510caa576bd765b41b0e90b08df527b110c4b56d/private_gpt/components/llm/llm_component.py#L22

looking at ...\py3.11\Lib\site-packages\llama_index\llms\llama_utils.py
you will find definitions of something like:

messages_to_prompt
completion_to_prompt

use llama_utils.py as an example and define copy&crafted inside your private_gpt\components\llm\llm_component.py

[kalle07]

hey

themperature i have found,
also the file lama_utils.py, but i dont get your idea ^^
the other paramerts (p and k maybe some more) are not that simpel to change ?

[piotrmasior]

                self.llm = LlamaCPP(
                    model_path=str(models_path / settings.local.llm_hf_model_file),
                    temperature=0.1,
                    # llama2 has a context window of 4096 tokens,
                    # but we set it lower to allow for some wiggle room
                    context_window=3900,

                    #here kwargs to pass to __call__()
                    generate_kwargs={"top_k": 100, "top_p": 1},

                    # All to GPU
                    model_kwargs={"n_gpu_layers": -1},
                    # transform inputs into Llama2 format
                    messages_to_prompt=messages_to_prompt,
                    completion_to_prompt=completion_to_prompt,
                    verbose=True,
                )

stop privateGPT and run again, if you pass wrong param it will respond with error that invalid param was passed, see reference for other params here:
https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.llama.Llama.__call__

for messages templates check - it will give you idea what I meant about crafting own messages_to_prompt and completion_to_prompt https://github.com/run-llama/llama_index/blob/dbefde942434dde6d1d2a3eef87ed366028da07b/llama_index/llms/llama_utils.py#L4

[piotrmasior]

and for completion_to_prompt I think something is buggy on llama_cpp.py level

    @llm_chat_callback()
    def stream_chat(
        self, messages: Sequence[ChatMessage], **kwargs: Any
    ) -> ChatResponseGen:
        prompt = self.messages_to_prompt(messages)
        completion_response = self.stream_complete(prompt, formatted=True, **kwargs)
        return stream_completion_response_to_chat_response(completion_response)

and later there is:

    @llm_completion_callback()
    def stream_complete(self, prompt: str, **kwargs: Any) -> CompletionResponseGen:
        self.generate_kwargs.update({"stream": True})
        is_formatted = kwargs.pop("formatted", False)
        if not is_formatted:
            prompt = self.completion_to_prompt(prompt)

what results with prompt = self.completion_to_prompt(prompt) never being called as formatted=True is always set... and you cannot override it easily so hue hue hue

[piotrmasior]

I was capable to fix it by simply changing llama_cpp.py formatted=False

but I guess it is super not elegant solution, works for experiments, cheers

[kalle07]

thy

i added p an k but other stuff is tu much fo me :D

but the response token seems very low, never the less that only 3 models run fine with prtvateGPT^^
#1205

allways at max 1200 characters as response thats is not much ... maybe ou have any idea ?

and if you have time...
the different between using "querydoc"(if you have megred some documents) und "llm" is heavy ... so querydoc uses only cpu all cores, llm uses one core and full GPU ?

[hossein890]

Hi, is there any update here? can we go beyond lama.cpp limitation in this case 4096 tokens in answer prompt ?

[piotrmasior]

When I started to look into lama.cpp and show them that they added bugs after a new feature was added to support monsterAPI, I was banned for being rude. I'm sorry, but that was the worst experience I've ever had with an open source community in my 18-year career. Just don't give a damn... ask the people at lama.cpp why they care so much about words but not about code. I do not have time to repair whole world. I give some advice here, Taskeweaver, Orca2, multiple other options, let it sink.