[Question]: How do I get the exact and full prompt that was passed to the LLM?

[sia-cerebras]

Question Validation

• I have searched both the documentation and discord for an answer.

context

I'm not sure query_wrapper_prompt is actually being used, as I see some conflicting information from llama_debug.
So I would like to know exactly what prompt is passed to the LLM when query_engine.query(...) method is called.

Example:
I first call query_engine.query("My query")

Then I call llama_debug.get_llm_inputs_outputs() which returns:

[[
CBEvent(event_type=<CBEventType.LLM: 'llm'>, payload={
<EventPayload.PROMPT: 'formatted_prompt'>: 'Context information is below.\n---------------------\n[retrieved documents]---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: My query\nAnswer: ', 
<EventPayload.ADDITIONAL_KWARGS: 'additional_kwargs'>: {}, 
<EventPayload.SERIALIZED: 'serialized'>: {'model_name': 'StabilityAI/stablelm-tuned-alpha-3b', 'context_window': 4096, 'max_new_tokens': 512, 'system_prompt': '', 'query_wrapper_prompt': "[INST]<<SYS>>\nYou are an AI assistant that answers questions in a friendly manner, based on the given source documents. <</SYS>>\n\n{query_str}[/INST]", 'tokenizer_name': 'StabilityAI/stablelm-tuned-alpha-3b', 'device_map': 'auto', 'stopping_ids': [], 'tokenizer_outputs_to_remove': [], 'tokenizer_kwargs': {'max_length': 4096}, 'model_kwargs': {'torch_dtype': torch.float16, 'load_in_8bit': True}, 'generate_kwargs': {'temperature': 0.0, 'do_sample': False}, 'class_name': 'HuggingFace_LLM'}}, time='09/20/2023, 15:11:02.595113', id_='c97ad2ff-f873-4f24-a525-bf5690521fa8'),
  
CBEvent(event_type=<CBEventType.LLM: 'llm'>, payload={
<EventPayload.PROMPT: 'formatted_prompt'>: 'Context information is below.\n---------------------\n[retrieved documents]---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: My query\nAnswer: ', 
<EventPayload.COMPLETION: 'completion'>: CompletionResponse(text='<<SYS>>\n\n[LLM response]', additional_kwargs={}, raw={'model_output': tensor([[    1,   518, ...,  287, 24413]], device='cuda:0')}, delta=None)}, time='09/20/2023, 15:11:21.680252', id_='c97ad2ff-f873-4f24-a525-bf5690521fa8')
]]

You can see that I have a query_wrapper_prompt that says [INST]<<SYS>>\nYou are an AI assistant that answers questions in a friendly manner, based on the given source documents. <</SYS>>\n\n{query_str}[/INST]"

While the 'formatted_prompt' is 'Context information is below.\n---------------------\n[retrieved documents]---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: My query\nAnswer: '

So it looks like different template is being used.

Question:

Is 'formatted_prompt' the prompt that is passed to the LLM?
If yes, why does it not use my query_wrapper_prompt?
If no, then what is the point of query_wrapper_prompt? where does it get used?

[dosubot[bot]]

🤖

Hello,

Based on the context you've provided, the 'formatted_prompt' is indeed the exact prompt that is passed to the LLM. It is created by using the format or format_messages method on a PromptTemplate or ChatPromptTemplate instance.

As for the 'query_wrapper_prompt', it's not explicitly mentioned in the context you've provided, so I can't provide a detailed explanation of its purpose. However, in general, prompts in the LlamaIndex framework are used to transform a query into another query given the existing context. They can also add instructions for formatting image output.

If 'query_wrapper_prompt' is a part of the LlamaIndex framework, it might be used in a different part of the process not covered in the provided context. For a complete understanding, I would recommend referring to the relevant documentation or code where 'query_wrapper_prompt' is used or defined.

If you could provide more information about where 'query_wrapper_prompt' is used in your code or its intended purpose, I might be able to provide a more detailed answer.

Best,
Dosu

Sources

• docs/core_modules/model_modules/prompts.md
• llama_index/indices/query/query_transform/prompts.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[haf-tech]

@dosu-bot

I would jump in here, because have a similar issue - or lack of understanding.

from llama_index.llms.llama_utils import (messages_to_prompt, completion_to_prompt)
from llama_index.llms import LlamaCPP

model_url = "https://huggingface.co/TheBloke/Llama-2-13B-chat-GGUF/resolve/main/llama-2-13b-chat.Q4_0.gguf"
model_local_path = os.getcwd() + '/work/llamaindex/models/llama-2-13b-chat.Q4_0.gguf'


llm = LlamaCPP(
    # You can pass in the URL to a GGML model to download it automatically
    model_url=model_url,
    # optionally, you can set the path to a pre-downloaded model instead of model_url
    model_path=model_local_path,
    temperature=0.1,
    max_new_tokens=256,
    # llama2 has a context window of 4096 tokens, but we set it lower to allow for some wiggle room
    context_window=3900,
    # kwargs to pass to __call__()
    generate_kwargs={},
    # kwargs to pass to __init__()
    # set to at least 1 to use GPU
    model_kwargs={"n_gpu_layers": 1},
    # transform inputs into Llama2 format
    messages_to_prompt=messages_to_prompt,
    completion_to_prompt=completion_to_prompt,
    verbose=True,
)

So defining LlampCPP with setting messages_to_prompt / completion_to_prompt from the llama_utils.
So my expectation is, that the Llama prompt template with the right instructions is used

[INST]
<<SYS>>
....
<<SYS>>
...

[/INST]

But in the debug output I see

Context information is below.\n---------------------\nfile_name: redacted.docs ....

So, is the expectation wrong, that the llama specific prompt template is used?
Is something missing?

Thanks in advance

[bradenkwebb]

Did either of you resolve this? I've got similar questions