Custom Embedding Model and LLM

[aniketmoha9]

How can I specify my local llm and local embedding model in any llamahub pack? I want use mistral or zephy as my llm model, and bge embedding model.

[dosubot[bot]]

🤖

Hi @aniketmoha9, great to see you again! I hope you're doing well. Let's take a look at your question.

To use your local LLM and local embedding model in any LlamaHub pack, you need to instantiate them and pass them to a ServiceContext. Here's how you can do it:

from llama_index import ServiceContext

# For local LLM
service_context_llm = ServiceContext.from_defaults(llm="local")

# For local embedding model
service_context_embedding = ServiceContext.from_defaults(embedding_model="local")

In the above code, "local" is a placeholder for your local LLM and embedding model. You need to replace it with the actual path or identifier of your local models.

However, the context provided does not specify how to use specific models like mistral, zephy, or bge. Therefore, the exact way to use these models in LlamaIndex is not clear from the given context.

For more specific instructions, you should refer to the documentation or usage guide of these models. If they are compatible with LlamaIndex, you should be able to use them in a similar way as shown in the code above.

For more information, you can refer to these similar issues:

• [Question]: llm and embed_model in service_context for index
• Querying Index using local models with no access to download from the hub(s)

If you need further assistance, please provide more details about the models you're trying to use and any error messages you're encountering.

Sources

• [Question]: llm and embed_model in service_context for index
• Querying Index using local models with no access to download from the hub(s)
• docs/understanding/using_llms/using_llms.md

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

The bot is nearly correct -- yea, you just set things in the service context, or pass in LLMs directly when needed

All LLMs supported
https://docs.llamaindex.ai/en/stable/module_guides/models/llms/modules.html

Some open-source LLM notebooks
https://docs.llamaindex.ai/en/stable/module_guides/models/llms.html#open-source-llms

All embeddings supported
https://docs.llamaindex.ai/en/stable/module_guides/models/embeddings.html#list-of-supported-embeddings