Can DocumentSummaryIndex be used with local langchain llm?

[GeorgeDittmar]

Been trying to see if I can set up a DocumentSummaryIndex but I keep getting an error message saying I need to set the OpenAI API key but I am passing in a LangChainLLM to my service context. Does the DocumentSummaryIndex only work with OpenAI right now?

Code to show you what I have set up, its possible I have a bug I am just not seeing lol.

# default mode of building the index
# set up ChromaVectorStore and load in data

def setup_llm():

    llm = HuggingFaceTextGenInference(
        inference_server_url="http://localhost:8080",
        max_new_tokens=2048,
        top_k=10,
        top_p=0.95,
        typical_p=0.95,
        temperature=0.7,
        repetition_penalty=1.03,
        streaming=True
    )
    return llm
llm = setup_llm()
llm_ind = LangChainLLM(llm=llm)
# create client and a new collection
db = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = db.get_or_create_collection("resume_summary")
# define embedding function
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")

vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
service_context = ServiceContext.from_defaults(llm=llm_ind, embed_model=embed_model, chunk_size=1024)


response_synthesizer = get_response_synthesizer(
    response_mode="tree_summarize"
)
doc_summary_index = DocumentSummaryIndex.from_documents(
    documents,
    service_context=service_context,
    response_synthesizer=response_synthesizer,
    show_progress=True,
)

[dosubot[bot]]

🤖

Hello @GeorgeDittmar! I'm Dosu, a friendly bot here to assist you with your LlamaIndex queries or issues while we wait for a human maintainer to join us. I'm designed to help you solve bugs, answer questions, and guide you on how to become a contributor. Looking forward to our interaction!

Based on the information you've provided and the similar issues I found in the LlamaIndex repository, it seems like the error message about setting the OpenAI API key is likely because the OpenAI LLM is being used somewhere in your pipeline, possibly as a default setting.

To use a local LangChainLLM, you need to instantiate a ServiceContext with the local LLM. If you are still receiving an error about the OpenAI API key, it's possible that another part of your pipeline is still using the OpenAI LLM. You should ensure that all parts of your pipeline are correctly configured to use the local LLM.

If you are using the ServiceContext.from_defaults() method, it might be using OpenAI as the default LLM. In this case, you should explicitly pass your local LLM to the ServiceContext:

from llama_index.llms import LangChainLLM
from llama_index import ServiceContext

llm = LangChainLLM()  # replace with your local LLM initialization
service_context = ServiceContext(llm=llm)

Please replace LangChainLLM() with your local LLM initialization. This will ensure that your local LLM is used instead of the default OpenAI LLM.

Also, based on the issues I found, it seems that setting a global service context at the beginning of your code might help:

from llama_index import set_global_service_context

set_global_service_context(service_context)

This should prevent the function from creating a ServiceContext using defaults and thus avoid the OpenAI key missing error.

Here are the similar issues I found that might help you:

1. Properly Loading Summary Index from Storage with Open Source LLM in Application Context
2. When customize the LLM, using VectorStoreIndex, it reports error
3. RetrieverQueryEngine with CustomRetriever and LLM other than OpenAI

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• [Question]: Properly Loading Summary Index from Storage with Open Source LLM in Application Context
• [Question]: When customize the LLM, using VectorStoreIndex, it reports error
• [Question]: RetrieverQueryEngine with CustomRetriever and LLM other than OpenAI
• docs/understanding/using_llms/using_llms.md
• llama_index/service_context.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[GeorgeDittmar]

from llama_index import set_global_service_context

set_global_service_context(service_context)

seemed to solve it so I guess I need to figure out where in the pipeline its still looking for OpenAI.