What should I do if I want to use a custom LLM

[mc112611]

If I don't use OpenAI's LLMs, but my own LLM and use the Transformers library for local inference. What should I do to achieve it.
I think it would be very convenient if Haystack could be highly compatible with the models in the huggingface hub.

[anakin87]

Hello, @mc112611!

In this section of the Haystack documentation, you can find information on using Hugging Face models.

Do you need further help?

[mc112611]

I'm using this LLM，
THUDM/chatglm2-6b

but the following error occurred

ValueError: Model THUDM/chatglm2-6b-32k-int4 is not supported - no matching invocation layer found. Currently supported invocation layers are: [<class 'haystack.nodes.prompt.invocation_layer.open_ai.OpenAIInvocationLayer'>, <class 'haystack.nodes.prompt.invocation_layer.azure_open_ai.AzureOpenAIInvocationLayer'>, <class 'haystack.nodes.prompt.invocation_layer.chatgpt.ChatGPTInvocationLayer'>, <class 'haystack.nodes.prompt.invocation_layer.azure_chatgpt.AzureChatGPTInvocationLayer'>, <class 'haystack.nodes.prompt.invocation_layer.anthropic_claude.AnthropicClaudeInvocationLayer'>, <class 'haystack.nodes.prompt.invocation_layer.cohere.CohereInvocationLayer'>, <class 'haystack.nodes.prompt.invocation_layer.hugging_face.HFLocalInvocationLayer'>, <class 'haystack.nodes.prompt.invocation_layer.hugging_face_inference.HFInferenceEndpointInvocationLayer'>, <class 'haystack.nodes.prompt.invocation_layer.sagemaker_base.SageMakerBaseInvocationLayer'>, <class 'haystack.nodes.prompt.invocation_layer.sagemaker_meta.SageMakerMetaInvocationLayer'>, <class 'haystack.nodes.prompt.invocation_layer.sagemaker_hf_infer.SageMakerHFInferenceInvocationLayer'>, <class 'haystack.nodes.prompt.invocation_layer.sagemaker_hf_text_gen.SageMakerHFTextGenerationInvocationLayer'>] You can implement and provide custom invocation layer for THUDM/chatglm2-6b-32k-int4 by subclassing PromptModelInvocationLayer.

This is Colab's notebook
Building_Chat_App.ipynb

I saw comments in the code. Haystack seems to only support models with text generation in the huggingface hub. Is there a way to make it compatible with most models?