Langchain Database Agent (mssql) with local LLM

[dinonovak]

Hi,

anyone has experience creating langchain setup with local LLM running on Mac M1 (using metal) ?
In your experience would would be the best model to use LLAMA, Vicuna, ... ?

[dinonovak]

I am using openhermes-2.5-mistral-7b.Q8_0.gguf

llm is defined as follows:
llm = LlamaCpp( model_path=model_path, temperature=0, n_gpu_layers=n_gpu_layers, n_batch=n_batch, n_ctx=n_ctx, callback_manager=callback_manager, verbose=True, )

agent is defined as follows:
`toolkit = SQLDatabaseToolkit(db=db, llm=llm)
toolkit.get_tools()

PREFIX = '''You are a SQL expert. You have access to a Microsoft SQL Server database.
Identify which tables can be used to answer the user's question and write and execute a SQL query accordingly.
'''

FORMAT_INSTRUCTIONS = """Please use the following format:

                 '''
                 Thought: 
                 Action: the action to take should be one of [{tool_names}]
                 Action Input: the input to the action
                 Observation: the result of the action
                 '''
                
                Provide the output in the Final Answer."""

SUFFIX = '''

Begin!

Previous conversation history:
{chat_history}

Instructions: {input}
{agent_scratchpad}
'''

agent_executor = create_sql_agent(
llm=llm,
toolkit=toolkit,
agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
verbose=True,
handle_parsing_errors=True,
agent_kwargs={
'prefix': PREFIX,
'format_instructions': FORMAT_INSTRUCTIONS,
'suffix': SUFFIX
}
)`

But I am usually getting "Could not parse LLM output: Action: sql_db_list_tables"
it looks like 'action_input' part is not created.

can you please help how to resolve it.

In very rare cases (as the same code is run multiple times), action input is created and chain executes successfully.

[dinonovak]

If I remove temperature setting from LLM I can see that sometimes it is varying in answer, so I get
Action: sql_db_list_tables Input: ""

which still does not confirm to what langchain is expecting