how can Agenta be integrated with ollama LLM platform

[kmx2]

I need use Agenta completely on local server with a local LLM platform ollama, how can do that?

[dosubot[bot]]

Hey @kmx2! I'm here to help you out with any issues or questions you have. Let's see how we can integrate Agenta with the local LLM platform Ollama.

To integrate Agenta with the Ollama LLM platform and use it completely on a local server, follow these steps:

1. Install the Agenta Python SDK and CLI:

   pip install agenta

2. Set up the Agenta web platform:

   • Clone the Agenta repository and navigate to the folder:

     git clone https://github.com/Agenta-AI/agenta.git
     cd agenta

   • Launch the Agenta server using Docker Compose:

     docker compose -f "docker-compose.yml" up -d --build

   • Verify the installation by opening your browser and going to http://localhost. If you see the Agenta web interface, you're good to go.

3. Set up environment variables:

   • AGENTA_API_KEY for cloud users (not needed for local server).
   • AGENTA_HOST set to http://localhost if you are self-hosting.

4. Write a custom application using the Agenta SDK:

   import agenta as ag
   import requests
   import json
   
   default_prompt = "Please write a joke about {subject}"
   
   url = "http://localhost:5000/generate"  # Replace with your local Ollama LLM server URL
   ag.config.register_default(prompt=default_prompt, temperature=0.8)
   
   @ag.entrypoint
   def generate(subject: str) -> str:
       prompt = ag.config.prompt.format(subject=subject)
       data = {
           "prompt": prompt,
           "temperature": ag.config.temperature
       }
       response = requests.post(url, data=json.dumps(data))
       return response.json()

   This code sets up a simple LLM app that generates jokes about a given subject using the Ollama LLM platform hosted locally. The ag.config.register_default function sets the default values for the prompt and temperature. The @ag.entrypoint decorator marks the function that will be called to generate the joke.

5. Deploy the application using the Agenta CLI:

   • Ensure that the Ollama LLM platform is running on your local server.
   • Deploy the application by running the following commands in the code folder:

     agenta init
     agenta variant serve app.py

By following these steps, you can integrate Agenta with the Ollama LLM platform and use it entirely on a local server ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[mmabrouk]

Hey @kmx2 ,

This should be simple to do. You need to create a custom application that calls your local ollama endpoint. You can see an example on how to create such a custom application here: https://docs.agenta.ai/guides/tutorials/deploy-mistral-model
In this case we are calling Mistral in Huggingface, but you can do exactly the same by calling Ollama locally.

[kmx2]

Hey @mmabrouk ,
Thanks for your reply. but my question about Agenta integrated with ollama is use ollama in the playground of Agenta , that means in Agenta menu setting->LLM keys , add configure link to local ollama platform.

[mmabrouk]

That's a great idea. We have in the pipeline adding a "Model zoo" configuration to add custom models in the setting.

[snip2D]

Hello @mmabrouk , could you please share an estimated timeline for when the 'Model Zoo' feature will be available? This feature really does look very useful, and we're eager to start using it.

[mmabrouk]

Hello @snip2D, we expect to have it in February.

[zcasanova]

What's the status of this? I would love to see support for Ollama in the model playground! Ollama has quickly become one of the most popular LLM inference frameworks. Adding an Ollama provider would open up new possibilities for the many developers who run models locally.

[mmabrouk]

Hey @zcasanova , this feature is planned for the next release (scheduled for tuesday 11.3.)

[mmabrouk]

@zcasanova @snip2D @kmx2

Ollama, bedrock and azure and any custom openai API model are now supported in Agenta starting v0.37.0