How does Theia AI establish communication with Open AI models when making function calls?

[sudhe-er]

Hi @JonasHelming,

Coder agent has a set of tools in Theia AI which simply does the operations like fetching working Directory structure, fetching files in a specific directory, creating a new file or overriding an existing file and so on.
So if we config a open ai model at the language model, how does that model is communicating with the Theia AI framework as one tool output should be sent back to the OpenAI models in order to invoke remaining set of tools using the concept of Function Calling. Have you guys implemented web sockets for this specific LLM provider as the communication should be back-and-forth manner.

And also we've a similar use case where 7 tools must be invoked in sequence in order to perform an operation, let's say saving a file in a specific folder in Theia AI. Here, each tool output should be sent back to my LLM, based on which remaining set of tools will be invoked. For this purpose, we implemented websockets in a backend API, however we came to know that Theia AI is not supporting Web Sockets.

I've 2 questions at last:

1. Have you guys implemented Websockets for OpenAI models?
2. Does the above use case need websockets or that can be achieved using HTTP based API calls?
   Thanks you in advance!

[sdirix]

Hi @sudhe-er,

I'm not sure I can follow you. We use the official OpenAI SDK to integrate with OpenAI. Specifically we are using the runTools convenience feature. There we can hand over tools as functions and the OpenAI SDK makes sure to call them when the LLM indicates to.

Internally this just calls the tool, places the result in a new message and sends a new request to OpenAI. All of these are just regular HTTP calls.

Internally we are using Theia RCP to bridge the function calls from the backend to the frontend, but this is only an implementation detail of Theia, unrelated to the OpenAI SDK.

AFAIK websockets are used for realtime use cases like voice which we do not support (yet) in Theia.

---

I don't understand where the question is coming from. You can simply implement the tools and register them in Theia and they will be available for the LLM. You can then instruct the LLM on how to use the tools and in which order.

[sudhe-er]

Thank you @sdirix for clarifying my doubt.

However, we've observed that Theia AI is not yet supporting AWS bedrock models, as we're using those models for the Agents we create in Their AI for some reasons like better reasoning capabilities and low latency function calls. We enabled AWS models via custom API from Theia Side, but we couldn't able to simulate the tool calls as like OpenAI models.

My main doubt is that if at all we want to use such AWS bedrock models for tool calls, do we also need to implement ai-aws kinda SDK into this Theia AI framework.
Please shed some light on this, it'd be very much helpful to us.

[sdirix]

I'm not familiar with AWS Bedrock APIs. If they offer an OpenAI compatible API, then you don't need any custom code in Theia. Instead you can reuse the OpenAI provider, see here.

If they require to call a custom AWS Api, then you need to implement an own AWS LanguageModel which serves as the bridge between Theia AI and AWS Bedrock. From Theia AI side you already get all the messages, tools etc. handed over to your LanguageModel implementation. You then need to map this to the provider specific API, in this case AWS bedrock.

All the LLM Apis are slightly different, so it depends on the complexity of the API on how much effort it is. See for example here for the OpenAI integration and here for the Anthropic integration.

Conceptually they are usually very similar: The LLM indicates it wants to call a tool via a message. You execute the tool and send a new request with the tool result appended as a different message.

Your API can be based on any technology, web sockets, https, web rtc, etc. etc.. For Theia AI this is not important. All you have to do is to implement that one "request" method of the LanguageModel interface.