Micro Services

[kavitharaju]

With the requirement to add new services for AI APIs in VE, it becomes necessary we have some changes in the architecture to enable support for extending the APP module wise and also deploying/developing them separately as needed.

Here is a Miro board where I have tried to capture various points we have discussed within different teams and meetings and also some new thoughts.

Consider this board and discussion as a place to continue and collaborate on the already started discussions.

[joelthe1]

This is looking good from a high-level @kavitharaju! I think we should do some more thinking around what our direction would be for APIs for fine-tuning models vs using it for inference-only. We might have some modifications (e.g. data flow, hardware requirements, etc.) based on that.

[kavitharaju]

Yes, we would need to do that. As of now, the design is only for inference. Whether the models are trained/fine-tuned ourselves or obtained from outside, we just keep the list of available models and let users use it.

But we can do more thinking on how to integrate the abilities to either add more models via some CI/CD processes happening outside of the API server or include provisions to do the fine-tunings and training via APIs. I think the former will be preferable. Something like this;
If there is an ML pipeline to train a model, say a TTS model handled by its own project, one of the final steps in it could be to POST that model to our API server, thus making it available for inference via APIs. This way the training can be handled outside. Even the hardware requirements could be dealt with separately in the project where the actual model building is happening.

[udkumar]

@kavitharaju
I am just adding some context for API gateways as it act as a crucial intermediary between clients and microservices in a microservices architecture, providing essential functions like routing & Load Balancing, security, Rate Limiting, monitoring, and policy enforcement.

I tried "KrakenD"

It is a flexible API gateway that can be used to implement a variety of features for your API. Uses a JSON configuration file to define your API endpoints and their associated middleware.

{
  "endpoints": [
    {
      "name": "users",
      "method": "GET",
      "path": "/users",
      "proxy": {
        "target": "http://localhost:8080/users"
      }
    }
  ]
}

Here Micro-service running with port "8080" and end-point id "/users" and KrakenD will run "krakend run --config krakend_conf.json" and make your API available at "8081" with endpoint "/users"

KrakenD also supports a number of other features, such as:

• Authentication and authorization: KrakenD can be used to implement authentication and authorization for your API endpoints.
• Rate limiting: KrakenD can be used to rate limit requests to your API endpoints.
• Transforming requests and responses: KrakenD can be used to transform requests and responses before they are sent to and from your backend services.
• Aggregating responses from multiple backend services: KrakenD can be used to aggregate responses from multiple backend services into a single response.

And many more we can find here