Look into Scaling Service of Containers with Kubernetes

[cedricdcc]

We have a container named sembench that interacts with another container housing our graph database. Our aim is to initiate multiple instances of pysembench and connect them seamlessly to the graph database container.

Objectives:

• Scaling Containers: We need to understand how we can efficiently scale the pysembench containers while ensuring they interact smoothly with the graph database container.
• Resource Monitoring Interface: We require a user-friendly interface to monitor the health and resource usage of each pysembench instance and the graph database container.

Possible routes of investigation

Given the complexity of our requirements, Kubernetes appears to be a suitable choice for container orchestration. Kubernetes offers robust features for scaling applications and managing resources efficiently. However, further investigation is needed to determine the specific configurations and best practices for our scenario.

Points to investigate

• Kubernetes Deployment Strategy: Explore different deployment strategies (e.g., Deployment, StatefulSet) to understand which aligns best with our scaling requirements.

• Service Discovery and Load Balancing: Investigate how Kubernetes handles service discovery and load balancing to ensure seamless communication between pysembench instances and the graph database container.

• Monitoring and Logging: Look into Kubernetes-native monitoring and logging solutions (e.g., Prometheus, Fluentd) to implement the desired resource monitoring interface effectively.

• Scalability Testing: Conduct scalability testing to determine the optimal number of pysembench instances based on resource utilization and performance metrics.

Action items to search and discuss

• : Research Kubernetes documentation and community resources to gain insights into best practices for scaling applications with multiple interconnected containers.
• : Experiment with Kubernetes configurations in a controlled environment to understand their impact on scalability and resource management.
• : Define requirements for the resource monitoring interface and explore potential solutions within the Kubernetes ecosystem.

[cedricdcc]

@marc-portier anything you want to add to this issue already of insights?

[marc-portier]

not at all, mainly do not currently think we have an issue nor desire into this direction, might be useful though, just don't see it yet.