Latency optimized node and traffic scheduling in Kubernetes

Containerized microservices have become the defacto standard for building modern applications providing, improved fault isolation, scalability and resilience. To ensure fault-tolerance and maintain good performance under fluctuating workloads, microservices should be scaled horizontally by deploying multiple replicas. High availability is achieved by placing these replicas across multiple servers. Kubernetes is a powerful orchestrator tool that manages the lifecycle of containerized applications, providing features for automated scaling, self-healing, service discovery and seamless deployment. The Horizontal Pod Autoscaler (HPA) is Kubernetes default tool for scaling the containterized microservices, known as Pods. Pods often communicate with each other in applications. During scaling, they should ideally be scheduled based on their communication frequency to minimize cross-Node communication latency. Reducing cross-Node communication can significantly improve response time and minimize cloud costs when the Kubernetes is deployed on a cloud provider. However, HPA relies on the default Kubernetes Scheduler to assign replica Pods to Nodes, which can sometimes lead to sub-optimal placement of replicas across Nodes, because it does not rely on Pods communication patterns (i.e., inter or intra node or inter or intra cluster communication). Aside from scaling, Kubernetes by default relies on iptables to route traffic to appropriate Pods. However, iptables uses a randomized algorithm and does not consider Node-to-Node latency, Pods' resource usage or the number of available replica Pods, resulting in overloaded Pods and long delays when requests are sent to remote Nodes. To address the problem of Kubernetes replica Pod placement and load-balaning, we introduced StornX. StornX is a Kubernetes Pod autoscaler and traffic scheduler that optimal spreads the Pod replicas within a cluster across Availability Zones (AZ), optimizes efficiency by co-locating additional replicas, and dynamically manages traffic distribution using Istio Service Mesh locality weights by directing requests to co-located replicas while also rerouting traffic away from Pods that have reached their threshold limits toward replicas with available resources. StornX is tested on AWS cloud environment with two microservice-based applications. The experimental results reveal that StornX reduces the response time, resource usage, number of replica Pods and reduces the cost in stress and load test scenarios. Furthermore, Availability Zone failure experiments show that StornX recovers faster and with lower performance degradation by dynamically redistributing traffic among the remaining replicas.