Kubernetes ClusterIP Resilience During Availability Zone Failures

Introduction #

I recently encountered a scenario where we needed to evaluate whether Kubernetes ClusterIP services provide sufficient resilience during availability zone failures. This analysis examines ClusterIP behavior during both node and AZ outages to determine if it meets our availability requirements.

Understanding ClusterIP: Creation and Traffic Flow #

To understand ClusterIP resilience during failures, we first need to grasp how ClusterIP services are created and how traffic flows through them in normal operations.

Service Creation Workflow #

When you create a ClusterIP Service¹, Kubernetes orchestrates a multi-step process involving several core components:

1. kube-apiserver² receives your kubectl apply request, validates it, and stores the Service object in etcd³ with an initially empty ClusterIP field.

2. kube-controller-manager⁴ watches for new Service objects and allocates a unique internal IP from the cluster’s service CIDR range, then updates the Service object with this assigned ClusterIP⁵.

3. kube-proxy⁶ running on every node detects the new Service and creates iptables rules that map the ClusterIP to the actual pod IPs behind the service.

Traffic Routing Mechanics #

Once established, traffic routing follows a straightforward path:

1. When a pod sends a request to a ClusterIP, it goes through the local node’s network stack
2. The kube-proxy on that node has pre-configured iptables rules that intercept traffic destined for ClusterIPs
3. These iptables rules perform load balancing and direct the traffic to one of the healthy backend pod IPs
4. The request reaches the target pod directly, bypassing any central load balancer

This distributed approach means each node independently handles routing decisions, which becomes crucial during failure scenarios.

Node Failure Recovery Process #

1. kube-apiserver stops receiving heartbeat signals from kubelet running on that node. The node is set to NotReady or Unknown state.
2. All pods running on this node are considered dead.
3. EndpointSlice Controller⁷ removes IP addresses of failed pods from the EndpointSlice object⁸ associated with the corresponding Service. It now contains only the IPs of healthy pods.
4. kube-proxy on all healthy nodes notice the changes to Service and EndpointSlice objects. Then it re-evaluates its local iptables rules for the affected Service and remove the failed pod IPs.

Availability Zone Failure Recovery Process #

1. The EKS control plane is distributed and replicated across multiple AZs, so the etcd cluster should have the latest state information.
2. The healthy k8s nodes in other AZs follow the process described in the previous section.
3. The AWS Load Balancer Controller will detect failure of all nodes in the AZ using its healthchecks and automatically stop routing traffic to them.
4. The EKS Cluster Autoscaler will notice the node group size shrink, and provision new nodes in the healthy AZs to replace the lost capacity (this can take time, as we know EC2 instance creation takes a while)
5. As new nodes are added, k8s will start provisioning new pods onto them to restore the original state.

Conclusion: Is ClusterIP resilient enough during AZ failures? #

Yes, ClusterIP services provide sufficient resilience for availability zone failures through Kubernetes’ built-in failover mechanisms and AWS EKS infrastructure.