The Kubernetes network can be pretty complex if you’re not familiar with the different components that are part of it. One of the most common components present in many clusters is kube-proxy, which translates Kubernetes services into actual network rules within the nodes that compose a Kubernetes cluster.

In a nutshell, kube-proxy is an agent that runs in every node and monitors Service resources present in your cluster, translating them into actual iptables or ipvs rules. It usually runs as a daemonset but can also be deployed directly on your servers.

I will assume that you are somewhat familiar with Kubernetes networking, the different types of Service, and understand how cloud controllers interact with provider APIs.

For the sake of simplicity, I’ll focus on iptables-based routing in this article.

Kubernetes services

A LoadBalancer service is basically an L4 load balancer that uses DNAT to redirect inbound traffic to backend pods. I will not go through details about how services of this type are processed by the cloud controller manager along with any cloud provider particularities here.

Traffic redirection is controlled via iptables rules on a host level and kube-proxy controls the creation and removal of chains over various iptables tables that are also managed by kube-proxy.

Each node runs the kube-proxy service that watches the API server for the addition or removal of service and endpoint objects. When it sees a new service, it will open up a new (random-ish) port on each node for it. When a client connects to a service IP, this endpoint redirects traffic to its own port using node-level routing, selects a backend and proxies traffic from the client to the destination process.

Communication happens on a cluster level as the apiserver assigns virtual IPs for services and kube-proxy keeps track of backends across different nodes.

When processes need to know the real client IP behind a request, we usually rely on service.spec.externalTrafficPolicy: Local, which explicitly tells kube-proxy to proxy requests only to node-local endpoints and not to other nodes to avoid one more NAT operation.

If there are no active local endpoints, packets sent to the local node are dropped. For example, if you are using AWS and ELBs for your load balancer services, target groups will mark these nodes as unavailable.

Details about source IP usage can be found in:
https://kubernetes.io/docs/tutorials/services/source-ip/

Iptables chains

As mentioned before, multiple chains are responsible for filtering and NAT between pods and services when a Service or Endpoint object is created. Let’s take a quick look at some of the most used ones:

KUBE-SERVICES
This is the entry point for service packets. Matches the destination IP:port and dispatch the packet to the corresponding KUBE-SVC-* chain.

KUBE-SVC-*
Acts as a balancer, which distributes packets to a KUBE-SEP-* chain. The number of KUBE-SEP-* chains is equal to the number of endpoints behind the service. Which KUBE-SEP-* will be used is chosen randomly.

KUBE-SEP-*
Represents a service endpoint. DNAT traffic by replacing service IP:port with pod’s endpoint IP:Port.

KUBE-MARK-MASQ
Adds a netfilter mark to packets destined for the service, which originate outside the cluster’s network. Packets with this mark will be altered in a POSTROUTING rule to use source network address translation (SNAT) with the node’s IP address as their source IP address.

KUBE-MARK-DROP
Adds a netfilter mark to packets, which do not have destination NAT enabled by this point. These packets will be discarded in the KUBE-FW chain.

KUBE-FW-*
Acts when a service is deployed with the LoadBalancer type and matches the destination IP with a service’s load balancer IP, distributing packets to the corresponding KUBE-SVC-* chains (externalTrafficPolicy: Cluster) or KUBE-XLB-* chains (externalTrafficPolicy: Local).

KUBE-NODEPORTS
Used for NodePort and LoadBalancer services. External sources can access a service via node port. It matches the node port and distributes packets to the corresponding KUBE-SVC-* chains (externalTrafficPolicy: Cluster) or KUBE-XLB-* chains (externalTrafficPolicy: Local).

KUBE-XLB-*
This chain is used when externalTrafficPolicy is set to Local. Packets are dropped if a node has no relevant endpoints.

Workflow

This is an incomplete representation of the network flows handled by kube-proxy but can give you an idea of the most popular paths:

  flowchart TD
   A[Node Traffic from network] -->|net.PREROUTING| B(KUBE-SERVICES)
   B --> C{Match svc clusterIP?}
   C -->|Yes| D{Masquerade all?}
   D -->|Yes| J[KUBE-MARK-MASK]
   J --> K[KUBE-SVC-*]
   K --> L[KUBE-SEP-*]
   L --> M{Same src and dst IP?}
   M -->|Yes| N[KUBE-MARK-MASQ]
   N --> O([DNAT to endpoint])
   M -->|No| O([DNAT to endpoint])
   D -->|No| K[KUBE-SVC-*]
   C -->|No| E{Match externalIP?}
   E -->|Yes| F[KUBE-MARK-MASQ]
   F --> H([Route to network])
   E -->|No| G[KUBE-NODEPORTS]
   G --> I([Evaluate service and node ports])