Published on

⚖️ ClusterIP vs LoadBalancer: Load Balancing Between Pods on K3s

Authors

⚖️ ClusterIP vs LoadBalancer: Load Balancing Between Pods on K3s

You scaled your API to three replicas. Who decides which replica answers a request, and who is allowed to send that request in the first place? In Kubernetes the answer is the Service, and the two types you will meet first are ClusterIP and LoadBalancer.

This post explains both, compares them, and then measures them. The experiment deploys a small FastAPI service with three Pods three times: with no Service at all, behind a ClusterIP and behind a LoadBalancer. Each one is called from a Pod, from the node itself and from a machine outside the node, by IP and by DNS name. Every Pod spec matches the FastAPI service I run in production on a single-node K3s cluster.

1. Kubernetes in two minutes

Kubernetes is a control loop for containers. You declare the state you want ("three copies of this image, each with 64 MiB of memory, restarted when its health check fails") and controllers keep changing the cluster until it matches.

The objects that matter for this post:

ObjectWhat it is
NodeA machine (VM or bare metal) that runs containers. My production cluster has exactly one.
PodThe smallest deployable unit: one or more containers sharing an IP address.
DeploymentKeeps N identical Pods running and replaces them during a rollout.
ServiceA stable name and address in front of a changing set of Pods.
IngressHTTP routing rules (host and path) that an Ingress controller such as Traefik applies.

The key fact is that Pods are disposable. Each one gets an IP from the Pod network (10.42.0.0/16 on K3s). When a Pod is deleted, rescheduled or replaced during a rollout, the new one gets a different IP. Three replicas therefore leave you with three questions:

  1. Discovery: how does a client find the Pods when their IPs keep changing?
  2. Load balancing: how are requests spread across them?
  3. Exposure: who can reach them: other Pods only, or the internet too?

A Service answers all three. The Service type mostly decides the third.

The types build on each other:

ClusterIP     a virtual IP + DNS name, reachable inside the cluster
  └─ NodePort      + a port (30000-32767) opened on every node
       └─ LoadBalancer  + an external address that forwards to that NodePort

Every LoadBalancer is also a NodePort and a ClusterIP. Keep that in mind; the experiment shows it.

2. What is ClusterIP?

ClusterIP is the default Service type. Kubernetes allocates a virtual IP from the Service range (10.43.0.0/16 on K3s) and creates a DNS record for it in CoreDNS:

<service>.<namespace>.svc.cluster.local  ->  10.43.242.43

Nothing listens on that IP. It is a virtual IP that kube-proxy matches in packet-rewriting rules installed on every node:

  1. The EndpointSlice controller watches Pods that match the Service's selector and are Ready, and writes their IPs into EndpointSlices.
  2. kube-proxy watches EndpointSlices and programs iptables (or IPVS/nftables) rules on every node.
  3. When a packet to 10.43.242.43:8000 leaves a Pod, the kernel picks one backend and rewrites the destination (DNAT) to that Pod's IP.

This is the rule my lab cluster generated for a three-Pod Service:

-A KUBE-SVC-R4EY... --comment "lab-clusterip/whoami:http -> 10.42.0.122:8000" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-5WOL...
-A KUBE-SVC-R4EY... --comment "lab-clusterip/whoami:http -> 10.42.0.126:8000" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-CK7Y...
                                                                               (the last endpoint takes whatever is left)

That is the whole load balancer: a random choice per new TCP connection with probabilities 1/3, then 1/2 of the rest, then everything that remains. Each Pod gets one third. Three properties follow:

  • It is layer 4. It balances connections, not HTTP requests, and never reads the payload.
  • It is cluster-internal. Pods and nodes have the rules; the rest of the world cannot route to 10.43.x.x.
  • It follows readiness. A Pod that fails its readiness probe leaves the EndpointSlice and stops receiving new connections.

A minimal ClusterIP Service, the shape used by every application in my GitOps repository:

apiVersion: v1
kind: Service
metadata:
  name: monopoly-api
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: monopoly-api
  ports:
    - name: http
      port: 8000
      targetPort: http

3. What is LoadBalancer?

type: LoadBalancer asks the infrastructure for an external address. Kubernetes does three things:

  1. It allocates a ClusterIP, exactly as above.
  2. It opens a NodePort on every node.
  3. It asks a load-balancer controller to provision something outside the cluster that forwards to the nodes, and writes that address into status.loadBalancer.ingress, shown as EXTERNAL-IP.

Kubernetes itself does not provide step 3. Who does depends on where the cluster runs:

EnvironmentWho implements type: LoadBalancerExternal address
AWS / GCP / Azurecloud controller managera cloud NLB/ALB with its own IP or hostname
Bare metalMetalLB, kube-vip, Cilium LB-IPAMan IP from a pool you assign
K3sServiceLB (formerly Klipper LB), built inthe node's own IP
No controller at allnobody<pending> forever

K3s's ServiceLB is simple, which makes it easy to inspect. For each LoadBalancer Service it creates a DaemonSet in kube-system. Each Pod declares a hostPort for every Service port and forwards what arrives there to the ClusterIP:

$ kubectl -n kube-system get pod -l svccontroller.k3s.cattle.io/svcname=whoami -o jsonpath=...
lb-tcp-8080 hostPort=8080 SRC_RANGES=0.0.0.0/0 DEST_PROTO=TCP DEST_PORT=8080 DEST_IPS=10.43.70.67

The external path is therefore node-ip:8080 → svclb Pod → ClusterIP → kube-proxy → one of the Pods. The ClusterIP's kube-proxy rules still do the balancing. LoadBalancer adds a way in.

Two fields only matter for LoadBalancer and NodePort:

  • externalTrafficPolicy: Cluster (default): any node accepts the traffic and may forward it to a Pod on another node. The source address is SNATed, so the Pod sees a node IP instead of the client's.
  • externalTrafficPolicy: Local: only nodes that run a ready Pod accept traffic, and the client IP is preserved. Balancing then happens per node instead of per Pod.

4. ClusterIP vs LoadBalancer

ClusterIPLoadBalancer
Reachable fromPods and nodes in the clusterthe cluster and outside it
Stable addressvirtual IP 10.43.x.x + DNS namethe same, plus an external IP/hostname
Also creates—a ClusterIP and a NodePort
Balancing algorithmkube-proxy: random per connection (iptables)the external LB to the nodes, then kube-proxy to the Pods
OSI layerL4 (TCP/UDP)L4 (TCP/UDP); some clouds offer L7 through annotations
Needs a controllernoyes (cloud CCM, MetalLB, ServiceLB...)
Costfreeon a cloud, one billed load balancer per Service
Client source IPthe calling Pod's IPSNATed unless externalTrafficPolicy: Local
TLS, host/path routingnono (L4); that is the Ingress controller's job
Ports on K3snone on the nodethe Service port becomes a hostPort on every node

The most important row is the first one. Both types balance across Pods the same way, because a LoadBalancer is a ClusterIP with an entrance added. You choose between them based on who must reach the service, not on how well they balance.

5. When to use each

Use ClusterIP for

  • Anything only other workloads call: databases, Redis, internal APIs, workers, gRPC backends.
  • Every HTTP service published through an Ingress. The Ingress controller is the single entrance, and each app stays a ClusterIP behind it.
  • Most Services. If you are unsure, ClusterIP is the safe default because it exposes nothing.

Use LoadBalancer for

  • The Ingress controller itself: Traefik, NGINX, Envoy Gateway. One LoadBalancer for the whole cluster.
  • Non-HTTP protocols an Ingress cannot route: raw TCP/UDP game servers, MQTT, SMTP, a database you must reach from outside, DNS.
  • Services that need their own IP or port, or a cloud L4 load balancer's features (static IPs, cross-zone, proxy protocol).
  • A quick external test of a service without writing Ingress rules.

The common pattern is to combine them: one LoadBalancer for the Ingress controller and ClusterIPs for everything else. My production cluster works this way. It runs eleven applications and has exactly one LoadBalancer Service, kube-system/traefik on 80/443. Every app is a ClusterIP behind a Traefik Ingress. On a cloud, giving every app its own LoadBalancer would mean paying for a dozen load balancers. On K3s it would open a dozen ports on the node's public IP.

Neither of them is the answer when clients need to address individual Pods (database replicas, Kafka brokers). For that you want a headless Service (clusterIP: None), which returns every Pod IP in DNS and does no balancing.

6. The experiment

6.1 Goal and setup

Deploy one FastAPI app with 3 Pods in three configurations and call each from three places:

Vantage pointRepresents
Podanother microservice inside the cluster (a curlimages/curl Pod in namespace lab-client)
Nodea shell on the VPS itself, host network, outside every Pod
Externalmy laptop, reaching the node by its public IP or a DNS name, like any internet client

Everything ran on my production cluster: a single-node K3s v1.36.3+k3s1 on an 8 GB Hetzner VPS in Helsinki (ubuntu-8gb-hel1-1), with its bundled Traefik 3.7.8 and ServiceLB. The external client was my laptop in Vietnam. The node's public IP appears below as <node-public-ip>.

The full source of every file is in Appendix B. The layout:

experiments/service-lb-lab/
├── app/                      FastAPI source + Dockerfile (whoami-api)
├── chart/                    Helm chart + values-{none,clusterip,loadbalancer}.yaml
└── scripts/run-experiment.sh calls every scenario from a Pod, the node and outside

6.2 The API

Each response names the Pod that served it, so counting responses shows how requests were spread. /api/fanout turns a Pod into a client so we can compare one TCP connection per request against a single keep-alive connection.

# app/main.py (abridged)
POD_NAME = os.getenv("POD_NAME", socket.gethostname())

@app.get("/api/health")
def health():
    return {"status": "ok"}

@app.get("/api/whoami")
def whoami(request: Request):
    return {
        "pod": POD_NAME,
        "pod_ip": POD_IP,
        "node": NODE_NAME,
        "service_mode": SERVICE_MODE,
        "client_ip": request.client.host,          # what the Pod really saw
        "x_forwarded_for": request.headers.get("x-forwarded-for"),
        "host_header": request.headers.get("host"),
        "served_by_this_pod": next(_served),
    }

@app.get("/api/fanout")
def fanout(url: str, n: int = 30, keepalive: bool = False):
    ...  # call `url` n times with httpx, return {"by_pod": {...}}

The image matches the production FastAPI service: python:3.13-slim, UID 10001, uvicorn with one worker on :8000.

6.3 The Helm chart

One chart with a single switch, serviceMode: none | clusterip | loadbalancer. The Deployment copies the production monopoly-api spec: named port http, /api/health startup, readiness and liveness probes, runAsNonRoot, read-only root filesystem with an emptyDir on /tmp, all capabilities dropped, RuntimeDefault seccomp, no service-account token, and the same 20m/64Mi → 500m/256Mi resources. The only deliberate change is replicas: 3. The Downward API passes in the Pod's name, IP and node:

# chart/templates/deployment.yaml (excerpt)
env:
  - name: POD_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - name: POD_IP
    valueFrom: { fieldRef: { fieldPath: status.podIP } }

The Service template renders nothing, a ClusterIP or a LoadBalancer:

# chart/templates/service.yaml
{{- $mode := lower .Values.serviceMode -}}
{{- if ne $mode "none" }}
apiVersion: v1
kind: Service
metadata:
  name: {{ include "whoami.name" . }}
spec:
  {{- if eq $mode "clusterip" }}
  type: ClusterIP
  {{- else }}
  type: LoadBalancer
  externalTrafficPolicy: {{ .Values.service.externalTrafficPolicy }}
  {{- with .Values.service.loadBalancerSourceRanges }}
  loadBalancerSourceRanges:
    {{- toYaml . | nindent 4 }}
  {{- end }}
  {{- end }}
  selector:
    {{- include "whoami.selectorLabels" . | nindent 4 }}
  ports:
    - name: http
      port: {{ if eq $mode "loadbalancer" }}{{ .Values.service.loadBalancerPort }}{{ else }}{{ .Values.service.port }}{{ end }}
      targetPort: http
{{- end }}

The LoadBalancer listens on 8080, not 8000, because ServiceLB turns the Service port into a hostPort on the node. Ports 80 and 443 already belong to Traefik. The ClusterIP release also enables an Ingress for whoami.lab.local, the same way production apps are published:

# chart/values-clusterip.yaml
serviceMode: clusterip
ingress:
  enabled: true
  host: whoami.lab.local

Install all three. On a public VPS, a LoadBalancer opens its port to the whole internet, so loadBalancerSourceRanges limits port 8080 to my laptop and the node. ServiceLB enforces it on the node.

cd experiments/service-lb-lab/chart
for m in none clusterip; do
  helm upgrade --install whoami . -n "lab-$m" --create-namespace -f "values-$m.yaml" --wait
done
helm upgrade --install whoami . -n lab-loadbalancer --create-namespace -f values-loadbalancer.yaml \
  --set 'service.loadBalancerSourceRanges={<my-ip>/32,<node-public-ip>/32}' --wait
NAMESPACE          NAME     TYPE           CLUSTER-IP      EXTERNAL-IP        PORT(S)
lab-clusterip      whoami   ClusterIP      10.43.11.229    <none>             8000/TCP
lab-loadbalancer   whoami   LoadBalancer   10.43.185.254   <node-public-ip>   8080:30579/TCP
kube-system        traefik  LoadBalancer   10.43.147.46    <node-public-ip>   80:30960/TCP,443:30616/TCP

EXTERNAL-IP is the node's own IP, as expected with ServiceLB, and Traefik is the only other LoadBalancer on the cluster. lab-none has no Service, only three Pods.

6.4 Run the matrix

NODE_IP=<node-public-ip> NODE_EXEC="ssh vps" ./scripts/run-experiment.sh 2>&1 | tee results.txt

Each row sends 300 requests, each on a new TCP connection unless stated otherwise. External DNS names are pinned to the node IP with curl --resolve, which works like an A record without touching a real zone.

6.5 Scenario 1: no Service

--- pod -> DNS name whoami.lab-none.svc.cluster.local
000  exit=6                                   # curl: could not resolve host
--- pod -> one Pod IP 10.42.0.44, 300 requests
    300 whoami-54799dfb49-n2ddv
--- node -> same Pod IP
    300 whoami-54799dfb49-n2ddv
--- external -> same Pod IP
000  exit=28                                  # timeout: 10.42.0.0/16 exists only on the node
--- replace that Pod, then call the old IP again
  whoami-54799dfb49-jltlz 10.42.0.56          # the replacement got a new IP
  whoami-54799dfb49-rqjh5 10.42.0.43
  whoami-54799dfb49-x2cgc 10.42.0.53
000  exit=28                                  # old IP: timeout

Without a Service there is no DNS name, no balancing (all 300 requests go to the one IP you picked), and no external access. Worst of all, the address breaks on the first Pod replacement. To balance, a client would have to list Pods through the Kubernetes API and pick one itself, which is what a Service already does.

6.6 Scenario 2: ClusterIP

--- pod -> nslookup whoami.lab-clusterip.svc.cluster.local
Name:    whoami.lab-clusterip.svc.cluster.local
Address: 10.43.11.229
--- pod -> http://whoami.lab-clusterip:8000, 300 requests
     99 whoami-84c49fc4bf-lkqvb
    100 whoami-84c49fc4bf-sj45h
    101 whoami-84c49fc4bf-vb2hl
--- node -> ClusterIP 10.43.11.229:8000
     95 whoami-84c49fc4bf-lkqvb
    107 whoami-84c49fc4bf-sj45h
     98 whoami-84c49fc4bf-vb2hl
--- node -> DNS name through the node's own resolver
Address: 10.43.11.229
--- node -> DNS name asked of CoreDNS (10.43.0.10) directly
Address: 10.43.11.229
--- external -> ClusterIP 10.43.11.229:8000
000  exit=28                                  # unreachable
  • From a Pod, the short name whoami.lab-clusterip resolves because the Pod's resolv.conf search list includes svc.cluster.local. Requests spread 99 / 100 / 101, about a third each. The choice is random, so a different run lands elsewhere: my first run on the same cluster gave 107 / 90 / 103.
  • From the node, the ClusterIP works because kube-proxy's rules run on the node too. The DNS name also resolves, and that is specific to my VPS. A node's resolver normally knows nothing about cluster.local; a plain K3s node I tested returned NXDOMAIN. My VPS has a systemd-resolved drop-in that routes ~cluster.local to CoreDNS (10.43.0.10), added so the host-level cloudflared connector can dial *.svc.cluster.local.
  • From outside, the ClusterIP is unreachable, as designed.

Keep-alive pins a client to one Pod. The same 300 requests from inside a Pod, once with a new connection per request and once over one reused connection:

{"keepalive": false, "by_pod": {"...lkqvb": 113, "...sj45h": 95, "...vb2hl": 92}}
{"keepalive": true,  "by_pod": {"...vb2hl": 300}}

kube-proxy chooses a backend once per connection. HTTP keep-alive, connection pools, gRPC and HTTP/2 all reuse connections, so one busy client can send everything to a single Pod while the other two sit idle. If you need balancing per request with long-lived connections, use an L7 proxy (Ingress, service mesh) or client-side balancing over a headless Service.

Getting a ClusterIP to the outside world through Ingress. The ClusterIP release has an Ingress for whoami.lab.local. Traefik is reached through its own LoadBalancer on :80:

--- external -> http://whoami.lab.local/api/whoami via Traefik (node IP :80)
    101 whoami-84c49fc4bf-lkqvb
     97 whoami-84c49fc4bf-sj45h
    102 whoami-84c49fc4bf-vb2hl
--- external -> same IP, unknown Host header
404

Traefik does not use the ClusterIP at all. It reads the EndpointSlices and balances across Pod IPs per HTTP request. That is the L7 balancing kube-proxy cannot do. The spread stays close to even (101 / 97 / 102 here, exactly 100 / 100 / 100 in my first run). The 404 for an unknown Host is a reminder that routing goes by the Host header: a DNS record that points at the node is not enough without a matching rule.

6.7 Scenario 3: LoadBalancer

--- pod -> http://whoami.lab-loadbalancer:8080
    107 whoami-5d878c65cb-4ncd2
    100 whoami-5d878c65cb-n8qk2
     93 whoami-5d878c65cb-vntsj
--- node -> <node-public-ip>:8080
     98 whoami-5d878c65cb-4ncd2
    104 whoami-5d878c65cb-n8qk2
     98 whoami-5d878c65cb-vntsj
--- external -> <node-public-ip>:8080        (ServiceLB hostPort)
                                             # no answer: dropped by loadBalancerSourceRanges
--- external -> <node-public-ip>:30579       (the NodePort it also created)
    116 whoami-5d878c65cb-4ncd2
     91 whoami-5d878c65cb-n8qk2
     91 whoami-5d878c65cb-vntsj
--- external -> http://lb.whoami.lab.local:8080 (A record -> node IP)
                                             # no answer: same filter

Inside the cluster and on the node, the LoadBalancer behaves like a ClusterIP. Every path ends in the same kube-proxy random choice, so the spreads look the same. A LoadBalancer does not balance better. It is reachable from more places.

The two empty external rows were the most useful result of the run. The node passed the source-range filter, but my laptop did not: the address the VPS saw for it was not the one I had allowed, most likely because my ISP sends traffic out through more than one NAT address. The same laptop still reached the NodePort, 300 out of 300. loadBalancerSourceRanges filters the LoadBalancer port only, and every LoadBalancer also opens a NodePort on every node. If you rely on source ranges, block the NodePort range (30000-32767) at the firewall too.

What the Pod sees as the client. Through Traefik:

{"pod": "...vb2hl", "client_ip": "10.42.0.27", "x_forwarded_for": "10.42.0.1", "host_header": "whoami.lab.local", ...}

client_ip is the Traefik Pod (10.42.0.27), and the X-Forwarded-For Traefik added is 10.42.0.1, the node's cni0 bridge, not my laptop. Traefik's own Service uses the default externalTrafficPolicy: Cluster, so the client address was rewritten by SNAT before Traefik ever saw it.

The LoadBalancer port was filtered for my laptop, so the comparison of the two policies comes from a local single-node K3s lab with the same chart. There, a caller at 172.30.0.1 was seen as:

# externalTrafficPolicy: Cluster (default)
{"pod": "...xsvqg", "client_ip": "10.42.0.1", ...}

# externalTrafficPolicy: Local
{"pod": "...crr4b", "client_ip": "172.30.0.1", ...}

With Cluster the real caller is replaced by the node's cni0 address because of SNAT. With Local the Pod sees the real address. On one node all three Pods are local, so Local still spread the load (101 / 103 / 96). On a multi-node cluster, nodes without a Pod would stop accepting traffic. Behind Cloudflare's proxy, the original client IP arrives in the CF-Connecting-IP header, so an app should read that header rather than the socket address.

6.8 Results at a glance

All numbers are from the VPS run, 300 requests per row.

ScenarioFrom a PodFrom the nodeFrom outside the node
No Service, DNS name❌ NXDOMAIN——
No Service, Pod IP✅ but 300 / 0 / 0, breaks on Pod replacement✅ 300 / 0 / 0❌ not routable
ClusterIP, DNS name✅ 99 / 100 / 101✅ only because my node routes cluster.local to CoreDNS❌
ClusterIP, virtual IP✅✅ 95 / 107 / 98❌
ClusterIP, one keep-alive connection⚠️ 300 / 0 / 0——
ClusterIP + Ingress (Traefik LB :80)——✅ 101 / 97 / 102, balanced per request
LoadBalancer, DNS name✅ 107 / 100 / 93——
LoadBalancer, node IP :8080—✅ 98 / 104 / 98🚫 filtered by loadBalancerSourceRanges
LoadBalancer, NodePort :30579——✅ 116 / 91 / 91, not filtered

7. Takeaways

  1. Without a Service you get no load balancing. You get Pod IPs that disappear on the next rollout.
  2. ClusterIP and LoadBalancer balance the same way: kube-proxy picks a random Pod for each new connection. LoadBalancer is a ClusterIP plus a NodePort plus an external entrance.
  3. Choose by audience. Use ClusterIP when only the cluster calls the service, and LoadBalancer when something outside must reach it directly over L4.
  4. Balancing is per connection, not per request. Keep-alive, gRPC and HTTP/2 can pin a client to one Pod. Use an L7 proxy if per-request balancing matters.
  5. For HTTP, use one LoadBalancer and many ClusterIPs. Put the Ingress controller behind the only LoadBalancer and get host routing, TLS and per-request balancing for every app behind it.
  6. On K3s, a LoadBalancer opens a port on the node's IP, and a NodePort next to it. loadBalancerSourceRanges only guards the first. On a public VPS that is a firewall decision as much as a Kubernetes one.
  7. Cluster DNS exists only inside Pods unless you route cluster.local to CoreDNS on the node, as my VPS does.

Acknowledgments

Thanks to Claude Opus 5.5 by Anthropic for help refining the writing of this post.

Appendix A: reproducing the lab without a VPS

Before touching the VPS, I validated the chart and script on K3s running in a privileged Docker container on a cgroup v1 host, which is also where the externalTrafficPolicy comparison in 6.7 comes from. Three adjustments were needed. They are specific to the lab and not needed on a real VPS:

docker network create --subnet 172.30.0.0/24 vpsnet
docker run -d --name vps-node --privileged --network vpsnet --ip 172.30.0.10 \
  rancher/k3s:v1.36.3-k3s1 server --tls-san 172.30.0.10 \
  --kubelet-arg=fail-cgroupv1=false \
  --kube-proxy-arg=conntrack-max-per-core=0
# and, if the sandbox forbids lowering oom_score_adj, add
#   restrict_oom_score_adj = true
# under [plugins.'io.containerd.cri.v1.runtime'] in
# /var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.tmpl and restart the container.

Clean up the experiment with:

for m in none clusterip loadbalancer; do helm uninstall whoami -n "lab-$m"; kubectl delete ns "lab-$m"; done
kubectl delete ns lab-client

Appendix B: full source

Every file of the experiment, ready to copy into this layout. Click a file name to expand it.

service-lb-lab/
├── app/
│   ├── main.py
│   ├── requirements.txt
│   └── Dockerfile
├── chart/
│   ├── Chart.yaml
│   ├── values.yaml
│   ├── values-none.yaml
│   ├── values-clusterip.yaml
│   ├── values-loadbalancer.yaml
│   └── templates/
│       ├── _helpers.tpl
│       ├── configmap.yaml
│       ├── deployment.yaml
│       ├── service.yaml
│       ├── ingress.yaml
│       └── NOTES.txt
└── scripts/
    └── run-experiment.sh
app/main.py
"""whoami-api: a tiny FastAPI service for comparing Kubernetes Service types.

Every response says which Pod answered, so a client can count how requests
were spread across replicas. `/api/fanout` lets a Pod act as an in-cluster
client: it calls a URL N times and returns the per-Pod tally.
"""

from __future__ import annotations

import os
import socket
from collections import Counter
from itertools import count

import httpx
from fastapi import FastAPI, Query, Request

POD_NAME = os.getenv("POD_NAME", socket.gethostname())
POD_IP = os.getenv("POD_IP", "unknown")
NODE_NAME = os.getenv("NODE_NAME", "unknown")
SERVICE_MODE = os.getenv("SERVICE_MODE", "unknown")

app = FastAPI(title="whoami-api")
_served = count(1)


@app.get("/api/health")
def health() -> dict[str, str]:
    return {"status": "ok"}


@app.get("/api/whoami")
def whoami(request: Request) -> dict[str, object]:
    return {
        "pod": POD_NAME,
        "pod_ip": POD_IP,
        "node": NODE_NAME,
        "service_mode": SERVICE_MODE,
        # The source address the Pod actually saw: shows kube-proxy/ServiceLB SNAT.
        "client_ip": request.client.host if request.client else None,
        "x_forwarded_for": request.headers.get("x-forwarded-for"),
        "host_header": request.headers.get("host"),
        "served_by_this_pod": next(_served),
    }


@app.get("/api/fanout")
def fanout(
    url: str = Query(..., description="Absolute URL of a /api/whoami endpoint"),
    n: int = Query(30, ge=1, le=1000),
    keepalive: bool = Query(False, description="Reuse one TCP connection for all calls"),
) -> dict[str, object]:
    """Call `url` n times from inside this Pod and count which Pod answered."""
    tally: Counter[str] = Counter()
    errors: Counter[str] = Counter()

    def one(client: httpx.Client) -> None:
        try:
            tally[client.get(url).json()["pod"]] += 1
        except Exception as exc:  # noqa: BLE001 - we want to report every failure kind
            errors[type(exc).__name__] += 1

    if keepalive:
        with httpx.Client(timeout=3.0) as client:
            for _ in range(n):
                one(client)
    else:
        # A fresh client per call means a fresh TCP connection, so kube-proxy picks again.
        for _ in range(n):
            with httpx.Client(timeout=3.0) as client:
                one(client)

    return {
        "caller": POD_NAME,
        "url": url,
        "requests": n,
        "keepalive": keepalive,
        "by_pod": dict(sorted(tally.items())),
        "errors": dict(errors),
    }
app/requirements.txt
fastapi==0.115.12
uvicorn==0.34.2
httpx==0.28.1
app/Dockerfile
FROM python:3.13-slim

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .

# Same runtime shape as monopoly-api: non-root UID 10001, uvicorn with one worker on :8000.
USER 10001:10001
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1", "--no-access-log"]
chart/Chart.yaml
apiVersion: v2
name: whoami-api
description: >-
  Three-replica FastAPI "whoami" service used to compare reaching Pods with no
  Service, a ClusterIP Service and a LoadBalancer Service (K3s ServiceLB).
type: application
version: 0.1.0
appVersion: "0.1.0"
chart/values.yaml
# Which Service sits in front of the Pods:
#   none          - no Service at all; clients must find and call Pod IPs themselves
#   clusterip     - type: ClusterIP; one stable virtual IP + DNS name, cluster-internal only
#   loadbalancer  - type: LoadBalancer; ClusterIP + NodePort + an external IP (ServiceLB on K3s)
serviceMode: clusterip

replicaCount: 3

image:
  repository: vanloc1808/whoami-api
  tag: "0.1.0"
  pullPolicy: IfNotPresent

containerPort: 8000

service:
  # In-cluster port, the same convention as monopoly-api (Service :8000 -> named port http).
  port: 8000
  # Only used when serviceMode=loadbalancer. ServiceLB publishes this port on every node IP,
  # so it must not collide with Traefik's 80/443.
  loadBalancerPort: 8080
  # Cluster: any node accepts traffic and kube-proxy may forward (and SNAT) it to any Pod.
  # Local: only nodes with a ready Pod answer, and the Pod sees the real client IP.
  externalTrafficPolicy: Cluster
  # Only used when serviceMode=loadbalancer: CIDRs allowed to reach the LoadBalancer port.
  # K3s ServiceLB enforces it on the node (SRC_RANGES), so the lab port need not be public,
  # e.g. --set 'service.loadBalancerSourceRanges={203.0.113.7/32}'. Empty = everyone.
  loadBalancerSourceRanges: []

# Optional: route a hostname to the Service through the cluster's Traefik Ingress controller.
ingress:
  enabled: false
  host: whoami.lab.local
  entrypoints: web

config:
  LOG_LEVEL: INFO

# Same budget as monopoly-api.
resources:
  requests:
    cpu: 20m
    memory: 64Mi
  limits:
    cpu: 500m
    memory: 256Mi
chart/values-none.yaml
serviceMode: none
chart/values-clusterip.yaml
serviceMode: clusterip
# The production pattern of this cluster: ClusterIP behind the shared Traefik Ingress.
ingress:
  enabled: true
  host: whoami.lab.local
chart/values-loadbalancer.yaml
serviceMode: loadbalancer
service:
  loadBalancerPort: 8080
  externalTrafficPolicy: Cluster
chart/templates/_helpers.tpl
{{- define "whoami.name" -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{- define "whoami.selectorLabels" -}}
app.kubernetes.io/name: {{ include "whoami.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}

{{- define "whoami.labels" -}}
{{ include "whoami.selectorLabels" . }}
app.kubernetes.io/component: api
app.kubernetes.io/part-of: service-lb-lab
app.kubernetes.io/managed-by: {{ .Release.Service }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
{{- end -}}
chart/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "whoami.name" . }}-config
  labels:
    {{- include "whoami.labels" . | nindent 4 }}
data:
  SERVICE_MODE: {{ .Values.serviceMode | quote }}
  {{- range $k, $v := .Values.config }}
  {{ $k }}: {{ $v | quote }}
  {{- end }}
chart/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "whoami.name" . }}
  labels:
    {{- include "whoami.labels" . | nindent 4 }}
spec:
  # Stateless, so unlike monopoly-api it can run several replicas and roll without downtime.
  replicas: {{ .Values.replicaCount }}
  revisionHistoryLimit: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      {{- include "whoami.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "whoami.labels" . | nindent 8 }}
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          # FastAPI (uvicorn, one worker) on :8000.
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: {{ .Values.containerPort }}
              protocol: TCP
          env:
            # Downward API: every response names the Pod, its IP and its node.
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          envFrom:
            - configMapRef:
                name: {{ include "whoami.name" . }}-config
          startupProbe:
            httpGet:
              path: /api/health
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /api/health
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /api/health
              port: http
            periodSeconds: 20
            timeoutSeconds: 3
            failureThreshold: 3
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: 16Mi
      terminationGracePeriodSeconds: 15
chart/templates/service.yaml
{{- $mode := lower .Values.serviceMode -}}
{{- if not (has $mode (list "none" "clusterip" "loadbalancer")) -}}
{{- fail (printf "serviceMode must be none, clusterip or loadbalancer (got %q)" .Values.serviceMode) -}}
{{- end -}}
{{- if ne $mode "none" }}
apiVersion: v1
kind: Service
metadata:
  name: {{ include "whoami.name" . }}
  labels:
    {{- include "whoami.labels" . | nindent 4 }}
spec:
  {{- if eq $mode "clusterip" }}
  type: ClusterIP
  {{- else }}
  type: LoadBalancer
  externalTrafficPolicy: {{ .Values.service.externalTrafficPolicy }}
  {{- with .Values.service.loadBalancerSourceRanges }}
  loadBalancerSourceRanges:
    {{- toYaml . | nindent 4 }}
  {{- end }}
  {{- end }}
  selector:
    {{- include "whoami.selectorLabels" . | nindent 4 }}
  ports:
    - name: http
      port: {{ if eq $mode "loadbalancer" }}{{ .Values.service.loadBalancerPort }}{{ else }}{{ .Values.service.port }}{{ end }}
      targetPort: http
      protocol: TCP
{{- end }}
chart/templates/ingress.yaml
{{- if .Values.ingress.enabled }}
{{- if eq (lower .Values.serviceMode) "none" }}
{{- fail "ingress.enabled needs a Service: set serviceMode to clusterip or loadbalancer" }}
{{- end }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "whoami.name" . }}
  labels:
    {{- include "whoami.labels" . | nindent 4 }}
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: {{ .Values.ingress.entrypoints }}
spec:
  rules:
    - host: {{ .Values.ingress.host }}
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: {{ include "whoami.name" . }}
                port:
                  name: http
{{- end }}
chart/templates/NOTES.txt
whoami-api: {{ .Values.replicaCount }} Pods, serviceMode={{ .Values.serviceMode }}

{{- if eq (lower .Values.serviceMode) "none" }}
No Service. List the Pod IPs and call one directly:
  kubectl -n {{ .Release.Namespace }} get pods -l app.kubernetes.io/instance={{ .Release.Name }} -o wide
{{- else }}
In-cluster DNS name:
  http://{{ include "whoami.name" . }}.{{ .Release.Namespace }}.svc.cluster.local:{{ if eq (lower .Values.serviceMode) "loadbalancer" }}{{ .Values.service.loadBalancerPort }}{{ else }}{{ .Values.service.port }}{{ end }}/api/whoami
{{- end }}
{{- if eq (lower .Values.serviceMode) "loadbalancer" }}
External address (EXTERNAL-IP is the node IP under K3s ServiceLB):
  kubectl -n {{ .Release.Namespace }} get svc {{ include "whoami.name" . }}
{{- end }}
scripts/run-experiment.sh
#!/usr/bin/env bash
# Calls whoami-api from three vantage points for each Service mode and counts
# which Pod answered:
#
#   pod       - a curl Pod inside the cluster (namespace lab-client)
#   node      - a shell on the node itself (host network, outside every Pod)
#   external  - this machine, outside the node, as an internet client would be
#
# Usage:
#   NODE_IP=<node ip> NODE_EXEC="ssh vps" ./run-experiment.sh
#   NODE_IP=172.30.0.10 NODE_EXEC="docker exec -i vps-node" ./run-experiment.sh   # local lab
#
# Requires the three releases from the README (namespaces lab-none,
# lab-clusterip, lab-loadbalancer) and kubectl pointed at the cluster.
set -uo pipefail

NODE_IP="${NODE_IP:?set NODE_IP to the node address clients use}"
NODE_EXEC="${NODE_EXEC:-ssh vps}"
N="${N:-300}"
LB_PORT="${LB_PORT:-8080}"
INGRESS_HOST="${INGRESS_HOST:-whoami.lab.local}"
CLIENT_NS=lab-client
CLIENT_IMAGE="${CLIENT_IMAGE:-curlimages/curl:8.17.0}"

hr() { printf '\n=== %s ===\n' "$*"; }

# Reads JSON lines on stdin, prints "count pod" per Pod plus failures.
tally() { grep -o '"pod":"[^"]*"' | cut -d'"' -f4 | sort | uniq -c; }

from_pod() { kubectl -n "$CLIENT_NS" exec lab-client -- sh -c "$1"; }
# The command goes over stdin so ssh (which joins its arguments into one string) cannot mangle the quoting.
from_node() { printf '%s\n' "$1" | $NODE_EXEC sh; }

# Loops run inside the target shell so that each request is a new process and a new TCP connection.
pod_loop() { from_pod "for i in \$(seq 1 $N); do curl -s -m 2 $* ; echo; done" | tally; }
node_loop() { from_node "for i in \$(seq 1 $N); do wget -q -T 2 -t 1 -O- $* ; echo; done" | tally; }
ext_loop() { for _ in $(seq 1 "$N"); do curl -s -m 2 "$@"; echo; done | tally; }

ensure_client() {
  kubectl get ns "$CLIENT_NS" >/dev/null 2>&1 || kubectl create ns "$CLIENT_NS" >/dev/null
  if ! kubectl -n "$CLIENT_NS" get pod lab-client >/dev/null 2>&1; then
    kubectl -n "$CLIENT_NS" run lab-client --image "$CLIENT_IMAGE" --restart=Never \
      --command -- sleep 86400 >/dev/null
  fi
  kubectl -n "$CLIENT_NS" wait --for=condition=Ready pod/lab-client --timeout=120s >/dev/null
}

ensure_client

hr "Inventory"
kubectl get pods -n lab-none -o wide --no-headers | awk '{print "lab-none         " $1, $6}'
kubectl get pods -n lab-clusterip -o wide --no-headers | awk '{print "lab-clusterip    " $1, $6}'
kubectl get pods -n lab-loadbalancer -o wide --no-headers | awk '{print "lab-loadbalancer " $1, $6}'
kubectl get svc -A -l app.kubernetes.io/part-of=service-lb-lab
kubectl -n kube-system get svc traefik

POD_IP_NONE=$(kubectl -n lab-none get pods -o jsonpath='{.items[0].status.podIP}')
CIP=$(kubectl -n lab-clusterip get svc whoami -o jsonpath='{.spec.clusterIP}')
NODEPORT=$(kubectl -n lab-loadbalancer get svc whoami -o jsonpath='{.spec.ports[0].nodePort}')

########################################################################
hr "1. No Service (lab-none)"
echo "--- pod -> DNS name whoami.lab-none.svc.cluster.local (expect NXDOMAIN)"
from_pod "curl -s -m 2 -o /dev/null -w '%{http_code}\n' http://whoami.lab-none.svc.cluster.local:8000/api/whoami; echo exit=\$?"
echo "--- pod -> one Pod IP $POD_IP_NONE, $N requests (always the same Pod)"
pod_loop "http://$POD_IP_NONE:8000/api/whoami"
echo "--- node -> same Pod IP (works: the node routes the Pod CIDR)"
node_loop "http://$POD_IP_NONE:8000/api/whoami"
echo "--- external -> same Pod IP (expect timeout: 10.42.0.0/16 is not routable off the node)"
curl -s -m 3 -o /dev/null -w '%{http_code}\n' "http://$POD_IP_NONE:8000/api/whoami"; echo "exit=$?"
echo "--- replace that Pod: its IP disappears and the replacement gets a new one"
kubectl -n lab-none delete pod -l app.kubernetes.io/instance=whoami --field-selector "status.podIP=$POD_IP_NONE" --wait=true >/dev/null
kubectl -n lab-none rollout status deploy/whoami --timeout=120s >/dev/null
kubectl -n lab-none get pods -o wide --no-headers | awk '{print "  " $1, $6}'
from_pod "curl -s -m 3 -o /dev/null -w '%{http_code}\n' http://$POD_IP_NONE:8000/api/whoami; echo exit=\$?"

########################################################################
hr "2. ClusterIP (lab-clusterip)"
echo "--- pod -> DNS lookup"
from_pod "nslookup whoami.lab-clusterip.svc.cluster.local 2>/dev/null | tail -n +4 || getent hosts whoami.lab-clusterip.svc.cluster.local"
echo "--- pod -> http://whoami.lab-clusterip:8000, $N requests, new connection each"
pod_loop "http://whoami.lab-clusterip:8000/api/whoami"
echo "--- a whoami Pod as client (/api/fanout): $N requests, new connection each vs ONE keep-alive connection"
SVC_URL="http://whoami.lab-clusterip:8000/api/whoami"
from_pod "curl -s 'http://whoami.lab-clusterip:8000/api/fanout?n=$N&keepalive=false&url=$SVC_URL'"; echo
from_pod "curl -s 'http://whoami.lab-clusterip:8000/api/fanout?n=$N&keepalive=true&url=$SVC_URL'"; echo
echo "--- node -> ClusterIP $CIP:8000 (kube-proxy rules exist on the node too)"
node_loop "http://$CIP:8000/api/whoami"
echo "--- node -> DNS name through the node's own resolver (NXDOMAIN unless it routes cluster.local to CoreDNS)"
from_node "nslookup whoami.lab-clusterip.svc.cluster.local 2>&1 | tail -n 2"
echo "--- node -> DNS name asked of CoreDNS (10.43.0.10) directly"
from_node "nslookup whoami.lab-clusterip.svc.cluster.local 10.43.0.10 2>&1 | tail -n 2"
echo "--- external -> ClusterIP $CIP:8000 (expect timeout)"
curl -s -m 3 -o /dev/null -w '%{http_code}\n' "http://$CIP:8000/api/whoami"; echo "exit=$?"
echo "--- external -> http://$INGRESS_HOST/api/whoami via Traefik (the shared LoadBalancer on :80)"
ext_loop --resolve "$INGRESS_HOST:80:$NODE_IP" "http://$INGRESS_HOST/api/whoami"
echo "--- external -> same IP, unknown Host header (expect Traefik 404)"
curl -s -m 3 -o /dev/null -w '%{http_code}\n' -H 'Host: nope.lab.local' "http://$NODE_IP/api/whoami"

########################################################################
hr "3. LoadBalancer (lab-loadbalancer)"
echo "--- pod -> http://whoami.lab-loadbalancer:$LB_PORT, $N requests"
pod_loop "http://whoami.lab-loadbalancer:$LB_PORT/api/whoami"
echo "--- node -> $NODE_IP:$LB_PORT"
node_loop "http://$NODE_IP:$LB_PORT/api/whoami"
echo "--- external -> $NODE_IP:$LB_PORT (ServiceLB hostPort), $N requests"
ext_loop "http://$NODE_IP:$LB_PORT/api/whoami"
echo "--- external -> NodePort $NODE_IP:$NODEPORT (a LoadBalancer is also a NodePort)"
ext_loop "http://$NODE_IP:$NODEPORT/api/whoami"
echo "--- external -> DNS name lb.$INGRESS_HOST -> $NODE_IP (A record), port $LB_PORT"
ext_loop --resolve "lb.$INGRESS_HOST:$LB_PORT:$NODE_IP" "http://lb.$INGRESS_HOST:$LB_PORT/api/whoami"
echo "--- external -> what the Pod sees as client_ip (externalTrafficPolicy=Cluster)"
curl -s -m 2 "http://$NODE_IP:$LB_PORT/api/whoami"; echo
echo "--- external -> client_ip through Traefik (X-Forwarded-For carries the original)"
curl -s -m 2 --resolve "$INGRESS_HOST:80:$NODE_IP" "http://$INGRESS_HOST/api/whoami"; echo

########################################################################
hr "4. LoadBalancer with externalTrafficPolicy=Local"
CHART_DIR="$(cd "$(dirname "$0")/../chart" && pwd)"
# --reuse-values keeps whatever else the release was installed with (e.g. loadBalancerSourceRanges).
helm upgrade whoami "$CHART_DIR" -n lab-loadbalancer --reuse-values \
  --set service.externalTrafficPolicy=Local --wait >/dev/null
sleep 5
echo "--- external -> $NODE_IP:$LB_PORT, client_ip is now the real caller"
curl -s -m 2 "http://$NODE_IP:$LB_PORT/api/whoami"; echo
echo "--- external -> $N requests (still spread: all 3 Pods are on this node)"
ext_loop "http://$NODE_IP:$LB_PORT/api/whoami"
helm upgrade whoami "$CHART_DIR" -n lab-loadbalancer --reuse-values \
  --set service.externalTrafficPolicy=Cluster --wait >/dev/null