Advanced Kubernetes
live from the paddock.

Stage 01 onboard telemetry

Application health and resource management

Ladies and gentlemen, the field leaves the garage for the first lap, and the question that opens EVERY broadcast is the simplest and the most brutal one: is the car in one piece? Is it on the pace? Before any elaborate strategy, it's telemetry that calls the shots. Buckle up, because Stage 01 is where we learn to READ the car.

Picture your e-commerce site on Black Friday. One of the API pods has deadlocked: the process is still standing, the container hasn't died, but it answers every incoming client with an HTTP 500. Kubernetes, on its own, only knows one thing: "the process is alive." It restarts nothing, and you bleed sales for hours until someone spots it on a graph. In the pod next door, a memory leak is quietly eating RAM until it blows up the whole node and takes the healthy neighbors down with it.

How it works

This is exactly what happened to Antonelli at Silverstone: started on pole (P1), crossed the line in P15. The car was alive (engine running, laps completed, race finished), but it wasn't on the pace. Process standing, 500 response. You need two different kinds of telemetry: one that screams "the car has stopped" and calls it into the garage to swap a part (liveness), and another that says "the car is still running but off the pace, pull it from the fight until it recovers" without calling it in for repair (readiness). And you need to define how much ERS energy and quota each component is entitled to, so no one steals the other's envelope (requests/limits).

The three probes are the telemetry the pit wall reads at the end of every sector. Liveness restarts the container after N failures (self-healing); readiness only pulls the pod out of the load balancer, without restarting it; startup protects slow-booting apps by suspending the other two while the app hasn't come up yet. Confusing liveness with readiness causes cascading restarts. It's the difference between calling the car into the garage (losing 22s) and just telling it to cede position and recover.

Requests and limits are the resource contract. requests is what the container reserves, the guaranteed floor the scheduler uses to decide which node the pod fits on (the guaranteed ERS energy per lap). limits is the ceiling it can't cross: blow past the memory limit and you get an OOMKill (the kernel kills the container); blow past the CPU limit and you get throttling (the kernel slows the process down to keep it inside its quota). Blowing the PU's thermal envelope burns the component; hitting the energy ceiling is the derate that reins in the power.

The combination of requests and limits produces the pod's QoS class: the sacrifice order for when the garage runs short on resources. Guaranteed (requests == limits on every container) is sacrificed last (the championship leader's car); Burstable (requests < limits) uses idle slack but yields under contention; BestEffort (no requests, no limits) is evicted first (the test car nobody protects).

One counterintuitive detail (a real case from Buffer): setting a CPU limit can actually make latency worse. The CFS quota algorithm can throttle a pod even at low average usage: small spikes burn through the 100ms quota and the process freezes until the next period. Buffer removed CPU limits from latency-sensitive services and saw p99 drop by 5 to 10x. In a multi-tenant setup you want limits for isolation; in a latency-sensitive service, sometimes less is more. And replicas in parallel multiply availability (A = MTBF / (MTBF + MTTR)): two pods at 99% reach roughly 99.99%, because the Service behaves like a logical OR: it only takes one Ready pod. The team with two cars: if one retires, the other still scores.

Key concepts in this stage

YAML and commands in practice

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector: { matchLabels: { app: api } }
  template:
    metadata: { labels: { app: api } }
    spec:
      containers:
        - name: api
          image: myregistry/api:1.0.0
          ports: [ { containerPort: 8080 } ]
          startupProbe:            # 1) protects a slow boot
            httpGet: { path: /healthz, port: 8080 }
            failureThreshold: 30   #    up to 30 x 10s = 5min to come up
            periodSeconds: 10      #    checks every 10s
          livenessProbe:           # 2) stopped? restart the container
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3    #    3 failures in a row -> restart
          readinessProbe:          # 3) ready? enters/leaves the load balancer
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 5
          resources:
            requests: { cpu: 100m, memory: 128Mi }  # guaranteed floor (scheduling)
            limits:   { memory: 256Mi }              # RAM ceiling; NO CPU limit
                                                     # (avoids CFS throttling)

Essential commands for this stage

Hands-on

Prove in practice that liveness restarts a stuck container, your first fast lap of the weekend:

1. Create a pod whose liveness probe points at a port that does NOT exist (port 9999). It will always fail. Use --restart=Always (with --restart=Never the kubelet won't restart it on a failed liveness probe).
2. Run kubectl get pod teste-liveness -w and watch the RESTARTS column climb every ~15s.
3. Confirm the reason: kubectl describe pod teste-liveness | grep -i liveness, and see "Liveness probe failed".
4. That's self-healing: the car automatically called into the garage because telemetry flagged a stall. Clean up with kubectl delete pod teste-liveness.

SECTOR 1 GREEN! We've crossed the first split and the car is in one piece, telemetry clean, resources under control. What a start! But now the track tightens: next up is the technical complex where we decide WHICH garage each car goes into. Strategy takes over. Hold on tight, Stage 02 is a pure precision corner.

Stage 02 the strategy complex

Advanced scheduling and node behavior

The track now demands a cool head. A fast car isn't enough: you need to put the RIGHT car in the RIGHT garage, and know who the team sacrifices when resources get tight. This is where the strategist wins or loses the Sunday. Radio's live with the pit wall!

You have a mixed cluster: some nodes have GPUs (expensive, for training models), most are ordinary. By default the scheduler drops a pod on any node it fits, so a random pod can occupy the GPU node and block the ML team, while a training job can land on an ordinary node with no GPU and never work. Worse: when a node runs out of memory, who does Kubernetes kill first? If it's the wrong pod, your critical service goes with it.

How it works

Think of the pit lane. There's the ordinary garage and the specialized one: the one with the thermal blankets for wet tyres, the GPU calibration rig. Taints are the sign that says "reserved garage, authorized cars only" (repels anyone without clearance). Affinity is the strategy call on the radio: "take that car to the GPU garage" (attracts). And eviction is the chief mechanic who, when the garage runs short on power, starts sacrificing the test car before touching the championship leader's car.

Taints and Tolerations repel. A taint is a mark on the node ("don't schedule pods here unless they tolerate X"); a toleration is the mark on the pod ("I can handle X, I can go there"). The effects: NoSchedule (won't schedule non-tolerant pods; ones already running stay), PreferNoSchedule (avoids it, but it's not absolute) and NoExecute (blocks new pods AND evicts non-tolerant pods already there; useful for draining a node for maintenance). Crucial point: a taint only repels, it doesn't attract. A pod that tolerates the GPU node can still end up on an ordinary one. To guarantee it only goes to the GPU, combine it with affinity.

Node Affinity is the positive counterpart ("I want to go to nodes with label X"): required (hard, no label means no scheduling) vs. preferred (soft, weighted, if none is found it schedules elsewhere anyway). Pod Affinity/Anti-Affinity, meanwhile, relates pods to each other: affinity pulls them together ("put the cache near the front-end"); anti-affinity keeps them apart ("never put two replicas of the database on the same node/zone"), never both cars in the same garage. Inter-pod affinity gets expensive for the scheduler above a few hundred nodes; for spreading with a numeric guarantee, prefer Topology Spread Constraints (maxSkew), which scales better.

Eviction under pressure is the node's self-protection. When memory, disk, or PIDs run short, the kubelet first tries to clean up (dead containers, unused images); if that's not enough, it evicts pods in this order: first the ones exceeding their requests (overshot BestEffort and Burstable), then by PriorityClass, and Guaranteed pods last. Details worth knowing: node-pressure eviction ignores the PodDisruptionBudget (it's an emergency); eviction ≠ OOMKill (OOMKill is the cgroup killing a container that went over its limit; eviction is the kubelet pulling pods to save the whole node). Placing pods under multiple constraints is bin packing, which is NP-hard. The scheduler uses a greedy heuristic (filter nodes → score them → pick one): fast, never optimal.

Key concepts in this stage

YAML and commands in practice

 # 1) Taint the GPU node (once, via kubectl):
 #    kubectl taint nodes gpu-node-1 nvidia.com/gpu=true:NoSchedule
apiVersion: v1
kind: Pod
metadata: { name: treino-ml }
spec:
  tolerations:                     # PART 1: tolerates the GPU taint (allows entry)
    - key: "nvidia.com/gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  affinity:
    nodeAffinity:                  # PART 2: AND requires a GPU node (forces it there)
      requiredDuringSchedulingIgnoredDuringExecution:   # hard: no label, no scheduling
        nodeSelectorTerms:
          - matchExpressions:
              - key: "accelerator"
                operator: In
                values: ["nvidia-gpu"]
  containers:
    - name: treino
      image: myregistry/treino:1.0
      resources:
        limits: { nvidia.com/gpu: 1 }   # requests 1 GPU (extended resource)

Essential commands for this stage

Hands-on

Watch a taint repel a pod in real time (works fine on a single-node minikube/kind cluster):

1. Find the node's name: kubectl get nodes.
2. Apply the taint: kubectl taint nodes <NODE_NAME> demo=sim:NoSchedule. Nothing without a toleration can be scheduled.
3. Launch an ordinary pod with no toleration: kubectl run repelido --image=nginx.
4. It stays Pending; check why: kubectl describe pod repelido | grep -A2 Events, which shows "had untolerated taint".
5. Clean up: remove the taint (note the trailing "-") with kubectl taint nodes <NODE_NAME> demo=sim:NoSchedule- and delete the pod.

SECTOR 2 COMPLETE, what a strategy masterclass! You put the right car in the right garage and you already know who the team sacrifices under pressure. But now comes the part that makes the pit crew's heart race: the race is live and you NEED to update the car without pulling anyone out of the fight. Welcome to the art of the pit stop. Stage 03, HERE WE GO!

Stage 03 the pit straight

Update strategies and rollouts

Friday, production is running, and you need to ship v2 without the public noticing. This is the sector of steel nerves: a mistimed pit stop throws away the lead. Here we learn to swap everything with the car still moving. BREATHE, and come with me.

Friday, 4pm. You need to ship v2 of the API. Take everything down at once to bring up the new version, and the service goes dark in the middle of the day. Ship it wrong with no fast way back, and the weekend turns into an on-call marathon. Worse still: if v2 is incompatible with v1 because of a database migration, running both at once corrupts data.

How it works

It's the pit stop dilemma in a long race. RollingUpdate is swapping the cars' setup one pit stop at a time, with the rest of the team still scoring points on track (the operation never stops). Recreate is the red flag: calls everyone into the garage at once, stops everything, but guarantees no car ever runs on half-old, half-new parts, there's never a mix of old and new spec on track at the same time.

RollingUpdate (the default) delivers zero downtime: it replaces pods gradually, spinning up a new ReplicaSet and scaling the old one down proportionally. Two parameters control the pace. maxUnavailable is the brake: how many pods can be unavailable during the swap (default 25%), guaranteeing a floor for availability. maxSurge is the power: how many pods above the desired count can exist temporarily (default 25%), the T-cars that speed up the round of swaps. With N replicas and the defaults, during a rollout you have at least 75% available and at most 125% running. You can't set both maxUnavailable and maxSurge to zero (the swap queue would jam).

Recreate guarantees full consistency, at the cost of downtime: it tears down all of v1 before bringing up v2. It guarantees the two versions never coexist, which you need when v1 and v2 are incompatible (a database migration that breaks the old version). The tuning trade-off: a high maxUnavailable speeds things up but shrinks capacity (latency spikes); a high maxSurge speeds things up but doubles pods, which doubles connections, which can saturate the database. Two recipes: availability-first (maxUnavailable: 0, generous maxSurge) or minimum-time under a budget (push the peak and accept the minimum tolerable degradation).

Guarantee tools: minReadySeconds (the pod must stay Ready for X seconds before it counts as available); progressDeadlineSeconds (default 600s; if the rollout stalls, it's flagged ProgressDeadlineExceeded, but the Deployment does NOT roll back on its own, you decide whether to go back); revisionHistoryLimit (default 10, how many revisions are kept for rollback). A rollout only fires on a change to .spec.template, scaling replicas alone doesn't count. More sophisticated strategies (Blue/Green, Canary) aren't native to Deployment; controllers like Argo Rollouts and Flagger add them with automated promotion gates.

Key concepts in this stage

YAML and commands in practice

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  annotations:
    kubernetes.io/change-cause: "Rollout v2.0.0"   # shows up in the rollout history
spec:
  replicas: 10
  minReadySeconds: 30           # a pod only "counts" after 30s stable Ready
  progressDeadlineSeconds: 600  # stalled for 10min, flags ProgressDeadlineExceeded
  revisionHistoryLimit: 10      # keeps 10 revisions for rollback
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0         # BRAKE: never drops below minimum capacity
      maxSurge: "50%"           # POWER: creates up to 5 extra pods to speed things up
  selector: { matchLabels: { app: myapp } }
  template:
    metadata: { labels: { app: myapp } }
    spec:
      containers:
        - name: app
          image: myregistry/myapp:2.0.0
          readinessProbe:        # without readiness, the rollout "lies" about being ready
            httpGet: { path: /healthz, port: 8080 }

Essential commands for this stage

Hands-on

Run a rollout and a rollback from scratch and watch the ReplicaSets:

1. Create it and wait until it's ready: kubectl create deployment web --image=nginx:1.25 --replicas=3 and kubectl rollout status deployment/web.
2. Update the image (fires the rollout because it changes the template): kubectl set image deployment/web nginx=nginx:1.27 and track it with rollout status.
3. See BOTH ReplicaSets (the old one at zero, the new one with 3): kubectl get rs -l app=web.
4. Roll back and confirm it's on 1.25 again: kubectl rollout undo deployment/web and kubectl describe deployment web | grep Image.
5. Clean up: kubectl delete deployment web.

WHAT A PIT STOP, LADIES AND GENTLEMEN! Sector 3 green, the car changed setup on the move and we still have the undo button in hand. But now we level up how the team is organized: how do you build EVERY car from the same mold, changing only the setup sheet? Helm takes the wheel. Stage 04!

Stage 04 behind the scenes at the factory

Package management with Helm Charts

Far from the cameras, the championship gets built at the factory. Same chassis all year, a different setup sheet for every circuit. Enough copy-pasting YAML and hoping you didn't forget anything. Helm is Kubernetes's apt, and it's about to organize your garage. Come into the workshop!

You've got dev, staging, and prod. Each environment needs the same ~15 YAML manifests, changing only replicas, image tag, and resources. You copy-paste everything three times. Then one day you change a label on the prod Deployment, forget to replicate it in staging, and spend the week chasing why "it works in staging but not in prod". That's a DRY violation and a factory for bugs.

How it works

Helm is Kubernetes's apt/yum. Think of the car's build mold: the chassis, the suspension philosophy, the aero architecture stay the same base all year. What changes from circuit to circuit is the setup sheet: low wing at Monza vs. high downforce at Monaco, tyre pressure, engine map. You don't redesign the car for every Grand Prix; you only swap the ingredients that vary (the values) inside the same mold (the template).

Anatomy of a Chart: Chart.yaml holds the metadata: version is the chart's version and appVersion is the version of the packaged application (both SemVer; keeping them separate helps governance). values.yaml holds the configurable defaults (replicas, image, port, resources), and everything that varies per environment lives here. templates/ are the manifests written as Go templates plus Sprig functions, where {{ .Values.replicaCount }} injects a value and helpers in _helpers.tpl avoid repeating logic.

The theoretical insight: a Helm template is a pure function T(values) → YAML: same inputs, same output, predictable and testable (helm template renders without applying). It's the simulator: you feed it the setup sheet and it hands back exactly how the car will turn out, without ever touching the track. That embodies two pillars of Infrastructure as Code: idempotence (applying it N times has the same effect as applying it once) and immutability (it doesn't mutate the existing object, it creates a new version and keeps the history, which is what makes rollback reliable).

Lifecycle: helm install (creates and records the release in a Secret) → helm upgrade (computes the diff and applies only the changes) → helm rollback (returns to a saved revision). Helm 3 has no central server (no Tiller): the client uses your kubectl credentials directly, so the user's RBAC limits what Helm can do. At scale, charts follow SemVer, get distributed through repositories (Artifact Hub, private ones), can be digitally signed (GPG + .prov) to protect the supply chain, and should embed least-privilege RBAC. Combined with GitOps (ArgoCD/Flux), every release becomes an auditable commit.

Key concepts in this stage

YAML and commands in practice

 # Chart.yaml: the package's identity
apiVersion: v2
name: myapp
version: 0.1.0          # CHART version (the mold)
appVersion: "1.0.0"     # packaged APPLICATION version (the cake)
type: application

Essential commands for this stage

Hands-on

See the "pure function" with your own eyes, without touching any cluster:

1. Create a sample chart: helm create demo.
2. Render with the default value: helm template demo | grep -i replicas (notice replicas: 1).
3. Render changing ONLY the input: helm template demo --set replicaCount=4 | grep -i replicas, and the output changes predictably.
4. Validate the chart: helm lint demo. You tested everything without touching any cluster: it ran in the simulator before the car went on track.

SECTOR 4 CLOSED with the garage running like clockwork! The factory is dialed in, every car comes off the same mold with the right setup sheet. Now we're back to Sunday's ADRENALINE: how do you ship v2 with an INSTANT undo button? The T-car is warmed up behind the scenes. Stage 05, Blue/Green. Hold on to that feeling!

Stage 05 the warmed-up T-car in the garage

Blue/Green deployment

Imagine having a spare car, same team, new setup, engine at temperature, READY to take over in the blink of an eye. That was the spirit of the spare-car era, before 2008, when a driver could switch to the spare on the grid and still start the race. That's the spirit of Blue/Green: flip the switch and be back in seconds. Radio's live in the garage!

You shipped v2, it passed every test, but in production, with real traffic and real data, a critical bug showed up 3 minutes later. With RollingUpdate, rolling back means running another full rollout, which takes minutes while customers suffer. You wanted an instant undo button.

How it works

It's the T-car (spare car), already built and warmed up behind the scenes. The Blue car is the one on track serving the public (traffic); Green is the spare: same team, new setup, engine already at temperature, ready to take over. Once you've validated Green, you swap cars instantly: the public sees Green right away. Something goes wrong? You're back on Blue in seconds, because it was never torn down, it stayed warm and waiting.

The core idea: keep two identical production environments. Blue is the current version serving 100% of the traffic; Green is the new version, ready and "warm", but with no traffic. Once Green is validated, you flip the switch and it takes over everything. In Kubernetes, the two sets of pods carry different labels (env: blue, env: green) and a single Service points to one of them via its selector. Promotion is just a patch to the Service's selector, and kube-proxy reconfigures routing in an instant. Rollback is the reverse patch. It's an atomic transition between two states (B → G) with deterministic rollback.

The traps: data/schema consistency is the Achilles' heel. If Green changes the database in an incompatible way, reverting to Blue breaks it. The fix is temporary dual compatibility (Martin Fowler): migrate the database to support both versions first, validate it, and only then switch the code. This is where the T-car analogy starts to strain: if the new car "rewrites the track layout" itself, meaning the database schema, going back to the old car doesn't help, the track has already changed underneath it. That's why backward-compatible migration comes before the switch.

Cost: two full environments means roughly 100% overhead during the transition (Etsy has run as high as 200% capacity). Good practice: scale the inactive environment to zero once things stabilize. Connection storm: switching 100% at once dumps the whole load on Green suddenly, and cold caches can cause a spike; mitigate it by warming Green up with shadow traffic beforehand. Classic anti-pattern: not treating Green as production all the time. If you let changes pile up waiting for the "right moment", the Blue-to-Green jump gets huge and risky. Tools: Argo Rollouts supports blueGreen natively; Flagger integrates with a service mesh (also called Red/Black at Netflix/Spinnaker).

Key concepts in this stage

YAML and commands in practice

 # Service initially pointing at BLUE
apiVersion: v1
kind: Service
metadata: { name: myapp-service }
spec:
  selector: { app: myapp, env: blue }   # <- the "switch": flips to green on promotion
  ports: [ { port: 80, targetPort: 8080 } ]

Essential commands for this stage

Hands-on

Build a toy Blue/Green and flip the switch while watching the endpoints:

1. Create two "environments": kubectl create deployment blue --image=nginx:1.25 and green --image=nginx:1.27.
2. Set the labels on the pod TEMPLATE (the Service selects pods): patch blue with env=blue and green with env=green (app=bg).
3. Expose a Service pointing at BLUE: kubectl expose deployment blue --name=bg-svc --port=80 --selector=app=bg,env=blue.
4. See which pod is getting traffic: kubectl get endpoints bg-svc (the blue pod's IP).
5. Flip the switch to GREEN: kubectl patch service bg-svc -p ... env green, and watch the endpoints change, nothing restarted.
6. Clean up: kubectl delete deploy blue green; kubectl delete svc bg-svc.

SECTOR 5 GREEN, a perfect switch flip! You've got the instant undo button in your pocket now. But Blue/Green throws 100% of the public onto v2 at once. What if the bug only shows up in 1 user out of 20? Then you don't risk both cars: you send ONE car to test the new part on track. The canary's in the cockpit. Stage 06!

Stage 06 one car tests the new part

Canary deployment

Ferrari doesn't put the new wing on both cars at once. They fit it to Leclerc's car, read the telemetry for a few laps, and only then release it to Hamilton. One feels the gas before anyone else: the old canary in the coal mine. It's the most surgical strategy on the grid, and it's our next corner. Telemetry live!

Blue/Green flips 100% of traffic at once. But what if v2 has a bug that only shows up under real load, in 1 out of every 20 users? Flipping everything at once exposes everybody at the same time. You wanted to test v2 with a small, real slice of users, measure it, and only then expand.

How it works

It's sending a single car to test the upgrade under race conditions before rolling it out to both. Ferrari doesn't put the new wing on both cars at once: they fit it to Leclerc's car for a few laps, read the telemetry (lap-time delta, tyre temperature, degradation), and only then release it to Hamilton too. If the delta gets worse, they pull the part off that car and the rest of the fleet was never exposed. The name comes from the "canary in the coal mine": a small fraction feels the toxic gas before anyone else does.

The core idea: instead of exposing everyone to v2, you send a small fraction of real users to it, monitor metrics, and only ramp up if things look fine. Blue/Green is "all or nothing" (100% at once); Canary is progressive (1% → 5% → 25% → 50% → 100%), validating at each step under real load. Both versions run side by side and you migrate traffic gradually.

Fine-grained traffic routing: done by a service mesh (Istio, Linkerd) or a weighted ingress, letting you set exact percentages (e.g., 90% v1 / 10% v2) independent of how many replicas exist, through sidecar proxies. In Istio, a VirtualService splits traffic between subsets (v1/v2) and you adjust the weights dynamically. What decides whether to promote or roll back are real-time metrics: error rate (5xx), latency (p95, p99), resource usage, business metrics. That closes an automatable feedback loop: if the canary's error rate stays below the threshold, increase the weight; if it violates it, roll back to v1 immediately.

Vocabulary worth knowing (deeper theory): a canary is a continuous A/B test, a statistical experiment between v1 (control) and v2, tested for significance (α = 0.05) via a z-test, chi-squared, or Mann-Whitney test (Netflix's Kayenta uses Mann-Whitney). Sequential tests (Wald's SPRT) stop early once there's enough evidence. Queueing theory: even 10% of traffic can have poor latency if that 10% nearly saturates the v2 pods (W = 1/(μ−λ) blows up near saturation). Multi-armed bandits (Thompson Sampling) allocate traffic adaptively. Real cases: Shopify sends ~5% for ~10min with automated analysis; Netflix automated the whole thing with Kayenta; Nubank uses feature flags plus canary for hundreds of deploys a day.

Key concepts in this stage

YAML and commands in practice

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: { name: myapp }
spec:
  hosts: [ myapp ]
  http:
    - route:
        - destination: { host: myapp, subset: v1 }
          weight: 90        # stable gets 90%
        - destination: { host: myapp, subset: v2 }
          weight: 10        # canary gets 10%, ramps up gradually

Essential commands for this stage

Hands-on

Without Istio installed, you can approximate a canary effect through the replica ratio behind the same Service (the balancing stays proportional to pod count):

1. Create v1 with 9 replicas and v2 with 1 (~10% of traffic lands on v2): kubectl create deployment app-v1 --image=nginx:1.25 --replicas=9 and app-v2 --image=nginx:1.27 --replicas=1.
2. Give both the same label app=canary, including on the pod template (via patch).
3. Expose a Service that covers BOTH versions: kubectl expose deployment app-v1 --name=canary-svc --port=80 --selector=app=canary.
4. Confirm 10 endpoints (9 v1 + 1 v2): kubectl get endpoints canary-svc -o wide.
5. "Promote" by adjusting replicas: kubectl scale deployment app-v2 --replicas=3 and app-v1 --replicas=1.
6. Clean up: kubectl delete deploy app-v1 app-v2; kubectl delete svc canary-svc.

SECTOR 6 DONE, canary approved! You now know how to ship without fear, the most surgical way on the grid. We've hit the second half of the race, and now the question changes scale: what happens when there isn't enough GARAGE space for the cars coming in? Who builds the pit on demand? Logistics takes the wheel. Stage 07, floor it on the back straight!

Stage 07 on-demand infrastructure power

Node scalability with Karpenter

In a flash sale, the HPA asks for 20 pods and... there's no garage for anyone. The cars queue up in the pit lane. It's like a back-to-back race weekend where the team has to build the structure on the spot, in the right place, with the cheapest material available. Karpenter is the fastest logistics manager in the paddock. Foot down!

So far you've scaled pods. But in a flash sale the HPA asks for 20 new pods and... there's no node to put them on. The pods sit Pending and the service can't keep up with demand. The traditional Cluster Autoscaler would solve it, but it's slow and depends on fixed node groups (ASGs) pre-configured by type/zone, a pain to maintain. And it often spins up a giant node for one small pod, wasting money.

How it works

It's the difference between two ways of building a garage in the pit lane. The Cluster Autoscaler is having fixed-size garage contracts, booked months in advance: a small car shows up and you're forced to open a giant garage, paying for the empty space. Karpenter is the logistics manager who looks at exactly how many cars, and what size, just arrived, and builds the right-sized garage on the spot, choosing even the cheapest material available, then tears it down the moment it's empty.

Karpenter = just-in-time node autoscaling. Built by AWS (open source, now multicloud), it provisions nodes sized to the pending pods, without fixed groups: faster (seconds to under a minute), picks the optimal instance (including Spot), and removes idle nodes on its own. Companies report savings of up to 30%. The flow: pods become unschedulable (the scheduler can't find a node) → Karpenter watches → calculates which instance best fits those pods → provisions it → the pods get scheduled. In reverse, it spots underutilized nodes and consolidates: moves the pods elsewhere and removes the idle node, or swaps an expensive On-Demand instance for a cheaper Spot one.

Configuration via NodePools (formerly called Provisioners): they define the rules for the nodes Karpenter is allowed to launch: instance types, zones, Spot vs. On-Demand, limits. Recommendation: restrict as little as possible, to give Karpenter room to optimize. Why it's clever: the underlying problem is multidimensional bin packing (NP-hard); it uses a greedy heuristic (filters out unfit types, sorts by cost/fit, does something close to First-Fit Decreasing). Key difference: the Cluster Autoscaler preserves homogeneity within a group (a small pod can trigger a big node); Karpenter does right-sizing, a small node for a small pod.

Spot and cost: Spot instances cost roughly 70-90% less, but can be pulled back with short notice (2 minutes on AWS). They're borrowed material the supplier can reclaim at any time: cheap, but you need a plan B. Karpenter handles this with proactive interruption: it detects the warning, marks the node as terminating, applies a taint, and drains the pods elsewhere before the cutoff. It also respects PodDisruptionBudgets, never consolidating or terminating nodes if that would violate the budget. Best practices: a PDB to control how many pods go down together, Topology Spread to distribute replicas, and separate NodePools for critical workloads (On-Demand) vs. tolerant ones (Spot).

Key concepts in this stage

YAML and commands in practice

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: default }
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]     # can use spot; falls back to on-demand if unavailable
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]                  # restricts architecture
      nodeClassRef: { name: default }        # points at the infra config (AMI, subnets...)
  limits: { cpu: "1000", memory: 1000Gi }    # cost ceiling: never exceeded
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized  # consolidates empty OR underutilized nodes
    consolidateAfter: 30s                          # hysteresis: waits 30s before acting

Essential commands for this stage

Hands-on

Karpenter runs in the cloud, but you can see the trigger it watches for (a pod Pending for lack of a node) in any cluster:

1. Ask for more CPU than any node has: kubectl create deployment faminto --image=nginx and kubectl set resources deployment faminto --requests=cpu=100.
2. The pod stays Pending (no node fits the request): kubectl get pods -l app=faminto.
3. See why, the signal Karpenter watches for: kubectl describe pod -l app=faminto | grep -A3 Events → "Insufficient cpu" (FailedScheduling).
4. Clean up: kubectl delete deployment faminto.

SECTOR 7 GREEN, what an overtake down the straight! You've solved GARAGE scaling. Now for the other side of the coin: sizing the PIT CREW to the queue of cars coming in, and pulling everyone back to zero in the quiet of the night. Scaling by event, not by effort. KEDA takes the stage. Stage 08!

Stage 08 event-driven scaling

Application scalability with KEDA

No car coming into the garage, zero mechanics standing idle on the jack for nothing. A queue of 40 orders shows up? Open 4 stations right now. It empties out? Pull everyone back. What triggers it isn't how hard a mechanic is sweating, it's how many cars are in the queue. Event-driven scaling. Let's get into KEDA!

You have workers consuming orders off a queue. The standard HPA only knows how to scale by CPU/memory, but your bottleneck isn't CPU, it's the size of the order queue. Worse: at 3am the queue sits empty, and you're still paying for 3 pods standing around with nothing to do. You wanted to scale by queue size and drop to zero when there's no work.

How it works

It's the pit crew sized to the queue of cars coming in. No car coming into the garage? Zero mechanics standing on the jack (the team doesn't pay people to do nothing). A car shows up? A crew steps up right away. A queue of 40 orders, each station handling 10 at a time? Open 4 stations. It empties out and stays empty for a while? Pull everyone back. What triggers it isn't one mechanic's effort (CPU), it's how many cars are in the queue (the event).

KEDA = event-driven autoscaling. It scales based on external events (RabbitMQ/SQS/Kafka queues, Prometheus metrics, cron, etc.) and enables scale-to-zero: no pod consuming resources when the queue is empty. It complements the HPA, it doesn't replace it. How it works: you create a ScaledObject (the central CRD) pointing at a target Deployment and defining a trigger (e.g., a RabbitMQ queue, 1 pod per 10 messages). KEDA creates an HPA behind the scenes. Empty queue → keeps the Deployment at 0 pods; a message arrives → activates 1 pod immediately; the queue grows → the HPA calculates the pods (40 messages = 4 pods); the queue empties out past the cooldownPeriod → back to zero. Only KEDA has the authority to go to zero; the HPA alone goes from 1 to N (minReplicas ≥ 1).

The theory (queueing theory): model events arriving at rate λ and each pod processing at rate μ; with c pods, capacity is c·μ. Stability condition: λ < c·μ (otherwise the queue grows without bound, and no autoscaling saves you). Little's Law (L = λ·W) relates average backlog L, arrival rate λ, and response time W: if the backlog is growing, either arrivals are too fast, or you're short on pods. The ideal autoscaler controls L to keep W within the SLO.

The challenge: oscillation/jitter (pods scaling up and down repeatedly), caused by feedback delay and static thresholds. KEDA/HPA mitigations: hysteresis (~10%: only scales up past 110% or down below 90% of target); stabilization windows and cooldownPeriod; a dead band (different thresholds for scaling up vs. down: scale up at 100 msgs but only scale down below 20). KEDA is cloud-agnostic (AKS, EKS, GKE, on-prem) and extensible via external gRPC scalers. Typical case: an e-commerce site scaling order workers by queue size during flash sales, back to zero overnight.

Key concepts in this stage

YAML and commands in practice

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: worker-scaler }
spec:
  scaleTargetRef: { name: meu-deployment }   # the Deployment that will be scaled
  minReplicaCount: 0        # scale-to-zero: zero pods when there's no work
  maxReplicaCount: 5        # safety ceiling
  cooldownPeriod: 60        # wait 60s of an empty queue before zeroing out
  pollingInterval: 15       # checks the queue every 15s
  triggers:
    - type: rabbitmq
      metadata:
        queueName: minha-fila
        protocol: amqp
        mode: QueueLength
        value: "10"         # target: 1 pod for every 10 messages in the queue

Essential commands for this stage

Hands-on

You can test KEDA without an external queue, using the cron trigger (which depends on nothing) to see scale-to-zero and reactivation:

1. Prerequisite: KEDA installed (helm install keda kedacore/keda -n keda --create-namespace).
2. Create any Deployment: kubectl create deployment cron-demo --image=nginx.
3. Apply a ScaledObject with a cron trigger: minReplicaCount 0, maxReplicaCount 3, ramping up at minute 0 and down at minute 30 of every hour (timezone America/Sao_Paulo).
4. Watch the Deployment: 0 outside the window; 3 inside it (kubectl get deployment cron-demo -w).
5. Clean up: kubectl delete scaledobject cron-demo; kubectl delete deployment cron-demo.

SECTOR 8 DONE, scaling fully under control! Garages and crew now breathe with demand. One sector left, and it's the most unforgiving one. You don't lose a race on the track here, you lose it OFF it, in scrutineering. It's the final security chicane. Maximum focus, because the wrong badge costs you the championship. Stage 09, the decisive lap!

Stage 09 the final chicane

Cluster security

The FIA doesn't forgive. We've seen cars win on track and get disqualified in scrutineering over a single regulation detail. It's the same in the cluster: one compromised container carrying the master key hands over the whole kingdom. Zero Trust, a badge with your name on it, a seal that expires on its own. It's the final corner, and it's where champions prove they deserve the trophy. FOCUS!

Your cluster runs multiple teams and services. One of them runs a dependency with a vulnerability and gets compromised. If that pod was using the default ServiceAccount with broad permissions (or worse, had a static AWS key baked into the container), the attacker just became king of the kingdom: reads secrets from every namespace, spins up mining pods, reaches into buckets. A single compromised container shouldn't be able to cost you the whole cluster.

How it works

It's FIA paddock access control. Every team member gets a badge that opens only their own zone: the tyre mechanic doesn't walk into the strategy room, the data engineer doesn't touch the PU (ServiceAccount plus least-privilege RBAC). Nobody walks around with the circuit's master key. Badges expire at the end of the weekend and areas lock themselves automatically (TLS and short-lived tokens). It's the same philosophy as the rulebook, Zero Trust: nobody passes a gate without a verified credential, not even the team principal. Kubernetes security rests on three pillars: identity, authorization, and encryption.

Identity (ServiceAccounts): every application should have its own ServiceAccount, never the default one. It gives the pod a token to talk to the API Server. Isolating identities per workload is what lets you contain the damage when a pod gets compromised: each role on the team with a badge bearing their own name, and if one falls into the wrong hands, you know exactly which zone was exposed and revoke just that one.

Authorization (RBAC): grants permissions through Roles (namespaced) or ClusterRoles (cluster-wide), tied to subjects via RoleBindings. The model is a graph: Subject → RoleBinding → Role → Permission. Least privilege in practice: prefer a namespaced Role over a ClusterRole; grant only the verbs you need (get, list, never *) and only the resources you need. Kubernetes RBAC is static and additive (it only adds permissions, there are no denials), which makes it auditable. For conditional policies ("deny pods without limits", "only the stable image tag"), use OPA/Gatekeeper with the Rego language (Policy as Code).

Federated cloud identity: applications need access to S3, buckets, Key Vault, without baking static keys into the container. The fix is federating the Kubernetes ServiceAccount with a cloud identity via OIDC, receiving short-lived tokens: IRSA (IAM Roles for Service Accounts) on EKS, Workload Identity on GKE and AKS. Real incidents prove the point: in the 2023 SCARLETEEL attack, the damage was contained because the pod's Role had a tightly scoped permission set; in the Tesla case (2018), a password-free dashboard plus broad AWS credentials on a pod led to cryptojacking. Encryption (cert-manager): automates the TLS lifecycle through CRDs (Issuer/ClusterIssuer issues; Certificate defines the CN, SANs, duration, and renewBefore), generates the CSR, signs it, fills in the Secret, and renews itself. Secret rotation uses an overlap window: old and new both valid at once until everyone has migrated, then the old one is invalidated.

Key concepts in this stage

YAML and commands in practice

apiVersion: v1
kind: ServiceAccount
metadata:
  name: analytics-sa
  namespace: analytics
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/S3Reader  # IRSA:
                                                       # federates this SA with an IAM Role
                                                       # (short-lived token, zero static keys)
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role                    # namespaced (limited to the analytics namespace), NOT ClusterRole
metadata: { name: analytics-read, namespace: analytics }
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]    # read-only on pods, no "*"
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]            # only get on configmaps
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding             # ties the Role to the subject (the SA)
metadata: { name: bind-analytics-read, namespace: analytics }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: analytics-read }
subjects:
  - kind: ServiceAccount
    name: analytics-sa
    namespace: analytics

Essential commands for this stage

Hands-on

Prove least privilege with kubectl auth can-i, no cloud required:

1. Create a namespace, an SA, and a read-only Role on pods: kubectl create namespace lab; kubectl create serviceaccount leitor -n lab; kubectl create role ler-pods --verb=get,list --resource=pods -n lab; kubectl create rolebinding bind-leitor --role=ler-pods --serviceaccount=lab:leitor -n lab.
2. CAN the SA list pods? kubectl auth can-i list pods -n lab --as=system:serviceaccount:lab:leitor → yes.
3. CAN the SA delete pods? (we didn't grant "delete") → no.
4. CAN the SA read Secrets? (we didn't grant that resource) → no.
5. Clean up: kubectl delete namespace lab. A badge that opens only one door: whoever steals it is stuck in that one room.

CHECKERED FLAG IN SIGHT! The car got through the final chicane without touching a thing, scrutineering signed off, and the trophy is YOURS. Nine sectors completed start to finish. You're no longer a passenger in this cluster: you're the driver in command of the pit wall. It's been an honor calling this race!