Zero-downtime Kubernetes upgrades with Talos and Cluster API

Kubernetes upgrades fail for a small number of predictable reasons, all of them configuration rather than bad luck. This post covers the setup I use to roll production clusters during working hours with no measurable impact on traffic.

Why upgrades drop traffic

Three failure modes account for most upgrade incidents. A node reboots while it is still receiving requests. A controller replaces every replica at once because nothing constrains it. A StatefulSet loses quorum because two members restart inside the same window. Each is caused by a missing constraint, and each is fixable before you start.

The setup

Talos provides the node operating system and Cluster API manages machine lifecycle. Talos images are immutable, so upgrading replaces the machine instead of mutating packages on a running host. Rollback is a reboot into the previous image rather than a package downgrade.

• Talos Linux: immutable nodes, API driven, no SSH access.
• Cluster API: MachineDeployments declare the target Kubernetes version.
• A rolling strategy with one surge node, so capacity never drops below baseline.

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      version: v1.30.2

maxSurge: 1 with maxUnavailable: 0 instructs Cluster API to provision a replacement node before draining an existing one. Scheduleable capacity stays constant for the duration of the roll.

Draining safely with PodDisruptionBudgets

Surge capacity alone is not enough. Draining a node calls the eviction API, which honours PodDisruptionBudgets. Any workload without one can have all of its pods evicted simultaneously, regardless of how many replicas it runs.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

A drain that blocks is usually correct behaviour: a PodDisruptionBudget is preventing an unsafe eviction. Fix the workload rather than forcing the drain.

Pre-flight checklist

• etcd snapshot taken, and the restore path verified against a scratch cluster.
• A PodDisruptionBudget on every workload with more than one replica, satisfiable at current scale.
• No kubelet or control plane certificates expiring inside the maintenance window.
• One canary node upgraded and observed for ten minutes before the roll continues.

Summary

Immutable nodes, one surge machine, accurate PodDisruptionBudgets and a checklist that is actually run. That combination removes the variance from cluster upgrades and makes them a routine operation rather than a scheduled event.