Kubernetes Upgrade Checklist: Plan Version Updates Without Breaking Production Workloads

Overview

Kubernetes version upgrades can affect the control plane, worker nodes, APIs, admission policies, networking, storage, observability, and deployment tooling. The safest approach is to treat the upgrade as a sequence of controlled changes with clear evidence at each step.

Before choosing a target version, record the following:

• Current Kubernetes version for the control plane and every node pool.
• Target version and the supported upgrade path offered by your provider or distribution.
• Cluster type: managed, self-managed, or part of a hosted Kubernetes platform.
• Critical workloads, maintenance windows, recovery objectives, and business owners.
• Versions of Helm, GitOps controllers, ingress components, service meshes, CSI drivers, CNIs, policy engines, monitoring agents, and backup tools.

Managed Kubernetes hosting may automate parts of the control-plane process, but application compatibility and node-pool planning remain your responsibility unless your service agreement explicitly says otherwise. For a control-versus-operations comparison, see Managed Kubernetes vs Self-Managed Kubernetes.

Use a simple upgrade record for every environment:

Checklist by scenario

Before any production upgrade

• Read the upgrade guidance for the exact Kubernetes distribution and provider. Confirm permitted version jumps, sequencing rules, maintenance behavior, and node image requirements.
• Define success and abort criteria. Examples include healthy control-plane checks, no unexpected increase in pending pods, successful application smoke tests, and a documented rollback decision time.
• Freeze unrelated infrastructure changes for the upgrade window. Avoid changing application images, network policy, storage classes, and cluster add-ons at the same time unless the change is required.
• Confirm access to cluster credentials, provider consoles, infrastructure-as-code repositories, monitoring, logs, and incident channels.
• Review capacity. Ensure the cluster can temporarily run replacement nodes while old nodes are drained. Check quotas, subnet capacity, IP availability, and autoscaling limits.

For an API deprecation or removal

API compatibility is one of the most important parts of a Kubernetes version upgrade. A manifest can appear valid in source control while an older API remains in a generated chart, a Helm release, a GitOps repository, or an operator-managed resource.

• Compare every Kubernetes API used by your manifests with the APIs served by the target version.
• Search source repositories, Helm templates, rendered manifests, CI pipelines, and deployment artifacts.
• Inspect live resources, not only files. Export and review objects created by older tools or controllers.
• Upgrade controllers and operators before the cluster when their compatibility guidance requires it.
• Replace deprecated API versions, then apply and observe the replacement resources in a non-production environment.

Do not assume that changing an apiVersion string is sufficient. Field names, defaults, validation rules, ownership behavior, and conversion support may also differ. Keep a list of affected resources and record the test that proves each one works.

For a workload-sensitive production cluster

• Test rolling updates, readiness and liveness probes, disruption budgets, autoscaling, scheduled jobs, ingress routes, persistent volumes, and network policies.
• Confirm that workloads tolerate node draining. Check replica counts, topology spread, anti-affinity, graceful termination, and termination handling.
• Verify that admission policies, image-signing checks, secrets integration, and identity bindings behave as expected.
• Test observability: metrics collection, log shipping, alert rules, tracing, dashboards, and on-call notifications.
• Ask application owners to approve a short validation script for each critical service.

For a data-bearing or stateful cluster

• Confirm recent backups of application data, cluster configuration, persistent volume data, and required secrets.
• Perform a restore test in an isolated environment. A backup that has not been restored should not be treated as a complete rollback plan.
• Document replication status, database failover behavior, storage-driver compatibility, and the order in which stateful components must be handled.
• Define whether rollback means reverting the cluster, restoring data, failing over to another environment, or redeploying from known-good artifacts. These are different actions with different risks.

Use the cloud backup RPO and RTO planning guide to align the upgrade plan with recovery objectives.

For a staged rollout

1. Upgrade a disposable or development cluster first.
2. Repeat the process in staging with production-like policies, data flows, and traffic patterns.
3. Upgrade a small production node pool or a low-risk cluster segment if your architecture supports it.
4. Observe workload health, error rates, scheduling, resource usage, and infrastructure events.
5. Continue only after the agreed validation period passes. Stop if abort criteria are met.

What to double-check

Version and support information

Build a small reference table before each upgrade rather than relying on an old runbook:

Supported versions and removed APIs change over time. Verify them against current Kubernetes documentation, your provider's release notes, and the release documentation for each critical add-on. Avoid copying a version table from an older article into a production runbook without checking its date and scope.

Operational checks

• Run a pre-upgrade health check for nodes, pods, events, storage, and control-plane access.
• Confirm that PodDisruptionBudgets do not prevent necessary draining or create an unsafe false sense of protection.
• Check for pending pods, unschedulable nodes, failed jobs, expired certificates, and overloaded resources before starting.
• Ensure the deployment pipeline can reproduce the intended state after the upgrade.
• Capture baseline metrics so post-upgrade behavior can be compared with a known-good period.

Common mistakes

• Upgrading directly in production: A staging rehearsal often exposes API, storage, admission, and scheduling problems at lower risk.
• Checking only application manifests: Operators, charts, generated files, and live resources can contain deprecated APIs that are absent from the main repository.
• Draining without capacity planning: Replacement nodes may fail to launch because of quotas, unavailable IP addresses, restrictive affinity, or insufficient headroom.
• Treating backups as rollback: Restoring data may take longer than reverting a node image, and some cluster changes cannot be reversed safely.
• Changing too many variables: Combining a Kubernetes upgrade with a networking redesign or major application release makes diagnosis harder.
• Ignoring add-ons: A healthy control plane does not prove that ingress, storage, policy, telemetry, or GitOps components are compatible.
• Skipping post-upgrade observation: Some failures appear only during rescheduling, autoscaling, certificate renewal, scheduled jobs, or a later deployment.

When to revisit

Review this checklist before every Kubernetes minor-version upgrade, node-image replacement, managed-cluster maintenance event, or major add-on update. Revisit it earlier when your provider changes its supported upgrade path, Kubernetes removes an API, a critical operator publishes compatibility guidance, or your cluster architecture changes.

At least once per planning cycle, refresh the following:

• The supported-version and upgrade-sequence table.
• API deprecation search commands and manifest-scanning tools.
• Provider procedures, maintenance policies, and rollback limitations.
• Backup restore evidence and recovery contacts.
• Critical workload smoke tests and acceptance criteria.
• Monitoring dashboards, alert thresholds, and post-upgrade observation windows.

For the next upgrade, create a change ticket with the target version, owners, maintenance window, preflight results, backup evidence, staged rollout plan, abort criteria, and validation results. Start with a non-production rehearsal, resolve every finding that affects production, and obtain explicit approval from both the platform owner and the owners of critical workloads. This turns a general Kubernetes upgrade checklist into a current, testable operating procedure.