Kubernetes cluster broken in production.Fixed this session.

CrashLoopBackOff means the container is starting, crashing, and Kubernetes is backing off exponentially between restart attempts. The engineer runs kubectl logs <pod> --previous to get the last crash output, kubectl describe pod <pod> to see the exit code and last termination reason, and kubectl get events --sort-by=.metadata.creationTimestamp to see cluster-level context. Exit code 1 typically means an application error on startup; exit code 137 means OOMKill; exit code 139 is a segfault in native code. For exit code 1, the application log in --previous reveals the root cause: a missing environment variable in the ConfigMap or Secret, a database connection refused because the secret has the wrong hostname, or a required file mount missing because the PVC is not bound. Most CrashLoopBackOff incidents are resolved within a single 4-hour session.

OOMKill events occur when a container exceeds its resources.limits.memory value and the kernel terminates it. The right fix is not simply raising the limit blindly - the engineer first determines whether the OOMKill is caused by a genuine memory leak or by legitimate memory growth that was underprovisioned. The process: kubectl top pods to see current live usage, Prometheus or Datadog pod memory metrics to see the growth curve over time, and kubectl describe pod to confirm the OOMKilled exit code. If the usage grows linearly over time, it is a leak - the engineer profiles the container process (pprof for Go, async-profiler for JVM, tracemalloc for Python) to identify the allocation source. If usage is stable at a high level, the limit is raised with a buffer and requests/limits ratio is set to 1:1.5 to allow occasional bursts without eviction.

A Helm upgrade that breaks services is rolled back using helm rollback <release-name> <revision-number> where the revision is the last successful release (visible with helm history <release-name>). The engineer verifies the rollback succeeded with helm status <release-name> and kubectl rollout status deployment/<name>. For the root cause investigation, the engineer runs helm diff upgrade (using the helm-diff plugin) between the broken and previous chart versions to see the exact Kubernetes manifest changes that caused the failure: a changed ConfigMap value breaking application config, an updated container image tag that has a startup error, a changed Service selector that stopped matching the running Pods, or a removed environment variable referenced by the application. The chart fix is applied to a staging environment first and smoke-tested before re-upgrading production.

A PVC stuck in Pending means Kubernetes cannot bind a PersistentVolume to it. The engineer runs kubectl describe pvc <name> to read the Events field which gives the exact reason: no PersistentVolume available with matching storage class, access mode, and capacity, the StorageClass provisioner not running or having permission errors (check the provisioner pod logs in kube-system), a topology constraint on the PVC (such as zone-affinity) that conflicts with the available storage in the cluster, or the storage quota for the namespace being exceeded. The fix depends on the root cause: creating a PV manually for static provisioning, fixing the StorageClass provisioner IAM permissions on AWS EBS CSI or GCP PD CSI, relaxing the zone constraint, or requesting a quota increase.

Every QuickHire Kubernetes engineer passes: a live cluster debugging exercise on a broken Kubernetes environment with a CrashLoopBackOff caused by a missing Secret and an OOMKilled pod from a Go memory leak, a written assessment covering Kubernetes internals (scheduler predicates and priorities, controller reconciliation loops, etcd and API server interaction), networking (CNI plugins, kube-proxy iptables vs IPVS, NetworkPolicy enforcement, Service mesh sidecar injection), storage (CSI drivers, StorageClass dynamic provisioning, StatefulSet PVC lifecycle), security (RBAC, PodSecurity admission, OPA Gatekeeper), and Helm packaging and release management, plus a portfolio review of production Kubernetes clusters managed at scale. Only the top 3% pass all stages. PM matches engineers to your cluster distribution (EKS, GKE, AKS, self-managed kubeadm) and your GitOps tooling (ArgoCD, Flux).

QuickHire Instant is built for immediate engineering execution - book a vetted developer and dedicated PM in under 10 minutes, pay per session with no contracts. QuickHire Enterprise is for long-term team partnerships: dedicated developers, managed engineering pods, staff augmentation at scale, and enterprise-grade contracts (MSA, NDA, SLA) with a dedicated account manager. Both models use the same vetted talent network. Visit enterprise.quickhire.services to explore Enterprise engagements.