The Drain That Would Not Finish: A Small Club Learns KubernetesOps

The Drain That Would Not Finish: A Small Club Learns KubernetesOps

Let me tell you about a small team at an imaginary light-sport flying club, because their week explains KubernetesOps better than a definition could. The club runs its booking site and aircraft-availability board on a modest Kubernetes cluster. Nobody on the team is a full-time infrastructure person; they are a volunteer web developer, a retired airline technician who loves spreadsheets, and a part-time operations coordinator. None of this happened to a real club. It is a made-up composite, assembled to show how the practices in the Ops Atlas entry on KubernetesOps play out in a small group.

Monday: the plan that looked simple

The cluster was a couple of versions behind, and the coordinator had finally blocked out an evening for a minor upgrade. The technician had read that the work goes in a set order, API server first, nodes later, one minor version at a time, with each node drained before its turn. They wrote the steps on a whiteboard in the clubhouse and felt ready.

Tuesday: the drain that would not finish

The control plane went smoothly. Then the first node refused to empty. The command sat there, printing the same retry message every few seconds. The developer’s first instinct was to delete the stuck pods by hand. The technician, who had spent decades following maintenance manuals, said no: forcing a deletion skips the protections that exist for good reasons, and the Atlas treats it as something to try only at the end, after the workload’s owner has agreed.

They looked at the message more carefully. The API was answering with a refusal code, which the Atlas describes as an eviction declined for now, with a retry as the right response. Then the technician noticed that the booking service had only one copy, and its protective budget allowed zero downtime. A drain could never succeed, since removing the single copy would break the rule.

Wednesday: fixing the cause, not the symptom

The fix had two parts. First, the booking service was scaled to two copies, so that one could move while the other kept serving. Second, the budget was changed to allow one copy to be unavailable. The technician also read about a setting for unhealthy pods, which the Atlas recommends setting so that a broken pod does not hold the drain hostage by default. The drain completed in minutes, and the upgrade carried on, node by node.

The coordinator asked why this had not shown up in testing. The answer was uncomfortable and common: the test cluster had no budgets at all. Somebody wrote that down as a task and gave it an owner and a Friday deadline.

Thursday: the bill that made sense at last

With the upgrade done, the technician turned to a question that had nagged for months: why did a cluster hosting two small sites need five machines? The technician exported the reserved and used figures. Reservations were generous; real peak usage was a fraction. The Atlas suggests sizing around the ninety-fifth percentile of usage with some headroom, and gathering recommendations from the vertical autoscaler in its off mode, which suggests numbers without changing anything. The team trimmed reservations for three services and let the Cluster Autoscaler, which removes machines only when they fall below half requested, take a node away over the following days.

One detail made the developer smile. Recent versions allow resizing a running pod’s resources in place, so some of these changes no longer required a restart, though the Atlas notes a quality-of-service class cannot change and shrinking memory is only best-effort.

Their working rule of thumb, borrowed from the Atlas’s rightsizing example, was to take the 95th-percentile usage, multiply by 1.3, and then divide by a target utilisation of 75 percent. It is not a law of nature, just a sane starting point, and it moved their sums from guesswork to arithmetic. They agreed to re-run the numbers each month for a quarter, and to keep the old reservations in the repository history in case anything went wrong.

The coordinator also asked a fair question during that conversation: what if usage doubles during the summer rally weekend? The answer was to rely on the horizontal autoscaler for bursts, to make sure its target metric had matching requests, and to avoid stacking the vertical autoscaler on the same metric, since the project itself advises against combining the two on the same CPU or memory signal.

Friday: writing it down

The last step was the one nobody finds exciting. The coordinator wrote a one-page runbook from the whiteboard photo, added the lesson about single-copy services and strict budgets, and set a calendar reminder for the next hop. The technician added a note to rehearse restoring from a backup before the next quarter, because a stored copy that has never been restored is closer to a rumour than a safeguard.

What the club took away

  • A drain that hangs is usually telling you about a rule, not a bug.
  • Single-copy services and zero-downtime budgets cannot coexist with maintenance.
  • Reservations deserve a review every so often, because they drive both placement and cost.
  • The runbook is the product of the week; the upgrade is only the occasion.

If you want more from this site

The same methodical mood runs through the club’s own checklist for inspecting landing gear. And if the club’s choice of tools interests you, a pragmatic way to discover software for your own toolkit pairs well with it. Small teams do well when they favour steady, written routines over heroics, whether the machine in question has wings or runs a website.