Here is a problem that will be familiar to anyone who keeps a small site or booking tool running for a light-sport business. The service works all morning, then every so often it vanishes: the page errors out, a few minutes later it is back, and nobody knows why. This write-up treats that as a case in SysOps diagnosis, the host-level discipline of working out what a server is really doing. It moves from symptom to evidence to cause to fix, and it stops at each step to ask what was actually observed rather than assumed.
The method follows the Ops Atlas, whose page on host-level triage in the Ops Atlas SysOps entry insists on gathering evidence before intervening. The scenario below is a generic composite, not a report of a particular incident.
The symptom
A web application behind a reservations page stops answering for two to five minutes, at irregular times, roughly twice a day. The process supervisor restarts it each time. Staff have been restarting the whole machine on bad days, which feels productive and tells you nothing.
The first mistake to avoid
The Atlas lists reaching for the restart button first among its warning signs, and the reason is simple: the bounce throws away the state you need. Memory contents, open connections and the kernel’s recent messages are all lost in the bounce. A better opening is to preserve the scene. Capture the process list, the last few hundred lines of the service log and the kernel message buffer before anyone touches the machine.
Gathering evidence
Three questions narrow the field quickly. Did the process exit on its own, or was it killed? What did the operating system say at that moment? And was the machine short of any resource when it happened? The service’s journal answers the first two. If the log shows an error about too many open files, that is a strong lead. If it shows nothing, the kernel’s own record may show that the memory-pressure killer ended the process.
The Atlas recommends a structured walk through resources, citing a method published in 2012 that asks, for each resource, about utilisation, saturation and errors, plus a ten-command quick survey from 2015 and kernel pressure-stall metrics introduced in 2018. A structured walk avoids the common trap of fixating on the first graph that looks bad.
Hypothesis A: the file-descriptor ceiling
Every open connection and file counts as a descriptor, and the Atlas notes that the soft limit for a systemd-managed service defaults to 1024, with a hard limit of 524288. A busy service holding many client connections can hit 1024 long before anything else is strained. Symptoms include refused connections and log lines about too many open files, with ordinary processor and memory use. The remedy is to raise the soft limit for that specific service, within reason, and to confirm the new value from the running process rather than from the configuration file. One caution from the Atlas: systemd marks one particular way of expressing this limit with the words “Do not use”, so check the current documentation for the form you plan to apply.
Hypothesis B: the memory killer
If the log is silent but the kernel messages mention the out-of-memory killer, the process was sacrificed to protect the machine. The tempting fix is to give the process the most protected score, minus 1000. The Atlas calls out why this fails: the kill does not disappear, it simply lands on another process, perhaps a more important one. The better fix is to find what consumed memory, whether a leak, a missing limit on a cache or a batch job running at the same hour, and address that.
A small evidence table
Before choosing, lay the facts side by side. Make three columns: what you saw, where you saw it and what it rules in or out. A line such as “service exits with status 137 at 14:02, from the journal, suggests a forced kill” is more useful than a paragraph of impressions. A line such as “memory available stayed above a third of the total, from the monitoring graph, argues against the memory killer” removes a suspect cleanly. The table forces you to separate observation from inference, and it gives a colleague everything needed to challenge your conclusion. In practice most diagnostic arguments end quickly once the table exists, because the disagreement turns out to be about a missing fact rather than a difference of opinion.
Distinguishing the two
- Look at the exit reason in the journal for the failure time.
- Search the kernel messages for out-of-memory records.
- Check the service’s descriptor count against its limit during a busy period.
- Compare available memory, not free memory, around the event.
Only one hypothesis will usually fit. If both fail to, widen the search to disk, network and scheduled jobs that fire at the same minute.
The fix and the proof
Suppose the evidence points to the descriptor limit. You raise it for that service, reload, and note the change in the shared log. Then you wait. A fix without a follow-up is a guess with extra steps. Check the service over several busy periods, record the peak descriptor count, and write down whether the symptom has stopped. If it has, you now know the cause and have a number to monitor.
Owners who have worked through the undercarriage checks laid out for light-sport aircraft will recognise the cautious, inspect-first temperament, since there recording measurements beats trusting a feeling. Fuel problems follow the same pattern of symptom, evidence and cause, as explored in common ways light-sport fuel systems trip pilots up.
What if neither hypothesis fits?
Sometimes the table rules out both suspects. Then the discipline is to widen the net methodically, not to improvise. Look at whether a scheduled job starts at the failure minute, whether the disk was briefly saturated, and whether a network dependency such as a database timed out first. Note that a symptom can be several things at once: a slow dependency can pile up connections, which then hits the descriptor ceiling, so the descriptor limit is a consequence rather than the root. Chasing the chain back to its first link is the real work, and the evidence table is what keeps you honest while you do it.
Lessons for the next time
The scenario teaches three things. Intermittent failures are usually resource ceilings, and the ceiling is often a default nobody chose. Preserving evidence costs five minutes and saves days. And a fix is complete only when you can show the symptom has gone and say why. Teams that work this way produce fewer repeat incidents, and that is a measurable outcome: count the repeats per quarter and watch the number fall.

