The failures that don't announce themselves

Most engineering instinct about failure is shaped by the loud kind. A service falls over. A request throws. An alert fires, somebody is paged, a ticket is opened, and a clock starts running. That failure is expensive for an afternoon and then it is finished.

The failures worth worrying about are quieter. They produce a number. The number is wrong. Nothing in the system disagrees with it, so it goes into a report, the report goes into a decision, and the decision gets made again every quarter for years.

Everything below comes from production systems our team has worked on directly. We describe the mechanism and not the organization, because the mechanism is the part that transfers.

A multiplier in the wrong place

A value multiplier was applied at the wrong layer of a conversion pipeline. The result was reported revenue one hundred times smaller than the revenue that had actually occurred. Seven production configurations carried the same mistake. It had been running for eight years.

There was no exception. No log line, no crash, no retry storm, no degraded health check. A figure that is wrong by a constant factor is still a figure. It has the right data type. It sits inside a plausible range, especially after everyone involved has spent a few years calibrating their expectations to it. Dashboards rendered it. People compared this quarter with last quarter, and both were wrong by the same factor, so the trend looked entirely sensible.

That last part is what makes scale errors so durable. Most monitoring watches for change. A defect that is constant produces no change to watch.

The drop with no receipt

An ingest path enforced a maximum payload size. Anything over the limit was discarded. Nothing downstream was told.

Think about what an operator can conclude from a missing record. Either the event never happened, or it happened and the pipeline threw it away. Those are different facts with different responses, and the system offered no way to tell them apart. There is a second problem underneath the first. The payloads most likely to exceed a size limit are the unusual ones: long sessions, heavy accounts, the tail where the interesting behavior lives. The filter was not random. It removed precisely the records an analyst would most want to see.

A discarded record should leave a receipt. A counter, a reason code, a single incremented metric. Once you can count discards you can alert on them. While the count does not exist, a system that is losing data looks exactly like a system that is working.

The configuration value that is a trap

The same size limit was operator-configurable, and the documentation invited people to tune it.

One environment set it to 4.5MB. That value was read with an integer parser, which stopped at the first character that was not a digit and returned 4. A four-byte cap. Every payload exceeded it. Everything was dropped.

A different environment set the key to a value the parser could not read at all. The parser returned 0. Zero was treated as "no limit", which disabled the cap completely.

Both outcomes are severe, and they are opposites. Both came from the one field that operators were encouraged to change by hand. A parser that returns a plausible number for an implausible input converts a typo into a silent production change. The remedy is dull and effective. Reject input you cannot parse. Log the rejection. Refuse to start rather than start wrong. If a setting has a magic value like zero, make that value explicit and separate from the value you fall back to when parsing fails.

Failing open is quieter than failing closed

A privacy feature masked sensitive fields before data left the device. The rule referenced a class by name. The class was later renamed, the reference stopped resolving, and masking silently failed open.

Consider the two ways that feature can break. If it masks too much, somebody sees a redacted field where they expected a value. They complain the same day and the ticket gets a high severity, because the damage is visible on a screen. If it masks too little, nothing looks wrong to anyone. The data flows. The reports fill in. Nobody files a ticket for a mask that silently did not happen.

Over-masking gets a P1 because it is visible. Under-masking is invisible by construction. Any control that can fail open needs a self-check that runs in production and makes noise on its own, because the population of humans who would otherwise report it is empty.

The failure the system already computed

The most frustrating shape in this list is the one where the information existed the whole time.

A function detects that it could not do the thing it was asked to do. It returns false, or an error object, or a count of what it skipped. The caller ignores the return value. The only place that result is ever read is a log statement that is compiled out of release builds.

The system knew. It measured its own failure, wrote the measurement into a variable, and dropped the variable on the floor. Search your codebase for ignored return values on functions whose names include words like send, flush, mask, persist, upload, or validate. It is a short search and it finds real things.

Why this class of defect survives so long

Three properties keep these alive.

The first is that nothing in the normal escalation path is triggered. No alert, no page, no ticket, no retro. Organizations get very good at the failures that generate work, and these generate none.

The second is that time confers trust. A report that has rendered every morning for four years feels more reliable than one built last month, when the opposite is closer to the truth. Age means the defect has had longer to become load-bearing.

The third is that the people best placed to catch the error are the ones least able to see it. An analyst comparing two numbers from the same broken pipeline sees a consistent story. The only way to break the illusion is to compare the pipeline against something outside it.

Three things worth checking this week

Reconcile what the application sends against what the report claims. Choose one metric that carries weight in a decision. Capture what your application actually emits for a known, bounded set of interactions. Then open the report that claims to describe those same interactions and compare absolute values, not trends. A factor error only shows up against ground truth.

Find the failures you compute and discard. Look for return values nobody checks, catch blocks that log at debug level and continue, and any place where a count of skipped or rejected items is calculated but never exported. Each one is a signal you already paid to produce.

Make "off" distinguishable from "broken". For every feature that can be disabled by configuration, ask how an operator would tell a disabled state from a misconfigured one. Emitting one line at startup that states the resolved configuration, including which values came from defaults and which were parsed from input, closes most of this gap for almost no effort.

None of this needs new tooling or a platform decision. Each item converts silence into a signal, which is the only real defense here. A system that cannot tell you it is failing will tell you nothing at all, and everyone will read that silence as good news.