Method · Technical support engineering · by Ayman Sbeiti — I support high-trust software platforms · Hiring?
Method
How I run a root-cause analysis.
Separate the trigger from the cause, demand a mechanism rather than a correlation, and fix the class of failure, not just the instance that got caught.
The distinction that decides the fix
The trigger
The change that exposed the failure. It tells you where to look.
Recent workflow modifications made escalations start failing.
The cause
What was wrong all along. This is what you fix.
A data-handling gap the modifications exposed. It had been possible the entire time.
The principles
- Separate the trigger from the cause. The change that exposed a failure tells you where to look, not necessarily what is wrong. Recent modifications can widen a pre-existing gap without being the gap.
- Demand a mechanism, not a correlation. The analysis is finished when the failure can be explained as a chain a smart non-specialist could follow, not when a suspect has been found.
- Check the boundaries between systems first. Two components can each behave correctly by their own rules while the failure lives in the unobserved gap between them.
- Verify the fix against the original failure. Reproduce, apply, and confirm the failure is gone and that nothing else moved.
- Fix the class, not the instance. Ask what made this failure possible and what made it invisible, then address both. The second fix is usually the one that outlives the bug.
- Write the dead ends down. The hypotheses that were eliminated, and what disqualified each, are what make the conclusion trustworthy.
Why this matters
The most consequential sentence in a root-cause analysis distinguishes what triggered a failure from what caused it. An analysis that stopped at "the change broke it, revert the change" would have shipped a wrong conclusion and left the real fragility in place.
The standard I hold a conclusion to is mechanical: the failure explained as a chain of behaviours, each system doing exactly what its rules say, ending in the observed symptom. The sending system transmitted what it was given. The receiving system rejected what didn't validate. Nothing surfaced the rejection. Both behaved correctly, and the failure lived between them.
That framing points the fix at the right layer: not blame for either system, but validation and visibility at the boundary. And the fix isn't done when the bug is. The lasting question is what made the failure invisible.
The rule
The class of silent failure is worse than any instance of it.