Debugging a Silently Failing Escalation Workflow

Context

This happened inside the story another case study on this site tells at full length: the HubSpot-to-Intercom migration, during its roughly two-week phased rollout in early July 2026. That piece is about keeping the support channel alive through a platform change. This one is about a single failure inside that window, the kind that doesn’t announce itself, and the investigation that found it.

Part of the migration’s redesign was automatic escalation: support issues that needed engineering created Jira tickets directly from Intercom, with context attached. Developers’ visibility into support work depended on that handoff working. My role covered the support side of the migration, including the escalation workflow’s design. So when the handoff broke, the investigation was mine.

Symptom

After recent workflow modifications, some escalations stopped creating Jira tickets. Three properties made it the worst kind of failure to see:

• It was silent. The automation appeared to trigger successfully; nothing errored where anyone was looking.
• It was partial. Most escalations still worked, so the workflow looked healthy at a glance.
• It looked like success. From the support side, an escalated conversation seemed handled. The ticket that should have existed simply never did, somewhere nobody could see.

Stakes

The support channel showed no known interruption. Customers’ conversations continued normally, as far as the rollout monitoring could tell, which is the same carefully bounded claim the migration case study makes. What broke was a different channel: the handoff from support to engineering. When an escalation silently failed, the customer’s conversation looked fine. What vanished was invisible to them: the ticket that should have reached a developer. That is where the cost lived: SLA exposure on the escalation path, reduced engineering visibility, and the risk of delayed investigation and duplicate customer contacts as issues stalled unseen.

Investigation

Signal to confirmed root cause took roughly one working day. The record, reconstructed with deliberately coarse timing:

The elimination order mattered. Ruling out the connection first meant the problem was in what was being sent, not how it travelled. The rollback testing was investigative, not corrective: I progressively reverted the recent workflow modifications, testing after each step. When the failure stopped occurring, I knew the problematic change lived in the portion most recently reverted, without yet knowing what about it was wrong. The workflow’s structure wasn’t the cause; something in that change’s data handling was. Comparing successful and failed payloads line by line found it: a required attribute arriving empty in the failed requests, present in every successful one.

Escalation decision

I deliberately did not escalate to engineering immediately, but that never meant sitting on a known-broken path quietly. Containment started at once: support was working a temporary manual watch on escalated issues from the moment the signal was characterized. What I held back was the engineering hand-off, because reporting “escalations are failing” without understanding why would have handed engineering a mystery instead of a problem. I owned the investigation end to end: reproduction, comparison, rollback testing, payload inspection, root-cause validation, and the design of the operational fix. Once the cause was understood, I brought the findings to the lead software engineer for review, with reproduction steps, the successful-versus-failed comparison, workflow observations, and the payload evidence already packaged. His implementation involvement in this episode belonged to the separate attachment-synchronization work the migration case study describes. No customer communication was needed. The failure was internal to the escalation path, and support’s manual watch covered affected issues in the meantime.

Root cause

Jira rejected any request that violated its schema: a required attribute arriving empty or missing, or a field mismatching the expected schema. Because the rejection happened only on Jira’s side, Intercom appeared to succeed while no ticket was created.

Both systems behaved according to their own rules. Intercom sent what the workflow produced, Jira rejected what failed validation, and nothing surfaced the rejection where support was looking. The failure lived in the gap between the two systems rather than inside either platform individually, and the recent workflow modifications had widened that gap without anyone seeing it.

Resolution

I designed the fix so that a bad payload would be caught before it ever left, and a recurrence would announce itself: I made the required attributes mandatory at the source, aligned the fields between Intercom and Jira, and added validation before escalation so a bad payload never left. I added error handling that produced a visible note inside the Intercom conversation, so the failure surfaced exactly where the person who owned the customer relationship was already looking, and a notification path so a failed escalation could no longer pass unseen. Additional testing followed, and validation continued over the following days as the rollout progressed.

What the team was told

Communication ran internally on two tracks while the investigation was open:

What changed afterward

The durable change was observability. Before, the workflow could fail without anyone knowing. After, a failed escalation announced itself instead of disappearing. Validation moved ahead of escalation instead of relying on the receiving system’s silence. And the clearest signal that trust returned: once the workflow was reliable, the integration was expanded with additional attributes, because developers were willing to put more weight on a handoff they could now see.

Reflection

This investigation was not primarily about fixing a bug. It was about restoring observability. The failure’s defining property was that it produced no evidence anywhere the team could see. So the investigation’s real work was creating visibility: comparing successful and failed executions, narrowing the workflow through rollback testing, isolating differences, inspecting payloads, and validating one hypothesis at a time until the gap between two correctly-behaving systems became visible.

• This failure survived because it was silent and partial at once. A loud failure would have been fixed the day the workflow changed; this one produced healthy-looking automations and a subset of missing tickets, and cost accumulated until the absence itself became the signal.
• With no error message anywhere, every useful step of this investigation was a diff. Working escalations against broken ones, the current workflow against its rolled-back version, successful payloads against failed ones. The empty required attribute only became visible as a difference between two JSON bodies.
• Holding the engineering hand-off until I had a diagnosis shortened the whole incident. Engineering received reproduction steps, the payload comparison, and a suspected root cause in one package instead of a mystery to re-investigate from scratch, while support’s manual watch covered the gap.
• The part of the fix that outlived the bug was the visibility. Mandatory attributes closed this instance; validation-before-send and failures that announce themselves changed what any future instance will look like.