Self-service diagnostics for API and webhook failures
Thirty-five webhooks failed and two were never sent. A fictional incident report shows what the customer could safely see, where the case split between two owners, and what evidence was still missing after recovery.
7 min read
Thirty-seven invoice webhooks went missing overnight, and they didn’t go missing the same way. Thirty-five reached the customer’s endpoint and failed there. Two were never sent at all.
That split decides recovery: what the customer can fix, what the platform must fix, and which button nobody should press. The incident below is fictional, with invented companies, people, IDs, times, and logs. It’s written up like a real incident report, because good self-service diagnostics mostly mean reading evidence in the right order.
9:40 AM – what the customer could see
Alex Okafor, a developer at Brightwater Freight, noticed no invoices had synced since 02:15. Brightwater’s billing provider, Kestrel Ledger, sends invoice webhooks to Brightwater’s endpoint. Kestrel had disabled that endpoint at 08:15.
Alex’s first instinct was the big button. Replay every event this month. Instead, Alex opened the delivery view, signed in to Brightwater’s own tenant. A tenant is one customer account’s isolated slice of a shared platform.
Here’s what that view returned, as diagnostic packet DP-0417:
The “Withheld” row matters as much as the others. Tenant scope should come from the authenticated session. Every lookup by event, endpoint, or delivery ID should confirm the object is the requester’s first. Beyond that baseline, what you expose is a policy choice:
The signature row is easy to misread. Kestrel signing with active key version 3 says what the sender did. It says nothing about whether Brightwater’s handler verified that signature.
Kestrel’s customer-safe summary said: “Your endpoint returned HTTP 500 to 35 events, so we disabled it at 08:15. Two more were queued and never sent, and we’re escalating those. Status codes alone don’t show why your handler failed.”
The split: two kinds of failure, two owners
That summary already contains the decision. The 35 failed events have evidence on both sides of the wire, so the customer can probably diagnose them. The two unsent events have no attempt at all, so nothing in Brightwater’s logs can explain them.
This is the rule that matters here. Work from permitted evidence, take the smallest safe step it supports, and escalate with a reproduction when the evidence runs out. If you can’t fill a field from evidence the customer may see, that empty field is your escalation trigger.
Each stage of a single delivery is a different fact, proven by a different party:
The two unsent events stop at row one, which makes them Kestrel’s problem. The 35 failures reached row two and came back with a 500, which is thin evidence. A 500 says the receiver failed, not why.
Success codes are thin too. Under the CloudEvents webhook specification, a 202 Accepted means delivery was accepted, but processing hasn’t happened or is unknown.
So the case now has two recovery owners. Alex owns the 35, and a Kestrel engineer will own the two.
9:50 AM – the receiver’s own evidence
Status codes couldn’t name a cause, so Alex pulled Brightwater’s own handler logs, sanitized as RX-0417. They showed a deploy at 01:50. For all 35 events, the handler verified the signature with key version 3, then threw a missing-configuration error before writing anything.
That’s an established cause, and it came from receiver evidence, not from the 500. Alex rolled back the deploy at 9:50 AM.
One caution travels with RX-0417. A valid signature shows the payload was signed with the signing secret and wasn’t altered. Anyone holding that secret could sign it. It also doesn’t prove the requester has authority over that tenant or resource.
Before any resend: reconciliation
The big button would have resent every event this month, including 1,103 that Kestrel recorded as acknowledged. An acknowledgment isn’t an invoice, so before any redelivery Alex ran reconciliation REC-0417 against Brightwater’s ERP by source event ID.
All 1,103 acknowledged events had exactly one matching invoice. None of the 37 window events had one. The arithmetic closes: 35 failed, plus 2 unsent, plus 1,103 reconciled, equals 1,140 events this month.
That turns a full replay from risky to plainly wrong. A replay is a consequential action, not a diagnostic. It needs its own authorization, a defined event list, and evidence of which business effects already happened. That’s how AI agent guardrails treat any action that changes state.
Alex re-enabled the endpoint and requested redelivery of exactly the 35 failed IDs as RD-552. Because redelivery changes state, Brightwater’s tenant admin, Ana Duarte, approved it separately. Two requests were refused:
- A replay of all 1,140 events, because REC-0417 shows 1,103 business records already exist.
- A view of Brightwater’s EU account, tenant tn_bw_eu_022, because Alex’s session isn’t authorized for it.
10:20 AM – one event, start to finish
A green check can hide five different facts. Here’s one of the 35, ev_bw_9184, followed through every state. The state names are this article’s shorthand, not product fields, and every value is part of the authored case.
Illustrative trace – HTTP 202 is only an acknowledgment; receiver processing and ERP recovery need their own evidence.
Approval and submission are separate rows because different people did them. The 202 and the invoice are separate rows because the first is Kestrel’s evidence and the second is Brightwater’s. That’s the answer-versus-outcome gap Computer support resolution describes: showing the log isn’t the fix.
Redelivery is only safe if a repeat does no harm. Idempotency means processing the same event twice has the same effect as processing it once. Logging event IDs isn’t enough on its own.
Treat it as a scoped requirement, not a guarantee. The duplicate check and the invoice write should commit together, keyed on event ID, and be tested against a crash between the two. In this case, Brightwater’s staging test covered that. If duplicate handling or prior outcomes are unknown, pause redelivery and assign reconciliation to a named owner.
10:41 AM – the other owner takes the unsent pair
Self-service stops when permitted evidence can’t name a cause, the fix needs authority the customer lacks, or another tenant is involved. The two unsent events hit the first condition, so they went to Kestrel with the reproduction attached.
Escalation packet EC-118 carried both event IDs, their queued timestamps, zero recorded attempts, and endpoint state. It also showed that 35 sibling events had been attempted. Sam Rivera, a Kestrel support engineer, accepted EC-118 at 10:41 without asking Alex to repeat anything.
Sam’s acceptance put the case with the right owner, but it recovered no invoices on its own. Sam found both events stuck in a queue state left by the endpoint disable and cleared it. REC-0417 already showed no invoices for either, so Ana approved RD-553 for those two IDs, and Alex submitted it.
Both attempts returned 202, and Brightwater’s queue log shows both processed. Alex confirmed INV-77344 and INV-77345 in the ERP, one per event. Only then did Sam close EC-118, noting the events had been held on Kestrel’s side.
Sam’s internal account could see tn_bw_eu_022, but that didn’t make it disclosable to Alex. The next step there belongs to the EU tenant admin, who opens the request from their own session.
What this report still can’t tell you
Every invoice is back, and the incident is closed. It isn’t fully explained, and the gaps are where a buyer should look hardest.
Sam’s fix cleared the symptom, and the closing note explains where the events sat. Neither is a reproduction. Without one, the next disable could strand events the same way.
When you assess a platform’s self-service diagnostics, ask where items like these live once the dashboard turns green. Track shown-versus-recovered events, blind replays prevented, and escalations accepted without re-explaining, with your own denominators. Then ask vendors for the evidence behind each, as part of your enterprise AI agent buying criteria.
If Computer, by DevRev is on your list, hold it to the same standard. Take a gap like Kestrel’s missing reproduction to a failed-delivery diagnostics session with DevRev, and work out what evidence would close it.
