# Debugging Production: A Systematic Approach That Actually Finds Bugs
By Govind Bajaj · 2025

Tags: debugging, production, methodology, troubleshooting, backend
A production payment webhook was failing silently. I found the bug in 20 minutes using this systematic approach. Here's the exact process I follow.
A production payment webhook was failing silently last month.

Payments were processing successfully, but the order fulfillment emails were not sending. The logs showed nothing. The webhook endpoint returned 200. Everything looked fine on the surface.

I found the bug in twenty minutes using a systematic approach. The issue was a race condition between two async database operations — one was completing before the other, and the email was being sent with incomplete data.

Here is the exact process I follow for any production bug.

## Step 1: Reproduce the Symptom

Before touching any code, confirm the bug is real and understand its exact symptoms.

Questions I ask: What exactly is the user seeing? When did it start? Does it happen every time or intermittently? What is the expected vs actual behavior?

For the webhook issue: Order confirmation emails were not sending. It started three days ago. It happened for about 30% of payments. Expected behavior: email sends within 30 seconds of payment. Actual: no email at all.

## Step 2: Check the Logs (All of Them)

Start with application logs. Look for errors, warnings, or unusual patterns around the time of failure.

Then check infrastructure logs. Server CPU, memory, disk. Database slow query logs. CDN error rates.

Then check external service logs. Payment provider webhooks. Email service delivery reports. Third-party API status pages.

For the webhook: Application logs showed successful webhook processing with no errors. But email service logs showed zero sends for affected orders. The gap was between webhook handling and email queuing.

## Step 3: Add Targeted Logging

If existing logs do not reveal the issue, add logging at key points in the suspected code path.

```ts
// Add targeted logging to trace execution
console.log("[WEBHOOK] Received payment:", paymentId)
console.log("[WEBHOOK] Order lookup result:", order ? "found" : "missing")
console.log("[WEBHOOK] Email queue result:", emailResult)
```

Deploy the logging, trigger a test payment, and observe. Remove the logging after the bug is fixed — temporary logging should not become permanent.

## Step 4: Isolate the Code Path

Trace the exact execution path from input to failure. For each step, ask: what data enters this step, what should happen, and what actually happens?

```
Webhook received -> Parse payment -> Find order -> Update status -> Queue email
     |                  |              |             |              |
   200 OK           Valid data     Order found   Status set    NOT QUEUED!
```

The breakdown was at the final step. The order update and email queue were both async, but the email was being queued before the order update completed.

## Step 5: Form a Hypothesis and Test It

Hypothesis: The email is being sent before the order status update completes, so the email template sees the old status and skips sending.

Test: Add an await before the email queue call.

```ts
// Before (buggy):
updateOrderStatus(orderId, "paid")
queueEmail(orderId, "confirmation") // Runs before update completes!

// After (fixed):
await updateOrderStatus(orderId, "paid")
await queueEmail(orderId, "confirmation")
```

## Step 6: Verify the Fix

Deploy the fix to staging first. Reproduce the original failure scenario. Confirm it is fixed. Then deploy to production with monitoring.

After the fix, I monitored webhook processing for 24 hours. All 147 test payments sent confirmation emails correctly.

## The Debugging Checklist

1. Reproduce the symptom — confirm the bug is real
2. Check all logs — application, infrastructure, external services
3. Add targeted logging — trace the exact code path
4. Isolate the failure point — what step breaks?
5. Form a hypothesis — what could cause this?
6. Test the fix — verify on staging before production
7. Monitor after deploy — ensure the fix holds

## Takeaways

- Never start fixing before you understand the problem — reproduction comes first
- Check application, infrastructure, and external service logs systematically
- Add temporary targeted logging to trace execution paths
- Async operations are a common source of race conditions — check your awaits
- Test fixes on staging with the same data patterns that caused the production bug
- Monitor for 24 hours after deploying a production fix
- This systematic approach finds most bugs in under 30 minutes