#30 Webhook Delivery System — Solution

✦ AI-Generated Solution · System Design · Comprehensive Reliable HTTP delivery of events to user-registered callback URLs at 1B events/day (~12K events/sec average, plan for 5–10× peak). The interview emphasis is REST API, caching, DB design, and the retry mechanism.

1. Requirements

Functional

• Register/update/delete a webhook: (event_id → callback_url) (assume unique 1:1 mapping per the prompt).
• On an event, POST the payload to the registered callback.
• Query webhook status and delivery logs (GET, with filtering + pagination).

Non-functional

• Reliable, at-least-once delivery with retries (consumers must be idempotent).
• Durable — never drop an accepted event.
• Secure — signed payloads, HTTPS, SSRF protection.
• Observable — per-delivery status, attempts, latency.
• Scale: 1B events/day.

2. Capacity

• 1B/day ÷ 86,400 ≈ ~12K events/sec average; design for ~60–100K/sec peak.
• Delivery is I/O-bound (waiting on slow/cold customer endpoints) → many concurrent workers, not CPU-heavy.
• Delivery logs dominate storage: ~1B rows/day → TTL + cold storage tiering.

3. Architecture

Decouple accept from deliver with a durable queue. Accepting an event is a fast, durable write; delivery (slow, retried) happens asynchronously in workers.

4. API Design

POST   /v1/webhooks            { "event_id": "...", "callback_url": "https://...", "secret": "..." }
GET    /v1/webhooks/{id}       -> config + status
DELETE /v1/webhooks/{id}
GET    /v1/webhooks/{id}/deliveries?status=failed&cursor=...&limit=50   -> delivery log (paginated)
POST   /v1/events             { "event_id": "...", "payload": {...} }    -> 202 Accepted (queued)

Use cursor-based pagination for delivery logs (offset pagination is too expensive at this volume).

5. Data Model

CREATE TABLE webhooks (
  id           UUID PRIMARY KEY,
  event_id     TEXT UNIQUE,         -- 1:1 mapping per problem statement
  callback_url TEXT NOT NULL,
  secret       TEXT NOT NULL,       -- for HMAC signing
  is_active    BOOLEAN DEFAULT true,
  created_at   TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE deliveries (
  id            UUID PRIMARY KEY,
  webhook_id    UUID,
  event_uid     UUID,               -- dedupe key, also sent as header for consumer idempotency
  status        TEXT,               -- PENDING|SUCCESS|FAILED|DEAD
  attempt_count INT DEFAULT 0,
  next_retry_at TIMESTAMPTZ,
  last_status   INT,                -- last HTTP code
  created_at    TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX idx_deliveries_lookup ON deliveries (webhook_id, status, created_at);

6. Caching

• Active webhook configs (the event_id → callback_url + secret map) are read on every event → cache in Redis for fast fan-out; invalidate on update/delete (write-through). This is the hot read path the interviewer probes.
• Do not cache delivery logs (write-heavy, append-only) — index them instead.

7. Retry Mechanism (the primary focus)

Implemented with message-queue primitives:

• Which failures retry: transient — 5xx, connection timeout, DNS failure → retry. Permanent — 4xx (except 429) → fail fast (don't retry a malformed/forbidden endpoint). 429 → honor Retry-After.
• Exponential backoff with jitter: delays like 1s, 5s, 30s, 5m, 30m, 2h … up to a max (e.g. 6–8 attempts), with random jitter to avoid synchronized retry storms.
• Implementation: a delay queue / scheduled re-enqueue (Kafka with a delay topic, SQS visibility timeout, or RabbitMQ TTL+DLX). Store attempt_count and next_retry_at on the delivery; a scheduler re-enqueues due retries.
• Dead Letter Queue: after max attempts → move to DLQ, mark DEAD, alert, and expose for manual replay.
• Idempotency: send a stable X-Webhook-Id/event_uid header so consumers can dedupe (delivery is at-least-once).
• Retry-storm protection: per-endpoint circuit breaker — if a callback is failing consistently, back off that endpoint specifically so one dead consumer doesn't starve workers.

def schedule_retry(delivery):
    delivery.attempt_count += 1
    if delivery.attempt_count > MAX_ATTEMPTS or permanent(delivery.last_status):
        dead_letter(delivery)                       # -> DLQ, alert
        return
    backoff = min(BASE * 2 ** delivery.attempt_count, MAX_BACKOFF)
    delivery.next_retry_at = now() + backoff + random_jitter()
    delay_queue.put(delivery, eta=delivery.next_retry_at)

8. Security

• Sign every payload: X-Signature = HMAC_SHA256(secret, body) so consumers verify authenticity and integrity. Include a timestamp to prevent replay.
• HTTPS only; validate callback URLs on registration to block SSRF (no private/loopback IP ranges, no metadata endpoints).

9. Scaling & Observability

• Partition the queue by webhook_id for ordering + parallelism; autoscale stateless delivery workers on queue depth.
• Tier delivery logs: hot (recent, queryable) → cold (object storage) via TTL.
• Metrics: delivery success rate, p99 latency, attempts distribution, DLQ size, per-endpoint failure rate (drives circuit breaking).

10. Summary