Deduplication

Webhook providers sometimes send the same event multiple times — due to retries, network issues, or provider bugs. Hookbase's deduplication system detects and silently drops duplicate events before they reach your destinations.

How It Works

When deduplication is enabled on a source, each incoming event is assigned a dedup key based on the configured strategy. This key is checked against a sliding time window. If a matching key is found, the event is dropped and returns 200 OK to the provider (so it stops retrying).

Enabling Deduplication

Enable dedup on a source via the API or dashboard:

curl -X PATCH https://api.hookbase.app/api/sources/src_abc123 \
  -H "Authorization: Bearer whr_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "dedupEnabled": true,
    "dedupStrategy": "provider_id",
    "dedupWindowHours": 24
  }'

Strategies

auto (Default)

Automatically detects the best strategy based on the source's provider setting:

Provider	Auto Strategy
github	Uses X-GitHub-Delivery header
stripe	Uses Stripe-Event-Id header (from data.id in payload)
shopify	Uses X-Shopify-Webhook-Id header
slack	Uses X-Slack-Request-Timestamp + payload hash
twilio	Uses MessageSid from payload
custom/generic	Falls back to payload_hash

provider_id

Extracts the event ID from provider-specific headers:

Provider	Header
GitHub	X-GitHub-Delivery
Stripe	Stripe-Idempotency-Key or event id
Shopify	X-Shopify-Webhook-Id
Slack	X-Slack-Request-Timestamp
Twilio	I-Twilio-Idempotency-Token

If the header is missing, the event is accepted (not dropped).

payload_hash

Computes a SHA-256 hash of the entire request body. Two events with identical payloads within the window are considered duplicates.

WARNING

Be careful with payload_hash if your payloads contain timestamps or other fields that change on every request — they'll never match.

idempotency_key

Uses a custom header (specified in dedupCustomHeader) as the dedup key. This gives you full control over what constitutes a duplicate.

curl -X PATCH .../sources/src_abc123 \
  -d '{
    "dedupEnabled": true,
    "dedupStrategy": "idempotency_key",
    "dedupCustomHeader": "X-Idempotency-Key"
  }'

Your webhook sender must include this header:

curl -X POST https://api.hookbase.app/ingest/myorg/mysource \
  -H "X-Idempotency-Key: order-12345-payment-confirmed" \
  -H "Content-Type: application/json" \
  -d '{"event": "payment.confirmed", "order_id": "12345"}'

none

Explicitly disables deduplication. Every event is accepted regardless of duplicates. This is the default when dedupEnabled is false.

Window Configuration

The dedupWindowHours controls how long dedup keys are remembered:

Value	Use Case
1	Fast-moving events where duplicates arrive within minutes
24	Standard — covers most retry scenarios (default)
72	Conservative — handles delayed retries
168	Maximum (7 days) — for providers with aggressive retry policies

TIP

Larger windows use more memory. For high-volume sources (>10k events/hour), keep the window as small as your use case allows.

Monitoring Duplicates

Dropped duplicates are tracked in the source's metrics. View them in the dashboard or via the API:

curl https://api.hookbase.app/api/analytics?sourceId=src_abc123 \
  -H "Authorization: Bearer whr_your_api_key"

The response includes duplicatesDropped in the source breakdown.

Examples

GitHub — Auto Detection

curl -X POST https://api.hookbase.app/api/sources \
  -H "Authorization: Bearer whr_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GitHub Webhooks",
    "slug": "github",
    "provider": "github",
    "dedupEnabled": true,
    "dedupStrategy": "auto",
    "dedupWindowHours": 24
  }'

Stripe — Provider ID

curl -X PATCH .../sources/src_stripe \
  -d '{
    "dedupEnabled": true,
    "dedupStrategy": "provider_id",
    "dedupWindowHours": 48
  }'

Custom API — Idempotency Key

curl -X PATCH .../sources/src_custom \
  -d '{
    "dedupEnabled": true,
    "dedupStrategy": "idempotency_key",
    "dedupCustomHeader": "X-Request-ID",
    "dedupWindowHours": 24
  }'

Related

• Sources API — Configure dedup fields
• Pipeline Architecture — Where dedup fits in the processing flow