Webhook Logging & Audit Trail

This use case demonstrates how to build a compliance-grade webhook audit system. All incoming webhooks are routed to both their normal operational destinations and a centralized logging destination that sends structured audit records to a data warehouse (BigQuery, Snowflake, or similar). The system filters out test events, encrypts PII fields, and produces structured records suitable for compliance audits.

Architecture

mermaid

flowchart LR
    GH[GitHub Webhooks] --> S1[Source: GitHub]
    ST[Stripe Webhooks] --> S2[Source: Stripe]
    SH[Shopify Webhooks] --> S3[Source: Shopify]
    S1 --> R1[Route: Normal]
    S1 --> R4[Route: Audit Log]
    S2 --> R2[Route: Normal]
    S2 --> R5[Route: Audit Log]
    S3 --> R3[Route: Normal]
    S3 --> R6[Route: Audit Log]
    R1 --> D1[Operational Destinations]
    R2 --> D2[Operational Destinations]
    R3 --> D3[Operational Destinations]
    R4 --> |Filter + Transform| DW[Data Warehouse]
    R5 --> |Filter + Transform| DW
    R6 --> |Filter + Transform| DW

Flow:

Webhooks arrive from multiple providers
Each event is routed to its normal operational destination (unchanged)
Each event is also routed to the audit logging pipeline
A filter skips test and health-check events
A transform creates structured audit records with encrypted PII
Audit records are delivered to the data warehouse endpoint

Step 1: Create Sources

Create sources for each provider you want to audit. These may already exist from your operational setup—you can reuse them.

GitHub Source

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/sources \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GitHub Production",
    "slug": "github-prod",
    "verificationConfig": {
      "type": "github",
      "secret": "your_github_webhook_secret"
    }
  }'

Stripe Source

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/sources \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Stripe Production",
    "slug": "stripe-prod",
    "verificationConfig": {
      "type": "stripe",
      "secret": "whsec_your_stripe_webhook_signing_secret"
    }
  }'

Shopify Source

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/sources \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Shopify Production",
    "slug": "shopify-prod",
    "verificationConfig": {
      "type": "shopify",
      "secret": "your_shopify_webhook_signing_secret"
    }
  }'

Step 2: Create the Data Warehouse Destination

Create a destination that sends audit records to your data warehouse ingestion endpoint:

BigQuery (via Streaming Insert API)

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/destinations \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "BigQuery Audit Logs",
    "type": "http",
    "config": {
      "url": "https://bigquery.googleapis.com/bigquery/v2/projects/YOUR_PROJECT/datasets/webhook_audit/tables/events/insertAll",
      "method": "POST",
      "headers": {
        "Authorization": "Bearer YOUR_GCP_ACCESS_TOKEN",
        "Content-Type": "application/json"
      }
    },
    "retryConfig": {
      "maxAttempts": 5,
      "backoffMultiplier": 2,
      "initialInterval": 2000
    }
  }'

Snowflake (via Snowpipe REST API)

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/destinations \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Snowflake Audit Logs",
    "type": "http",
    "config": {
      "url": "https://YOUR_ACCOUNT.snowflakecomputing.com/v1/data/pipes/WEBHOOK_AUDIT_PIPE/insertReport",
      "method": "POST",
      "headers": {
        "Authorization": "Bearer YOUR_SNOWFLAKE_JWT",
        "Content-Type": "application/json"
      }
    },
    "retryConfig": {
      "maxAttempts": 5,
      "backoffMultiplier": 2,
      "initialInterval": 2000
    }
  }'

TIP

For either data warehouse, you may prefer to set up an intermediate HTTP endpoint (such as a Cloudflare Worker or AWS Lambda) that handles authentication token refresh and batch inserts. Point the Hookbase destination at that intermediate endpoint for simpler credential management.

Step 3: Create the Audit Filter

Filter out test events, health checks, and ping events that should not appear in audit logs:

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/filters \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Skip Test & Health Events",
    "conditions": [
      {
        "field": "headers.x-github-event",
        "operator": "neq",
        "value": "ping"
      },
      {
        "field": "type",
        "operator": "not_matches",
        "value": "^test_.*"
      },
      {
        "field": "headers.x-shopify-topic",
        "operator": "neq",
        "value": "app/uninstalled"
      },
      {
        "field": "headers.x-hookbase-test",
        "operator": "not_exists"
      }
    ],
    "logic": "AND"
  }'

This filter ensures:

GitHub ping events (sent when a webhook is first configured) are excluded
Stripe test events (prefixed with test_) are excluded
Shopify app/uninstalled lifecycle events are excluded
Any events sent with the X-Hookbase-Test header (used for pipeline testing) are excluded

Step 4: Enable Field Encryption

Before creating audit transforms, enable field encryption to protect PII fields. This ensures that sensitive data like email addresses, names, and customer IDs are encrypted at rest in your data warehouse.

bash

curl -X PATCH https://api.hookbase.app/api/organizations/{orgId}/sources/src_github_prod \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "encryptionConfig": {
      "enabled": true,
      "fields": [
        "sender.email",
        "pusher.email"
      ],
      "algorithm": "aes-256-gcm"
    }
  }'

bash

curl -X PATCH https://api.hookbase.app/api/organizations/{orgId}/sources/src_stripe_prod \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "encryptionConfig": {
      "enabled": true,
      "fields": [
        "data.object.customer_email",
        "data.object.billing_details.email",
        "data.object.billing_details.name",
        "data.object.billing_details.phone",
        "data.object.shipping.name",
        "data.object.shipping.phone"
      ],
      "algorithm": "aes-256-gcm"
    }
  }'

bash

curl -X PATCH https://api.hookbase.app/api/organizations/{orgId}/sources/src_shopify_prod \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "encryptionConfig": {
      "enabled": true,
      "fields": [
        "customer.email",
        "customer.first_name",
        "customer.last_name",
        "customer.phone",
        "shipping_address.name",
        "shipping_address.phone",
        "billing_address.name",
        "billing_address.phone"
      ],
      "algorithm": "aes-256-gcm"
    }
  }'

WARNING

Field encryption happens at the source level before any transforms or routing. This means the encrypted values will appear in both operational and audit routes. If your operational destinations need plaintext PII, configure encryption only on the audit-specific transforms instead. See the Field Encryption guide for details.

Step 5: Create Audit Transforms

Each transform creates a structured audit record from the provider's payload. The audit record schema is consistent across all providers.

Audit Record Schema

json

{
  "event_id": "hookbase-generated unique ID",
  "source": "provider name",
  "source_event_type": "original event type from provider",
  "timestamp": "ISO 8601 timestamp",
  "payload_hash": "SHA-256 hash of original payload for integrity verification",
  "headers": {
    "content_type": "content type header",
    "signature_header": "provider signature for non-repudiation",
    "provider_event_id": "provider's own event identifier"
  },
  "delivery_status": "received",
  "audit_metadata": {
    "hookbase_source_id": "source ID in Hookbase",
    "hookbase_org_id": "organization ID",
    "ingested_at": "when Hookbase received the event",
    "payload_size_bytes": "approximate payload size",
    "encryption_applied": true
  }
}

GitHub Audit Transform

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/transforms \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GitHub → Audit Record",
    "type": "jsonata",
    "config": {
      "expression": "{\n  \"event_id\": \"audit_\" & $string($millis()) & \"_gh_\" & headers.\"x-github-delivery\",\n  \"source\": \"github\",\n  \"source_event_type\": headers.\"x-github-event\" & ($exists(action) ? \".\" & action : \"\"),\n  \"timestamp\": $now(),\n  \"payload_hash\": $hash($string($), \"sha256\"),\n  \"headers\": {\n    \"content_type\": headers.\"content-type\",\n    \"signature_header\": headers.\"x-hub-signature-256\",\n    \"provider_event_id\": headers.\"x-github-delivery\"\n  },\n  \"delivery_status\": \"received\",\n  \"audit_metadata\": {\n    \"hookbase_source_id\": \"src_github_prod\",\n    \"hookbase_org_id\": \"{orgId}\",\n    \"ingested_at\": $now(),\n    \"payload_size_bytes\": $length($string($)),\n    \"encryption_applied\": true,\n    \"repository\": repository.full_name,\n    \"sender\": sender.login\n  }\n}"
    }
  }'

Stripe Audit Transform

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/transforms \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Stripe → Audit Record",
    "type": "jsonata",
    "config": {
      "expression": "{\n  \"event_id\": \"audit_\" & $string($millis()) & \"_st_\" & id,\n  \"source\": \"stripe\",\n  \"source_event_type\": type,\n  \"timestamp\": $fromMillis(created * 1000),\n  \"payload_hash\": $hash($string($), \"sha256\"),\n  \"headers\": {\n    \"content_type\": headers.\"content-type\",\n    \"signature_header\": headers.\"stripe-signature\",\n    \"provider_event_id\": id\n  },\n  \"delivery_status\": \"received\",\n  \"audit_metadata\": {\n    \"hookbase_source_id\": \"src_stripe_prod\",\n    \"hookbase_org_id\": \"{orgId}\",\n    \"ingested_at\": $now(),\n    \"payload_size_bytes\": $length($string($)),\n    \"encryption_applied\": true,\n    \"stripe_api_version\": api_version,\n    \"resource_type\": data.object.object,\n    \"resource_id\": data.object.id\n  }\n}"
    }
  }'

Shopify Audit Transform

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/transforms \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Shopify → Audit Record",
    "type": "jsonata",
    "config": {
      "expression": "{\n  \"event_id\": \"audit_\" & $string($millis()) & \"_sh_\" & headers.\"x-shopify-webhook-id\",\n  \"source\": \"shopify\",\n  \"source_event_type\": headers.\"x-shopify-topic\",\n  \"timestamp\": created_at ? created_at : $now(),\n  \"payload_hash\": $hash($string($), \"sha256\"),\n  \"headers\": {\n    \"content_type\": headers.\"content-type\",\n    \"signature_header\": headers.\"x-shopify-hmac-sha256\",\n    \"provider_event_id\": headers.\"x-shopify-webhook-id\"\n  },\n  \"delivery_status\": \"received\",\n  \"audit_metadata\": {\n    \"hookbase_source_id\": \"src_shopify_prod\",\n    \"hookbase_org_id\": \"{orgId}\",\n    \"ingested_at\": $now(),\n    \"payload_size_bytes\": $length($string($)),\n    \"encryption_applied\": true,\n    \"shop_domain\": headers.\"x-shopify-shop-domain\",\n    \"shopify_api_version\": headers.\"x-shopify-api-version\",\n    \"resource_id\": $string(id)\n  }\n}"
    }
  }'

Step 6: Create Audit Routes

For each source, create an audit route alongside the existing operational routes. These routes all share the same filter and data warehouse destination but use provider-specific transforms.

GitHub Audit Route

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/routes \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GitHub → Audit Log",
    "sourceId": "src_github_prod",
    "destinationId": "dst_warehouse01",
    "filterId": "flt_audit_skip_test",
    "transformId": "tfm_gh_audit01",
    "enabled": true
  }'

Stripe Audit Route

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/routes \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Stripe → Audit Log",
    "sourceId": "src_stripe_prod",
    "destinationId": "dst_warehouse01",
    "filterId": "flt_audit_skip_test",
    "transformId": "tfm_st_audit01",
    "enabled": true
  }'

Shopify Audit Route

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/routes \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Shopify → Audit Log",
    "sourceId": "src_shopify_prod",
    "destinationId": "dst_warehouse01",
    "filterId": "flt_audit_skip_test",
    "transformId": "tfm_sh_audit01",
    "enabled": true
  }'

Step 7: Test the Pipeline

Send a test event and verify the audit record is produced correctly.

Send a Test Stripe Event

bash

curl -X POST https://api.hookbase.app/ingest/your-org/stripe-prod \
  -H "Content-Type: application/json" \
  -H "Stripe-Signature: t=1234567890,v1=test_signature" \
  -d '{
    "id": "evt_audit_test_001",
    "type": "invoice.paid",
    "api_version": "2024-11-20.acacia",
    "created": 1707649200,
    "data": {
      "object": {
        "id": "in_test_audit_001",
        "object": "invoice",
        "customer": "cus_abc123",
        "customer_email": "[email protected]",
        "amount_paid": 4999,
        "currency": "usd",
        "status": "paid",
        "billing_details": {
          "email": "[email protected]",
          "name": "Jane Smith",
          "phone": "+1-555-0123"
        }
      }
    }
  }'

Expected Audit Record:

json

{
  "event_id": "audit_1707649200000_st_evt_audit_test_001",
  "source": "stripe",
  "source_event_type": "invoice.paid",
  "timestamp": "2026-02-11T14:00:00Z",
  "payload_hash": "a1b2c3d4e5f6...sha256hash",
  "headers": {
    "content_type": "application/json",
    "signature_header": "t=1234567890,v1=test_signature",
    "provider_event_id": "evt_audit_test_001"
  },
  "delivery_status": "received",
  "audit_metadata": {
    "hookbase_source_id": "src_stripe_prod",
    "hookbase_org_id": "{orgId}",
    "ingested_at": "2026-02-11T14:00:00Z",
    "payload_size_bytes": 412,
    "encryption_applied": true,
    "stripe_api_version": "2024-11-20.acacia",
    "resource_type": "invoice",
    "resource_id": "in_test_audit_001"
  }
}

Verify the Test Event is Filtered

Send an event with the test header to confirm it is skipped:

bash

curl -X POST https://api.hookbase.app/ingest/your-org/stripe-prod \
  -H "Content-Type: application/json" \
  -H "Stripe-Signature: t=1234567890,v1=test_signature" \
  -H "X-Hookbase-Test: true" \
  -d '{
    "id": "evt_should_be_filtered",
    "type": "charge.succeeded",
    "created": 1707649200,
    "data": {
      "object": {
        "id": "ch_test",
        "amount": 100,
        "currency": "usd"
      }
    }
  }'

This event should not appear in the data warehouse (filtered out by the x-hookbase-test header condition), but will still be delivered to any operational routes that do not use the audit filter.

Check Deliveries:

bash

curl https://api.hookbase.app/api/organizations/{orgId}/deliveries?limit=10 \
  -H "Authorization: Bearer YOUR_TOKEN"

Compliance Tips

Data Retention

Configure retention policies to meet your compliance requirements:

bash

curl -X PATCH https://api.hookbase.app/api/organizations/{orgId} \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "retentionConfig": {
      "eventRetentionDays": 90,
      "deliveryRetentionDays": 90,
      "auditLogRetentionDays": 365
    }
  }'

TIP

Even if you purge events from Hookbase after 90 days, the audit records in your data warehouse provide the long-term compliance trail. Set your data warehouse retention independently based on your regulatory requirements (GDPR typically requires ability to delete on request; SOC 2 and PCI-DSS recommend 1+ years of audit logs).

Non-Repudiation

The audit record preserves the original webhook signature in headers.signature_header. This provides cryptographic proof that the event was actually sent by the claimed provider and was not tampered with. Store the original signing secrets securely so you can verify signatures after the fact if needed.

Immutability

Configure your data warehouse table or bucket as append-only to prevent modification of audit records:

BigQuery: Use time-partitioned tables with require_partition_filter and restrict UPDATE/DELETE permissions
Snowflake: Use Time Travel with a long retention window and restrict TRUNCATE/DELETE grants
S3/R2: Enable Object Lock with compliance mode for write-once storage

Access Control

Use scoped API keys for the audit pipeline to minimize blast radius:

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/api-keys \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Audit Pipeline Service Account",
    "scopes": ["events:read", "deliveries:read"],
    "expiresAt": "2027-02-11T00:00:00Z"
  }'

Regular Auditing with Hookbase Audit Logs

Use Hookbase's built-in audit logs to track who changed pipeline configuration:

bash

curl https://api.hookbase.app/api/organizations/{orgId}/audit-logs?limit=50 \
  -H "Authorization: Bearer YOUR_TOKEN"

This helps answer questions like:

Who modified the audit filter?
When was a source encryption config changed?
Which user disabled an audit route?

Production Considerations

1. Circuit Breaker for Data Warehouse

Protect against data warehouse downtime or rate limiting:

bash

curl -X PATCH https://api.hookbase.app/api/organizations/{orgId}/destinations/dst_warehouse01 \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "circuitBreakerConfig": {
      "enabled": true,
      "failureThreshold": 10,
      "windowSeconds": 60,
      "resetTimeoutSeconds": 300
    }
  }'

2. Failover Destination

Set up a failover to a secondary storage location (e.g., an R2 bucket or S3 bucket) to ensure no audit records are lost:

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/destinations \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Audit Log Failover (S3)",
    "type": "http",
    "config": {
      "url": "https://YOUR_LAMBDA.execute-api.us-east-1.amazonaws.com/prod/audit-logs",
      "method": "POST",
      "headers": {
        "X-API-Key": "YOUR_FAILOVER_API_KEY",
        "Content-Type": "application/json"
      }
    }
  }'

Apply failover to each audit route:

bash

curl -X PATCH https://api.hookbase.app/api/organizations/{orgId}/routes/rte_gh_audit01 \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "failoverConfig": {
      "enabled": true,
      "destinationId": "dst_warehouse_failover",
      "triggerAfterAttempts": 3
    }
  }'

3. Delivery Failure Notifications

Get alerted immediately when audit log delivery fails:

bash

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/notification-channels \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Audit Pipeline Alerts",
    "type": "slack",
    "config": {
      "url": "https://hooks.slack.com/services/YOUR/COMPLIANCE/WEBHOOK"
    },
    "events": ["delivery.failed", "destination.circuit_breaker.opened"],
    "filters": {
      "destinationIds": ["dst_warehouse01"]
    }
  }'

Field Encryption - Encrypt PII fields before storage
Audit Logs - Track configuration changes in Hookbase
Transforms - JSONata syntax and advanced examples
Filters - Skip test events and route based on content
Failover - Ensure no audit records are lost
Circuit Breaker - Protect data warehouse endpoints
Notification Channels - Get alerted to pipeline failures
Plans & Limits - Data retention and storage limits per plan

Webhook Logging & Audit Trail ​

Architecture ​

Step 1: Create Sources ​

GitHub Source ​

Stripe Source ​

Shopify Source ​

Step 2: Create the Data Warehouse Destination ​

BigQuery (via Streaming Insert API) ​

Snowflake (via Snowpipe REST API) ​

Step 3: Create the Audit Filter ​

Step 4: Enable Field Encryption ​

Step 5: Create Audit Transforms ​

Audit Record Schema ​

GitHub Audit Transform ​

Stripe Audit Transform ​

Shopify Audit Transform ​

Step 6: Create Audit Routes ​

GitHub Audit Route ​

Stripe Audit Route ​

Shopify Audit Route ​

Step 7: Test the Pipeline ​

Send a Test Stripe Event ​

Verify the Test Event is Filtered ​

Compliance Tips ​

Data Retention ​

Non-Repudiation ​

Immutability ​

Access Control ​

Regular Auditing with Hookbase Audit Logs ​

Production Considerations ​

1. Circuit Breaker for Data Warehouse ​

2. Failover Destination ​

3. Delivery Failure Notifications ​

Related Guides ​

Webhook Logging & Audit Trail

Architecture

Step 1: Create Sources

GitHub Source

Stripe Source

Shopify Source

Step 2: Create the Data Warehouse Destination

BigQuery (via Streaming Insert API)

Snowflake (via Snowpipe REST API)

Step 3: Create the Audit Filter

Step 4: Enable Field Encryption

Step 5: Create Audit Transforms

Audit Record Schema

GitHub Audit Transform

Stripe Audit Transform

Shopify Audit Transform

Step 6: Create Audit Routes

GitHub Audit Route

Stripe Audit Route

Shopify Audit Route

Step 7: Test the Pipeline

Send a Test Stripe Event

Verify the Test Event is Filtered

Compliance Tips

Data Retention

Non-Repudiation

Immutability

Access Control

Regular Auditing with Hookbase Audit Logs

Production Considerations

1. Circuit Breaker for Data Warehouse

2. Failover Destination

3. Delivery Failure Notifications

Related Guides