Webhook Processing Pipeline

Relevant source files

• apps/sim/app/api/cron/renew-subscriptions/route.ts
• apps/sim/app/api/schedules/execute/route.test.ts
• apps/sim/app/api/schedules/execute/route.ts
• apps/sim/app/api/webhooks/[id]/route.ts
• apps/sim/app/api/webhooks/cleanup/idempotency/route.ts
• apps/sim/app/api/webhooks/route.ts
• apps/sim/app/api/webhooks/trigger/[path]/route.test.ts
• apps/sim/app/api/webhooks/trigger/[path]/route.ts
• apps/sim/app/api/workflows/[id]/execute/route.ts
• apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/terminal/utils.ts
• apps/sim/app/workspace/[workspaceId]/w/[workflowId]/hooks/use-workflow-execution.ts
• apps/sim/app/workspace/[workspaceId]/w/[workflowId]/utils/workflow-execution-utils.ts
• apps/sim/background/schedule-execution.ts
• apps/sim/background/webhook-execution.ts
• apps/sim/background/workflow-execution.ts
• apps/sim/executor/execution/block-executor.ts
• apps/sim/executor/execution/executor.ts
• apps/sim/executor/execution/types.ts
• apps/sim/executor/handlers/workflow/workflow-handler.test.ts
• apps/sim/executor/handlers/workflow/workflow-handler.ts
• apps/sim/executor/index.ts
• apps/sim/executor/types.ts
• apps/sim/hooks/use-execution-stream.ts
• apps/sim/lib/logs/execution/trace-spans/trace-spans.test.ts
• apps/sim/lib/logs/execution/trace-spans/trace-spans.ts
• apps/sim/lib/webhooks/deploy.ts
• apps/sim/lib/webhooks/processor.ts
• apps/sim/lib/webhooks/utils.ts
• apps/sim/lib/workflows/executor/execution-core.ts
• apps/sim/lib/workflows/executor/execution-events.ts
• apps/sim/stores/terminal/console/store.ts
• apps/sim/stores/terminal/console/types.ts
• helm/sim/templates/cronjobs.yaml

Purpose and Scope

This document describes the complete request flow for incoming webhook requests, from initial HTTP reception through validation, authentication, and job queueing. The webhook processing pipeline handles all external webhook events that trigger workflows in Sim, implementing provider-specific verification, rate limiting, billing checks, and safe background execution dispatch.

For webhook authentication mechanisms (signature verification, HMAC validation), see 13.2 Webhook Authentication. For provider subscription lifecycle management (Teams, Telegram, Airtable), see 13.3 Provider Subscriptions. For idempotency and duplicate prevention, see 13.4 Webhook Deduplication.

---

Architecture Overview

The webhook processing pipeline is a multi-stage funnel that validates, authenticates, and dispatches incoming webhook requests. The system handles URL verification challenges, signature validation, rate limiting, and credential set fan-out before queuing execution jobs.

Diagram 1: Webhook Processing Pipeline Overview

Sources: apps/sim/app/api/webhooks/trigger/[path]/route.ts:42-182, apps/sim/lib/webhooks/processor.ts106-191

---

Request Entry Point

API Route Handler

The webhook API route is the single entry point for all external webhook requests. The route implementation is split between GET and POST handlers:

Method	Purpose	Handler Location
GET	URL verification challenges (Microsoft Graph, WhatsApp)	apps/sim/app/api/webhooks/trigger/[path]/route.ts:26-40
POST	Webhook payload processing and execution dispatch	apps/sim/app/api/webhooks/trigger/[path]/route.ts:42-200

Both handlers generate a unique requestId via generateRequestId() for request tracing throughout the pipeline apps/sim/app/api/webhooks/trigger/[path]/route.ts:27,62.

Diagram 2: Route Method Handling

Sources: apps/sim/app/api/webhooks/trigger/[path]/route.ts:26-200, apps/sim/lib/core/utils/request.ts1-10

---

Processing Phases

Phase 1: Body Parsing and Early Challenges

The first phase parses the request body and handles provider verification challenges.

Body Parsing (apps/sim/lib/webhooks/processor.ts106-156)

• Clones request to read body as text (preserves original for signature verification) apps/sim/lib/webhooks/processor.ts112-113
• Detects content type: application/json or application/x-www-form-urlencoded apps/sim/lib/webhooks/processor.ts128-130
• For form data, checks for payload parameter containing JSON string (common in Slack) apps/sim/lib/webhooks/processor.ts132-135

Challenge Handling (apps/sim/lib/webhooks/processor.ts158-191)

• Slack: Returns { challenge: body.challenge } if body.type === 'url_verification' via handleSlackChallenge apps/sim/lib/webhooks/processor.ts164-167
• Microsoft Graph: Echoes validationToken query parameter as plain text apps/sim/lib/webhooks/processor.ts172-179
• WhatsApp: Validates hub.verify_token and returns hub.challenge via handleWhatsAppVerification() apps/sim/lib/webhooks/processor.ts181-188

Diagram 3: Body Parsing and Challenge Handling

Sources: apps/sim/lib/webhooks/processor.ts106-191 apps/sim/lib/webhooks/utils.server.ts30-102

---

Phase 2: Webhook Lookup and Credential Set Fan-out

The lookup phase retrieves all webhooks matching the request path. This supports credential set fan-out, where multiple webhooks share the same path but use different credentials for billing separation.

Lookup Implementation (apps/sim/lib/webhooks/processor.ts225-266) The findAllWebhooksForPath() function joins webhook, workflow, and workflowDeploymentVersion tables:

• webhook.path matches request path apps/sim/lib/webhooks/processor.ts244
• webhook.isActive is true apps/sim/lib/webhooks/processor.ts245
• webhook.deploymentVersionId matches active deployment apps/sim/lib/webhooks/processor.ts247-250

Fan-out Handling (apps/sim/app/api/webhooks/trigger/[path]/route.ts:103-200) When multiple webhooks are found:

1. Process each webhook independently in a loop apps/sim/app/api/webhooks/trigger/[path]/route.ts:106.
2. Verify authentication for each credential individually via verifyProviderAuth() apps/sim/app/api/webhooks/trigger/[path]/route.ts:107-113.
3. Skip failed webhooks, continuing to the next one apps/sim/app/api/webhooks/trigger/[path]/route.ts:115-118.
4. Return a combined success response if multiple succeeded apps/sim/app/api/webhooks/trigger/[path]/route.ts:193-199.

Sources: apps/sim/lib/webhooks/processor.ts225-266 apps/sim/app/api/webhooks/trigger/[path]/route.ts:103-200

---

Phase 3: Authentication and Signature Verification

Authentication verifies the webhook request originated from the claimed provider. The implementation supports multiple signature schemes via verifyProviderAuth() apps/sim/lib/webhooks/processor.ts303-676

Environment Variable Resolution (apps/sim/lib/webhooks/processor.ts274-297) Before verification, the system resolves {{VARIABLE}} references in providerConfig:

1. Fetch workspace environment variables (decrypted) via getEffectiveDecryptedEnv() apps/sim/lib/webhooks/processor.ts278
2. Call resolveEnvVarReferences() to substitute {{VAR}} with actual values apps/sim/lib/webhooks/processor.ts288
3. Use resolved config for signature validation.

Authentication Methods:

Provider	Signature Header	Algorithm	Config Key
Microsoft Teams	Authorization: HMAC ...	HMAC-SHA256	hmacSecret
Twilio Voice	X-Twilio-Signature	HMAC-SHA1	authToken
GitHub	X-Hub-Signature-256	HMAC-SHA256	webhookSecret
Typeform	Typeform-Signature	HMAC-SHA256	secret
Linear	Linear-Signature	HMAC-SHA256	webhookSecret

Sources: apps/sim/lib/webhooks/processor.ts274-676 apps/sim/lib/webhooks/utils.server.ts483-538 apps/sim/executor/utils/reference-validation.ts37

---

Phase 4: Preprocessing - Rate Limits and Billing

Preprocessing validates that the webhook execution is authorized and within limits. This phase delegates to the central preprocessExecution() function apps/sim/lib/execution/preprocessing.ts74-450

Preprocessing Checks (apps/sim/lib/webhooks/processor.ts682-745)

1. Actor Resolution: Determine billing user (workspace billed account or credential owner).
2. Rate Limiting: Check per-user or per-workspace request quota via RateLimiter apps/sim/lib/execution/preprocessing.ts255-291
3. Billing Validation: Verify active subscription via checkServerSideUsageLimits() apps/sim/lib/execution/preprocessing.ts322-376
4. Deployment Status: Confirm workflow is deployed apps/sim/lib/execution/preprocessing.ts207-252

Sources: apps/sim/lib/webhooks/processor.ts682-745 apps/sim/lib/execution/preprocessing.ts1-450

---

Phase 5: Background Job Queueing

The final phase queues the webhook execution as a background job.

Payload Construction (apps/sim/lib/webhooks/processor.ts913-924) The system builds a WebhookExecutionPayload containing the sanitized headers, body, and correlation IDs.

Execution Strategy (apps/sim/lib/webhooks/processor.ts959-1024)

• Production: Dispatch to Trigger.dev using tasks.trigger('webhook-execution') apps/sim/lib/webhooks/processor.ts983-999
• Development: shouldExecuteInline() triggers executeWebhookJob() directly in the request thread for faster debugging apps/sim/lib/webhooks/processor.ts967-981

Diagram 4: Job Queueing Flow

Sources: apps/sim/lib/webhooks/processor.ts747-1191 apps/sim/lib/core/async-jobs/index.ts7-9 apps/sim/background/webhook-execution.ts136-165

---

Background Execution Job

The background execution job (webhookExecution task) runs the workflow in an isolated worker.

Job Flow (apps/sim/background/webhook-execution.ts136-165)

1. Idempotency Check: Verify request hasn't been processed using webhookIdempotency.executeWithIdempotency() apps/sim/background/webhook-execution.ts160-164
2. Preprocessing: Re-run billing and timeout resolution via preprocessExecution() apps/sim/background/webhook-execution.ts196-209
3. Provider-Specific Processing:
   • Airtable: Fetch actual record changes via fetchAndProcessAirtablePayloads() apps/sim/background/webhook-execution.ts316-324
   • Teams: Fetch full message and attachments via formatTeamsGraphNotification() apps/sim/lib/webhooks/utils.server.ts148-481
4. File Processing: Upload attachments to blob storage using WebhookAttachmentProcessor.processAttachments() apps/sim/background/webhook-execution.ts80-90
5. Workflow Execution: Run executeWorkflowCore() with DAG executor apps/sim/background/webhook-execution.ts405-412

Sources: apps/sim/background/webhook-execution.ts1-663 apps/sim/lib/webhooks/utils.server.ts148-481 apps/sim/lib/workflows/executor/execution-core.ts17-19

---

Key Components Reference

Core Functions

Function	Location	Purpose
parseWebhookBody()	apps/sim/lib/webhooks/processor.ts106-156	Parse JSON/form-encoded body, clone for signature verification.
handleProviderChallenges()	apps/sim/lib/webhooks/processor.ts158-191	Handle URL verification challenges (Slack, Graph, WhatsApp).
findAllWebhooksForPath()	apps/sim/lib/webhooks/processor.ts225-266	Query webhooks by path, support credential set fan-out.
verifyProviderAuth()	apps/sim/lib/webhooks/processor.ts303-676	Verify HMAC signatures or token authentication.
checkWebhookPreprocessing()	apps/sim/lib/webhooks/processor.ts682-745	Run rate limit, billing, and deployment checks.
queueWebhookExecution()	apps/sim/lib/webhooks/processor.ts747-1191	Build payload and dispatch to Trigger.dev.
executeWebhookJob()	apps/sim/background/webhook-execution.ts136-165	Background job entry point with idempotency.
formatWebhookInput()	apps/sim/lib/webhooks/utils.server.ts744-1400	Transform provider payloads to trigger outputs.

Database Schema

The webhook processing pipeline interacts with these key tables:

Table	Purpose	Key Columns
webhook	Webhook registration records	id, path, provider, workflowId, providerConfig
workflow	Workflow metadata	id, userId, workspaceId, isDeployed
workflowDeploymentVersion	Deployment snapshots	id, workflowId, isActive
account	OAuth account credentials	id, userId, provider, accessToken

Sources: apps/sim/lib/webhooks/processor.ts1-1191 apps/sim/background/webhook-execution.ts1-663 apps/sim/lib/webhooks/utils.server.ts1-1500