Background Execution & Job Queue

Relevant source files

• apps/sim/app/api/auth/oauth/disconnect/route.test.ts
• apps/sim/app/api/files/multipart/route.ts
• apps/sim/app/api/files/presigned/batch/route.ts
• apps/sim/app/api/knowledge/[id]/documents/[documentId]/chunks/route.ts
• apps/sim/app/api/knowledge/[id]/documents/[documentId]/route.test.ts
• apps/sim/app/api/knowledge/[id]/documents/[documentId]/route.ts
• apps/sim/app/api/knowledge/[id]/documents/route.test.ts
• apps/sim/app/api/knowledge/[id]/documents/route.ts
• apps/sim/app/api/knowledge/[id]/documents/upsert/route.ts
• apps/sim/app/api/knowledge/[id]/route.test.ts
• apps/sim/app/api/knowledge/[id]/route.ts
• apps/sim/app/api/knowledge/route.test.ts
• apps/sim/app/api/knowledge/route.ts
• apps/sim/app/api/knowledge/search/route.test.ts
• apps/sim/app/api/knowledge/search/route.ts
• apps/sim/app/api/knowledge/search/utils.test.ts
• apps/sim/app/api/knowledge/search/utils.ts
• apps/sim/app/api/knowledge/utils.test.ts
• apps/sim/app/api/knowledge/utils.ts
• apps/sim/app/api/schedules/execute/route.test.ts
• apps/sim/app/api/schedules/execute/route.ts
• apps/sim/app/api/webhooks/trigger/[path]/route.test.ts
• apps/sim/app/api/webhooks/trigger/[path]/route.ts
• apps/sim/app/api/workflows/[id]/execute/route.ts
• apps/sim/app/workspace/[workspaceId]/knowledge/hooks/use-knowledge-upload.ts
• apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/terminal/utils.ts
• apps/sim/app/workspace/[workspaceId]/w/[workflowId]/hooks/use-workflow-execution.ts
• apps/sim/app/workspace/[workspaceId]/w/[workflowId]/utils/workflow-execution-utils.ts
• apps/sim/background/schedule-execution.ts
• apps/sim/background/webhook-execution.ts
• apps/sim/background/workflow-execution.ts
• apps/sim/executor/execution/block-executor.ts
• apps/sim/executor/execution/executor.ts
• apps/sim/executor/execution/types.ts
• apps/sim/executor/handlers/workflow/workflow-handler.test.ts
• apps/sim/executor/handlers/workflow/workflow-handler.ts
• apps/sim/executor/index.ts
• apps/sim/executor/types.ts
• apps/sim/hooks/use-execution-stream.ts
• apps/sim/lib/chunkers/json-yaml-chunker.ts
• apps/sim/lib/knowledge/chunks/service.ts
• apps/sim/lib/knowledge/chunks/types.ts
• apps/sim/lib/knowledge/service.ts
• apps/sim/lib/knowledge/types.ts
• apps/sim/lib/logs/execution/trace-spans/trace-spans.test.ts
• apps/sim/lib/logs/execution/trace-spans/trace-spans.ts
• apps/sim/lib/messaging/email/mailer.test.ts
• apps/sim/lib/messaging/email/unsubscribe.test.ts
• apps/sim/lib/tokenization/constants.ts
• apps/sim/lib/tokenization/estimators.ts
• apps/sim/lib/tokenization/index.ts
• apps/sim/lib/webhooks/processor.ts
• apps/sim/lib/workflows/executor/execution-core.ts
• apps/sim/lib/workflows/executor/execution-events.ts
• apps/sim/stores/terminal/console/store.ts
• apps/sim/stores/terminal/console/types.ts
• apps/sim/tools/knowledge/create_document.ts
• apps/sim/tools/knowledge/index.ts
• apps/sim/tools/knowledge/search.ts
• apps/sim/tools/knowledge/types.ts
• apps/sim/tools/knowledge/upload_chunk.ts
• apps/sim/tools/knowledge/upsert_document.ts
• apps/sim/trigger.config.ts
• apps/sim/vitest.setup.ts
• helm/sim/templates/cronjobs.yaml

Purpose & Scope

This document describes the implementation of background execution and asynchronous job processing within Sim Studio. The system leverages Trigger.dev for job orchestration, enabling long-running workflows to execute outside the constraints of HTTP request/response cycles. This architecture is essential for handling webhooks, scheduled tasks, and large-scale AI agent workflows that may exceed standard serverless or edge function timeouts.

---

System Architecture

Sim Studio uses a distributed background job architecture. When a workflow execution is triggered asynchronously (either via a specific header, a webhook, or a schedule), the system offloads the processing to a background worker.

Execution Modes

1. Synchronous (SSE): Real-time streaming for interactive UI sessions using useExecutionStream. apps/sim/app/workspace/[workspaceId]/w/[workflowId]/hooks/use-workflow-execution.ts:157-157
2. Asynchronous (Background): Managed by Trigger.dev for reliable, non-interactive execution. apps/sim/background/workflow-execution.ts136-165
3. Inline (BullMQ/Internal): Direct execution using BullMQ or local workers for specific internal tasks. apps/sim/app/api/workflows/[id]/execute/route.ts:7-8

Key Components

• Trigger.dev SDK: Used to define tasks (task) and manage background execution lifecycles. apps/sim/background/workflow-execution.ts4
• Job Processors: Dedicated handlers for different trigger sources (workflow-execution, webhook-execution, schedule-execution). apps/sim/background/webhook-execution.ts136 apps/sim/background/schedule-execution.ts3
• Idempotency Service: Prevents duplicate processing of the same webhook payload using webhookIdempotency. apps/sim/background/webhook-execution.ts149-164
• Logging Session: Manages trace spans and execution logs during background runs. apps/sim/background/webhook-execution.ts188-193

Sources: apps/sim/app/api/workflows/[id]/execute/route.ts:7-8, apps/sim/background/workflow-execution.ts4 apps/sim/background/webhook-execution.ts149-164 apps/sim/background/schedule-execution.ts3

---

Trigger.dev Integration

The integration is centralized in the apps/sim/background directory. Each file defines a Trigger.dev task that wraps the core execution logic. Configuration for the Trigger.dev build process, including external packages, is managed in trigger.config.ts.

Natural Language to Code Entity Mapping (Trigger.dev)

Concept	Code Entity	File Path
Job Definition	task()	apps/sim/background/workflow-execution.ts4
Webhook Task	executeWebhookJob	apps/sim/background/webhook-execution.ts136
Schedule Task	TriggerScheduleExecution	apps/sim/background/schedule-execution.ts35
Job Correlation	AsyncExecutionCorrelation	apps/sim/background/workflow-execution.ts7

Trigger.dev Task Flow

The following diagram illustrates how an external trigger (Webhook) moves from an API request into a background task.

Sources: apps/sim/app/api/webhooks/trigger/[path]/route.ts:161-168, apps/sim/background/webhook-execution.ts183-208 apps/sim/lib/workflows/executor/execution-core.ts218-225 apps/sim/lib/webhooks/processor.ts38

---

Webhook Processing & Queueing

Webhooks are the primary source of background jobs. The system includes a pipeline for validation, challenge handling, and fan-out to multiple workflows.

The Processing Pipeline

1. Challenge Handling: Responds to provider-specific verification (e.g., Slack, Microsoft Graph, WhatsApp). apps/sim/lib/webhooks/processor.ts158-191
2. Signature Verification: Validates HMAC or provider-specific signatures using verifyProviderAuth. apps/sim/app/api/webhooks/trigger/[path]/route.ts:107-113
3. Fan-out: If multiple workflows share a webhook path (Credential Sets), findAllWebhooksForPath iterates through all matching records. apps/sim/app/api/webhooks/trigger/[path]/route.ts:85-100
4. Idempotency Check: Uses IdempotencyService.createWebhookIdempotencyKey to generate a hash of the provider, headers, and body. apps/sim/background/webhook-execution.ts149-154

Natural Language to Code Entity Mapping (Webhooks)

Concept	Code Entity	File Path
Body Parsing	parseWebhookBody	apps/sim/lib/webhooks/processor.ts106-156
Challenge Logic	handleProviderChallenges	apps/sim/lib/webhooks/processor.ts158
Verification	verifyProviderWebhook	apps/sim/lib/webhooks/processor.ts36
Attachment Handling	WebhookAttachmentProcessor	apps/sim/background/webhook-execution.ts14

Sources: apps/sim/lib/webhooks/processor.ts106-156 apps/sim/app/api/webhooks/trigger/[path]/route.ts:85-100, apps/sim/background/webhook-execution.ts149-154 apps/sim/lib/webhooks/processor.ts158-191

---

Scheduled Executions

Schedules are managed via a cron-based polling mechanism that dispatches jobs to Trigger.dev.

Workflow Schedule Lifecycle

1. Polling: External triggers hit the schedule execution endpoint.
2. Due Selection: Queries workflowSchedule for records requiring execution. apps/sim/background/schedule-execution.ts1
3. Locking: Atomically updates lastQueuedAt via applyScheduleUpdate to prevent duplicate dispatch. apps/sim/background/schedule-execution.ts63-77
4. Job Enqueue: Dispatches the ScheduleExecutionPayload to the background worker. apps/sim/background/schedule-execution.ts47
5. Failure Tracking: Tracks consecutive failures. If they exceed MAX_CONSECUTIVE_FAILURES (3), the schedule is disabled. apps/sim/background/schedule-execution.ts33

Sources: apps/sim/background/schedule-execution.ts63-77 apps/sim/background/schedule-execution.ts79-96 apps/sim/background/schedule-execution.ts139-157 apps/sim/background/schedule-execution.ts33

---

Preprocessing & Security

Before any background job executes the workflow core, it must pass through preprocessExecution. This ensures that background jobs respect the same security and billing constraints as synchronous ones. apps/sim/background/webhook-execution.ts196-208

Preprocessing Steps

• Deployment Resolution: Loads the specific workflowDeploymentVersion for execution. apps/sim/background/webhook-execution.ts21
• Billing Validation: Resolves the actorUserId for credit consumption. apps/sim/background/webhook-execution.ts198
• Timeout Resolution: Sets execution limits using createTimeoutAbortController. apps/sim/background/webhook-execution.ts8
• File Hydration: Processes attachments and base64 data for block inputs. apps/sim/background/webhook-execution.ts54-65

Sources: apps/sim/background/webhook-execution.ts196-208 apps/sim/background/webhook-execution.ts54-65 apps/sim/background/webhook-execution.ts8

---

Execution Correlation & Logging

To provide a unified view in the "Terminal" and "Logs" UI, background jobs use a correlation object. apps/sim/background/webhook-execution.ts32-48

AsyncExecutionCorrelation

The LoggingSession class uses this correlation to aggregate logs across different stages of the background job (preprocessing, execution, and finalization). apps/sim/background/webhook-execution.ts188-193

Sources: apps/sim/background/webhook-execution.ts32-48 apps/sim/background/schedule-execution.ts46-61 apps/sim/background/webhook-execution.ts188-193