Webhook deduplication prevents duplicate workflow executions when external providers retry webhook deliveries or send duplicate events. The system uses an idempotency service to identify and skip redundant webhook requests, ensuring each unique event executes exactly once even if the provider sends it multiple times.
External webhook providers often retry failed deliveries or send duplicate notifications due to network issues, timeout constraints, or platform-specific behaviors. Without deduplication, each retry would trigger a new workflow execution, leading to duplicate data processing and inconsistent state.
The system addresses this through the IdempotencyService, which manages request hashes and state using either Redis or PostgreSQL storage.
Sources: apps/sim/lib/core/idempotency/service.ts45-52 apps/sim/triggers/generic/webhook.ts53-61
The deduplication logic is integrated into the webhook processing pipeline. It can be triggered either automatically by provider-specific identifiers or manually via a user-configured deduplication field in the webhook trigger settings.
Deduplication Architecture Diagram
Sources: apps/sim/lib/core/idempotency/service.ts53-66 apps/sim/app/api/webhooks/route.ts177-200
The IdempotencyService is a universal utility used to prevent duplicate operations. It supports two storage backends, determined by the environment configuration:
REDIS_URL is set, providing high-performance atomic operations via SET NX.idempotency_key table with onConflictDoNothing.The service generates a normalizedKey by combining a namespace, provider, and a unique identifier.
Sources: apps/sim/lib/core/idempotency/service.ts53-82 packages/db/migrations/meta/0147_snapshot.json442-475
For generic webhooks, users can define a Deduplication Field. This is a dot-notation path to a unique field within the incoming JSON payload (e.g., event.id or order_number).
The generic_webhook trigger includes an idempotencyField sub-block. If configured, the system extracts the value at this path from the webhook body and uses it as the identifier for the IdempotencyService.
| Sub-Block ID | Type | Description |
|---|---|---|
idempotencyField | short-input | Dot-notation path to the unique identifier in the payload. |
Sources: apps/sim/triggers/generic/webhook.ts53-61 apps/sim/blocks/blocks/generic_webhook.ts21
To prevent race conditions where two identical requests arrive simultaneously, the system uses an "Atomic Claim" pattern.
Atomic Claim Sequence Diagram
Sources: apps/sim/lib/core/idempotency/service.ts159-213 apps/sim/lib/core/idempotency/service.ts215-235
Idempotency keys are not stored indefinitely. By default, the system retains keys for 7 days. A dedicated cron endpoint handles the removal of expired keys to maintain database performance.
GET /api/webhooks/cleanup/idempotency.cleanupExpiredIdempotencyKeys performs batch deletion of records where the createdAt timestamp exceeds the TTL.Sources: apps/sim/lib/core/idempotency/service.ts40 apps/sim/app/api/webhooks/cleanup/idempotency/route.ts12-30 apps/sim/lib/core/idempotency/service.ts145-149
| Class/Function | File | Role |
|---|---|---|
IdempotencyService | apps/sim/lib/core/idempotency/service.ts53-66 | Core logic for checking and claiming idempotency keys. |
atomicallyClaimRedis | apps/sim/lib/core/idempotency/service.ts177-213 | Uses Redis SET NX for distributed atomic locking. |
atomicallyClaimDb | apps/sim/lib/core/idempotency/service.ts215-235 | Uses PostgreSQL ON CONFLICT DO NOTHING for atomic locking. |
genericWebhookTrigger | apps/sim/triggers/generic/webhook.ts4-151 | Defines the UI and schema for user-defined deduplication fields. |
idempotencyKey | packages/db/migrations/meta/0147_snapshot.json442-475 | Database schema for persisting idempotency state. |
Sources: apps/sim/lib/core/idempotency/service.ts53-235 apps/sim/triggers/generic/webhook.ts4-151