Batch Processing

Relevant source files

• .cursor/rules/otel-metrics.mdc
• apps/webapp/app/env.server.ts
• apps/webapp/app/v3/runEngine.server.ts
• apps/webapp/app/v3/telemetry/compactMetricExporter.server.ts
• apps/webapp/app/v3/telemetry/loggerExporter.server.ts
• apps/webapp/app/v3/tracer.server.ts
• apps/webapp/test/utils/tracing.ts
• docker/config/grafana/provisioning/dashboards/runs-replication.json
• internal-packages/database/prisma/migrations/20250919123905_remove_unnecessary_foreign_key_constraints_task_run_execution_snapshot/migration.sql
• internal-packages/run-engine/src/engine/index.ts
• internal-packages/run-engine/src/engine/locking.ts
• internal-packages/run-engine/src/engine/statuses.ts
• internal-packages/run-engine/src/engine/systems/checkpointSystem.ts
• internal-packages/run-engine/src/engine/systems/delayedRunSystem.ts
• internal-packages/run-engine/src/engine/systems/dequeueSystem.ts
• internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
• internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
• internal-packages/run-engine/src/engine/tests/attemptFailures.test.ts
• internal-packages/run-engine/src/engine/tests/batchTriggerAndWait.test.ts
• internal-packages/run-engine/src/engine/tests/checkpoints.test.ts
• internal-packages/run-engine/src/engine/tests/getSnapshotsSince.test.ts
• internal-packages/run-engine/src/engine/tests/helpers/executionStateMachine.ts
• internal-packages/run-engine/src/engine/tests/helpers/snapshotTestHelpers.ts
• internal-packages/run-engine/src/engine/tests/locking.test.ts
• internal-packages/run-engine/src/engine/tests/waitpoints.test.ts
• internal-packages/run-engine/src/engine/types.ts
• internal-packages/run-engine/src/engine/workerCatalog.ts

This document describes the batch processing system in Trigger.dev's RunEngine, which enables fair, efficient processing of large sets of task runs through a 2-phase streaming API with Deficit Round Robin (DRR) scheduling. For information about triggering individual runs, see Task Triggering Services. For information about general queue management, see Queue Management.

Purpose and Architecture

The batch processing system provides a mechanism to trigger and process multiple task runs as a coordinated group. It implements a 2-phase streaming API that allows runs to be added incrementally to a batch, and uses DRR scheduling to ensure fair processing across multiple concurrent batches.

Core Components:

Component	File Location	Purpose
BatchQueue	internal-packages/run-engine/src/batch-queue/index.js	DRR-based queue managing batch item processing
BatchSystem	internal-packages/run-engine/src/engine/systems/batchSystem.ts	High-level batch operations (initialize, stream, complete)
WaitpointSystem	internal-packages/run-engine/src/engine/systems/waitpointSystem.ts	Handles BATCH type waitpoints for blocking parent runs
RunEngine	internal-packages/run-engine/src/engine/index.ts76-388	Orchestrates all batch-related components

The batch system is initialized in the RunEngine constructor and operates independently from the main run queue, with its own consumer pool and concurrency controls.

Sources: internal-packages/run-engine/src/engine/index.ts332-362 internal-packages/run-engine/src/engine/types.ts76-90

Batch Processing Flow

Sources: internal-packages/run-engine/src/engine/index.ts232-234 internal-packages/run-engine/src/engine/systems/batchSystem.ts327-330

2-Phase Streaming API

The batch processing API operates in two distinct phases to support incremental addition of items:

Phase 1: Initialize and Stream

1. Initialize Batch - Creates batch metadata without items
   • Generates unique batch ID
   • Creates associated BATCH waitpoint
   • Sets initial concurrency limits based on plan type
2. Stream Items - Adds items incrementally (1-by-1 or in chunks)
   • Items enqueued to DRR queue immediately
   • No limit on streaming duration
   • Supports up to STREAMING_BATCH_MAX_ITEMS items per batch

Phase 2: Finalize and Process

3. Complete Batch - Marks batch as ready for processing
   • Signals that no more items will be added
   • Enables DRR scheduler to fully process the batch
   • Triggers completion check when all items finish

The separation between streaming and finalization allows the system to begin processing items immediately while still accepting new items, improving overall throughput.

Sources: apps/webapp/app/env.server.ts542-548 internal-packages/run-engine/src/batch-queue/types.js

Deficit Round Robin (DRR) Scheduling

The batch processing system uses DRR scheduling to ensure fair resource allocation across multiple concurrent batches. This prevents large batches from monopolizing processing capacity at the expense of smaller batches.

DRR Parameters

Parameter	Environment Variable	Default	Purpose
Quantum	BATCH_QUEUE_DRR_QUANTUM	5	Items dequeued per batch per round
Max Deficit	BATCH_QUEUE_MAX_DEFICIT	50	Maximum accumulated deficit before reset
Master Queue Limit	BATCH_QUEUE_MASTER_QUEUE_LIMIT	Varies	Max batches in master queue

DRR Algorithm:

1. Each batch starts with deficit = 0
2. In each round, deficit += quantum
3. Dequeue min(deficit, available items) from batch
4. Subtract dequeued count from deficit
5. If deficit > maxDeficit, reset to 0
6. Move to next batch

This ensures:

• Small batches complete quickly without starvation
• Large batches get predictable processing rates
• Fair distribution of worker capacity

Sources: internal-packages/run-engine/src/engine/index.ts341-345 apps/webapp/app/v3/runEngine.server.ts171-175

Concurrency and Rate Limiting

The batch processing system implements multiple layers of concurrency control and rate limiting to prevent resource exhaustion:

Per-Batch Concurrency

Each batch has an independent concurrency limit set at batch creation time:

This limit controls how many items from a single batch can execute simultaneously.

Sources: apps/webapp/app/env.server.ts548 apps/webapp/app/v3/runEngine.server.ts185

Global Rate Limiting

An optional global rate limiter restricts processing across all consumers:

Setting	Environment Variable	Default	Description
Refill Rate	BATCH_RATE_LIMIT_REFILL_RATE	100	Tokens added per interval
Max Tokens	BATCH_RATE_LIMIT_MAX	1200	Token bucket capacity
Refill Interval	BATCH_RATE_LIMIT_REFILL_INTERVAL	"10s"	Time between refills

The rate limiter uses a token bucket algorithm to smooth processing rates and prevent bursts that could overwhelm the system.

Sources: apps/webapp/app/env.server.ts545-547 apps/webapp/app/v3/runEngine.server.ts187-189

Consumer Configuration

The batch processing system runs multiple consumers that process items from the worker queues:

Setting	Environment Variable	Default	Purpose
Consumer Count	BATCH_QUEUE_CONSUMER_COUNT	2	Number of parallel consumers
Consumer Interval	BATCH_QUEUE_CONSUMER_INTERVAL_MS	100	Polling interval in milliseconds
Default Concurrency	BATCH_CONCURRENCY_LIMIT_DEFAULT	1	Default per-batch concurrency

Consumers can be disabled independently from the main RunEngine worker using BATCH_QUEUE_WORKER_ENABLED.

Sources: apps/webapp/app/v3/runEngine.server.ts180-182 internal-packages/run-engine/src/engine/index.ts348-351

Batch Waitpoints and Parent Blocking

Batch processing integrates with the waitpoint system to enable batchTriggerAndWait patterns where a parent run blocks until all batch items complete.

BATCH Waitpoint Type

When a parent run calls batchTriggerAndWait:

1. A BATCH waitpoint is created with the batch ID
2. Parent run blocks on this waitpoint via WaitpointSystem.blockRunWithWaitpoint()
3. Each batch item run completes independently
4. When the last item completes, BatchSystem.tryCompleteBatch() is called
5. If all items are done, the BATCH waitpoint is completed
6. WaitpointSystem.completeWaitpoint() unblocks the parent run

The batch completion check is scheduled via the worker catalog job tryCompleteBatch:

Sources: internal-packages/run-engine/src/engine/workerCatalog.ts47-51 internal-packages/run-engine/src/engine/systems/waitpointSystem.ts366-497

Queue Architecture and Sharding

The batch processing system uses a multi-level queue architecture with sharding for scalability:

Two-Stage Processing

The system optionally uses two-stage processing with blocking pop:

1. Master Queue → Worker Queue: DRR scheduler moves items to worker queues
2. Worker Queue → Consumers: Consumers block-wait on worker queues

This is enabled via workerQueueBlockingTimeoutSeconds:

When enabled:

• Reduces Redis polling overhead
• Provides backpressure mechanism
• Allows fair distribution across consumers

Sources: apps/webapp/app/v3/runEngine.server.ts177-179 internal-packages/run-engine/src/engine/types.ts82

Sharding Configuration

Master queue sharding distributes batches across multiple Redis keys:

• Shard Count: BATCH_QUEUE_SHARD_COUNT (default varies by deployment)
• Key Pattern: {keyPrefix}batch-queue:master:{shardIndex}
• Distribution: Batches assigned to shards via hash of batch ID

Sharding benefits:

• Reduces lock contention on single Redis keys
• Enables horizontal scaling
• Improves throughput for high batch volumes

Sources: internal-packages/run-engine/src/engine/index.ts346 apps/webapp/app/v3/runEngine.server.ts176

Configuration Reference

Environment Variables

Variable	Default	Type	Description
STREAMING_BATCH_MAX_ITEMS	1000	int	Maximum items per batch
STREAMING_BATCH_ITEM_MAXIMUM_SIZE	3,145,728	int	Max item size in bytes (3MB)
BATCH_CONCURRENCY_LIMIT_DEFAULT	1	int	Default per-batch concurrency
BATCH_QUEUE_DRR_QUANTUM	5	int	DRR quantum value
BATCH_QUEUE_MAX_DEFICIT	50	int	Maximum deficit accumulation
BATCH_QUEUE_MASTER_QUEUE_LIMIT	varies	int	Max batches in master queue
BATCH_QUEUE_SHARD_COUNT	varies	int	Number of master queue shards
BATCH_QUEUE_CONSUMER_COUNT	2	int	Number of consumer workers
BATCH_QUEUE_CONSUMER_INTERVAL_MS	100	int	Consumer polling interval
BATCH_QUEUE_WORKER_ENABLED	false	bool	Enable batch consumers
BATCH_RATE_LIMIT_REFILL_RATE	100	int	Rate limiter refill rate
BATCH_RATE_LIMIT_MAX	1200	int	Rate limiter max tokens
BATCH_RATE_LIMIT_REFILL_INTERVAL	"10s"	string	Rate limiter refill interval
BATCH_METADATA_OPERATIONS_FLUSH_INTERVAL_MS	1000	int	Metadata flush interval
BATCH_METADATA_OPERATIONS_FLUSH_ENABLED	"1"	string	Enable metadata flushing

Sources: apps/webapp/app/env.server.ts542-558

RunEngine Configuration

The batch processing system is configured during RunEngine initialization:

The batch system is separate from the main run queue but shares the same tracer and meter for observability.

Sources: apps/webapp/app/v3/runEngine.server.ts161-190 internal-packages/run-engine/src/engine/index.ts332-362

Metadata Operations

The batch processing system tracks metadata for each batch to support monitoring and completion detection:

Flush Configuration

Metadata operations (item count, completion status, etc.) are batched and flushed periodically:

• Flush Interval: BATCH_METADATA_OPERATIONS_FLUSH_INTERVAL_MS (default: 1000ms)
• Flush Enabled: BATCH_METADATA_OPERATIONS_FLUSH_ENABLED (default: "1")
• Logging: BATCH_METADATA_OPERATIONS_FLUSH_LOGGING_ENABLED (default: "1")

This batching reduces database write load during high-throughput batch processing.

Completion Detection

The system tracks:

• Total items in batch
• Items started
• Items completed
• Items failed

When all items reach a terminal state, the batch waitpoint is completed via the tryCompleteBatch worker job.

Sources: apps/webapp/app/env.server.ts556-558 internal-packages/run-engine/src/engine/workerCatalog.ts47-51