Every publish now also pushes an immutable per-publish tag alongside the
mutable one, named after the snapshot date and commit (e.g.
`22-bookworm-20260812-45444a7`), so previously published digests stay
tag-referenced after republishes. Shipped CLI releases pin those
digests, so they must remain resolvable indefinitely.

Merging triggers a republish; the fresh tag-protected digests will then
be pinned by #4602 before it merges.

## Summary

Fair queue consumers could leak the per-tenant concurrency slots that
gate admission. Slots were freed on some paths and skipped on others,
and once enough leaked slots accumulated for a tenant, every queue that
tenant owned stopped being served until someone cleared the set by hand.
This PR frees slots on every path and, more importantly, makes the
remaining failure modes self-healing.

## Design

The fix applies one rule uniformly: releasing a concurrency slot is
best-effort cleanup and must never block the message's primary state
transition. Blocking completion re-delivers the message, which
duplicates customer work; blocking a retry loses the attempt increment,
so the message can circle forever; blocking a reclaim strands the
message in flight. A leaked slot is the better failure in every one of
those trades because it is the only one that is recoverable. A failed
release is therefore logged and the transition proceeds.

Leaked slots then heal through two mechanisms:

- `reserve` re-admits a message that is already a member of its own
concurrency set, since re-admitting it does not increase concurrency. A
message whose earlier release failed can no longer be blocked by its own
leftover slot.
- A reconcile loop periodically removes any set member with no in-flight
record (interval configurable via `reconcileIntervalMs`, default 60s).
The check-and-remove is atomic, and it is sound because a message is
always registered in flight before its slot is reserved, so a member
with no in-flight record can only be a leak. This also covers leaks this
PR cannot prevent directly, such as a release that resolves the wrong
concurrency group from queue metadata.

Ordering hardening from earlier revisions stays: slots are released
before the in-flight record needed to describe them is discarded, the
release Lua scripts write the message back to the queue before removing
it from in-flight (Lua does not roll back on error), and dangling
in-flight entries with no payload are dropped instead of being rescanned
forever.

Every guard test was verified to fail without its specific fix,
including the duplicate-execution case: completing a message while its
slot release fails used to re-deliver and re-execute it.

…4613)

## Summary

Projects that configure their own `metricExporters` or `metricReaders`
in `trigger.config.ts` were losing task metrics on nearly every run, and
seeing an unexplained `Failed to flush tracingSDK` alongside
`OTLPExporterError: Bad Request` in their run logs. Spans and logs kept
working, so the runs otherwise looked healthy.

## Root cause and fix

Every configured exporter gets its own `PeriodicExportingMetricReader`,
and `meterProvider.forceFlush()` fans out across all readers with
`Promise.all`, so two collections can land on the same millisecond.
`@opentelemetry/host-metrics` divides by the elapsed interval to compute
`process.cpu.utilization`
([common.ts](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/packages/host-metrics/src/stats/common.ts)),
so a zero interval yields `0/0`. `JSON.stringify(NaN)` is `null`, and a
collector rejects `"asDouble": null` with a 400 that drops the
**entire** request, not just the offending point.

`flush()` and `shutdown()` now walk the metric readers one at a time, so
collections can no longer share a timestamp. Each reader is isolated, so
one failing reader cannot skip the readers behind it, and every failure
is logged with the reader that produced it. The first error is still
rethrown, so callers see failures exactly as before.

As a second layer, non-finite data points are dropped just before our
own export, so a metric that divides by zero cannot take the rest of the
batch with it. Exporters and readers supplied through
`trigger.config.ts` are untouched by that filter and still receive raw
data.

The trade-off is that configured exporters now flush after the built-in
one rather than alongside it, so flush latency is the sum rather than
the max.

An internal test package's dependency on core was replaced with a local
helper, because core now needs that package in `devDependencies` and the
two together formed a workspace cycle.

## Verification

Tested against a real collector in a container: a batch containing a
`NaN` reading is rejected with a 400 without the fix and accepted with
it, and a single flush is asserted to collect from one reader at a time.

## Summary

Adds execution-window product surfaces for both declarative and
imperative schedules.

- Declarative schedules can set `window` through `schedules.task()`,
with support for whole-minute, hour, and percentage values.
- Imperative schedules can create, update, clear, and inspect windows
through the API and dashboard.
- Schedule API responses preserve `nextRun` as the nominal CRON time and
expose `nextRunEffectiveAt` as the stable assigned time.
- The dashboard displays configured windows alongside assigned
upcoming-run times.
- Deploy output summarizes declarative schedules and suggests adding a
wider window when the default 60-second placement range is used.

## Design

Window validation remains authoritative on the server and ensures each
window is compatible with the schedule cadence. Omitting a window uses
the default 60-second range, while explicit zero-duration windows remain
supported.

Deployment summaries are derived from the deployment's stored task
metadata, so they reflect the declarations associated with that
deployment.

…ion settings (#4611)

## Summary

Since the move from the Remix compiler to Vite
([#4188](#4188)), the
"App version" on the organization settings page shows `v0.0.0` unless
the image was built from a semver release tag (which bakes in
`BUILD_APP_VERSION`). Self-hosted builds and any image built from `main`
are affected. This restores the real version.

## Root cause

The Vite SSR bundle resolves workspace packages to TS source via the
`@triggerdotdev/source` condition, so `@trigger.dev/core`'s `VERSION`
constant is bundled as its raw `"0.0.0"` placeholder.
`scripts/updateVersion.ts` still stamps the real version at build time,
but only into the packages' dist output, which the bundle no longer
reads. The old Remix compiler bundled the stamped dist, which is why
this used to work.

The fix is a small Vite plugin that applies the same substitution to the
source version modules of `@trigger.dev/core` and `@trigger.dev/sdk`
during bundling. Beyond the settings page, this also restores real
values in the `trigger-version` request header and the version
attributes the bundled packages emit.

Verified by building the server bundle and confirming the VERSION
constants carry the package versions, with no `"0.0.0"` occurrences left
in the build output.

The generated deploy Containerfile now starts from the prebuilt base
images published by base-images/ (`triggerdotdev/node` and
`triggerdotdev/bun` on DockerHub, pinned by digest) instead of
installing system packages during every project's build. Uncustomized
projects run no apt at all and their base layers are identical across
every project, so worker nodes cache one copy fleet-wide. The build
stage uses the -build toolchain variant for uncustomized and
package-only projects; projects with image instructions build FROM base
so instructions and their downloads run exactly once.

### Notes

- User packages install in their own sorted RUN with --allow-downgrades
(a pin of a preinstalled package is a downgrade against the prebuilt
base), preceded by a dpkg repair whenever instructions came first, since
apt-get install refuses to run on state a dpkg -i instruction left
broken.
- Deployed runtime images inherit newer package versions than today's
live-archive installs (the published bases upgrade everything to their
snapshot), plus the base images' OCI labels. Runtime env, user, workdir,
and entrypoint are unchanged.

…database (#4616)

## Summary

The notifications admin list loaded every interaction row for the
notifications on the current page just to show three per-notification
counters (seen, clicked, dismissed), then counted them in memory. On
notifications with many interactions this made the page slow to load and
heavy on memory, even though only 20 notifications are shown.

## Fix

Compute the counters in a single grouped aggregate in the database
instead, returning one row per notification rather than one row per
interaction:

```sql
SELECT "notificationId",
  COUNT(*) AS seen,
  COUNT(*) FILTER (WHERE "webappClickedAt" IS NOT NULL) AS clicked,
  COUNT(*) FILTER (WHERE "webappDismissedAt" IS NOT NULL OR "cliDismissedAt" IS NOT NULL) AS dismissed
FROM "PlatformNotificationInteraction"
WHERE "notificationId" IN (...)
GROUP BY "notificationId"
```

Behavior is unchanged; notifications with no interactions report zero.

)

## Summary

Clarifies when to add a changeset or a `.server-changes/` file. The
friction that keeps coming up is treating these as "I touched a public
package or a server app, so I owe a note." They are user-facing release
notes that go straight into the changelog customers read, not a catalog
of every change.

The guidance now leads with the real test: would a user or customer care
about this change? Add a note when the change is something they would
notice, act on, or want to hear about. Skip it otherwise, even when a
public package or server app is touched, for example:

- internal-only or admin-only changes, refactors, test-only changes,
chores
- performance or query tuning with no user-visible behavior change
- public packages that are not consumed independently (e.g.
`@trigger.dev/redis-worker`), where a version bump means nothing to a
user

Anyone who wants the exact history reads the commits.

Updates every place that encoded the old "touched a package or app, so
add a note" rule so they agree: `AGENTS.md`,
`.server-changes/README.md`, `CONTRIBUTING.md`, `CHANGESETS.md`,
`.claude/rules/server-apps.md`, and `.claude/REVIEW.md` (the last drives
automated review flagging, so it stops flagging exactly the changes the
new guidance says to skip). Also handles the mixed-PR case where the
package change needs no changeset but the server change is user-facing.

## What

A relation with `onDelete: Cascade | SetNull` whose child FK column has
no index makes every parent delete fire a cascade that sequentially
scans the whole child table. That has shipped three times recently and
had to be fixed after the fact (#4554 `ProjectAlert.channelId`, #4555
`EnvironmentVariableValue.valueReferenceId`, #4588
`PersonalAccessToken.userId`).

This adds a schema-aware CI guard that catches the next one before it
merges.

## How

`apps/webapp/scripts/fkCascadeIndexGuard.ts` parses both Prisma schemas
(`@trigger.dev/database`, `@internal/run-ops-database`) and flags any
`onDelete: Cascade | SetNull` relation whose leading FK scalar is not
the leading column of some index (`@@index` / `@@unique` / `@@id` /
field-level `@id`/`@unique`) on the child model. A leading FK column
lets the cascade's `WHERE fk = $1` use the index instead of a seq scan.

It is modeled on the existing `runOpsLegacyGuard` (same `--check` gate,
same baseline-regenerate pattern), and it is lighter: it only reads
`schema.prisma` as text, so its CI job needs no Prisma client generation
and no raised heap.

## Why a baseline, not a hard rule

Not every unindexed cascade FK is a live bug. When the parent is only
ever soft-deleted, the cascade never fires, so the missing index is
harmless. Hard vs soft delete lives in application code
(`parent.delete()` vs `parent.update({ deletedAt })`), not in the
schema, and a `deletedAt` column proves neither direction. So the guard
makes no such judgment: it flags every unindexed cascade FK uniformly
and carries a baseline of the 72 currently-accepted cases. Only
violations **not** in the baseline fail `--check`.

The value is the forcing function: a newly added cascade FK stops CI and
makes the author answer "is the parent ever hard-deleted?" Add the index
if yes; regenerate the baseline with a reason if no.

## Wiring

- `apps/webapp/package.json`: `guard:fk-cascade-index` script
(regenerate with no args, gate with `-- --check`).
- `.github/workflows/fk-cascade-guard.yml`: the reusable workflow.
- `.github/workflows/pr_checks.yml`: runs on webapp-affecting changes,
aggregated into `all-checks`.

## Verification

- The three already-fixed columns are correctly seen as indexed (absent
from the baseline).
- `--check` passes on the current schemas (72 baselined, 0 new).
- A synthetic new unindexed cascade FK fails with exit 1 and an
actionable message.
- Adding `@@index([fk])`, or a composite leading with the FK, clears it.
No false positives.
- `oxfmt` and `oxlint` clean on the new script.

## Rollback

Pure tooling addition, no runtime code, no schema or data change. Revert
to remove.

… to the columns dequeue uses (#4619)

## Summary

The worker-version resolve path fetched every column of every
`BackgroundWorkerTask` for a worker (`include: { tasks: true }`), plus
full `WorkerDeployment` and `TaskQueue` rows, just to match one task at
dequeue. That pulls large JSON columns none of this path reads (task
`payloadSchema`/`config`/`queueConfig`/`description`, deployment
`externalBuildData`/`buildServerMetadata`/`errorData`/`git`, queue
`rateLimit`), so each resolve transfers and deserializes far more than
it uses.

## Fix

Replace the includes with explicit `select`s of only the columns dequeue
reads, in both the passthrough resolver and the app resolver:

- task: `id`, `slug`, `machineConfig`, `retryConfig`,
`maxDurationInSeconds`
- deployment: `id`, `friendlyId`, `imageReference`, `imagePlatform`
- queue: `id`, `name` (the queue matcher keys on both)

The shared `ResolvedWorkerVersion` element types narrow to match
(mirrored in the cache), which also shrinks each cached worker-version
entry.

## Impact

The `tasks` read fetches every task of a worker to match one, so its
cost scales with task count and payload-schema size. For a worker with
~70 registered tasks, dropping the unread columns cuts the per-query
transfer roughly:

| Task shape | Before | After | Reduction |
|---|---|---|---|
| Light (no payload schema, small config) | ~28 KB | ~14 KB | ~54% |
| Typical (mixed schemas / config) | ~62 KB | ~14 KB | ~77% |
| Schema-heavy (large `payloadSchema`) | ~200 KB | ~14 KB | ~93% |

The `after` size is roughly fixed because the kept columns are small;
the win grows with how heavy the dropped JSON is. Narrowing `deployment`
(four JSON columns off a single row) and `queues` saves further on top.

No behavior change: pure read-shape narrowing, no flag and no schema
change, so rollback is a plain revert. Verified with a red/green
run-engine test that asserts the resolved task, deployment, and queue
carry only the used columns, plus the queue feature-matrix runs (batch,
retry-policy, machine-preset, plain trigger) that exercise the kept
columns.

…#4621)

## Summary

When resolving the current worker for a development environment,
`findCurrentWorkerFromEnvironment` loaded the entire `BackgroundWorker`
row,
including the large `metadata` JSON, even though it only ever returns a
handful
of small fields. It is a frequently-run query, so the wasted payload
adds up:
every call pulled data it immediately threw away.

## Fix

Add a `select` to the development-environment lookup listing exactly the
fields
the function returns (`id`, `friendlyId`, `version`, `sdkVersion`,
`cliVersion`,
`supportsLazyAttempts`, `engine`). The query plan is unchanged, still a
single-row indexed lookup; only the row width shrinks. No behavior
change: the
dropped columns were never read.

…#4622)

## Summary

After a deployment promotion or rollback, newly triggered runs could
keep dispatching onto the previously deployed version for up to 30
seconds. Runs now resolve the current version fresh on every dequeue, so
a promotion or rollback takes effect immediately.

## Fix

The dequeue path resolved the worker version through a 30s in-process
cache that nothing invalidated on promotion, and it loaded the worker's
entire task and queue set only to keep the single row matching the run.
Both go away: the resolve now fetches just the matched task and queue by
unique index and reads them fresh, so there is no cache left to serve a
stale version.

```
- cache.get(env:current)              # 30s TTL, never invalidated -> stale
- worker + ALL tasks + ALL queues
+ worker + one task WHERE slug=...  + one queue WHERE id/name=...   # fresh
```

A kill-switch env var (`RUN_OPS_WORKER_VERSION_FRESH_READ_ENABLED`,
default on) falls back to the old cached path without a code deploy.

Verified end-to-end on an isolated stack: a run triggered after a
mid-stream promotion now dequeues onto the new version, with the
previous stale behavior reproduced first.

…are pushed (#4625)

<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C045W9WM3E1/p1786732623292829?thread_ts=1786732623.292829&cid=C045W9WM3E1)_

**Before:** you pause an environment, then a deploy lands (or a
background worker is created, or an admin changes the
concurrency/burst-factor). The environment starts picking up runs again
even though the dashboard still shows it as paused.

**After:** a paused environment stays paused until it is resumed, no
matter what else pushes its concurrency limit.

Pausing an environment sets `paused` in the database and writes a `0`
env concurrency limit into the run queue — the `0` is the only thing
that actually stops dequeueing. Any caller that pushed the limit without
an explicit value (`finalizeDeployment`, `createBackgroundWorker`, the
two admin environment routes) rewrote the real limit and silently
un-paused the environment.

## ✅ Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

`apps/webapp/test/pauseEnvironment.server.test.ts` gains two
`containerTest` cases that wire a real `RunEngine` (real Redis) in place
of the stubbed app singleton and assert the actual run-queue env limit:

- pause a PRODUCTION env → limit is `0` → run the real
`FinalizeDeploymentService` → limit is still `0`, plus a control on a
running env in the same test proving that deploy path really does push
the limit (so the `0` can't just mean "nothing happened").
- pause → resume → the real limit is restored, so the clamp can't
regress resuming.

Both cases fail on `main` (`expected 17 to be +0` and `expected +0 to be
17`) and pass with this change. `pnpm run typecheck --filter webapp` is
clean.

---

## Changelog

Fix paused environments starting to run work again after a deploy.

---

## How

The clamp lives in the shared `updateEnvConcurrencyLimits` helper in
`apps/webapp/app/v3/runQueue.server.ts`, so every present and future
caller is covered: when no explicit limit is passed and the environment
is paused, `0` is written instead of the stored maximum. An
explicitly-passed limit still wins, which is what pausing itself relies
on. The resume path now passes the post-update environment state (its
in-memory copy was read before the un-pause and would otherwise be
clamped back to `0`), and the helper no longer mutates the caller's
environment object — that aliasing made a pause followed by a resume on
the same object write `0` twice. The existing `!paused` guards in
`allocateConcurrency` and the queue-level guard in
`createBackgroundWorker` are left in place as defence in depth, and
queue-level `TaskQueue.paused` behaviour is untouched.

---------

Co-authored-by: Claude <noreply@anthropic.com>

…nce (maxWait + tx-start retry) (#4623)

## What

Makes two transaction-resilience behaviors real and env-var
configurable, defaults set to the good values, so we can tune during and
after the Aug 15 database patch window without a redeploy:

- **maxWait 2s → 10s** (TRI-12982): how long Prisma waits to borrow a
connection before it can `BEGIN`. A restart freeze holds the pool full,
and the only thing that errored was transaction starts giving up at 2s.
- **Retry transaction-start P2028-at-acquisition** (TRI-12984): when
Prisma can't borrow a connection within `maxWait` it raises P2028
(`Unable to start a transaction in the given time`) and **no SQL ran**,
so retrying is safe. Scoped narrowly: only that error (never P2024
pool-exhaustion), 2 attempts, jittered backoff, and a token-bucket
budget so a mass freeze can't amplify into a retry storm.

## Env vars (`DATABASE_*` convention)

Generic defaults:

| var | default |
|---|---|
| `DATABASE_TRANSACTION_MAX_WAIT_MS` | `10000` |
| `DATABASE_TRANSACTION_START_RETRY_ENABLED` | `true` (kill switch) |
| `DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS` | `2` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS` | `250` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST` | `100` |

Per-writer-pool overrides, each falling back to the generic when unset
(same pattern as the per-client pool/connect-timeout work):
`RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all 7 knobs each). Transactions
only open on writer pools, so those are the only pools with their own
knobs. Each pool gets its **own** token bucket, so a storm on one pool
can't drain another's retry budget.

## Design

- The retry primitives live in `internal-packages/database` and never
read `process.env` (IoC): a P2028-at-acquisition classifier, a
`TokenBucketRetryBudget`, and `withTransactionStartRetry`, folded into
the `$transaction` helper via a new `startRetry` option. Config is
resolved at the app boundary and threaded in.
- The `$transaction` helper is the chokepoint (wraps the whole
transaction), not the per-statement `$allOperations` extension.
- The run engine's writes go through `PostgresRunStore`'s own
`.$transaction(...)`, not the webapp helper, so both the helper and the
two `PostgresRunStore` sites apply maxWait + retry (sharing the per-pool
config). Builds on the `options?: { timeout, maxWait }` seam added in
#4514.
- Webapp `$transaction` call sites get the default `maxWait` + retry
injected at one merge point, so no call site needed editing.

## Evidence

- Unit red/green in `internal-packages/database`: reverting the helper
wiring turned the acquisition-retry test red (`Unable to start a
transaction in the given time`), re-applying it green. Full package
suite 25/25. Covers: classifier (P2028-acq yes, P2024 no, in-tx P2028
no), retry (retry-then-succeed, no-retry P2024, stop at maxAttempts,
disabled, budget-exhausted, jitter bounds), token bucket, and
`$transaction` wiring.
- Typecheck clean: webapp, run-store, run-engine.
- Full-stack run: bounded queue-ay pass (15 projects, real dev runs
through the run-engine `PostgresRunStore` transaction path). 13 pass;
the 2 failures are one documented known-failure and one
stale-worker-state flake that passes 2/2 with this change active on a
fresh app.
- Boots cleanly with per-pool overrides set.

## Configuration & rollout

Ship **inert** first (zero behavior change), then flip to the good
values **live via env** — no redeploy needed for either.

### Inert — behaves exactly as today

```
DATABASE_TRANSACTION_MAX_WAIT_MS=2000            # Prisma's built-in default (change defaults to 10000)
DATABASE_TRANSACTION_START_RETRY_ENABLED=false   # disable the new retry entirely
```

`maxWait=2000` is what every path used before (Prisma's default; the
run-store sites and the helper passed no maxWait). `retry=false`
short-circuits `withTransactionStartRetry` to a single run and makes the
serialization-retry exclusion a no-op. Verified on the pooler-freeze
rig: identical fail-fast P2028 at ~2003ms with zero retries —
byte-for-byte current behavior, across all pools.

### Production ("good") — the baked defaults

Rely on defaults (nothing to set) or set explicitly:

```
DATABASE_TRANSACTION_MAX_WAIT_MS=10000
DATABASE_TRANSACTION_START_RETRY_ENABLED=true
DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS=3      # 3 attempts (2 retries); ~30s acquisition tolerance covers a ~20-25s freeze
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS=50
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS=250
DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC=50
DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST=100
```

Per-pool overrides `RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all seven knobs each) are
optional and fall back to the generic set — not needed for v1; the
generic set covers the control-plane, run-ops, and run-ops-legacy writer
pools. Readers open no transactions and take nothing.

**Guardrail:** the retry only engages when a pool's `pool_timeout` >
`maxWait`. Prod is fine (`DATABASE_POOL_TIMEOUT=60` >> 10). Do not set
any writer pool's `pool_timeout` at or under `maxWait`, or saturation
failures flip from retryable P2028 to non-retryable P2024 and the retry
silently stops helping.

### Rollback

Env flip (set inert) or revert. Retry only fires where no SQL ran, and
the per-pool token bucket caps a storm. No migration.

refs TRI-13295, TRI-12982, TRI-12984