Skip to content

feat(cli): default database workflows to the bundled pg-delta engine - #6102

Open
avallete wants to merge 43 commits into
developfrom
feat/upgrade-pg-delta-next
Open

feat(cli): default database workflows to the bundled pg-delta engine#6102
avallete wants to merge 43 commits into
developfrom
feat/upgrade-pg-delta-next

Conversation

@avallete

@avallete avallete commented Aug 6, 2026

Copy link
Copy Markdown
Member

Bundles @supabase/pg-delta and @supabase/pg-topo into the CLI and makes the in-process engine the default for pg-delta-backed db diff, db pull, and the declarative schema workflows (db schema declarative generate|sync). The legacy edge-runtime implementation remains available as an explicit SUPABASE_USE_PG_DELTA_NEXT=false opt-out; there is no automatic fallback or runtime package download — a next-engine failure propagates instead of silently switching implementations.

This branch is reconciled with the TypeScript database-command port on develop:

  • shared database bootstrap owns shadow-database setup and receives the complete local configuration, including database webhooks
  • pgAdmin uses the native TypeScript differ-container path, while db diff --use-pg-schema remains the sole Go delegation
  • --project-ref resolution stays in the TypeScript command path and is rejected for delegated modes that cannot honor it
  • local diffs target the running local database, while migration and declarative baselines use isolated shadow databases
  • declarative flows repair missing extension prerequisites, scope schema filters, and warn when unsupported object kinds would be omitted
  • when a legacy declarative tree cannot converge on the next engine, sync offers a staged recovery flow: it generates a next-compatible export into a sibling <declarative_schema_path>-next directory (never inside the active tree — trailing separators in the configured path are normalized, and staging into or under the active directory is rejected) and prints adoption commands built from the configured path
  • both compatibility gates (shadow-load failure and plan-refuse) render one shared recovery template: diagnosis, evidence, an explicit do-not-apply hazard line, and the staged-upgrade commands carried as the error suggestion — so a deliberate gate prints actionable guidance instead of the generic "rerun with --debug" footer. Non-interactive runs get exactly one recommended recovery (the staged regenerate); interactive prompts keep in-place extension repair as an advanced choice with a "may surface another gap" hint
  • db pull when already in sync keeps Go's "No schema changes found" message and non-zero exit, but replaces the debug footer with an explanatory suggestion (recorded in go-cli-divergences.md)
  • migration files preserve pg-delta's transaction=false directive so later push, reset, and up operations keep those plans outside CLI-owned transactions
  • migrations that call pg_net while local webhooks are disabled now surface targeted guidance to enable [experimental.webhooks]

Default-engine snapshots, plans, and diagnostics use an isolated v2 artifact layout. Extraction errors are refused; coverage gaps warn by default and can be promoted to failures with --strict-coverage on db diff, db pull, and the declarative commands (a TS-only flag, recorded in docs/go-cli-divergences.md). Generated SQL and file segmentation may differ from the legacy renderer — the compatibility contract is successful execution and convergence on a subsequent diff.

The branch pins published @supabase/pg-delta@1.0.0-alpha.40 and @supabase/pg-topo@1.0.0-alpha.5 from the npm registry, and patches the libpg-query parser packaging so the WASM asset is embedded in compiled Bun binaries (with an integration test verifying the patched asset survives compilation).

A live local-convergence suite (legacy-pgdelta-next.live.test.ts) exercises the next engine against a real Docker stack. It is gated on the configured live environment plus a Docker probe (describeDockerLive), so it runs only under the cli-e2e-ci harness — machines that merely expose Docker stay inert.

The earlier dogfooding playbook, ADR, and glossary documents were trimmed out of this PR to keep it scoped to the engine change. Linked-project, TLS-required, and pooler/SNI acceptance still needs a provisioned data-plane environment because the shared live harness does not create project Postgres.

@avallete
avallete requested a review from a team as a code owner August 6, 2026 07:15
@avallete avallete added the run-live-e2e-ci Execute the supabox live e2e tests and report back label Aug 6, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8078b53b04

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts Outdated
Comment thread apps/cli/tests/helpers/live.ts Outdated
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Supabase CLI preview

npx --yes https://pkg.pr.new/supabase/cli/supabase@f7e27f1e5613a40d486e914cc0a775d5d3a3d57d

Preview package for commit f7e27f1.

@avallete

avallete commented Aug 6, 2026

Copy link
Copy Markdown
Member Author

Provisioned validation against exact head 4b697a2e5e0d5090ece6b319e6a1498db91afbcf: internal PG17 run.

  • Project provisioning reached the data plane, and the direct database endpoint connected over TLS with sslmode=require (SUPABASE_LIVE_DB_URL was set).
  • legacy-pgdelta-next.live.test.ts passed both tests: the full convergence scenario (260.2s) and explicit legacy opt-out smoke (25.8s).
  • The aggregate live project finished 19/22 tests. Its three failures were outside this PR's diff: an existing start-status timing assertion, an existing db-pull assertion, and an existing db-diff test colliding on host port 54320 while live files ran concurrently.

The labeled dispatch check itself cannot reach the internal repository because its GitHub App is not installed there, so this run was dispatched manually against the same SHA.

@avallete
avallete marked this pull request as draft August 6, 2026 08:12
@avallete
avallete marked this pull request as ready for review August 7, 2026 17:06
@avallete
avallete marked this pull request as draft August 7, 2026 17:07
Comment thread apps/cli/package.json Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

const exists = yield* fs.exists(w.path).pipe(

P2 Badge Detect collisions by migration version, not full path

When a multi-segment pg-delta plan is generated in the same second as an existing migration with a different name—or overlaps future-dated segments from a prior run—checking only w.path misses the collision because the filenames differ. This writes multiple files with the same 14-digit version; local migration loading accepts both, but schema_migrations.version is a primary key, so a pull can fail while repairing history and later push/reset operations can fail while applying the duplicate version. Check every candidate version against all existing migration filenames rather than only the generated pathname.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts Outdated
Comment thread apps/cli-go/internal/db/diff/shadow.go Outdated
@blacksmith-sh

This comment has been minimized.

@avallete
avallete marked this pull request as ready for review August 7, 2026 18:33
@avallete avallete removed the run-live-e2e-ci Execute the supabox live e2e tests and report back label Aug 7, 2026
Comment thread apps/cli/package.json Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d4861957ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-files.ts Outdated
Comment thread apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts Outdated
@avallete
avallete marked this pull request as draft August 8, 2026 06:46
@avallete
avallete marked this pull request as ready for review August 8, 2026 10:22

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

for (const w of set) {
const exists = yield* fs.exists(w.path).pipe(
Effect.mapError(
(cause) =>

P1 Badge Check migration version collisions across all filenames

When a multi-unit plan assigns a version already used by another migration with a different name, this exact-path check reports no collision and writes a second <same-version>_*.sql file. This can occur when a migration was created in the same second as the pull/diff, or when a later plan unit's future-dated timestamp overlaps an existing file; db pull then upserts that version in schema_migrations, replacing the existing history row while both local files remain. Check for any ${version}_*.sql entry before accepting the generated set, rather than only each proposed pathname.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: db045256e7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…lta-next

# Conflicts:
#	apps/cli/src/legacy/commands/bootstrap/SIDE_EFFECTS.md
#	apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md
#	apps/cli/src/legacy/commands/db/diff/diff.handler.ts
#	apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts
#	apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md
#	apps/cli/src/legacy/commands/db/pull/pull.handler.ts
#	apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts
#	apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts
#	apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md
#	apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.layers.ts
#	apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md
#	apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.layers.ts
#	apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts
#	apps/cli/src/legacy/commands/db/start/start.handler.ts
#	apps/cli/src/legacy/shared/legacy-db-config.toml-read.ts
#	apps/cli/src/legacy/shared/legacy-migration-apply.ts
Comment thread apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cea474d0b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts Outdated
Comment thread apps/cli/src/legacy/commands/db/diff/diff.handler.ts Outdated
Comment thread apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 23814e76d7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.ts
Comment thread apps/cli/src/legacy/shared/db-bootstrap/recreate-local-database.ts
@avallete avallete changed the title feat(cli): bundle pg-delta for database workflows feat(cli): default database workflows to the bundled pg-delta engine Aug 14, 2026
- read the declarative export manifest only under the next engine, so a
  stale or malformed .pgdelta-export.json can no longer break the
  SUPABASE_USE_PG_DELTA_NEXT=false escape hatch
- surface planSchemaFiles skipped statements as coverage diagnostics:
  warned by default, blocking under --strict-coverage
- warn when a manifest-less declarative directory keeps unmanaged files
  the next writer cannot prune, advising a clean regenerate
- converge pg_net in both directions: PG14 db reset now runs the same
  drop-before-conditional-create as fresh setup, and start-on-existing-
  volume drops pg_net when webhooks are disabled unless an applied
  migration installed it
- gate the schema_paths transition warning on the resolved next engine;
  migra and the legacy opt-out still substitute declared-schema targets
- warn once when --strict-coverage is ignored by the legacy engine
- redact unparseable connection strings with the shared redactor instead
  of a naive :password@ regex
- record deferred symlink/port-allocation findings in
  docs/roadmap/pg-delta-next-follow-ups.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a621690aff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts
Comment thread apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c3c4643150

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/cli/src/legacy/shared/legacy-pg-net-guidance.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b2b7cb9ebb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts Outdated
…lta-next

# Conflicts:
#	apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.ts
#	apps/cli/src/legacy/shared/db-bootstrap/container-lifecycle.unit.test.ts
#	apps/cli/src/legacy/shared/db-bootstrap/docker-create-args.ts
Comment thread apps/cli/src/legacy/commands/db/reset/reset.integration.test.ts Dismissed
Comment thread apps/cli-go/internal/utils/misc.go

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2cc84ab035

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts Outdated
Comment thread apps/cli/src/legacy/commands/db/diff/diff.command.ts
Comment thread apps/cli/tests/helpers/live.ts Outdated
- Normalize trailing separators (and trailing '.' segments) in
  legacyResolveStagedDeclarativeDir so a configured
  declarative_schema_path like './schemas/' stages a sibling
  './schemas-next' instead of nesting '-next' inside the active tree,
  where the printed rm -rf adoption command would destroy both copies.
- Harden the sync staging guard to reject any staged directory that
  resolves inside the active declarative directory, not just equality.
- Gate describeDockerLive on the configured live environment as well as
  the Docker probe, so live suites stay inert on machines that merely
  expose Docker; document the gate in AGENTS.md.
- Record the TS-only --strict-coverage flag in go-cli-divergences.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
avallete and others added 2 commits August 15, 2026 09:13
CodeQL flagged the trailing-separator regex for ambiguous nested
quantification (potential exponential backtracking on long separator
runs). Trim trailing separators and '.' segments with an explicit
linear scan instead; behavior is unchanged and covered by the same
unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dogfooding follow-up for the pg-delta next compatibility gates:

- Both gates (shadow-load failure and plan-refuse) now render one shared
  template: diagnosis + evidence + an explicit do-not-apply hazard line,
  with the staged-upgrade commands carried on the error's suggestion so
  Output.fail prints them instead of the generic 'rerun with --debug'
  footer. A deliberate gate no longer reads as a crash.
- Non-interactive runs get exactly one recommended recovery (the staged
  regenerate). The extension.sql alternative was a false trail there:
  each hand-added declaration only unlocks the next refusal.
- Interactive prompts keep repair as an advanced choice labelled with
  the full <declarative_schema_path>/extension.sql path and a 'may
  surface another gap' hint, and the repair gate now also offers the
  staged export as the recommended option.
- db pull in-sync keeps Go's message and non-zero exit but replaces the
  debug footer with an explanatory suggestion; recorded as a deliberate
  divergence in go-cli-divergences.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f7e27f1e56

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


/** Default declarative schema dir. */
const DEFAULT_DECLARATIVE_DIR_SEGMENTS = ["supabase", "database"] as const;
const DEFAULT_DECLARATIVE_DIR_SEGMENTS = ["supabase", "schemas"] as const;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve the previous default for existing declarative trees

When an existing project generated its declarative schema under the former default supabase/database and omitted declarative_schema_path, this unconditional default switch makes generate/sync look only in supabase/schemas. A non-interactive sync now reports no schema, while --yes can generate a replacement tree from the selected database and silently ignore edits that existed only in the old declarative tree. Detect the former default when it already exists or provide an explicit migration rather than abandoning the established filesystem input.

AGENTS.md reference: apps/cli/AGENTS.md:L288-L294

Useful? React with 👍 / 👎.

Comment on lines +303 to +304
const adoptionCommand = (declarativeDir: string, stagedDir: string): string =>
` rm -rf ${shellQuoteArgument(declarativeDir)} && mv ${shellQuoteArgument(stagedDir)} ${shellQuoteArgument(declarativeDir)}`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reject adoption paths that contain project metadata

When declarative_schema_path resolves to the supabase directory itself—for example, the documented under-supabase setting is "."—the staged recovery command becomes rm -rf supabase && mv supabase-next supabase. Following the command printed by either compatibility gate deletes config.toml, all migrations, seeds, and other project metadata before replacing the directory with only the staged export. Refuse declarative roots that contain these protected project files before emitting or executing this adoption flow.

AGENTS.md reference: apps/cli/AGENTS.md:L288-L294

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants