Skip to content

test(e2e): finalize settings suite stabilization#5885

Open
BillLeoutsakosvl346 wants to merge 1 commit into
e2e/settings-playwrightfrom
e2e/07-stabilization
Open

test(e2e): finalize settings suite stabilization#5885
BillLeoutsakosvl346 wants to merge 1 commit into
e2e/settings-playwrightfrom
e2e/07-stabilization

Conversation

@BillLeoutsakosvl346

@BillLeoutsakosvl346 BillLeoutsakosvl346 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • complete the Step 7 objective-to-proof audit and acceptance dossier for all settings navigation, authorization, persona, and workflow contracts
  • harden the pre-0266 SSO migration audit with a real database rehearsal and rollout inputs for GitHub environments, Docker Compose, and Helm
  • make auth failure diagnostics fail closed, retain safe failure artifacts, and graduate Settings E2E to a required provider-neutral CI job
  • correct the free-organization admin fixture and model post-lapse workspace detachment from persisted state
  • make unscoped --repeat-each run every Playwright project sequentially on one stack, rather than repeating only the terminal dependency project
  • fix the docs chat tool's cross-Zod schema type failure while preserving local runtime validation

Verification

  • frozen install, format, lint, architecture/policy checks, migration safety, schema drift, type-check, Helm/Compose validation, and sandbox bundle integrity passed
  • monorepo tests passed, including 13,761 Sim tests, 50 DB tests, and the new docs schema tests
  • real pre/post-0266 SSO migration rehearsal passed
  • production Sim build passed
  • clean E2E cache miss: 256/256 passed; verified build stored
  • cache-hit repeat qualification: 512/512 passed across all six projects
  • definitive retry-free five-repeat gate: 1,280/1,280 passed against one guarded stack
  • leak scans completed and Stripe/MCP fakes recorded no unexpected requests

Manual follow-up after umbrella merge

  • exercise this required workflow on the later umbrella-to-staging PR before changing branch protection
  • perform the documented protected SSO quiescence/audit/backfill/TXT rollout before enabling migration 0266 and domain verification in staging or production

@cursoragent
test(e2e): finalize settings suite stabilization
fbbc9f2 
Co-authored-by: Cursor <cursoragent@cursor.com>
@BillLeoutsakosvl346
BillLeoutsakosvl346 requested a review from a team as a code owner July 23, 2026 08:16
@vercel

vercel Bot commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Jul 23, 2026 8:21am

Request Review

@cursor

cursor Bot commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown

PR Summary

Medium Risk
Production/staging migrations gain a blocking SSO preflight that can halt deploys if secrets or legacy data are wrong; required full E2E on every PR increases CI cost and blast radius of harness regressions.

Overview
This PR finalizes Step 7 settings E2E stabilization and makes Settings E2E a blocking, provider-neutral CI job (full suite, SSO migration rehearsal, shared cache-mount runners). PR CI no longer skips docs/markdown-only changes so required checks cannot go missing.

Migration 0266 SSO hardening adds a shared legacy-provider audit, a one-shot preflight in migrate.ts (advisory lock, SSO_PROVIDER_WRITES_QUIESCED, approved provider IDs), db:rehearse-sso-migration, and env wiring for GitHub environments, Docker Compose, and Helm. SSO enterprise docs describe the new rollout sequence.

E2E harness changes include the freeOrganizationAdmin persona and modeling post-lapse workspace detachment from persisted DB state; unscoped --repeat-each now runs all six Playwright projects on one stack; fail-closed auth failure screenshots; selective trace-off for people/SSO workflows; and acceptance docs (STABILIZATION.md, README gate).

Docs Ask AI replaces inline Zod on searchDocs with a jsonSchema + custom validate module and Vitest coverage to fix cross-package schema typing.

Reviewed by Cursor Bugbot for commit fbbc9f2. Bugbot is set up for automated code reviews on this repo. Configure here.

@greptile-apps

greptile-apps Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR stabilizes the Settings E2E suite and hardens the SSO migration rollout.

  • Promotes Settings E2E to a required, provider-neutral CI job and expands repeat qualification across all Playwright projects.
  • Adds fail-closed authentication diagnostics, scenario/persona corrections, and broader settings contract coverage.
  • Adds a pre-0266 SSO audit, migration rehearsal, and rollout inputs for GitHub Actions, Docker Compose, and Helm.
  • Extracts the docs chat search-tool schema into a locally validated JSON schema to resolve cross-Zod typing.

Confidence Score: 3/5

This PR should not merge until the Helm migration init container can start when optional or retired SSO preflight Secret keys are absent.

The new non-optional secretKeyRef entries are resolved before the migration code runs, so a missing one-shot key can block every app pod even when the database no longer needs the preflight.

helm/sim/templates/deployment-app.yaml

Important Files Changed

Filename Overview
packages/db/scripts/migrate.ts Adds the one-shot pre-0266 SSO audit and mutation lock before the normal migration pipeline.
packages/db/sso-provider-audit.ts Centralizes legacy-provider validation, linked-account summaries, and migration-journal detection.
helm/sim/templates/deployment-app.yaml Injects Helm preflight values, but mandatory Secret key references can prevent migrated pods from starting.
apps/sim/e2e/scripts/options.ts Expands unscoped repeat qualification into ordered, single-project Playwright invocations.
apps/sim/e2e/scripts/run.ts Runs expanded Playwright invocations sequentially against one guarded E2E stack.
apps/docs/lib/search-tool-schema.ts Defines a strict AI SDK JSON schema with matching local runtime validation.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Deploy[Deployment starts] --> Init[Migration init container]
  Init --> Resolve[Resolve SSO preflight environment]
  Resolve --> Audit{Migration 0266 pending?}
  Audit -- Yes --> Gate[Quiescence and provider audit]
  Gate --> Migrate[Apply migrations]
  Audit -- No --> Migrate
  Migrate --> App[Start application container]
Loading

Reviews (1): Last reviewed commit: "test(e2e): finalize settings suite stabi..." | Re-trigger Greptile

greptile-apps[bot]
greptile-apps Bot reviewed Jul 23, 2026
View reviewed changes
Comment thread helm/sim/templates/deployment-app.yaml
Comment on lines +57 to +70
{{- if .Values.migrations.ssoPreflight.existingSecret }}
valueFrom:
secretKeyRef:
name: {{ .Values.migrations.ssoPreflight.existingSecret | quote }}
key: {{ .Values.migrations.ssoPreflight.providerWritesQuiescedKey | quote }}
{{- else }}
value: {{ .Values.migrations.ssoPreflight.providerWritesQuiesced | quote }}
{{- end }}
- name: SSO_AUDIT_APPROVED_PROVIDER_IDS
{{- if .Values.migrations.ssoPreflight.existingSecret }}
valueFrom:
secretKeyRef:
name: {{ .Values.migrations.ssoPreflight.existingSecret | quote }}
key: {{ .Values.migrations.ssoPreflight.auditApprovedProviderIdsKey | quote }}

@greptile-apps greptile-apps Bot Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Optional secret keys block startup

When migrations.ssoPreflight.existingSecret omits either preflight key, these non-optional secretKeyRef entries prevent the migration init container from starting, causing CreateContainerConfigError even after migration 0266 is applied or when no provider approvals are required. Mark these references optional or emit them only for the one-shot rollout.

Knowledge Base Used: Database schema and migrations (packages/db)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@BillLeoutsakosvl346