feat(knowledge): let the S3 connector sync with a saved AWS credential by waleedlatif1 · Pull Request #9934 · simstudioai/sim

waleedlatif1 · 2026-11-12T11:54:15Z

Summary

The S3 knowledge connector can now sign with a saved AWS credential owned by the knowledge base's workspace or organization: access keys, an assumed role, or a keyless OIDC role. The STS session token is SigV4-signed as x-amz-security-token.
Key connectors can declare an additive auth.savedCredential alternative. S3 declares aws, the only provider that resolves today. Key-only config fields (apiKeyOnly: Access Key ID, custom endpoint) are hidden and not sent when a credential is used.
A saved credential signs only for Amazon S3 in a commercial or GovCloud AWS region, where every S3 host is s3.<region>.amazonaws.com. These are refused with a clear message:
- a custom endpoint (R2/MinIO) or the auto region, so an AWS session never reaches a non-AWS host;
- China and isolated-partition regions, whose S3 hosts are on other domains.
R2/MinIO keep using access keys.
Resolution forwards the knowledge base's own scope to the same owner check and STS exchange the AWS tools use. Setup checks the acting principal with the existing connector credential check and a provider match. Create and access updates resolve the credential's service through one shared helper, so the provider match cannot be skipped. During a sync, STS sessions are reused until 5 minutes before they expire.
Validation errors are redacted against every secret the resolved token carries. For a saved AWS credential that means the secret key and session token, not just the empty bearer.
A connector that stores its key cannot be moved onto a saved credential later. To switch, create the connector again. This keeps a retired key from silently taking over if the credential is deleted.
Setup error codes:
- When AWS refuses a credential, setup returns a 400.
- When STS is throttled or the IdP is unreachable, setup returns a 500 instead of blaming the credential, matching how AWS tools return 503 for transient failures. The failure message and the transient check are shared with the AWS tools.
Audit metadata records these connectors as authMode: 'savedCredential'.
Setup is behind the global knowledge-connector-saved-credentials flag (env fallback KNOWLEDGE_CONNECTOR_SAVED_CREDENTIALS). Sync is not gated. The v2 catalog and credentialId descriptions say the option works once the deployment enables it.
The v2 catalog contract gains optional savedCredential / apiKeyOnly. OpenAPI, CLI API and CLI docs are regenerated, and the connector docs are updated.
Data drains are not changed. Org-owned saved AWS credentials, including keyless ones, already resolve. Drains still need three things first:
- an additive credential reference on data_drains, deploy-gated because older workers parse destination_credentials directly;
- org credential authorization at drain create and update;
- new contract and UI surface.
GCS, Azure Blob and Sentinel also need new resources in the shared Google keyless and Microsoft app resolvers (Cloud Storage, storage.azure.com, monitor.azure.com). These are left for a follow-up.

Type of Change

Testing

connectors/s3/s3.test.ts:
- SigV4 Authorization matches @smithy/signature-v4 reference values, with and without a session token.
- Covers refusal of a custom endpoint, a non-AWS region, and China/ISO regions, and signing for a GovCloud host.
- Reverting the guards turns these tests red.
access-token.test.ts:
- The knowledge base scope is forwarded and the session is seeded with an empty bearer token.
- A stored key bypasses AWS; refusals are wrapped as non-transient and throttling as transient.
- Run tokens are reused and renewed.
connector-access.test.ts: refuses moving a stored-key connector onto a saved credential, and allows moving a credential-backed one to another credential.
connectors.test.ts (application):
- Refuses another service's credential on S3 create.
- AWS refusal maps to 400 and throttling to 500 (the guard revert was watched turning red).
- A saved AWS session's secret key and session token are redacted from settings-validation results and exceptions. This test was red before the fix.
orchestration/connectors.test.ts: the same redaction on connector create, for results and exceptions. This test was also red before the fix.
auth.test.ts covers the auth-source decisions; saved-credential-rollout.test.ts covers the setup gate.
The neighbouring knowledge, catalog, sync-engine, deletion and AWS-tools suites pass. These checks pass: check:openapi, check:cli-api, check:cli-docs, check:mcp-operations, check:deployment-config, check:file-names, check:test-patterns, check:comment-hygiene.

Deploy notes

Leave knowledge-connector-saved-credentials off until this release runs on both the app and Trigger.dev workers. Older workers unschedule a key connector that has no stored key.
Once any connector uses a saved credential, rolling back to an earlier release is unsafe even with the flag off. Older workers unschedule those connectors as credential-removed, and they stay idle until someone edits them or syncs them manually.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
Backward compatible: existing S3 rows keep their stored key; stored secret shapes unchanged

vercel · 2026-11-12T11:54:21Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Nov 12, 2026 2:54pm UTC

greptile-apps · 2026-11-12T11:57:36Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge, with no new actionable issues found.

Summary

Adds saved AWS credentials as an alternative to stored keys for the S3 knowledge connector.

S3 connectors can sign requests with a saved AWS credential.
Knowledge base syncs resolve and reuse saved AWS sessions.
Saved-credential setup is gated and keeps stored-key connectors separate.
The connector catalog describes saved credentials and key-only settings.
AWS credential failure messages and retry checks come from shared helpers.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[S3 connector] --> B{Authentication source}
  B -->|Stored key| C[Decrypt stored key]
  B -->|Saved credential| D[Check knowledge base scope]
  D --> E[Resolve AWS keys or role session]
  E --> F[Reject custom endpoints and unsupported regions]
  C --> G[Sign S3 request]
  F --> G
  G --> H[Sync documents]

_{Reviews (5) · Last reviewed commit: "test(llm): mock AWS credentials through ..." · Reviewed by Greptile}

cubic-dev-ai

All reported issues were addressed across 44 files

_{Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-11-12T12:36:14Z

@greptile

waleedlatif1 · 2026-11-12T12:36:16Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-11-12T12:36:23Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 46 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-11-12T13:29:15Z

@greptile

waleedlatif1 · 2026-11-12T13:29:16Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-11-12T13:29:21Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 46 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

The S3 knowledge connector can now sign with keys a saved AWS credential resolved to, read from the run's syncContext. STS session tokens are sent as x-amz-security-token and included in the SigV4 signed headers. A saved credential signs only for Amazon S3 in an AWS region: a custom endpoint is refused so an AWS session never reaches an S3-compatible host. Inline keys sign exactly as before.

…dential A key-based connector may now declare a saved-credential alternative (auth.savedCredential); Amazon S3 declares a saved AWS credential. A row that names a credential resolves it for the knowledge base's own scope through the same owner check and STS exchange AWS tools use, and the resolved session is seeded into syncContext. Rows that store a key keep using it, and key-only connectors are unchanged. - connectorUsesCredential/connectorHasAuthSource decide the auth source in one place for the resolver, sync engine, validation and setup. - resolveConnectorAccessToken takes the knowledge base scope. - A run token reuses an STS session until shortly before it expires and re-seeds syncContext when it is replaced. - Setup with a saved credential is gated by the global knowledge-connector-saved-credentials flag: workers from an earlier release read a key connector without a key as credential-removed and unschedule it, so the flag stays off until the release is live on the app and Trigger.dev. - The connector catalog and v2 contract gain optional savedCredential and apiKeyOnly fields; generated OpenAPI, CLI API and CLI docs updated. - Adds a central mock for @/lib/internal/aws/credentials.

…onnector When knowledge-connector-saved-credentials is on, the add-connector modal lets Amazon S3 authenticate with a saved AWS credential instead of its access keys, hides the key-only Access Key ID and custom endpoint fields, and never sends values typed into them. The edit modal keeps the credential replaceable for a connector that syncs with one, including after its credential was removed. Inline keys stay the default.

…redential The access update accepted a credentialId for a key connector that still stored its key. The key stayed on the row and took over again, silently, once the credential was deleted. Switching auth source now means creating the connector anew, as it did before saved credentials.

…sistently - Share the AWS failure message and transient check with the AWS tools, so STS throttling or an IdP outage during setup is no longer a 400. - Refuse another service's credential on S3 creation, as access updates do. - Return an empty access token for a saved AWS credential, not its id. - Record saved-credential connectors as authMode savedCredential in audit. - Narrow savedCredential.provider to aws, the only provider that resolves. - Derive create's key-vs-credential choice from connectorUsesCredential and rename the run-token refresh.

…ovCloud S3 China regions serve S3 under amazonaws.com.cn and the isolated partitions under their own domains, so a saved credential for them failed with an opaque DNS error. Refuse them with a clear message instead.

Catalog and connector credentialId descriptions say saved credentials work once the deployment enables them, the connector credentialId field covers saved credentials, and the flag notes that a release rollback is unsafe once any connector uses one. Regenerates OpenAPI, CLI API and CLI docs.

…tial service check A saved AWS credential resolves to an empty bearer with its secret key and session token in awsSession, so redacting accessToken alone left them in provider validation text. Both validation sites now redact every secret the resolved token carries (connectorTokenSecrets). Setup and access updates now share requireConnectorCredentialService, so a key connector's saved-credential setup refuses a missing service config instead of skipping the provider check.

…-credential edits on rollout - Reuse a saved AWS credential without a session token for the whole run instead of reloading it on every page and hydration - Hide the edit form's saved-credential replacement while saved-credential setup is disabled, matching the server gate - Classify a connector whose credential service has no config as internal, not caller-fixable - State in the v2 catalog that apiKey connectors with savedCredential accept credentialId

… not an OAuth service

waleedlatif1 · 2026-11-12T14:14:00Z

@greptile

waleedlatif1 · 2026-11-12T14:14:01Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-11-12T14:14:08Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 48 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-11-12T14:53:59Z

@greptile

waleedlatif1 · 2026-11-12T14:54:00Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-11-12T14:54:07Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 48 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}