feat(observability): harden error ingest - #1972

Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/lib/error-ingest/error-ingest-scrubber.ts, line 69:

<comment>When a protocol-relative credential contains an unescaped `@`, this pattern redacts only up to the first `@` and leaks the remaining password suffix. Match the complete userinfo through the authority delimiter, including `@` inside the password, before replacing it.</comment>

<file context>
@@ -64,8 +64,13 @@ const URL_KEY_PATTERN = /(?:^|[._-])(?:http[-_.]?target|request[-_.]?(?:target|u
-const SENSITIVE_ASSIGNMENT_PATTERN = /((?:access[-_.]?token|api[-_.]?key|authorization|client[-_.]?secret|cookie|credential|id[-_.]?token|password|passwd|private[-_.]?key|refresh[-_.]?token|secret|session[-_.]?token|signature|token)\s*[:=]\s*)(["']?)(?:(Bearer|Basic|Digest)\s+)?([^\s"'&,;}\]]+)\2/gi;
+// The scheme is optional so protocol-relative references (`//user:pass@host`)
+// lose their userinfo credentials just like absolute URLs.
+const URL_AUTH_PATTERN = /((?:[a-z][a-z\d+.-]*:)?\/\/)(?:[^/@\s]+):(?:[^/@\s]+)@/gi;
+// The optional quote around the key (backreference \2) covers serialized JSON
+// embedded in message strings (`{"password":"..."}`), which the bare-key form
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/lib/safe-request-context.ts, line 195:

<comment>When a URL contains percent-encoded path data, this call leaves it unsanitized: `alice%40example.com` and JWTs with `%2E` bypass the scrubber and enter `context.url`. Decode the pathname safely before calling `scrubString`.</comment>

<file context>
@@ -156,23 +164,37 @@ function safeHeadersFromEntries(entries: Iterable<readonly [string, string]>): {
+    // value-shaped secrets (emails, JWTs, tokens) — so it goes through the
+    // same string scrubber and size bound as every other retained string.
+    return {
+      path: scrubString(parsed.pathname, { remainingCharacters: MAX_CONTEXT_STRING_LENGTH }),
+      queryParameterCount: [...parsed.searchParams.keys()].length,
+    };
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/lib/safe-request-context.ts, line 168:

<comment>When a cookie name percent-encodes PII or JWT delimiters, this call scrubs the escapes instead of the decoded name, so the sensitive name remains in `cookies.names`. Decode each name safely before applying `scrubString`.</comment>

<file context>
@@ -156,23 +164,37 @@ function safeHeadersFromEntries(entries: Iterable<readonly [string, string]>): {
         const cookieName = cookie.split("=", 1)[0]?.trim();
-        if (cookieName !== "") cookieNames.add(cookieName);
+        if (cookieName !== "") {
+          cookieNames.add(scrubString(cookieName, { remainingCharacters: MAX_CONTEXT_STRING_LENGTH }));
+        }
       }
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/lib/error-ingest/error-ingest-policy.ts, line 208:

<comment>When an error-ingest policy is loaded from environment configuration, this parser rejects the schema's existing `{selector: true}` overrides because it now requires string values, while the shared schema also rejects the new policy fields. Update the shared schema and migration/serialization contract together; otherwise final scrubbing and the new policy controls cannot be configured through the supported API.</comment>

<file context>
@@ -188,22 +198,22 @@ function parseOverrideKeys(value: unknown, field: "dropKeys" | "urlKeys"): reado
     }
-    if (enabled !== true) {
-      throw new ErrorIngestPolicyConfigError("Error-ingest scrub overrides must be enabled explicitly");
+    if (typeof selector !== "string" || Buffer.byteLength(selector, "utf8") > MAX_OVERRIDE_KEY_BYTES || !SAFE_OVERRIDE_KEY.test(selector)) {
+      throw new ErrorIngestPolicyConfigError("Unsupported error-ingest scrub override key");
     }
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/lib/error-ingest/error-ingest-policy.ts, line 203:

<comment>The new override shape in `parseOverrideKeys` (dotless rule-id keys with string selector values) conflicts with the shared config schema in `packages/shared/src/config/schema.ts`, which still validates `dropKeys`/`urlKeys` as `yupRecord(<dotted-selector>, yupBoolean().isTrue())`. New configs like `{ dropEmail: "user.email" }` (already used in the new tests) are rejected by schema validation because the value must be boolean `true`; previously-valid configs like `{ "user.email": true }` now throw `ErrorIngestPolicyConfigError` at policy parse time. Update the shared schema to accept the new `{ dotlessId: "selector" }` shape (and reject the old boolean shape) so config validation and policy parsing agree.</comment>

<file context>
@@ -188,22 +198,22 @@ function parseOverrideKeys(value: unknown, field: "dropKeys" | "urlKeys"): reado
-    if (Buffer.byteLength(key, "utf8") > MAX_OVERRIDE_KEY_BYTES || !SAFE_OVERRIDE_KEY.test(key)) {
+  const keys = new Set<string>();
+  for (const [ruleId, selector] of entries) {
+    if (!SAFE_OVERRIDE_RULE_ID.test(ruleId)) {
       // Never echo a configured key: configuration values are not guaranteed
       // to be harmless labels and policy errors must remain payload-free.
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/lib/error-ingest/error-ingest-protocol-adapter.test.ts, line 56:

<comment>The added comment says "The other five rejections are events/unknown items that no OTLP signal may claim," but only three rejected items are events/unknown (d `rate_limited`, e `rejected`, h `dropped`). The other two rejected items (b `filtered` log, c `filtered` span) are the ones the signals claim. "five" should be "three".</comment>

<file context>
@@ -53,18 +53,21 @@ describe("error-ingest protocol adapter", () => {
         { category: "span", reason: "sampling", quantity: 1 },
       ],
     });
+    // Each OTLP signal counts only its own item types: the filtered log for
+    // `logs`, the sampled-out span for `traces`. The other five rejections are
+    // events/unknown items that no OTLP signal may claim.
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/app/api/latest/analytics/events/batch/route.test.tsx, line 72:

<comment>The comment claims the payload is "one byte over the cap once serialized", but `{ blob: "x".repeat(64000) }` serializes to 64,011 bytes — 11 over the cap (the `{"blob":"` prefix and trailing `"}` add 11 bytes, and the route's `isPlainObjectWithinLimit` uses `Buffer.byteLength(serialized)`). The assertion itself is correct (still over the cap, so it rejects), but the comment's boundary claim is inaccurate and would mislead anyone reasoning about the exact boundary. Correct the figure or make the payload serialize to exactly cap+1.</comment>

<file context>
@@ -1,6 +1,87 @@
+  });
+
+  it("applies the serialized-size cap to versioned batches only", async () => {
+    // One byte over the cap once serialized (the JSON envelope around the
+    // object adds its own bytes on top of the repeated payload).
+    const oversized = { blob: "x".repeat(CUSTOM_TELEMETRY_MAX_ITEM_DATA_BYTES) };
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/lib/error-ingest/error-ingest-protocol-adapter.ts, line 533:

<comment>In `otlpPartialSuccess` the per-signal counts are obtained via `summarizeErrorIngestOutcomes(...)`, which computes a batch `status`/`reason` (an extra loop over `outcomes.slice(1)`) that is discarded — only `.counts` is used. Extract a `countErrorIngestOutcomes` helper that just builds the counts in one pass and use it here (and in `createErrorIngestProtocolProjection` where the summary status is actually needed).</comment>

<file context>
@@ -531,11 +516,21 @@ function boundedErrorMessage(counts: ErrorIngestBatchCounts, maxBytes: number):
+  // Each OTLP signal reports only its own item types. Counting the whole batch
+  // would let a rejected transaction surface as `rejectedLogRecords` (and vice
+  // versa) in mixed batches, telling the client the wrong signal failed.
+  const counts = summarizeErrorIngestOutcomes(items.filter((item) => isSignalItemType(item.itemType, signal))).counts;
   const rejectedItems = rejectedItemCount(counts);
   const errorMessage = boundedErrorMessage(counts, maxErrorMessageBytes);
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At packages/template/src/lib/hexclave-app/apps/implementations/error-scope.ts, line 37:

<comment>This PR caps initial `breadcrumbs` in `copyScopeData` to close the `createErrorScope(initial)` bypass, but initial `eventProcessors` are still copied unbounded (`[...data.eventProcessors]`, no slice). A scope seeded with more than `MAX_EVENT_PROCESSORS` processors is therefore not capped like breadcrumbs; the new per-source check in `processErrorEvent` then drops the entire capture. Apply the same bound here (`eventProcessors.slice(0, MAX_EVENT_PROCESSORS)`) so oversized initial data is trimmed to the newest `MAX_EVENT_PROCESSORS` processors instead of failing the whole event.</comment>

<file context>
@@ -30,11 +30,15 @@ function copyScopeData(data: ErrorScopeData | undefined): ErrorScopeData {
+    // initial data must obey the same bound addBreadcrumb enforces, otherwise
+    // createErrorScope(initial) becomes a bypass that emits arbitrarily large
+    // error payloads. Newest breadcrumbs win, matching addBreadcrumb.
+    ...data.breadcrumbs === undefined ? {} : { breadcrumbs: data.breadcrumbs.slice(-MAX_BREADCRUMBS).map((breadcrumb) => ({ ...breadcrumb, ...breadcrumb.data === undefined ? {} : { data: { ...breadcrumb.data } } })) },
     ...data.level === undefined ? {} : { level: data.level },
     ...data.fingerprint === undefined ? {} : { fingerprint: [...data.fingerprint] },
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At packages/template/src/lib/hexclave-app/apps/implementations/error-capture.ts, line 356:

<comment>`boundExceptionValues` sizes each entry from the value BEFORE the primary's `mechanism` is added in `buildErrorEventDataFromNormalized`, and `boundExceptionValue` does not truncate adapter-supplied fields other than type/value/stacktrace. The shipped `exception.values` can therefore exceed the 32KB budget that the new test asserts as an upper bound. Count the final primary mechanism (and any remaining value fields) in the byte accounting so the budget is a real invariant of the serialized array.</comment>

<file context>
@@ -280,19 +336,24 @@ export function buildErrorEventData(error: unknown, options: BuildErrorEventData
+  // Bounded at the single choke point every capture path funnels through, so
+  // neither a deep automatic cause chain nor an adapter-supplied chain can
+  // push the event past the shared item-data budget.
+  const exceptionValues = boundExceptionValues(options.exceptionValues ?? [exceptionValueFromNormalized(normalized)]);
   const primaryExceptionIndex = exceptionValues.length - 1;
   const exceptions = exceptionValues.map((exception, index) => index === primaryExceptionIndex
</file context>