feat(redact): detection-guided zoom — real credential redaction 49% → 84%, ~free on ordinary screens - #6239
Conversation
…> 85%
Credential redaction on real screens is limited by RESOLUTION, not by what the
model knows. On a captured frame carrying 23 credentials the shipped path
redacts 12; crop the same window, magnify it, and the same model redacts 11 of
11. A ~17px credential becomes ~7.5px in model space once 1920x1080 is squeezed
to 512, and 2x2 tiling only lifts it to ~11px.
Denser uniform tiling does not fix this. Measured on the real benchmark: 3x3
buys 6 points for 2x the compute, and 4x4 REGRESSES to 34% because real tokens
run 147-490px wide and smaller tiles sever them mid-string.
So magnify without shrinking the window: re-inspect a few text-bearing regions
as 640px NATIVE-resolution crops (1.25x downscale to the 512 input leaves
glyphs ~13.6px, and 640 still contains a 490px token whole). Where to crop
comes free from the passes already run — the model fires Email/Person on the
columns beside a credential even when it misses the token, so its own
detections are a text-presence map. No OCR, no accessibility tree, no second
model.
Measured on 26 captured frames carrying 136 real credentials, scored against
OCR-mined gold with production semantics (label==Secret, score>=0.50, >=50%
coverage):
baseline 2x2 66/136 (49%) 1.00x CPU
+2 zooms 109/136 (80%) 1.28x
+4 zooms 116/136 (85%) 1.41x <- saturates
+6 zooms 116/136 (85%) 1.44x
Confirmed through the production adapter on real frames: 35 -> 62 secret
regions (+77%) at 1.34x CPU. Other PII classes are unaffected — real core
recall holds at 13/16 agnostic and 11/16 class-correct, identical to baseline.
PRECISION. Guided zoom alone added 6 secret boxes on frames with no
credentials; rendering every one showed they are all EMAIL addresses
relabelled under magnification, which the secrets-only default policy would
black out against intent. A confidence floor on zoom detections cannot separate
them (0.60 -> 82%/8 strays, 0.80 -> 58%/4) because a misread email scores as
high as a real credential once magnified. A cross-pass class vote does: drop a
Secret box that any pass calls Email/Url. That costs zero recall and removes a
third of the strays.
Guided zoom runs AFTER the tile loop, which keeps Guard C structural —
extend_wrapped_secrets still runs only on window 0 (the whole frame), so
magnified crops can never seed the continuation walk.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Detection runs on the FULL captured frame — cropping the input would change the scale, which is the entire variable under test. Only the OUTPUT is cropped, to the two credential tables, so the user's taskbar, editor, branch name and URL bar stay out of a public repo. Shows the light-themed window going from 3 blacked to all 12, and honestly shows the two residual misses: one 39-char hex, and a tail on the longest JWT where the box is narrower than the token. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Does "redacted" actually mean covered?The benchmark counts a credential as redacted when a Measured on the same 136 real credentials — for every token the pipeline counts
So the headline is not hiding partial covers: when a credential is detected it is Harness: 🤖 Generated with Claude Code |
Where the remaining 15% goes — and one thing this PR gets wrongCharacterised every credential still unredacted after guided zoom (20 of 136):
The headline understates the win14 of the 20 are the Per family after guided zoom:
Excluding hex: 96/102 = 94%, against a 52% baseline on the same subset. The Correction: "zero recall cost" for the class vote was aggregate-onlyI reported the cross-pass class vote as costing no recall. That holds in Harness: 🤖 Generated with Claude Code |
Limitation: generalisation is not established by this evidenceAll headline numbers come from one frame set — 26 captured frames, largely Scored anyway, for direction only:
Consistent with the main result, but n=8 across 3 unique values is not evidence The honest position: credential-bearing frames are rare in this capture Two things that would raise confidence and are not blockers: capture on a second 🤖 Generated with Claude Code |
…on ordinary screens
Guided zoom fired on any frame with detections, which is nearly every frame:
the model reliably emits Person/Email on ordinary UI. So users paid 4 extra
forward passes on screens containing no credential at all, on every captured
frame.
Credentials cluster — an API-keys page lists a column of them — so "if you found
one, look harder nearby" is the cheap trigger. Measured on 214 ordinary captured
frames and 26 credential frames:
always (before) 85% recall 214/214 ordinary frames zoomed 7 strays
gated (this) 84% recall 3/214 ordinary frames zoomed 3 strays
One point of recall buys a 99% reduction in how often the extra passes run, so
the average cost on ordinary screens goes from a flat 1.41x to ~1.00x. Stray
boxes also drop 7 -> 3, because most of the email-relabelling happened on frames
that no longer get magnified — the trigger improves precision as a side effect.
Note the trigger is a detected Secret at the adapter's confidence threshold, not
a sub-threshold signal: infer_window already filters at conf_threshold, so a
weak response is not observable here. Thresholds of 0.10/0.20/0.30 measured
identically for exactly that reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Update: guided zoom now only fires when a Secret was already found (~1.00× CPU on ordinary screens)Pushed Guided zoom was firing on any frame with detections, which is nearly all of them Measured on 214 ordinary captured frames and 26 credential frames:
One point of recall buys a 99% reduction in how often the extra passes run, One correction to how I'd have described this: the trigger is a detected Harness: 🤖 Generated with Claude Code |
Trigger verified through the production adapterRan the A/B (
Credential frames are unaffected by the trigger (+27 secret regions, same as On timing: the ordinary-frame arms measured 1356 ms vs 1502 ms/frame here 🤖 Generated with Claude Code |
louis030195
changed the title
The "1.41× CPU" framing is wrong for the shipped worker — it costs throughput, not CPU shareChecked how the redaction worker actually schedules this The worker is under an adaptive CPU governor. Each frame it acquires a So longer per-frame work does not raise steady-state CPU share — the governor That reframes the trade for this PR:
I'd been describing the cost as CPU because that is what I measured in a 🤖 Generated with Claude Code |
1 participant
Credential redaction is limited by RESOLUTION, not by what the model knows
On a captured frame carrying 23 credentials the shipped path redacts 12. Crop
the same window, magnify it, and the same model redacts 11 of 11. A ~17px
credential becomes ~7.5px in model space once 1920×1080 is squeezed into 512,
and 2×2 tiling only lifts it to ~11px.
Detection runs on the full frame — cropping the input would change the scale,
which is the whole variable. Only the output is cropped, to keep a real desktop
out of a public repo. The two residual misses are visible in the lower panel.
Denser uniform tiling does not fix it
Measured on the real benchmark with the shipped model:
4×4 regresses because real tokens run 147–490px wide, so smaller tiles sever
them mid-string. Uniform tiling cannot magnify small glyphs and keep long tokens
whole at the same time.
What this does
Re-inspect a few text-bearing regions as 640px native-resolution crops —
a 1.25× downscale to the 512 input leaves glyphs ~13.6px, and 640 still contains
a 490px token whole.
Where to crop comes free from the passes already run: the model fires
Emailon the endpoint column andPersonon the name column even when itmisses the token between them, so its own detections are a text-presence map.
No OCR, no accessibility tree, no second model.
Results
Real benchmark — 26 captured frames, 136 credentials, gold from OCR, production
semantics (
label == Secret, score ≥ 0.50, ≥50% coverage):Through the production adapter on real credential frames: 35 → 62 secret
regions (+77%) at 1.34× CPU (
examples/rfdetr_guided_zoom_check.rs— CPU ismeasured in Rust because the adapter pins
with_intra_threads(2); a Pythonnumber would not be the cost a user pays).
Other classes are unaffected: real core recall holds at 13/16 agnostic and
11/16 class-correct, identical to baseline.
Precision, and the honest cost
Guided zoom alone added 6 secret boxes on frames with no credentials. Rendering
every one showed they are all email addresses relabelled under
magnification, which a secrets-only policy would black out against intent.
A confidence floor on zoom detections cannot separate them — 0.60 → 82%/8
strays, 0.80 → 58%/4 — because a misread email scores as high as a real
credential once magnified. A cross-pass class vote does: drop a
Secretboxthat any pass calls
Email/Url. Zero recall cost, removes a third of thestrays. Net cost: +4 stray boxes per 214 clean frames.
Guard C
Guided zoom runs after the tile loop, so
extend_wrapped_secretsstill runsonly on window 0 (the whole frame) and magnified crops can never seed the
continuation walk. A test asserts guided windows are never
(0,0,w,h)so afuture reorder cannot silently break it.
Tests
5 added, all passing: window bounds/budget, no-detections-no-cost, densest-region
selection, class-vote demotion, and the Guard C invariant.
Two pre-existing CI failures are unrelated to this change and also fail on
main:test-ubuntu(accessibility_late_materialization_testinscreenpipe-db, a crate with no dependency onscreenpipe-redact) andCoverage dashboards current(the same check fails onmain; the core coveragereport does not track
screenpipe-redactand regenerating it locally producesno change).
🤖 Generated with Claude Code