Skip to content

Harden docs-review substitution suggestions#118

Merged
theletterf merged 1 commit into
mainfrom
fab/harden-docs-review-suggestions
Apr 28, 2026
Merged

Harden docs-review substitution suggestions#118
theletterf merged 1 commit into
mainfrom
fab/harden-docs-review-suggestions

Conversation

@theletterf
Copy link
Copy Markdown
Member

Summary

  • Make the docs-review suggestion-block guidance conditional on a pre-output checklist.
  • Add a hard rule that prevents GitHub suggestion blocks when the original line or replacement contains Elastic {{...}} substitution syntax.
  • Recompile the generated gh-aw lock file so the stricter prompt ships with the workflow.

Test plan

  • Ran make compile successfully with 0 errors and 0 warnings.
  • Checked lints for .github/workflows/gh-aw-docs-review.md.

Made with Cursor

Require a pre-output checklist before creating suggestion blocks so Elastic substitution syntax is never included in apply-ready review suggestions.

Co-authored-by: GPT-5.5 <gpt-5.5@openai.com>
Made-with: Cursor
@theletterf theletterf requested a review from a team as a code owner April 28, 2026 07:13
@theletterf theletterf requested a review from cotti April 28, 2026 07:13
@theletterf theletterf self-assigned this Apr 28, 2026
@theletterf theletterf added the fix label Apr 28, 2026
@theletterf theletterf merged commit 076f02b into main Apr 28, 2026
3 of 4 checks passed
@theletterf theletterf deleted the fab/harden-docs-review-suggestions branch April 28, 2026 07:18
theletterf added a commit that referenced this pull request May 5, 2026
Updates the pinned gh-aw extension version from v0.71.1 to v0.71.4
(latest release at time of writing) and recompiles all 10 agentic
workflow lock files.

The bump is driven by two upstream fixes that landed in v0.71.2:

1. **github/gh-aw#28672** (PR github/gh-aw#28738) — `shared/apm.md`
   checkout step fails in private caller repos with metadata-only
   token. Without this fix, the docs-content-internal install of
   the docs-quality-sweep orchestrator (and of any other gh-aw
   workflow imported via APM) fails before the agent starts because
   `actions/checkout` cannot fetch the private caller repo. v0.71.2
   adds the missing `contents: read` permission to the shared APM
   job.

2. **github/gh-aw#28691** (PR github/gh-aw#28736) — Safe-output
   sanitizer escapes template delimiters inside GitHub `suggestion`
   blocks, breaking Elastic substitution syntax such as
   `{{fleet-server}}` when an author clicked **Commit suggestion**.
   The sanitizer now preserves fenced code-block contents verbatim,
   so the docs-review prompt no longer needs the defensive guidance
   added in PRs #116 and #118 to avoid emitting suggestion blocks
   whenever the line includes `{{...}}`.

`gh-aw-docs-review.md` is updated accordingly: the dedicated
"substitution syntax pre-output checklist" section is removed, and
the surrounding sentence about "passing the pre-output checklist
below" / "protected substitution syntax" is simplified back to the
plain "prefer apply-ready suggestions when they map cleanly" rule.
The recently-hardened lower-priority guidance (avoid stale-content
nits, etc.) is left in place.

Version pin updated in five places to keep local development, CI
lint, the weekly auto-recompile job, and the workflow APM imports
in sync:

- `Makefile`
- `scripts/run-gh-aw-compile.sh`
- `.github/workflows/check-aw-updates.yml`
- `.github/workflows/pre-commit.yml`
- `imports:` field of every gh-aw-*.md that uses the shared APM
  workflow (frontmatter, applies-to, openings, style, review)

`make compile` produces 0 errors, 0 warnings against v0.71.4.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
theletterf added a commit that referenced this pull request May 5, 2026
…re noop (#133)

The openings sweep noop'd on its first real run on
elastic/docs-content despite the skill being available, because
the agent stochastically invoked the skill with a non-exact form
the first time:

  11:33:22  ✗ skill(docs-page-opening-optimizer)        Skill not found
  11:33:27  ● noop  "docs-page-opening-optimizer skill unavailable..."
  11:33:33  ✓ skill(skill: docs-page-opening-optimizer)        succeeded

The prompt's "abort with noop on skill failure" rule fired on the
first non-exact attempt, before the agent retried with the correct
`skill(skill: …)` form and got results. By the time the exact
invocation succeeded, the noop was already committed.

Frontmatter, applies-to, and style produced fix-issues fine in the
same orchestrator run. The difference: the frontmatter sweep
prompt explicitly tolerates partial skill failures ("if one fails,
report only the other"); applies-to and style happened to land on
the exact form first try. Openings was the unlucky one.

Lift the docs-review pattern (PRs #117/#118) where skill
availability is only declared after a confirmed exact-form
failure. Apply to the three sweeps that had the strict
abort-on-first-failure rule:

- openings (was hit by this in real run 25373751367)
- applies-to (would hit it next time the agent picks the wrong form)
- style (same)

Each now spells out the procedure: try the exact form, retry
once on failure, only noop after the second exact-form failure.
A single non-exact failure is not sufficient evidence.

The frontmatter prompt already had a softer fallback ("merge what
works, note any failure in Notes"); coherence and staleness use
different mechanisms (>50% MCP failure threshold and full-repo
deterministic respectively); typos doesn't import any skill.
None of those need this change.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant