Skip to content

feat: Add agentready assessment workflow #5989

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ntkathole wants to merge 2 commits into from
Closed

feat: Add agentready assessment workflow #5989

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ad22bf4
feat: Add agentready assessment workflow
ntkathole Feb 18, 2026
49bfe0e
fix: Temp disable pull_request_target
ntkathole Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions .github/workflows/agentready-assessment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
name: AgentReady Assessment

on:
pull_request:
types: [opened, synchronize, reopened]
push:
branches: [main, master]
workflow_dispatch:

jobs:
assess:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ github.sha }}

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'

- name: Install AgentReady
run: |
pip install agentready

- name: Run AgentReady Assessment
continue-on-error: true
run: |
agentready assess . --verbose

- name: Upload Assessment Reports
uses: actions/upload-artifact@v4
if: always()
with:
name: agentready-reports
path: ${{ github.workspace }}/.agentready/
include-hidden-files: true
retention-days: 30

- name: Comment on PR
if: always() && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const path = require('path');
const agentreadyDir = path.join(process.env.GITHUB_WORKSPACE, '.agentready');

if (!fs.existsSync(agentreadyDir)) {
console.log('No .agentready directory found');
return;
}

const mdFiles = fs.readdirSync(agentreadyDir)
.filter(f => f.startsWith('report-') && f.endsWith('.md'))
.sort()
.reverse();

if (mdFiles.length === 0) {
console.log('No markdown report found');
return;
}

const report = fs.readFileSync(path.join(agentreadyDir, mdFiles[0]), 'utf8');

await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: report
});
Comment on lines +73 to +78
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 PR comment spam: createComment creates a new comment on every push instead of updating existing one

The workflow triggers on pull_request with types [opened, synchronize, reopened], meaning it runs on every push to the PR branch. The "Comment on PR" step uses github.rest.issues.createComment() which creates a new comment each time, rather than finding and updating an existing AgentReady comment.

Impact and suggested approach

For an active PR with many pushes, this will flood the PR with duplicate AgentReady assessment comments — one per push. This makes the PR conversation noisy and hard to follow.

The standard pattern for bot comments in GitHub Actions is to:

  1. Search for an existing comment with a known marker (e.g., <!-- agentready-assessment -->)
  2. If found, update it with github.rest.issues.updateComment()
  3. If not found, create a new one with github.rest.issues.createComment()

This is the approach used by most CI bots (coverage reporters, size checkers, etc.) to keep PR conversations clean.

Prompt for agents 
In .github/workflows/agentready-assessment.yml, lines 51-78, modify the "Comment on PR" step's script to find and update an existing comment instead of always creating a new one. The approach:

1. Add a marker string like `<!-- agentready-assessment -->` to the comment body.
2. Before creating a comment, list existing comments on the PR using `github.rest.issues.listComments()` and search for one containing the marker.
3. If found, use `github.rest.issues.updateComment({ comment_id: existingComment.id, ... })` to update it.
4. If not found, use `github.rest.issues.createComment()` to create a new one.

Example pattern:

const marker = '<!-- agentready-assessment -->';
const body = marker + '\n' + report;
const { data: comments } = await github.rest.issues.listComments({
  issue_number: context.issue.number,
  owner: context.repo.owner,
  repo: context.repo.repo,
});
const existing = comments.find(c => c.body && c.body.includes(marker));
if (existing) {
  await github.rest.issues.updateComment({
    comment_id: existing.id,
    owner: context.repo.owner,
    repo: context.repo.repo,
    body: body,
  });
} else {
  await github.rest.issues.createComment({
    issue_number: context.issue.number,
    owner: context.repo.owner,
    repo: context.repo.repo,
    body: body,
  });
}
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


Loading