skill-comply: Failed temporal step can still contribute evidence to a downstream dependency

Project

ECC — Everything Claude Code
Component: skills/skill-comply

skill-comply evaluates whether coding agents follow skills, rules, and agent definitions. It classifies tool-call events against expected steps, checks temporal constraints, and produces compliance results.

Summary

An event classified for a temporal step that fails its own ordering constraint can still be reused by a downstream after_step check.

In grader.py, when the referenced step is not present in resolved, the grader falls back to the raw classifier result:

after_events = resolved.get(step.detector.after_step)
if after_events is None:
    after_events = classified.get(step.detector.after_step, [])

A step is only added to resolved after it successfully passes:

detected = len(matched) > 0

if detected:
    resolved[step.id] = matched

This allows the following situation:

parent event was classified
+
parent step failed its own temporal constraint
+
parent is absent from resolved
+
child falls back to classified[parent]
+
child can still pass

As a result, an event belonging to a failed step can still contribute evidence to a downstream dependency check.

Minimal deterministic reproduction

Consider three steps:

C

A
must occur BEFORE C

B
must occur AFTER A

And three events:

C @ T0
A @ T1
B @ T2

Classifier output:

{
    "C": [0],
    "A": [1],
    "B": [2],
}

A fails

A is required to occur before C.

But:

C @ T0
A @ T1

so A occurs after C.

Therefore:

A = FAIL

and A is not inserted into resolved.

B still passes

B requires:

after_step = A

The grader first checks:

resolved.get("A")

Since A failed, it is not there.

The grader then falls back to:

classified.get("A", [])

which still contains:

A @ T1

Because:

B @ T2

occurs after T1, B passes.

Current result

C = PASS
A = FAIL
B = PASS

So a classified event from a step that failed its own temporal constraint can still be used by a downstream step.

Impact

This can overstate compliance.

For this example, if after_step is intended to require a successfully satisfied parent step, the consistent result would be:

C = PASS
A = FAIL
B = FAIL

compliance = 1 / 3

while the current behavior can produce:

C = PASS
A = FAIL
B = PASS

compliance = 2 / 3

The effect can also propagate through longer dependency chains.

For example:

A = FAIL
B depends on A
C depends on B
D depends on C

If B passes using A's raw classified event, later steps may continue passing:

A = FAIL
B = PASS
C = PASS
D = PASS

This can make a workflow appear substantially more compliant than its prerequisite failures suggest.

The behavior does not require an LLM classification error. Once the classification mapping above is supplied, the result follows deterministically from the grader logic.

Semantic clarification

If after_step is intentionally defined as:

occur after any event classified as that step

then the current behavior may be intentional.

However, the grader separately tracks classified events and successfully resolved steps, and a failed step is deliberately not added to resolved.

That makes this behavior worth clarifying because a failed step can still influence downstream dependency satisfaction through the fallback.

If after_step is intended to depend on successful step satisfaction, the current result is inconsistent with that behavior.

Important repair constraint

The fix should not simply remove the fallback to classified.

ECC already contains a regression case supporting a forward reference where a step depends on another step declared later in the specification:

step_a declared first
step_a after_step = step_b
step_b declared later

That supported behavior should remain intact.

The implementation details can vary. The key requirement is:

A step that fails its own temporal constraint should not satisfy a downstream dependency if after_step requires successful parent-step satisfaction, while valid forward references should continue to work.

Minimum regression requirements

A fix should preserve at least these behaviors:

1. The failed-parent case above no longer allows B to pass.

2. The existing valid later-declared forward-reference case still passes.

3. Existing valid compliant/TDD traces remain green.

Regression test

A focused regression test can capture the issue:

def test_failed_parent_cannot_satisfy_child_dependency():
    # C @ T0
    # A @ T1, but A must occur before C -> A fails
    # B @ T2, and B must occur after A

    classification = {
        "C": [0],
        "A": [1],
        "B": [2],
    }

    result = grade(spec, trace)

    detected = {
        step.step_id: step.detected
        for step in result.steps
    }

    assert detected["C"] is True
    assert detected["A"] is False
    assert detected["B"] is False

This should be added alongside the existing forward-reference regression case so that fixing this bug cannot silently remove supported forward-reference behavior.

Validation result

Focused reproduction of the current behavior:

C = PASS
A = FAIL
B = PASS

Expected behavior if after_step requires successful parent-step satisfaction:

C = PASS
A = FAIL
B = FAIL

The repair should produce the expected result while preserving the existing valid forward-reference case.

Source note

This finding was rechecked against the public main source on September 13, 2026.

The exact branch-head commit SHA was not available through the retrieval path used for this analysis, so the reproduction should be rerun against a pinned commit before preparing a merge-ready patch.

Method attribution

Identified using ADECAS — Assured Decision, Evidence, Capability and Authority System.

Only the externally reproducible finding, impact, repair constraints, and regression boundary are included here. Internal analysis methods and research assets are not included.