Skip to content

Benchmark coverage: close the gaps found by the consistency-type audit #94

Description

@jsam

The coverage audit (docs/internals/verified-completions/docs/COVERAGE-AUDIT.md, PR #90) checked the corpus against the full space of logic-resolvable internal inconsistency. Verdict: the measured core is solid (12 families, n=30 each) but NOT exhaustive. Two workstreams close the gap:

1. Corpus-only - rules already exist, no engine change:

  • disjoint class family (entity in two mutually exclusive classes) - rule V3
  • domain violation family (attribute on impossible entity type) - rule V4
  • identity families (distinct-yet-merged, shared unique identifier) - rules I1/I2
  • self-causation flavors (acyclic rule already catches self-edges)

2. New rules + corpus families - pure logic, engine capabilities sufficient (Datalog + comparisons + aggregation, all shipped):

  • sum/partition overflow: budget 2000 = flights 1500 + hotel 800 (sum aggregation exists; the benchmark doc wrongly filed this as out of reach - corrected in the audit)
  • percentage partitions exceeding 100
  • act-after-death / before-birth (activity date vs lifespan)
  • age-date arithmetic (born 1990, 'was 20 in 2005')
  • duration mismatch (3-hour flight, 10:00 to 12:00)
  • point-in-interval (meeting 'during' a dated conference, outside its dates)
  • symmetric-relation polarity ('X married Y' + 'Y not married to X')
  • same-time vs strict order
  • relation functionality (single direct manager - needs a seed decision)
  • direction 3-cycles and opposite-direction pairs, distance symmetry
  • effect-before-cause via event dates
  • taxonomic subclass violations (subclass_of as extractor DATA + one fixed rule)

Explicitly excluded (documented in the audit, revisit deliberately): general quantifiers, counterfactuals, sarcasm, world-falsity without a ground-truth KG, unit conversion, closed-world absence, conditional policies, calendar semantics, threshold spatial reasoning.

Every new rule follows the one-clause-per-detection-relation discipline (#91) and every new family ships at n=30 with clean twins and controls, wired through full_bench.py.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestverified-completionsVerified Completions: OpenAI-compatible endpoint with consistency checking

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions