Repository navigation
Fix false positives in quote and repeat-change checkers - #41
Open
Lazarus-coder wants to merge 1 commit into
Open
Lazarus-coder wants to merge 1 commit into
Lazarus-coder wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
QuoteExplanationCheckercurrently accepts an unexplained curly-quoted phrase, andRepeatChangeCheckeraccepts an unchanged request when only its whitespace differs. Both produce false positives in strict and loose evaluation.This change normalizes curly double quotes before the existing quote checks. It also compares whitespace-split words consistently: the first word must differ and every remaining word must match. Empty token lists are rejected, including an empty response to a single-word request. Existing case-sensitive comparisons and whitespace-insensitive matching are preserved.
Adds 33 regression subcases covering valid and invalid responses, straight/curly/mixed quotes, whitespace variations, changed tails, and single-word requests.
Before / after
The following results agree across direct checker calls and both strict and loose evaluation, using the actual package and evaluator:
“Hello”"Hello"“Hello” A greeting."Write a poem about rain\n"(trailing newline)" Write a poem about rain "(surrounding spaces)Compose a poem about rainThe repeat examples use
Write a poem about rainas the request.Validation
Python 3.12.14, dependencies installed with
uv sync --locked --group dev, and local NLTK resources:1c40f0c: 12 failing subcases; 21 of the 33 added subcases already pass.python -m pytest instructions_test.py -q -k 'quote_unquote or repeat_change': 5 passed, 35 subtests passed after the fix.python -m pytest -q: 74 passed, 176 subtests passed.uv pip checkandgit diff --check: passed.Model score impact has not been measured. This preserves the existing lexical quote heuristic.
Fixes #38.