Repository navigation
Conversation
dmcppcheck
left a comment
There was a problem hiding this comment.
AI review
I found two new problems; neither causes a wrong match result.
- One failed result object is reused for every call with the same pattern.
CompiledPattern.failedis a singleMatchResult, andmatch()hands that same object back on every failed call with that pattern.MatchResultis an ordinary mutable object. If an addon sets or changes an attribute on a failed result, every later failed match with that pattern returns the changed value. Before this PR each failed call got a new object, so this is a new way for state to leak between calls. - The
|=fix only covers|=as a whole word.|=is literal only when it is the entire word. Inside!!|=,**|=or an alternation such asa|=|b, the'|' in psplit runs first, as it did before. So!!|=still means "!!or=" and matches any token, including a|=token. Behaviour is unchanged here, but the PR description says the quirk is fixed.
Apart from the two documented behaviour changes, a differential fuzz of old vs new match() found no difference in results.
Testing
- I built
cppcheckinpr/and ranpython3 -m pytest -q test/addon --cppcheck-binary=./cppcheck: 114 passed, 3 xfailed. - Differential fuzz (
scratch/fz/fuzz.py): this loadsbase/addons/cppcheckdata.pyandpr/addons/cppcheckdata.pyside by side. It uses mock linked tokens forf ( a , b ) { a |= b ; a = a | b ; if ( a || ! b ) { ... } ... }. It generated 3×200k random patterns from a vocabulary of literals,%…%classes,!!x,**x, alternations, bindings (@x), empty words and|=, and compared truthiness,endand the bindings for every token and forNone.- Every difference involved one of the two documented changes: a whole-word
|=, or a whitespace-only pattern. - After filtering those out, there were 0 differences.
- No exceptions or hangs.
- Every difference involved one of the two documented changes: a whole-word
- Reusing the failed result (mock tokens from the fuzz script):
Output:
r = m.match(None, 'a@x'); r.x = 'stale' print(m.match(toks[0], 'a@x').x)
old fresh failed .x = None new fresh failed .x = stale |=inside compound words:match(tok_or_assign, '!!|=')returns True with both old and new.match(tok_or_assign, 'a|=|b')returns False with both.
Automated review of ec8567f by Claude. The pull request was built and tested locally.
| words = [bind_split(word) for word in pattern.split()] | ||
| self.atoms = [(self.compile_atom(p), b) for p, b in words] | ||
| keys = [b for p, b in words] + ['end'] if words else [] | ||
| self.failed = MatchResult(False, keys=keys) |
There was a problem hiding this comment.
AI review
match() returns this one MatchResult from every failed call with this pattern, and MatchResult is mutable. If an addon assigns to a failed result (r = match(tok, 'a@x'); r.x = ...), every later failed match with the same pattern returns that value: I reproduced .x == 'stale' on a later, unrelated failed match. Before this PR each failed call got a new object. A simple fix is to store only keys and build MatchResult(False, keys=self._failed_keys) on failure, which is still much cheaper than before. Another option is to make MatchResult reject attribute assignment.
There was a problem hiding this comment.
I dont think this is an issue. The MatchResult shouldn't be modified.
| return cls._match_literal(p) | ||
| if p in patterns: | ||
| return patterns[p] | ||
| if '|' in p: |
There was a problem hiding this comment.
AI review
|= is literal only when it is the whole word. In !!|=, **|= or an alternation such as a|=|b, this '|' in p split still runs before the !!/** handling. So !!|= still means !! OR = and matches every token, including |= itself; I verified this against old and new. Also, |= can't be one of the options in an alternation. The PR description says the |= quirk is fixed. Either handle the !!/** prefixes before splitting on |, as the C++ Token::Match does for !!, or narrow the claim in the description.
dmcppcheck
left a comment
There was a problem hiding this comment.
AI review
I found one new problem, a small regression: match(tok, None) now raises AttributeError. Before this PR it returned a failed result. The old code started with if not pattern: return MatchResult(False), and that guard is gone, so CompiledPattern(None) calls None.split(). No in-tree addon calls match(), but out-of-tree addons that pass an optional pattern would now crash.
The description is also missing some behaviour changes. The latest commit changed more than the two cases it lists: !!||, !!|, **|| and **| used to split on |, so !!|| and !!| matched every token. Now they mean "not ||", "not |" and so on. These changes are improvements, but the description should list them.
The other problems I saw (the shared failed result, |= inside alternations) are already in comments.md.
Testing
- I built
cppcheckinpr/, then ranpython3 -m pytest -q test/addon --cppcheck-binary=./cppcheck: 117 passed, 3 xfailed. All of the new tests pass:test_literal_or_assign,test_whitespace_only_pattern,test_negation_*andtest_find_forward_alternatives. - I compared old and new behaviour by loading
base/addons/cppcheckdata.pyandpr/addons/cppcheckdata.pyside by side, using a mock token (scratch/t.py):class T: pass t = T(); t.str = 'a'; t.next = None; t.link = None m.match(t, None) t2 = T(); t2.str = '||'; t2.next = None; t2.link = None for pat in ['!!||', '!!|', '**|', '!!|=']: m.match(t2, pat)
python3 -I t.pygives:base None False pr None EXC AttributeError 'NoneType' object has no attribute 'split' base !!|| True pr !!|| False base !!| True pr !!| True base **| False pr **| False base !!|= True pr !!|= True- With
Noneas the pattern,match()now raises an exception. Before, it returned a failed match. !!||changed meaning: before, it matched every token, including||; now it is "not||". The PR description doesn't mention this.
- With
Automated review of c59f7f1 by Claude. The pull request was built and tested locally.
| # pattern string -> CompiledPattern | ||
| _pattern_cache = {} | ||
|
|
||
| def match(token, pattern): |
There was a problem hiding this comment.
AI review
The old match() started with if not pattern: return MatchResult(False). Without that guard, match(tok, None) now raises AttributeError: 'NoneType' object has no attribute 'split' in CompiledPattern.__init__; before, it returned a failed result. I reproduced this against base and PR. A short guard here keeps the old behaviour:
if not pattern:
return MatchResult(False)That guard also keeps falsy patterns out of _pattern_cache.
dmcppcheck
left a comment
There was a problem hiding this comment.
AI review
I found two new problems, both caused by the new pattern cache. Neither produces a wrong result for a fixed pattern string used by an addon that leaves patterns alone.
- The cache has no size limit, and patterns built at run time get slower.
_pattern_cachestores every distinct pattern string forever. An addon that builds patterns at run time, such asmatch(tok, '%s ( ' % name), now pays the compile cost on every new string and keeps every compiled pattern alive. With 200k distinct patterns,match()is about 5× slower than before and the cache ends up with 200k entries. - Compiled patterns copy entries of the module-level
patternsdict.compile_atom()copiespatterns[p]when the pattern is compiled. If an addon adds a%foo%entry tocppcheckdata.patternsafter a pattern using%foo%was compiled, the cached pattern still treats%foo%as a literal. Before this PR the dict was looked up on every call, so the new entry was used.
The guard that C7 asked for is back. !!|= and **|= now work, but |= still can't be one of the options in an alternation (a|=|b).
Earlier comments
| # | Comment | By | Where | Status |
|---|---|---|---|---|
| 1 | Shared mutable failed MatchResult is returned from every failed call with the same pattern |
AI review | general | 💬 Answered: Author: MatchResult shouldn't be modified |
| 2 | The |= fix only works when |= is the whole word, not in !!|=, **|= or alternations |
AI review | general | ✅ Fixed |
| 3 | match(tok, None) raises AttributeError |
AI review | general | ✅ Fixed |
| 4 | Description doesn't list the changed meaning of !!||, !!|, **|| and **| |
AI review | general |
@pfultz2, please fix, resolve or reply to the open comments: #4.
Testing
- Built
cppcheckinpr/. - Ran
python3 -m pytest -q test/addon --cppcheck-binary=./cppcheck: 117 passed, 3 xfailed. - To compare old and new behaviour, I loaded
base/addons/cppcheckdata.pyandpr/addons/cppcheckdata.pyside by side with mock tokens forx = y ;. Run withpython3 -I scratch/t.py:
for m in (b, p):
print(m.__name__, bool(m.match(toks[0], '%foo% =')), end=' ')
m.patterns['%foo%'] = lambda t: t if t.isName else None
print(bool(m.match(toks[0], '%foo% =')))
for m in (b, p):
t0 = time.time()
for i in range(200000):
m.match(toks[0], 'name%d =' % i)
print(m.__name__, 'dyn', round(time.time()-t0, 2), len(getattr(m, '_pattern_cache', {})))Output:
b False True
p False False <- new %foo% entry ignored, the cached pattern is stale
b dyn 0.21 0
p dyn 1.03 200001 <- 5x slower, 200k cache entries kept
Automated review of 955b562 by Claude. The pull request was built and tested locally.
| return m | ||
|
|
||
| # pattern string -> CompiledPattern | ||
| _pattern_cache = {} |
There was a problem hiding this comment.
AI review
This cache has no size limit. An addon that builds pattern strings at run time, such as match(tok, name + ' ('), pays the compile cost for every new string and keeps every compiled pattern for the rest of the run. I measured 200k distinct patterns: 1.03 s against 0.21 s before this PR, with 200,001 cache entries left at the end. functools.lru_cache(maxsize=...) on a compile_pattern(pattern) helper would put a limit on the memory. Building patterns at run time would still be slower than before, so the docstring should tell addon authors to use constant pattern strings.
There was a problem hiding this comment.
Added the lru cache.
| if p in cls._literal_atoms: | ||
| return cls._match_literal(p) | ||
| if p in patterns: | ||
| return patterns[p] |
There was a problem hiding this comment.
AI review
This copies the entry of the module-level patterns dict when the pattern is compiled, and the result is then cached. If an addon adds an entry such as cppcheckdata.patterns['%foo%'] = ... after a pattern using %foo% was compiled, the cached pattern still treats %foo% as a literal. Before this PR the dict was looked up on every call. In my test match(tok, '%foo% =') stays False in the PR after %foo% is registered, while base returns True. Two possible fixes: clear _pattern_cache when patterns changes, or look up patterns[p] lazily inside the matcher.
There was a problem hiding this comment.
User arent meant to modify cppcheckdata.patterns.
dmcppcheck
left a comment
There was a problem hiding this comment.
AI review
I found no new problems in how match() behaves. A differential fuzz of base against PR match() found no differences outside the documented behaviour changes. All of the addon tests pass.
One small inaccuracy, and it is new: the comment in the new test_negation_alternatives says that !!a|b means "not a, or b", as in the C++ Token::Match. That is wrong. C++ checks the !! prefix before alternation and compares the rest of the word, a|b, literally (firstWordEquals). So in C++, struct !!Point|Foo matches struct Point, while the test asserts that it does not.
Earlier comments: the |= fix inside !!/**, the None guard and the size limit of the cache are all fixed. The author answered the comments about the shared failed result and about changing patterns. Nobody has answered the request to describe the !!||/!!|/**|/**|| changes. The current description mentions them, but it says they "no longer split on |", which doesn't spell out the new meaning ("not ||" and so on).
Comments
| # | Comment | By | Where | Status |
|---|---|---|---|---|
| 1 | One mutable failed MatchResult is shared by every failed call with the same pattern |
AI review | addons/cppcheckdata.py:1671 |
💬 Answered: Author: MatchResult shouldn't be modified |
| 2 | The |= fix only works when |= is the whole word (not in !!|=, **|=) |
AI review | addons/cppcheckdata.py:1643 |
✅ Fixed |
| 3 | The description doesn't spell out the new meaning of !!||, !!|, **|| and **| |
AI review | general | |
| 4 | match(tok, None) raises AttributeError |
AI review | addons/cppcheckdata.py:1765 |
✅ Fixed |
| 5 | The pattern cache has no size limit | AI review | addons/cppcheckdata.py:1695 |
✅ Fixed |
| 6 | Compiled patterns copy entries of patterns, so later changes to the dict are ignored |
AI review | addons/cppcheckdata.py:1703 |
💬 Answered: Users aren't meant to modify cppcheckdata.patterns |
| 7 | The comment says the C++ Token::Match reads !!a|b as "not a, or b", which is wrong |
AI review | test/addon/test_cppcheckdata.py:726 |
🆕 New |
@pfultz2, please fix, resolve or reply to the open and new comments: #3, #7.
Testing
- Build: I built
cppcheckinpr/withmake -j8 cppcheck, and it succeeded. - Addon tests:
python3 -m pytest -q test/addon --cppcheck-binary=./cppcheckgave 123 passed, 2 xfailed. - Differential fuzz (
scratch/d.py, run withpython3 -I d.py):- Setup: it loads
base/addons/cppcheckdata.pyandpr/addons/cppcheckdata.pyside by side. It uses mock linked tokens forf ( a , b ) { a |= b ; a = a | b ; if ( a || ! b ) { x [ 1 ] ; } } %. It runs 100k random patterns of 1 to 4 words, built from literals,%…%classes,(*)/{*},!!x,**x, alternations, bare!!/**and%. For every token and forNone, it compares whether the pattern matches and which tokenendis. - Result: after leaving out the documented changes (patterns containing
|=,!!||,!!|,**|), there were 0 differences, and no exceptions. - Tokens that are not real C tokens: with the whole vocabulary, the only other differences come from tokens whose text is
**or!!. Oldmatch_atomcomparedtoken.str == pfirst, so they matched. C and C++ never produce such tokens, so this doesn't matter in practice.
- Setup: it loads
- C++ claim in the test comment: I checked it by reading
Token::Matchinlib/token.cpp. The!!branch, which callsfirstWordEquals, runs before the alternation branch. I didn't buildtestrunnerto confirm it at run time.
Automated review of b14b932 by Claude. The pull request was built and tested locally.
| assert cppcheckdata.match(find_token(match_cfg, '=').previous, '%var% !!|=') | ||
|
|
||
| def test_negation_alternatives(self, match_cfg): | ||
| # '!!a|b' is "not a, or b", as in the C++ Token::Match |
There was a problem hiding this comment.
AI review
The C++ Token::Match does not read !!a|b this way. It checks for the !! prefix before alternation and compares the rest of the word literally against the token (firstWordEquals(p, tok->str()) in lib/token.cpp). So in C++, !!Point|Foo means "not the token Point|Foo", and struct !!Point|Foo matches struct Point. This test asserts the opposite. The Python behaviour isn't new, since base also splits on | first, but this comment claims the two agree. Please drop the "as in the C++ Token::Match" part, or say that the two differ here.
cppcheckdata.match()re-parsed its pattern string on every call: it split the pattern, split off the@bindings, andmatch_atom()ran through a chain ofin/startswithchecks for each word before doing the actual comparison. On a failed match it also rebuilt the list of binding keys and allocated a newMatchResult. Since an addon typically matches the same handful of pattern strings against every token, almost all of this work was repeated.This PR adds a
CompiledPatternclass that compiles a pattern string once into a list of per-word matcher functions, plus a precomputedMatchResultfor the failed case.match(token, pattern)keeps its signature and looks the compiled pattern up in a cache keyed by the pattern string. The internalmatch_atom()helper is removed; its logic is nowCompiledPattern.compile_atom().Two behavior changes, both for degenerate patterns:
end = None). It now fails, the same as an empty pattern.|=is now a literal word. Previously the|in it was also read as an alternation of''and=, so the pattern|=matched a=token as well. Also!!||,!!|,**||and**|no longer split on|either. (The C++Token::Matchstill has that quirk; the Python side no longer shares it.)Performance
Measured on the first configuration of a
lib/token.cppdump (49,349 tokens), Python 3.12 patterns (
if (*) {,%name% (*) ;,%var%@lhs = %var%@op1 | %var%@op2 ;,struct|class %name% {,%name% !!(,for (*) {*},%comp%,::, …) matched against every token, ~592k calls:200,000 calls of
match(tok, '%name% (*) ;')on one token (per-call overhead, no matching work):One-time compile cost per distinct pattern (not cached per word; happens once per pattern string):
if (*) {%var%@lhs = %var%@op1 | %var%@op2 ;struct|class %name% {**{