Skip to content

Fix SuspiciousContentChecker.Match to detect a pre-defined string when the text starts with it #18693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
daxian-dbw merged 3 commits into from
Dec 2, 2022
Merged

Fix SuspiciousContentChecker.Match to detect a pre-defined string when the text starts with it #18693

daxian-dbw merged 3 commits into from
Dec 2, 2022

Conversation

@daxian-dbw
Copy link
Member

@daxian-dbw daxian-dbw commented Nov 30, 2022

PR Summary

Fix SuspiciousContentChecker.Match to detect a pre-defined string when the text starts with it.

PR Context

The method SuspiciousContentChecker.Match has a flaw: when the text starts with a string that is defined in LookupHash(uint h), it cannot detect the string.

PS:1> $t = [psobject].Assembly.GetType('System.Management.Automation.ScriptBlock+SuspiciousContentChecker')
PS:2> $t::Match("Add-Type") -eq $null
True
PS:3> $t::Match(" Add-Type")      ## it can detect the string when text doesn't start with it.
Add-Type

NOTE: The SuspiciousContentChecker.Match check is a defense-and-depth thing, so this bug is NOT a security vulnerability.

PR Checklist

@ghost ghost assigned anmenaga Nov 30, 2022
@daxian-dbw daxian-dbw assigned daxian-dbw and unassigned anmenaga Nov 30, 2022
@daxian-dbw daxian-dbw added Backport-7.3.x-Consider CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log labels Nov 30, 2022
iSazonov
iSazonov reviewed Dec 1, 2022
View reviewed changes
Copy link
Collaborator

@iSazonov iSazonov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we take into account only ASCII chars and - I suggest adding more tests with using specific chars - ASCII non-letters, Unicode, surrogates in different positions in prefix, in word, in suffix.
Also now we check start position and I'd expect we check the transition for the 19th position too.

And maybe xUnit test would be more fast for the internal API (also exclude reflection - it is not good for public test) .

@iSazonov iSazonov mentioned this pull request Dec 1, 2022
Closed
22 tasks
@daxian-dbw
Copy link
Member Author

daxian-dbw commented Dec 1, 2022 •
edited
Loading

After chatting with @TravisEz13, my understanding is that this defense-and-depth feature is sort of broken (high chance of collision) and it was decided to not fix it. So, adding more tests for it doesn't seem worth the effort in my opinion.

But I'm totally open to add some more tests. What do you mean by "we check start position and I'd expect we check the transition for the 19th position too"? Can you provide an example string that you want to be added to the tests?

About xUnit test, I don't think it's necessary unless it turns out to be hard to test some specific characters. PowerShell has the `u{xxx} syntax for representing Unicode characters, so we should be fine using Pester with a tiny bit of reflection.

@iSazonov
Copy link
Collaborator

iSazonov commented Dec 1, 2022

But I'm totally open to add some more tests. What do you mean by "we check start position and I'd expect we check the transition for the 19th position too"? Can you provide an example string that you want to be added to the tests?

Sorry, typo, 29th - max word size. So a test could check starting with 30th position.
I ask about tests with non-ASCII chars based on your comment in my PR.

After chatting with @TravisEz13, my understanding is that this defense-and-depth feature is sort of broken (high chance of collision) and it was decided to not fix it. So, adding more tests for it doesn't seem worth the effort in my opinion.

We could reduce collisions with simple trick - take into account word length.

@pull-request-quantifier-deprecated
Copy link

pull-request-quantifier-deprecated bot commented Dec 1, 2022

This PR has 29 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

Quantification details 

Label      : Extra Small
Size       : +28 -1
Percentile : 11.6%

Total files changed: 2

Change summary by file extension:
.cs : +1 -1
.ps1 : +27 -0

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

  • Fast and predictable releases to production:
    • Optimal size changes are more likely to be reviewed faster with fewer
      iterations.
    • Similarity in low PR complexity drives similar review times.
  • Review quality is likely higher as complexity is lower:
    • Bugs are more likely to be detected.
    • Code inconsistencies are more likely to be detected.
  • Knowledge sharing is improved within the participants:
    • Small portions can be assimilated better.
  • Better engineering practices are exercised:
    • Solving big problems by dividing them in well contained, smaller problems.
    • Exercising separation of concerns within the code changes.

What can I do to optimize my changes

  • Use the PullRequestQuantifier to quantify your PR accurately
    • Create a context profile for your repo using the context generator
    • Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
    • Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
    • Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
  • Change your engineering behaviors
    • For PRs that fall outside of the desired spectrum, review the details and check if:
      • Your PR could be split in smaller, self-contained PRs instead
      • Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

  • One line was added: +1 -0
  • One line was deleted: +0 -1
  • One line was modified: +1 -1 (git diff doesn't know about modified, it will
    interpret that line like one addition plus one deletion)
  • Change percentiles: Change characteristics (addition, deletion, modification)
    of this PR in relation to all other PRs within the repository.

Was this comment helpful? 👍  :ok_hand:  :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

@daxian-dbw
Copy link
Member Author

daxian-dbw commented Dec 1, 2022

I have added the requested tests. But be noted that, those are just smoke tests, and are by no means intended for covering the Match implementation.

@iSazonov
Copy link
Collaborator

iSazonov commented Dec 2, 2022

LGTM.

@daxian-dbw daxian-dbw merged commit ae3c316 into PowerShell:master Dec 2, 2022
@daxian-dbw daxian-dbw deleted the match branch December 2, 2022 06:26
@TravisEz13
Copy link
Member

TravisEz13 commented Jan 10, 2023

/backport to release/v7.3.2

@github-actions github-actions bot mentioned this pull request Jan 10, 2023
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TravisEz13 TravisEz13 TravisEz13 approved these changes

@iSazonov iSazonov iSazonov approved these changes

Assignees

@daxian-dbw daxian-dbw

Labels

Backport-7.3.x-Done CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log Extra Small

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@daxian-dbw @iSazonov @TravisEz13 @anmenaga @adityapatwardhan