Skip to content

Conversation

@vexx32
Copy link
Collaborator

@vexx32 vexx32 commented Apr 26, 2020

PR Summary

Currently, we have some flawed assumptions around what defines a mutable value type. Rather than make a more complicated series of checks and making assumptions, we can simply look at whether the type is defined with the IsReadOnlyAttribute that the compiler uses to guarantee a value type is immutable.

As a fallback, it's much safer and less complex to enumerate all public and private fields on the type, and verify whether they're all read-only.

PR Context

Resolves #12430. See the issue for discussion and further context.

PR Checklist

@ghost ghost assigned anmenaga Apr 26, 2020
@vexx32 vexx32 force-pushed the DetectMutableStructs branch 4 times, most recently from db778ef to 3ca57b0 Compare April 26, 2020 04:14
@vexx32
Copy link
Collaborator Author

vexx32 commented Apr 26, 2020

@PoshChan please retry all

@PoshChan
Copy link
Collaborator

@vexx32, successfully started retry of PowerShell-CI-static-analysis, PowerShell-CI-Windows, PowerShell-CI-macOS, PowerShell-CI-Linux

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should compare a performance type.IsPrimitive and type.IsEnum vs IsDefine() before remove old code.

Comment on lines 2040 to 2051
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The custom attribute targets old runtimes.
It is for edge scenarios to target old runtimes with modern compiler.
It will lose value over time.
So the addition could only have value for Windows PowerShell but not for PowerShell Core and we can remove it.

Copy link
Collaborator Author

@vexx32 vexx32 Apr 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Binary modules targeting netstandard2.0 will still generate the custom attribute as the compiler can't be sure the attribute will be present when the DLL is loaded.

At what point are we considered to have completely dropped support for netstandard2.0 modules? Doing so effectively means we no longer support cross-compatibility with WinPS modules at all, and module authors have to pick either 5.1 or the new version to ship modules for.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing so effectively means we no longer support cross-compatibility with WinPS modules at all

No :-) In this case, we do not need to look back. What is current status of the code?

  • no security issues
  • no performance issues
  • no functional issues (do you know real broken scenarios? I think they don’t exist at all because it’s all for performance)

So only our intention is to benefit from modern runtime to reduce false positives.
I think "world wide plan" is to move community to .Net 6.0 LTS. That means Netstandard2.1. In any case, the old code will decrease over time, and if we did not address (the performance) it earlier, then it does not make sense to address it to the future paying with performance due to new code without an urgent need.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I follow you through to the conclusion there. Are you saying we should consciously break the compatibility with netstandard2.0 here and now, or are you saying we should leave behind the old, ineffective code as a fallback?

If you're saying it's not worth changing at all... I don't really agree. PS has to handle mutable from immutable structs a bit differently in a few cases that I see in the code. Being able to yield less false positives from this method should (in theory) allow the engine to take the more efficient code path more often -- though, certainly, the cases where we need to be working with structs in PS directly aren't that common.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you saying we should consciously break the compatibility with netstandard2.0 here and now, or are you saying we should leave behind the old, ineffective code as a fallback?

We can not break "compatibility with netstandard2.0" because the method is only for performance.
Yes, I ask you do not see back because it makes no sense - we have not real issues and real scenario to address and measure - measure it mandatory for performance changes. False positives on the old way will be reduced over time by themselves.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does the method return for the type before and after the change?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the method was returning true in this case before this change, then the method is actually wrong. That's part of why I made this PR, because the assumptions this method was using are very flawed. We can pretty easily verify that datetime is immutable:

image

None of its fields can be modified; it's not mutable. Any methods on the type simply return a new instance, and don't modify the original. The fact that it's decorated with the appropriate compiler attribute also confirms this.

Copy link
Collaborator Author

@vexx32 vexx32 May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have code that is relying on these false assumptions somehow, I'm happy to fix that too. I intend to investigate the CI failures tonight.

By your own comments, though, this method should only be used for optimizations. I doubt the CI failure is actually caused by relying on the logic, but I will verify.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CIs fail on this. To save your time - it is Datetime.

The only thing that is going to change with datetime is that when you assign one to a variable it will be done faster.

The current behavior is that it will be detected as mutable and a pointless copy will be done to ensure changes aren't persisted. But it's immutable, there will be no changes, so no copy is required.

Copy link
Collaborator

@iSazonov iSazonov May 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vexx32 Please revert the questionable change. The PR will be merged for non-LTS version and will affecte most of users only after next PowerShell LTS - in the time this will be not actual I believe.

Comment on lines 2050 to 2059
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a private field is not readonly but a property (used the field) is?
I think we should revert the code and enhance a code for methods below to take into account new readonly methods.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The private field can still mutate, e.g., if the type provides a method that modifies the private field. Without checking all the fields, we cannot be sure the struct is immutable.

Also, even if we could check it more heuristically like you suggest, it's likely that doing more complex logic would be more expensive, since we'd need to make several different checks and verify a wider scope of things than just checking all the fields.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please re-read my counterexample:

What is a private field is not readonly but a property (used the field) is?

using System;
struct S
{
    private int a;
    S(int p)
    {
        a = 1;
    }
    public int GetInt => a;
    public readonly int GetIntMethod()
    {
        // we could a = 2 if not readonly
        return a;
    }
}

You code return true but must false. See then compare with private readonly int a;.

Also the method is all for performance so we shouldn't address all rare cases. We should only reduce false positives in original code.

even if we could check it more heuristically like you suggest

I mean new C# 8.0 feature https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/builtin-types/struct#readonly-instance-members

Copy link
Collaborator Author

@vexx32 vexx32 Apr 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, but that's a limited case, and doesn't account for (as an example) a method being provided that mutates the private field. We don't have a way to ensure that a settable field is not being set, so it will reduce the false positives more effectively to simply check for settable fields. Relying on the attribute is good but will still result in false negatives (the method returning false when the value type is in fact mutable) which I think we should definitely avoid.

I'm not sure I necessarily agree with your reasoning that it's only worth improving in certain ways. If we choose to compare lists of properties and methods as well as fields, in many cases we will end up needing to make more reflection calls than only iterating the fields.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but that's a limited case

Whole the method is limited case because it is only for value types and most of calls will fast return on first condition type.IsPrimitive.
But it is BINDER code that is very-very sensitive and we need strong motivation to change it. Currently no use case exists we should address and no reliable tests and measures too.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, that's why the current flawed assumptions are concerning.

But if it helps, I can add a case for primitives/enums back in, it most likely would short-circuit things a bit quicker.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the short-circuiting for enums and primitives back in and tidied things up a little bit.

@vexx32 vexx32 force-pushed the DetectMutableStructs branch from 3ca57b0 to e1a93df Compare May 1, 2020 14:55
@vexx32 vexx32 force-pushed the DetectMutableStructs branch 2 times, most recently from db17a39 to 5f21a13 Compare May 1, 2020 16:21
@vexx32
Copy link
Collaborator Author

vexx32 commented May 1, 2020

@PoshChan please retry all

@PoshChan
Copy link
Collaborator

PoshChan commented May 1, 2020

@vexx32, did not find any matching pull request checks

@ghost ghost added the Review - Needed The PR is being reviewed label May 27, 2020
@ghost
Copy link

ghost commented May 27, 2020

This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days.
Mainainer, Please provide feedback and/or mark it as Waiting on Author

@SteveL-MSFT SteveL-MSFT added this to the 7.2-Consider milestone Dec 9, 2020
@ghost ghost removed this from the 7.2-Consider milestone Dec 9, 2020
@ghost
Copy link

ghost commented Dec 9, 2020

Open PRs should not be assigned to milestone, so they are not assigned to the wrong milestone after they are merged. For backport consideration, use a backport label.

@rjmholt rjmholt merged commit 1223518 into PowerShell:master Dec 10, 2020
@ghost ghost removed the Review - Needed The PR is being reviewed label Dec 10, 2020
@rjmholt
Copy link
Collaborator

rjmholt commented Dec 10, 2020

Merging this in so we can get early validation for it in the next preview

@rjmholt rjmholt added CL-BreakingChange Indicates that a PR should be marked as a breaking change in the Change Log CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log and removed CL-BreakingChange Indicates that a PR should be marked as a breaking change in the Change Log CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log labels Dec 10, 2020
@vexx32 vexx32 deleted the DetectMutableStructs branch December 10, 2020 21:48
@iSazonov iSazonov added this to the 7.2.0-preview.2 milestone Dec 11, 2020
@iSazonov iSazonov added the CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log label Dec 11, 2020
@ghost
Copy link

ghost commented Dec 15, 2020

🎉v7.2.0-preview.2 has been released which incorporates this pull request.:tada:

Handy links:

@ghost ghost mentioned this pull request Dec 15, 2020
@iSazonov
Copy link
Collaborator

iSazonov commented Dec 16, 2020

Merging this in so we can get early validation for it in the next preview

@rjmholt Have you a plan for the validation? I still have concerns about the implementation.

@rjmholt rjmholt added the Review - Committee The PR/Issue needs a review from the PowerShell Committee label Dec 16, 2020
@rjmholt
Copy link
Collaborator

rjmholt commented Dec 16, 2020

@iSazonov can you summarise your concerns here for other readers? With the changes in a release we might be able to get more users trying this out. We can then set up a decision on whether to revert this, require changes or leave it as is before the next release.

@iSazonov
Copy link
Collaborator

@iSazonov can you summarize your concerns here for other readers?

Oh, every step in this change raises questions. You can see it in my comments above.

  • we did not measure performance on the hot path, we can get a regression.
  • we changed the current behavior without a conclusion
  • we added new behavior without impact analysis.

Ideally, we should have seen good perf win in some scenarios. What is the scenarios and where is perf tests? Do we cover all new .Net features?

@rjmholt rjmholt added the WG-Engine core PowerShell engine, interpreter, and runtime label Aug 20, 2021
@adityapatwardhan adityapatwardhan removed Review - Committee The PR/Issue needs a review from the PowerShell Committee Review - Maintainer The PR/issue needs a review from the PowerShell repo Maintainers labels Jun 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CL-BreakingChange Indicates that a PR should be marked as a breaking change in the Change Log CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log WG-Engine core PowerShell engine, interpreter, and runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve mutable struct detection

9 participants