PSOptimizedProvider objects - Experimental feature #8995

powercode · 2019-02-26T22:48:55Z

PR Summary

Adds support for deriving from PSObject to create optimized CodeProperties, and uses this to implement the attached properties for FileSystem/Registry-Provider and the output of Get-Content.

This makes the FileSystemProvider roughly twice as fast getting items, and about 5 times faster getting content.

Memory usage of getting all item under c:\windows is 415Mb before the change, and 103Mb after.

To try it out:

Enable-ExperimentalFeature PSOptimizedProvider

PR Context

PSNoteProperties are quite expensive. To provide core scenarios with a more performant alternative, this PR adds types derived from PSObject that are tailored for common scenarios in the providers.

The providers add six PSNoteProperties to each item, adding massively to the memory usage, and substantially to the time it takes to create the PSObjects wrapping the items.

The same thing happens with Get-Content where the attached properties can inflate the PSObjects many times over.

The gist of the change is the addition of an interface, IPSObjectExtendedMemberInfo and an attribute,
PSExtensionMemberAttribute, that the derived PSObjects can use to communicate with the ETS about what members to expose.

CoreAdapter has a new member, AddExtensionProperties that adds members with the attribute, and the interface provides a member

T GetMember<T>(MemberNamePredicate predicate)

that can be used to quickly determine if an object provides a property matching the predicate.

PR Checklist

powercode · 2019-02-27T08:10:51Z

Not sure why that test fails. It passes locally 🤔

src/Microsoft.PowerShell.Commands.Management/commands/management/ContentCommandBase.cs

iSazonov · 2019-03-01T05:17:46Z

Verify Clear-Item with -WhatIf

It still fail. I don't see why - need to debug.

powercode · 2019-03-01T14:21:59Z

@iSazonov That seems like a genuine bug from my side. And windows only is easy to debug for me.

iSazonov · 2019-03-01T15:50:40Z

Perhaps I don't understand you comment but the test failed on Windows
https://powershell.visualstudio.com/PowerShell/_build/results?buildId=14744&view=logs&jobId=35656138-dd59-55dc-1472-6feeb4c97f57&taskId=41fef26b-4f6c-56d4-3d5d-1873b1a12853&lineStart=853&lineEnd=854&colStart=1&colEnd=1

Update: previous test seems works so only WhatIf parameter doesn't work.

powercode · 2019-03-03T13:00:12Z

@iSazonov, @daxian-dbw Do you know how to test global experimental features?

In this case, I would like to run all test with the feature both enabled and disabled, to make sure I didn't break anything in the old behavior, and that is works with the optimizations.

I know about the flags to Start-PSPester, and I can run them locally, but I don't know how to enable running both for a large set of tests.

To you have any ideas on how to handle this?
Since I'm changing the binder, PSObject and ProviderBase, it would be good to have quite a lot of testing on it :)

iSazonov · 2019-03-03T17:01:38Z

There is TestMetadata.json file.
See #7419

daxian-dbw · 2019-03-03T19:12:53Z

@powercode Here is the TestMetadata.json file: https://github.com/PowerShell/PowerShell/blob/master/test/tools/TestMetadata.json
Please look at it along with the Pester test example: https://github.com/PowerShell/PowerShell/blob/master/test/powershell/engine/ExperimentalFeature/ExperimentalFeature.Basic.Tests.ps1

TestMetadata.json allows you to define the experimental features and the corresponding tests to run. If an empty array is used for an experimental feature, that means all tests will run for the experimental feature.
When starting to Pester tests, we first run all tests without any experimental feature enabled, so the stable code gets exercised by all tests.
Then for each experimental feature defined in TestMetadata.json, Start-PSPester sets up the config file to enable the feature, and run the indicated tests with that feature enabled.

Sorry that I didn't update the testing doc with this. Will do that soon.

stale · 2019-04-26T17:01:51Z

This PR has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs within 10 days.
Thank you for your contributions.
Community members are welcome to grab these works.

powercode · 2019-05-20T09:01:36Z

Keeping this alive. Will get back to it after PSConfEU

PoshChan · 2019-06-07T22:37:26Z

@powercode, your last commit had 2 failures in PowerShell-CI-windows
Get-ItemProperty.Should be able to access a property using the Path and name switches

Expected $null, but got C.
at <ScriptBlock>, D:\a\1\s\test\powershell\Modules\Microsoft.PowerShell.Management\Get-ItemProperty.Tests.ps1: line 36
36: 	$output.PSDrive | Should -Be $testprovider

String cmdlets.Select-String.Network path

Expected 2, but got 0.
at <ScriptBlock>, D:\a\1\s\test\powershell\Modules\Microsoft.PowerShell.Utility\string.tests.ps1: line 56
56:             (select-string -LiteralPath $fileNameAsNetworkPath "b").count | Should -Be 2

PoshChan · 2019-06-07T22:39:11Z

@powercode, your last commit had 4 failures in PowerShell-CI-macos
Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: \foo:bar.txt

Expected exactly 'Hello', but got $null.
at <ScriptBlock>, /Users/vsts/agent/2.152.1/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246
246:                 $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: /foo:

Expected exactly 'Hello', but got $null.
at <ScriptBlock>, /Users/vsts/agent/2.152.1/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246
246:                 $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: :bar

Expected exactly 'Hello', but got $null.
at <ScriptBlock>, /Users/vsts/agent/2.152.1/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246
246:                 $files[0] | Get-Content | Should -BeExactly "Hello"

Get-ItemProperty.Should be able to access a property using the Path and name switches

Expected $null, but got /.
at <ScriptBlock>, /Users/vsts/agent/2.152.1/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/Get-ItemProperty.Tests.ps1: line 36
36: 	$output.PSDrive | Should -Be $testprovider

PoshChan · 2019-06-07T22:40:07Z

@powercode, your last commit had 4 failures in PowerShell-CI-linux
Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: \foo:bar.txt

Expected exactly 'Hello', but got $null.
at <ScriptBlock>, /home/vsts/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246
246:                 $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: /foo:

Expected exactly 'Hello', but got $null.
at <ScriptBlock>, /home/vsts/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246
246:                 $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: :bar

Expected exactly 'Hello', but got $null.
at <ScriptBlock>, /home/vsts/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246
246:                 $files[0] | Get-Content | Should -BeExactly "Hello"

Get-ItemProperty.Should be able to access a property using the Path and name switches

Expected $null, but got /.
at <ScriptBlock>, /home/vsts/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/Get-ItemProperty.Tests.ps1: line 36
36: 	$output.PSDrive | Should -Be $testprovider

PoshChan · 2019-11-09T23:29:38Z

src/System.Management.Automation/FormatAndOutput/DefaultFormatters/FileSystem_format_ps1xml.cs

    internal sealed class FileSystem_Format_Ps1Xml
    {
        internal static IEnumerable<ExtendedTypeDefinition> GetFormatData()
        {


@powercode, your last commit had 4 failures in PowerShell-CI-macos
Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: \foo:bar.txt

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /Users/runner/runners/2.160.0/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: /foo:

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /Users/runner/runners/2.160.0/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: :bar

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /Users/runner/runners/2.160.0/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

Get-ItemProperty.Should be able to access a property using the Path and name switches

Expected $null, but got /. at <ScriptBlock>, /Users/runner/runners/2.160.0/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/Get-ItemProperty.Tests.ps1: line 36 36: $output.PSDrive | Should -Be $testprovider

PoshChan · 2019-11-09T23:32:00Z

src/System.Management.Automation/FormatAndOutput/DefaultFormatters/FileSystem_format_ps1xml.cs

    internal sealed class FileSystem_Format_Ps1Xml
    {
        internal static IEnumerable<ExtendedTypeDefinition> GetFormatData()
        {


@powercode, your last commit had 3 failures in PowerShell-CI-linux
Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: \foo:bar.txt

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /home/vsts/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: /foo:

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /home/vsts/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: :bar

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /home/vsts/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

PoshChan · 2019-11-09T23:32:02Z

src/System.Management.Automation/FormatAndOutput/DefaultFormatters/FileSystem_format_ps1xml.cs

    internal sealed class FileSystem_Format_Ps1Xml
    {
        internal static IEnumerable<ExtendedTypeDefinition> GetFormatData()
        {


@powercode, your last commit had 2 failures in PowerShell-CI-windows
Get-ItemProperty.Should be able to access a property using the Path and name switches

Expected $null, but got C. at <ScriptBlock>, D:\a\1\s\test\powershell\Modules\Microsoft.PowerShell.Management\Get-ItemProperty.Tests.ps1: line 36 36: $output.PSDrive | Should -Be $testprovider

String cmdlets.Select-String.Network path

Expected 2, but got 0. at <ScriptBlock>, D:\a\1\s\test\powershell\Modules\Microsoft.PowerShell.Utility\string.tests.ps1: line 56 56: (select-string -LiteralPath $fileNameAsNetworkPath "b").count | Should -Be 2

PoshChan · 2019-11-10T00:37:22Z

src/System.Management.Automation/FormatAndOutput/DefaultFormatters/FileSystem_format_ps1xml.cs

    internal sealed class FileSystem_Format_Ps1Xml
    {
        internal static IEnumerable<ExtendedTypeDefinition> GetFormatData()
        {


@powercode, your last commit had 3 failures in PowerShell-CI-macos
Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: \foo:bar.txt

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /Users/runner/runners/2.160.0/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: /foo:

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /Users/runner/runners/2.160.0/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: :bar

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /Users/runner/runners/2.160.0/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

PoshChan · 2019-11-10T00:40:12Z

src/System.Management.Automation/FormatAndOutput/DefaultFormatters/FileSystem_format_ps1xml.cs

    internal sealed class FileSystem_Format_Ps1Xml
    {
        internal static IEnumerable<ExtendedTypeDefinition> GetFormatData()
        {


@powercode, your last commit had 3 failures in PowerShell-CI-linux
Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: \foo:bar.txt

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /home/vsts/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: /foo:

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /home/vsts/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

Basic FileSystem Provider Tests.Validate basic FileSystem Cmdlets.Get-Content on Unix succeeds with folder and file with colon: :bar

Expected exactly 'Hello', but got $null. at <ScriptBlock>, /home/vsts/work/1/s/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 246 246: $files[0] | Get-Content | Should -BeExactly "Hello"

ghost · 2020-05-27T02:04:11Z

This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days.
Mainainer, Please provide feedback and/or mark it as Waiting on Author

ghost · 2020-06-11T23:00:43Z

This pull request has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 15 days. It will be closed if no further activity occurs within 10 days of this comment.

powercode force-pushed the Perf/FileInfoPsProps branch 2 times, most recently from 17ed0b3 to 1fc03a1 Compare February 26, 2019 23:56

daxian-dbw self-assigned this Feb 27, 2019

iSazonov reviewed Feb 27, 2019

View reviewed changes

src/Microsoft.PowerShell.Commands.Management/commands/management/ContentCommandBase.cs Outdated Show resolved Hide resolved

powercode force-pushed the Perf/FileInfoPsProps branch 2 times, most recently from b6dd06d to 073cbc5 Compare March 1, 2019 01:19

iSazonov mentioned this pull request Mar 4, 2019

Fix tab completion for filename contains literal wildcard character #7407

Closed

11 tasks

This was referenced Mar 12, 2019

Get-ChildItem -Name adds provider NoteProperty members, Get-ChildItem -Name -Recurse does not #9120

Closed

Performance problem: -Name slows down Get-ChildItem -Recurse #9119

Closed

powercode force-pushed the Perf/FileInfoPsProps branch from 528df8d to 9eecd8b Compare March 27, 2019 16:08

stale bot added the Stale label Apr 26, 2019

daxian-dbw removed the Stale label Apr 26, 2019

SteveL-MSFT added the Experimental Experimental Feature label Jun 4, 2019

powercode force-pushed the Perf/FileInfoPsProps branch 2 times, most recently from 64cf96d to 5943ad2 Compare June 7, 2019 22:19

TravisEz13 added the Review - Waiting on Author label Oct 5, 2019

powercode added 2 commits November 9, 2019 23:58

Using derived PSObject for filesystem/registry PS* properties.

f65bef6

extracting a variable for readability.

b0b85a0

powercode force-pushed the Perf/FileInfoPsProps branch from 5943ad2 to b0b85a0 Compare November 9, 2019 23:14

PoshChan reviewed Nov 9, 2019

View reviewed changes

Adding missing PSDrive properties

27d9cc6

PoshChan reviewed Nov 10, 2019

View reviewed changes

powercode added 2 commits November 12, 2019 07:57

Adding back content psobject and experimental feature

60da5d7

Work in progress

c14e213

ghost added the Review - Needed The PR is being reviewed label May 27, 2020

TravisEz13 added Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept and removed Review - Needed The PR is being reviewed Review - Waiting on Author labels May 27, 2020

ghost added the Stale label Jun 11, 2020

ghost closed this Jun 22, 2020

OwnageIsMagic mentioned this pull request Aug 17, 2023

Get-ChildItem (-Recurse) and (-Recurse -Name) return inconsistent #20129

Closed

This pull request was closed.

PSOptimizedProvider objects - Experimental feature #8995

PSOptimizedProvider objects - Experimental feature #8995

Uh oh!

Conversation

powercode commented Feb 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Context

PR Checklist

Uh oh!

powercode commented Feb 27, 2019

Uh oh!

Uh oh!

iSazonov commented Mar 1, 2019

Uh oh!

powercode commented Mar 1, 2019

Uh oh!

iSazonov commented Mar 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

powercode commented Mar 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iSazonov commented Mar 3, 2019

Uh oh!

daxian-dbw commented Mar 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stale bot commented Apr 26, 2019

Uh oh!

powercode commented May 20, 2019

Uh oh!

PoshChan commented Jun 7, 2019

Uh oh!

PoshChan commented Jun 7, 2019

Uh oh!

PoshChan commented Jun 7, 2019

Uh oh!

PoshChan Nov 9, 2019

Choose a reason for hiding this comment

Uh oh!

PoshChan Nov 9, 2019

Choose a reason for hiding this comment

Uh oh!

PoshChan Nov 9, 2019

Choose a reason for hiding this comment

Uh oh!

PoshChan Nov 10, 2019

Choose a reason for hiding this comment

Uh oh!

PoshChan Nov 10, 2019

Choose a reason for hiding this comment

Uh oh!

ghost commented May 27, 2020

Uh oh!

ghost commented Jun 11, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

powercode commented Feb 26, 2019 •

edited

Loading

iSazonov commented Mar 1, 2019 •

edited

Loading

powercode commented Mar 3, 2019 •

edited

Loading

daxian-dbw commented Mar 3, 2019 •

edited

Loading