Repository navigation
Improve performance of Get-ChildItem #3575
Description
Steps to reproduce
Type 'dir -r >/tmp/dir-output' at the top of a directory subtree containing around 5K files.
Expected behavior
Command completes quickly.
The comparable Unix command
find . -print0 | xargs -0 ls -al >/tmp/dir-output
consistently takes less than 300ms to complete.
Actual behavior
Command takes between 3.7 and 4.1 seconds to complete, both using wall clock and
Measure-Command { dir -r >/tmp/dir-output }
Environment data
$PSVersionTable
For me, this returns no result.
Reacted by WSLUser, Richard Lim and Alexander Sklar
Activity
iSazonov commentedon Apr 16, 2017
Marcel Weiher (@mpw) Thanks for your report!
Native utilities on Unix and Windows use low level APIs and work with raw data. PowerShell work on top the APIs and add extra code levels (convert all in PSObjects). So it will always be slower. Of course we try to do things faster.
mpw commentedon Apr 20, 2017
Hmmm...an order of magnitude is rather significant, especially considering that the Unix version was doing significantly more work, splitting the task between two processes, communicating paths between the processes, parsing strings and therefore also doing full path evaluation for each file.
My bad, I was actually going for something different when I hit upon this by accident. The more efficient Unix version would of course be ls -lR, and that executes in
real 0m0.062s
user 0m0.029s
sys 0m0.028s
So PowerShell is actually more than 50 times slower. I realize layers can add overhead, but should simple CPU processing be able to add almost 2 orders of magnitude to filesystem/kernel operations?
Sample attached. Surprisingly informative despite all the jitted code not showing up.
powershell_2017-04-21_002049_AH6L.sample.txt
iSazonov commentedon Apr 21, 2017
What PowerShell versions you use for testing?
Currently we moved to .Net Cor 2.0 and since Alfa.19 you can test based on this version.
[-]"dir -r" is slow on macOS[/-][+]"dir -r" is slow (not just on macOS)[/+]mpw commentedon Apr 21, 2017
I just tried the same command with PowerShell on Windows (running emulated in Parallels) and it also took 3 seconds. While there is likely some virtualization overhead, it almost certainly isn't 50x.
This was a fresh install of Windows + VS downloaded just yesterday.
[-]"dir -r" is slow (not just on macOS)[/-][+]"dir -r" is slow (not just) on macOS[/+]powercode commentedon Apr 22, 2017
There are many layers of abstraction involved. PSDrives, PSProviders, Get-Childitem with generalized filtering, a generic, programmable, formatting system, an object pipeline with extremely powerful (and expensive) parameter binding and all this on top of the CLR. So we are comparing apples to oranges.
That said: it's too slow. We are in need of a performance push for the core scenarios, and ls -R is definitely one of those.
I was actually profiling this last night, as I also feel the pain.
iSazonov commentedon Oct 8, 2017
CoreFX has performance issues with file system too.
powercode commentedon Oct 10, 2017
Is is also interesting to see how much of it is formatting.
What times to you get if you pipe the output to Out-Null?
That should give us the raw powershell provider filesystem times.
Is also something to compare against. Just getting the filenames, without stat-ing each file.
I think .NET is quite inefficient when doing this, stat-ing each file instead of getting that at the same time as enumerating the files.
Maybe this should be fixed in System.IO.Directory, with another method that enumerates items returning FileSystemInfo instead of string.
13 remaining items
[-]"dir -r" is slow (not just) on macOS[/-][+]Improve performance of `Get-ChildItem`[/+]joeyaiello commentedon Dec 7, 2020
Unfortunately, this is a large body of work that we wouldn't want to do until we embark on a hypothetical FileSystemProvider v2, as opposed to band-aiding the existing FileSystemProvider.
iSazonov commentedon Dec 8, 2020
Joey Aiello (@joeyaiello) Please see #12834 - we can improve performance for most scenarios today (only current limitation for new approach is Depth parameter - the feature request has been opened by me in .Net Runtime repository).
microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 23, 2023
This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.