Repository navigation
Rewrite FileSystemProvider as v2 #5785
Description
The current FileSystemProvider is complex to maintain (small changes introduce regressions), expects NTFS, and not optimized. We should consider writing a new v2 version that works side-by-side with the current implementation kept for backwards compatibility.
Reference related issues that we can't address in current provider due to compatibility concerns.
Related #5699
Reacted by xtqqczze and Mikey Lombardi (He/Him)
Activity
iSazonov commentedon Jan 4, 2018
I saw that .Net Core team is working on optimization in this area. I think we get these improvements in 2.1.0.
Also we already have #5002 - I've been looking into this and I think our globbing code is a weakness.
lzybkr commentedon Jan 4, 2018
If going down this route, you should consider making it easier to write any provider - the current interface is too difficult.
SteveL-MSFT commentedon Jan 4, 2018
Jason Shirk (@lzybkr) agree, created #5789
iSazonov commentedon Feb 5, 2018
Note: in .Net Core 2.1 File enumeration extensibility. is expected. dotnet/corefx#26806
powercode commentedon Feb 5, 2018
They have done a ton of work to make it both faster and more flexible. No more kernel roundtrips for each FileInfo.
JoeSalmeri commentedon Oct 22, 2019
Steve Lee (@SteveL-MSFT)
Sorry, I did not mean to imply that the problem was your doing or something you created and I
certainly do appreciate all you have done since joining the team!
My issue is with the answers provided regarding the problem.
I have been in the field for 32 years and have programmed in at least 25 different programming
languages during my career.
In all cases, there is one common theme and answer to the question, "Why do we write programs?"
We program so that we can automate processes and produce reliable, consistent, and repeatable results.
There are very few cases or reasons why we would want to write code which produced results which were not repeatable or consistent (random number generator comes to mind) as that would defeat the purpose of automating the tasks.
If we were discussing what the expected behavior was versus what the actual behavior was that
is something that could be addressed (but maybe not desired) by changing your/my expectations
to match the actual behavior.
That is NOT the case here. The same code run multiple times does not produce the same results.
Before taking the time to write up that bug and all the various situations, I spent CONSIDERABLE
time trying various syntax (trailing slashes, *, ., and others) to try and find a single syntax
that would produce a consistent result which could be used in code which would cover the use cases.
Any good developer is not going to write new code using existing code which does not produce repeatable results because then their code also becomes unreliable and unstable.
I'll bet that most PS developers have taken the route that I have which is to use external tools to
perform that functionality in their code so that they can get consistent repeatable results.
I take breaking code issues quite seriously too, however, when the original code is broke in the first
place, it makes sense to fix it regardless of whether it would break "other" code.
You can argue as to whether you want to label this as a bug or whether it was a design flaw, however, that does not change the fact that the existing code does NOT produce consistent results.
Imagine if the existing Copy-Item had a bug where it deleted the file after it copied it. Using the
answer you provided, that would not be addressed because it might break existing code that depended on that bug. I cannot imagine anyone considering that a good reason or answer.
It's been a while since I wrote all that up and reviewed it, however, there were also examples where
data loss occurs because there was no warning or confirmation. That certainly goes against the
basic PS design principles of not doing harm without a warning.
While I appreciate that you opened the issue for FileSystemProvider V2, I am quite skeptical that it will ever occur based on your response here, your comment about "if #5785 ever gets done", and the fact that these issues have gone unaddressed pretty much since the beginning of PS.
Regardless of whether the existing code is fixed or whether a V2 is created, there should still
be a desired to provide this basic functionality in the product, especially one that is marketed
as the replacement for all of the other command shells.
I really hope that someone take up the task of addressing this most basic and core functionality.
Sincerely,
Joe
SteveL-MSFT commentedon Dec 7, 2020
Related #3575
JoeSalmeri commentedon Dec 7, 2020
Steve Lee (@SteveL-MSFT)
Is FileSystemProvider v2 every going to be on the roadmap?
Seems like it just keeps getting bumped.
11 remaining items
doctordns commentedon Jun 7, 2022
Again - getting angry is pointless. Now that PowerShell is open source, you can see what the team is doing and what its priorities are. The bug you refer to is one I have never seen in practice - but I've only been using PowerShell for just short of 20 years. I can tell you, there are some tough tradeoffs that have to be made, but you can see the process (unlike in other WIndows components that are totally opaque in terms of the changer (The feedback hub is just a black hole).
The challenge here is that if the team 'fixes' this problem, it is likely to be a complex change (eg File System Provider (V2)) and would not be cheap, It also would almost certainly be a breaking change that could (would?) break a LOT of existing scripts that depend on the current behavior. A targeted fix for just the original issue could be considered but this too has the potential to break stuff. And for me breaking stuff, without a really good reason, is just not acceptable. Frankly, I would prefer to leave this as it is, with well-documented workarounds, than to try to fix it. I do get that mileage may vary on this!
Like so many issues here in the repo, this is one I suspect we may just close as "won't fix - by design" which is what I am thinking today. But can you convince the team that this is a bug that NEEDS fixing? Is there enough user demand for this change? Right now, I am not seeing the need to make this change today. But please convince me I am not looking in the right places.
JoeSalmeri commentedon Jun 8, 2022
Thomas Lee (@doctordns)
I too have been using PS for a very long time.
I may certainly be wrong here but your comment leads me to believe that you didn't even read the bug report I referenced where I detailed 12 basic usage cases comparing and contrasting Copy-Item with, cmd copy, xcopy, and robocopy and how broken and unusable Copy-Item is for all but the most basic use case.
Closing an issue as by-design says that someone purposely designed Copy-Item to NOT be able to produce the same results when the same command and input is used. You REALLY believe that someone would PURPOSELY do something like that?
Anyone that worked for my company that had that mentality would be fired.
Who writes programs that cannot produce the same consistent results?
Isn't the very reason we write programs is so that we can automate a process and produce the same results with the same input?
theJasonHelmick commentedon Jun 9, 2022
Hello everyone -- I appreciate the conversation. Steve entered this issue as "We should consider " - which we continue to do. This is complex work that raises many risks that could be far worse than the original problem. Joe S (@JoeSalmeri), I certainly appreciate the frustration this causes, however this is not work we have committed to yet until we understand the best path forward . This work will continue to be evaluated by the Working group and I encourage folks to provide technical feedback that could support a solution.
To help, I will discuss with the WG and get some current status information and report in the next couple of weeks.
iSazonov commentedon Jun 9, 2022
In fact, the functionality is divided into four parts - cmdlets itself, Globbing, SessionState/SessionStateInternal.
Only Globbing could be easily isolated and replaced with new experimental code.
In addition, there are several classes that make internal APIs public to PowerShell SDK (like PathIntrinsics). This practically blocks changes of internal APIs if we strictly intend to avoid breaking changes.
As for cmdlets, I tried to strictly state PowerShell concepts in the RFC and found inconsistencies and deviations in the implementations. If we intend to conform strictly to PowerShell concepts and path manipulation rules, it will inevitably lead to a lot of breaking changes.
Among other things, there are performance problems which should have been solved if we are going to do anything in this area (i.e. path/file operations). This also requires changes on all four levels.
We could of course isolate the new code as an experimental feature starting with the cmdlet level, but as you understand we would have to re-design and write all four levels of code from scratch and this would inevitably affect all providers. So initially we should have written several thousand new tests for all providers and their cmdlets to avoid regressions. At the very least we need these new tests to open the door to innovation.
JoeSalmeri commentedon Jun 9, 2022
Jason Helmick (@theJasonHelmick)
Thank you Jason for your thoughtful reply, I really appreciate it.
I believe, but could certainly be wrong, that part of the complexity is because Copy-Item handles multiple providers which have some similarities are also very different.
For example, the env, and alias don't have the "directory" concept that the FileSystem Provider has.
When you discuss with the WG could you please present the items I provided in my bug report
#10458
as it contains a detailed review of 12 use cases and compares Copy-Item, cmd copy, xcopy, and robocopy as well as a nice summary the findings for each use case and copy method.
Thank You!
JoeSalmeri commentedon Jun 9, 2022
Ilya (@iSazonov)
I agree about not breaking existing code with ONE exception.
When the existing Copy-Item cannot be run with the same options and produce the same results then it it should break existing code, HOWEVER, since people write programs to provide consistent results with the same input, it seems pretty clear that everyone that runs into these issues switches to another method like xcopy, robocopy, rsync, etc. because Copy-Item cannot produce a consistent result.
Makes me wonder if what we really need is a Copy-FileItem or something similar instead of being lumped into a generic Copy-Item.
iSazonov commentedon Jun 10, 2022
My personal preference is to accept the inevitable.
This is reminiscent of the IE6 story. It was so popular that it was itself a standard, while violating all public standards. When its popularity declined, it became a serious problem. MSFT was forced to release 5 (!) versions of IE each introducing a new compatibility mode. This made IE so complicated that MSFT threw it out and said it would use Chrome!
Speaking of pwsh, there are about 50 breaking changes tagged in 2 1/2 years, plus probably as many more in the previous 2 1/2 years before 2016, plus things that were deliberately cut out (like workflow, etc.). As a result, we see a huge number of incompatibilities with Windows PowerShell.
We accept this as inevitable.
The policy today is that all innovation is in pwsh only, Windows PowerShell is frozen.
Maybe it's time to take a new leap and say pwsh 7 is frozen, the last LTS will be 7.4 and all innovations will be in pwsh 8.
These innovations could modernize virtually all pwsh subsystems.
Of course, that would require a lot of effort. But I don't see how we could make it easier and cheaper within version 7. It's more of a dead-end way to transform IE6.
JoeSalmeri commentedon Jun 14, 2022
Ilya (@iSazonov)
Interesting comparison.
One big difference that I see though.
While IE 6 may have violated all kinds of standards, back then it was what people coded too so it worked.
In the case of the Copy-Item issue I reported, it is basically unusable in all but the most simple use cases forcing people to use alternative non PS solutions like robocopy, xcopy, rsync, etc.
I can understand using non PS solutions for things that are not what I would consider "the basics" but for a product that will most likely replace the old command prompt which has existed since the beginning of time one would expect the basic features to work in PS.
Overall PS is awesome, but it is disappointing that these flaws exist in such a core feature. I remember when I first started using PS before I wrote up my 12 use case bug report where I figured I must be doing something wrong until I did some research and found many people searching for how to get it working.
I hope that your suggestion sticks and that PS 8 is all about making sure the basics work and addressing any areas where performance is not as good.
jdhitsolutions commentedon Jul 6, 2022
The Cmdlet Working Group has reviewed this issue. While a new provider would/could provide benefits the costs of re-writing the provider are high. We are marking this issue as Won't Fix
JoeSalmeri commentedon Aug 18, 2022
Jeff Hicks (@jdhitsolutions)
One would think that the FileSystemProvider is a key part of powershell and therefore it would be considered something that was important to make sure it worked properly in the basic use cases.
PowerShell is such a GREAT product overall but the desire to cram more features in when some of the basics are so badly broken defies logic.
JamesWTruher commentedon Apr 15, 2024
also this: #20651