Skip to content

Conversation

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented Jun 21, 2018

PR Summary

Continue #6929
Replace #7100

Cleanup code to replace Utils.FileExists(), Utils.DirectoryExists() and Utils.ItemExists() methods with .Net Core methods.

PR Checklist


This change is Reviewable

@iSazonov iSazonov force-pushed the cleanup-file-exists-2 branch 3 times, most recently from 27fa002 to a62483f Compare June 21, 2018 14:33
@iSazonov
Copy link
Collaborator Author

WIP - still 6 commits upcoming...

@SteveL-MSFT SteveL-MSFT changed the title Cleaup Utils.FileExists(), Utils.DirectoryExists() and Utils.ItemExists() methods WIP: Cleaup Utils.FileExists(), Utils.DirectoryExists() and Utils.ItemExists() methods Jun 21, 2018
@daxian-dbw daxian-dbw self-assigned this Jun 21, 2018
@iSazonov iSazonov force-pushed the cleanup-file-exists-2 branch 2 times, most recently from e525be4 to bb1aa8a Compare June 22, 2018 09:40
@iSazonov
Copy link
Collaborator Author

iSazonov commented Jun 22, 2018

@daxian-dbw Please look the issue with Unix compiling.

843/home/travis/build/PowerShell/PowerShell/src/System.Management.Automation/help/UpdatableHelpSystem.cs(1650,106): error CS0117: 'EnumerationOptions' does not contain a definition for 'MatchCasing' [/home/travis/build/PowerShell/PowerShell/src/System.Management.Automation/System.Management.Automation.csproj]

Yesterday the commit (UpdatableHelpSystem.cs) was compiled without errors. Today it failed. I tryed compile the code on CI Appveyor and get the same error.
Maybe we get updated version of .Net Core with a bug?

Locally I can compile without problem and IntelliSense works well in VS Code.

Tracking Issue https://github.com/dotnet/corefx/issues/30626

@iSazonov
Copy link
Collaborator Author

I have only one commit related to Unix but can not push it until travis-ci problem will be resolved.

@iSazonov iSazonov requested a review from TravisEz13 as a code owner June 25, 2018 04:38
@iSazonov
Copy link
Collaborator Author

I tested locally all versions - msi and zip packages - 2.1.300 and 2.1.301 and the project is compiled well.
So problem is somewhere else.
I wonder how and why the problem is arised on both CIs.

@iSazonov iSazonov force-pushed the cleanup-file-exists-2 branch from 8c2fa88 to 4c84b21 Compare June 29, 2018 06:31
@iSazonov
Copy link
Collaborator Author

Rebased to move to .Net Core 2.1.1.

@daxian-dbw
Copy link
Member

'EnumerationOptions' does not contain a definition for 'MatchCasing'

Ah, this is probably because the compiler resolves EnumerationOptions to System.Management.EnumerationOptions instead of System.IO.EnumerationOptions. Reference to the package System.Management was added to System.Management.Automation project some time back.

You can use the full type name System.IO.EnumerationOptions and it should work.

The CI failure now is different though, it's complaining about Utils.FileExists not being present.

@iSazonov iSazonov force-pushed the cleanup-file-exists-2 branch 3 times, most recently from 4b4fe4d to f281765 Compare July 2, 2018 10:23
@iSazonov iSazonov changed the title WIP: Cleaup Utils.FileExists(), Utils.DirectoryExists() and Utils.ItemExists() methods Cleaup Utils.FileExists(), Utils.DirectoryExists() and Utils.ItemExists() methods Jul 2, 2018
@iSazonov
Copy link
Collaborator Author

iSazonov commented Jul 2, 2018

@daxian-dbw Thanks for help!

The PR is ready for review.

@iSazonov iSazonov force-pushed the cleanup-file-exists-2 branch from f281765 to c53aaa3 Compare July 6, 2018 03:57
@iSazonov
Copy link
Collaborator Author

iSazonov commented Jul 6, 2018

Rebased to resolve a merge conflict.

@iSazonov iSazonov changed the title Cleaup Utils.FileExists(), Utils.DirectoryExists() and Utils.ItemExists() methods Cleanup Utils.FileExists(), Utils.DirectoryExists() and Utils.ItemExists() methods Jul 6, 2018
Copy link
Member

@adityapatwardhan adityapatwardhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Signed of with 1 non-blocking comment.

{
}
catch (UnauthorizedAccessException)
if (!Directory.Exists(drive.Root))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be

return !DirectoryExists(drive.Root)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems original code is easier to understand.

@iSazonov
Copy link
Collaborator Author

@daxian-dbw Can we merge the PR?

@daxian-dbw
Copy link
Member

@iSazonov Let me quickly go through the changes.

}
}
#else
if (Utils.FileExists(path))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part was in windows-only section. You moved it above at the beginning of the method, but that's not made "compile-only-windows"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assumed that we will always receive the path with the right case. (Seems we get the path in code not from user). So the first block will always return a value on all platforms. The code below will only search the file on Unix with ignoring case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it makes sense to to always have that check on all plats. #closed

// We use 'FileInfo.Attributes' (not 'FileInfo.Exist')
// because we want to get exceptions
// like UnauthorizedAccessException or IOException.
if ((int)item.Attributes != -1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please explain how do you decide which Utils.DirectoryExists or Utils.FileExists can be replaced with Directory.Exists and File.Exists (swallow all exceptions), and which ones need to continue throw exceptions? I wonder why we want to get exceptions in this method.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to keep the logic of the previous code. And I tried to make as few changes as possible. I'm afraid that this PR will grow many times if we try to correct the previous logic in all places where Utils.DirectoryExists or Utils.FileExists exist.

Here I'm afraid that we'll get issues in LoadStringFromPath() .

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you think this method GetFilePath(string path) should throw the exception from checking the file existence?

I tried to keep the logic of the previous code

The previous code just used Utils.FileExists and Utils.DirectoryExists like everywhere else. Here, you didn't replace them to File.Exists and Directory.Exists but check the existence in a way that may throw exception. I want to know what your decision is based on.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Utils.FileExists and Utils.DirectoryExists can throw but File.Exists and Directory.Exists don't throw.
If calling code catch exceptions and ignore them we can safely replace Utils.FileExists and Utils.DirectoryExists with File.Exists and Directory.Exists.
Here calling code don't catch and ignore exceptions so I used item.Attributes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If calling code catch exceptions and ignore them we can safely replace Utils.FileExists and Utils.DirectoryExists with File.Exists and Directory.Exists.
Here calling code don't catch and ignore exceptions so I used item.Attributes.

Does this mean for all the places you changed to File.Exists or Directory.Exists, you have checked that the calling code actually caches exceptions and ignore them?

Copy link
Collaborator Author

@iSazonov iSazonov Jul 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I did this commit by commit.

[daxian-dbw] #Closed

if (fspDynamicParam != null && fspDynamicParam.FollowSymlink)
{
tracker = new InodeTracker(directory.FullName);
tracker = new InodeTracker(((DirectoryInfo)fsinfo).FullName);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need to cast fsinfo to DirectoryInfo to access FullName?

Copy link
Collaborator Author

@iSazonov iSazonov Jul 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@daxian-dbw daxian-dbw Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a quick check of the properties, and it looks the casting is not needed.

void Main()
{
	FileSystemInfo info1 = GetFileSystemInfo(@"C:\arena\temp.txt");
	FileSystemInfo info2 = GetFileSystemInfo(@"C:\arena");
	
	Console.WriteLine("info1.Name: {0}", info1.Name);
	Console.WriteLine("info1.FullName: {0}", info1.FullName);
	
	Console.WriteLine("info2.Name: {0}", info2.Name);
	Console.WriteLine("info2.FullName: {0}", info2.FullName);
}

static FileSystemInfo GetFileSystemInfo(string path)
{
    if (File.Exists(path))
	{
            return new FileInfo(path);
	}
	else if (Directory.Exists(path))
	{
	    return new DirectoryInfo(path);
	}

	return null;
}

Console Output:

info1.Name: temp.txt
info1.FullName: C:\arena\temp.txt
info2.Name: arena
info2.FullName: C:\arena

Copy link
Collaborator Author

@iSazonov iSazonov Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we don't need casting because the value is evaluated in object creation time and here we only retrive the value.

Fixed.
[daxian-dbw] #Closed

fileInfo.Name,
fileInfo.FullName,
((FileInfo)fsinfo).Name,
((FileInfo)fsinfo).FullName,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, do you need to cast?

Copy link
Collaborator Author

@iSazonov iSazonov Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we don't need casting because the value is evaluated in object creation time and here we only retrive the value.

Fixed.
[daxian-dbw] #Closed

}
else
WriteItemObject(fileInfo, path, false);
WriteItemObject((FileInfo)fsinfo, path, false);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to cast here.

Copy link
Collaborator Author

@iSazonov iSazonov Jul 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.
[daxian-dbw] #Closed

path = NormalizePath(path);

return Utils.DirectoryExists(path);
return Directory.Exists(path);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question, why we ignore all exceptions here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This behavior corresponds to the xml description of the IsItemContainer method. We use result in MakePath or to get DirectoryInfo/FileInfo - so we always get specific error later. Seems we could cleanup using IsItemContainer() - it turns out we make a lot of extra kernel calls.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I guess it makes sense to swallow all exceptions here. #closed

{
isContainer = true;
fileSystemInfoShell = PSObject.AsPSObject(new DirectoryInfo(path));
fileSystemInfoShell = PSObject.AsPSObject((DirectoryInfo)fsinfo);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to cast here

Copy link
Collaborator Author

@iSazonov iSazonov Jul 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.
[daxian-dbw] #Closed

{
// Maybe the path is a file name so try a FileInfo instead
fileSystemInfoShell = PSObject.AsPSObject(new FileInfo(path));
fileSystemInfoShell = PSObject.AsPSObject((FileInfo)fsinfo);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto.
Actually, this if/else block can be removed.

Copy link
Collaborator Author

@iSazonov iSazonov Jul 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.
[daxian-dbw] #Closed

}

if (fileSystemInfoShell != null)
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fileSystemInfoShell still could be null, right?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? We call PSObject.AsPSObject(fsinfo) where fsinfo != null.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, you moved the code within if (fsinfo != null). I missed that. #Closed


// If the directory exists, just return it.
// Otherwise fallback to previous behavior and let provider handle it.
if (Directory.Exists(userPath))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please explain why having this change? And this might not be the right place for the new code, because a new collection is just allocated above, but you might return without using the collection at all.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this PR I went through so many code ways that I do not remember how I got here.
As far as I see the idea was simple - do not perform an analysis if the path already exists.

Fixed - I return without using the collection.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a quick browsing of the code after this, it seems the new code you added treats the userPath as a literal path without consulting context.SuppressWildcardExpansion. Say userPath is /blah/a[bc], and it happens to have a folder named a[bc].

I'm not that familiar with the file system code, so there could be other concerns on this early return. I suggest to focus on the topic of this PR, and leave optimization to a separate issue/PR.

Copy link
Collaborator Author

@iSazonov iSazonov Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code was removed.
[daxian-dbw] #Closed

path = NormalizePath(path);

PSObject results = new PSObject();
PSObject fileSystemInfoShell = null;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This variable now should be moved in if (fsinfo != null), since it's only used in the if block.

Copy link
Collaborator Author

@iSazonov iSazonov Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.
[daxian-dbw] #Closed


// If the directory exists, just return it.
// Otherwise fallback to previous behavior and let provider handle it.
if (Directory.Exists(userPath))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a quick browsing of the code after this, it seems the new code you added treats the userPath as a literal path without consulting context.SuppressWildcardExpansion. Say userPath is /blah/a[bc], and it happens to have a folder named a[bc].

I'm not that familiar with the file system code, so there could be other concerns on this early return. I suggest to focus on the topic of this PR, and leave optimization to a separate issue/PR.

@iSazonov
Copy link
Collaborator Author

@daxian-dbw Could you please continue?

@TravisEz13
Copy link
Member


src/System.Management.Automation/engine/SessionStateDriveAPIs.cs, line 966 at r1 (raw file):

                        if (systemDriveInfo.DriveType == DriveType.NoRootDirectory)
                        {
                            if (!Directory.Exists(drive.Root))

Is this going to suppress IO and UnAuthorized Exceptions like the old code?

@TravisEz13
Copy link
Member


src/System.Management.Automation/namespaces/FileSystemProvider.cs, line 136 at r1 (raw file):

                if (isContainer)
                {
                    return new DirectoryInfo(path);

Why don't you cast fsinfo to a DirectoryInfo like you do in a later line?

Dir((DirectoryInfo)fsinfo, recurse, depth, nameOnly, returnContainers, tracker);

Copy link
Member

@TravisEz13 TravisEz13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 9 of 9 files at r1.
Reviewable status: :shipit: complete! all files reviewed

@iSazonov
Copy link
Collaborator Author

Is this going to suppress IO and UnAuthorized Exceptions like the old code?

Yes, we preserve old code logic.

Why don't you cast fsinfo to a DirectoryInfo like you do in a later line?

Here we create objects of different types.
In Dir() you referenced we have above if (isContainer) so we really have DirectoryInfo object and can cast to the type.

@daxian-dbw
Copy link
Member

daxian-dbw commented Jul 19, 2018

@iSazonov Will do today. Sorry for being tardy in reviewing this PR (need to spend much time on releasing this week). Didn't get to it today ☹️ Tomorrow for sure.

// We use 'FileInfo.Attributes' (not 'FileInfo.Exist')
// because we want to get exceptions
// like UnauthorizedAccessException or IOException.
if (fileList.Length > 0 && (int)fileList[0].Attributes != -1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original code doesn't check for existence for the files returned from Direcotry.GetFiles. Why do you add the check now? Can't we trust direcotryInfo.GetFiles?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also respond to this comment: #7129 (comment)

Copy link
Member

@daxian-dbw daxian-dbw Jul 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iSazonov Can you please respond to the first comment in this thread (#7129 (comment))?

That's the only open comment left now :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, GitHub sometimes tricky hides comments and I can not find where to type response :-)

You are right - the check is dead code - I removed it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks! #Closed

@daxian-dbw daxian-dbw merged commit e385481 into PowerShell:master Jul 25, 2018
@daxian-dbw daxian-dbw added the Issue-Code Cleanup the issue is for cleaning up the code with no impact on functionality label Jul 25, 2018
@iSazonov iSazonov deleted the cleanup-file-exists-2 branch July 26, 2018 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Issue-Code Cleanup the issue is for cleaning up the code with no impact on functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants