Skip to content

Conversation

@daxian-dbw
Copy link
Member

@daxian-dbw daxian-dbw commented Jun 22, 2018

PR Summary

The major refactoring changes are:

  • In ModuleIntrisic.cs, remove unneeded Windows-PowerShell-only code.
  • In ModuleUtils.cs
    • use the new API Directory.GetDirectories(string path, string searchPattern, EnumerationOptions enumerationOptions) and Directory.GetFiles(string path, string searchPattern, EnumerationOptions enumerationOptions) to enumerate files and sub-directories within a directory path.
    • remove the unused parameter bool force from GetDefaultAvailableModuleFiles(bool force, bool isForAutoDiscovery, ExecutionContext context)
    • refactor the method GetModuleVersionsFromAbsolutePath. Add more comments and rename the method name.
  • In ModuleCmdletBase.cs, refactor the method GetModuleForNonRootedPaths to GetModuleForNames to simply its implementation.
  • In PSModuleInfo.cs
    • group the declarations of Declared*Exports fields together
    • rename _detected*Exports fields to Detected*Exports to group them together. They are internal fields and used outside PSModuleInfo.

Pref improvement

NOTE: Use Windows PowerShell default module paths to exercies the code with more modules.
The following measurement are done without the crossgen'ed assemblies.

There is some perf improvement after this refactoring change:

  • For Get-Module -ListAvailable, there is about 36% speed improvement for 94 default modules.
  • For Get-Module -ListAvailable -All, there is about 14% speed improvement for totally 600 module files.
  • For Get-Module <name> -ListAvailable -List, there is over 17x speed improvement for finding 13 modules from 600 modules. This is because we now filter names using the module file before creating a PSMdouleInfo object.

Before

PS> $env:PSModulePath = "C:\Users\dongbow\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\windows\system32\WindowsPowerShell\v1.0\Modules"
PS> $time = foreach ($i in 1..20) { Measure-Command { gmo -ListAvailable } | % TotalMilliseconds }
PS> $time | Measure-Object -Maximum -Minimum -Average | select Count, Maximum, Minimum, Average

Count  Maximum  Minimum    Average
-----  -------  -------    -------
   20 701.3139 623.5225 642.628685

PS> $time = foreach ($i in 1..20) { Measure-Command { gmo -ListAvailable -All } | % TotalMilliseconds }
PS> $time | Measure-Object -Maximum -Minimum -Average | select Count, Maximum, Minimum, Average

Count   Maximum   Minimum     Average
-----   -------   -------     -------
   20 1369.1893 1243.1195 1310.147175

PS> $time = foreach ($i in 1..20) { Measure-Command { gmo windows* -ListAvailable -All } | % TotalMilliseconds }
PS> $time | Measure-Object -Maximum -Minimum -Average | select Count, Maximum, Minimum, Average

Count   Maximum   Minimum    Average
-----   -------   -------    -------
   20 1535.3805 1292.2188 1353.62972

After

PS> $env:PSModulePath = "C:\Users\dongbow\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\windows\system32\WindowsPowerShell\v1.0\Modules"
PS> $time = foreach ($i in 1..20) { Measure-Command { gmo -ListAvailable } | % TotalMilliseconds }
PS> $time | Measure-Object -Maximum -Minimum -Average | select Count, Maximum, Minimum, Average

Count Maximum  Minimum  Average
----- -------  -------  -------
   20 438.338 393.7476 410.62353

PS> $time = foreach ($i in 1..20) { Measure-Command { gmo -ListAvailable -All } | % TotalMilliseconds }
PS> $time | Measure-Object -Maximum -Minimum -Average | select Count, Maximum, Minimum, Average

Count   Maximum   Minimum Average
-----   -------   -------   -------
   20 1341.5443 1055.1771 1120.4748

PS> $time = foreach ($i in 1..20) { Measure-Command { gmo windows* -ListAvailable -All } | % TotalMilliseconds }
PS> $time | Measure-Object -Maximum -Minimum -Average | select Count, Maximum, Minimum, Average

Count Maximum Minimum Average
----- ------- -------   -------
   20 78.0931 61.9288 66.194415

PR Checklist

@daxian-dbw daxian-dbw requested a review from rjmholt June 22, 2018 19:42
@daxian-dbw daxian-dbw requested a review from BrucePay as a code owner June 22, 2018 19:42
@daxian-dbw daxian-dbw removed the request for review from BrucePay June 22, 2018 19:42
@daxian-dbw daxian-dbw added the Area-Maintainers-Documentation specific to documentation in this repo label Jun 22, 2018
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any chance we could turn the else here into a continue while we're at it? (I noticed foundModule only gets used after this)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method GetModuleForRootedPaths is massively refactored to change the behavior of Get-Module <directory-path> -ListAvailable and Get-Module <directory-path> -ListAvailable -All. The new behvaior will make them the same as if running Get-Module -ListAvailable and Get-Module -ListAvailable -All when $env:PSModulePath == <directory-path>.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hooray for this and the one above!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we give the type here instead of the var? The ternary method call makes it less than obvious.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the nature of the change here? Why do we not need to do the duplicate checks now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method GetModuleForRootedPaths is massively refactored. Please review again. Thanks!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The formatter for Get-Module -ListAvailable does logic like this, and I think it would be very helpful as a property or method on PSModuleInfo in general. I know we've had to do the same logic in PowerShellEditorServices at some point.

I feel like we ought to factor it into a method. That and I'm just averse to such a large inline delegate.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The OrderBy call was previous OrderBy(m => m.Name). However, simply ordering by name will cause the modules from the same parent folder to be divided by some modules in more nested sub-folders. I have moved this delegate to a method in ModuleUtils.cs.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe String.IsNullOrEmpty is better here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is removed now.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume rather than throw this returns null or ""?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method is removed now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method returns an empty string. I changed to use string.IsNullOrEmpty below.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this method return a Version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It returns List<Version>. But the method is removed now.

Copy link
Collaborator

@rjmholt rjmholt Jun 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic in this method looks very similar to the Import-Module local load logic:

private PSModuleInfo ImportModule_LocallyViaName(ImportModuleOptions importModuleOptions, string name)
{
try
{
bool found = false;
PSModuleInfo foundModule = null;
string cachedPath = null;
string rootedPath = null;
// See if we can use the cached path for the file. If a version number has been specified, then
// we won't look in the cache
if (this.MinimumVersion == null && this.MaximumVersion == null && this.RequiredVersion == null && PSModuleInfo.UseAppDomainLevelModuleCache && !this.BaseForce)
{
// See if the name is in the appdomain-level module path name cache...
cachedPath = PSModuleInfo.ResolveUsingAppDomainLevelModuleCache(name);
}
if (!string.IsNullOrEmpty(cachedPath))
{
if (File.Exists(cachedPath))
{
rootedPath = cachedPath;
}
else
{
PSModuleInfo.RemoveFromAppDomainLevelCache(name);
}
}
if (rootedPath == null)
{
// Check for full-qualified paths - either absolute or relative
rootedPath = ResolveRootedFilePath(name, this.Context);
}
bool alreadyLoaded = false;
if (!String.IsNullOrEmpty(rootedPath))
{
// TODO/FIXME: use IsModuleAlreadyLoaded to get consistent behavior
// TODO/FIXME: (for example checking ModuleType != Manifest below seems incorrect - cdxml modules also declare their own version)
// PSModuleInfo alreadyLoadedModule = null;
// Context.Modules.ModuleTable.TryGetValue(rootedPath, out alreadyLoadedModule);
// if (!BaseForce && IsModuleAlreadyLoaded(alreadyLoadedModule))
// If the module has already been loaded, just emit it and continue...
PSModuleInfo module;
if (!BaseForce && Context.Modules.ModuleTable.TryGetValue(rootedPath, out module))
{
if (RequiredVersion == null
|| module.Version.Equals(RequiredVersion)
|| (BaseMinimumVersion == null && BaseMaximumVersion == null)
|| module.ModuleType != ModuleType.Manifest
|| (BaseMinimumVersion == null && BaseMaximumVersion != null && module.Version <= BaseMaximumVersion)
|| (BaseMinimumVersion != null && BaseMaximumVersion == null && module.Version >= BaseMinimumVersion)
|| (BaseMinimumVersion != null && BaseMaximumVersion != null && module.Version >= BaseMinimumVersion && module.Version <= BaseMaximumVersion))
{
alreadyLoaded = true;
AddModuleToModuleTables(this.Context, this.TargetSessionState.Internal, module);
ImportModuleMembers(module, this.BasePrefix, importModuleOptions);
if (BaseAsCustomObject)
{
if (module.ModuleType != ModuleType.Script)
{
string message = StringUtil.Format(Modules.CantUseAsCustomObjectWithBinaryModule, module.Path);
InvalidOperationException invalidOp = new InvalidOperationException(message);
ErrorRecord er = new ErrorRecord(invalidOp, "Modules_CantUseAsCustomObjectWithBinaryModule",
ErrorCategory.PermissionDenied, null);
WriteError(er);
}
else
{
WriteObject(module.AsCustomObject());
}
}
else if (BasePassThru)
{
WriteObject(module);
}
found = true;
foundModule = module;
}
}
if (!alreadyLoaded)
{
// If the path names a file, load that file...
if (File.Exists(rootedPath))
{
PSModuleInfo moduleToRemove;
if (Context.Modules.ModuleTable.TryGetValue(rootedPath, out moduleToRemove))
{
RemoveModule(moduleToRemove);
}
foundModule = LoadModule(rootedPath, null, this.BasePrefix, null, ref importModuleOptions,
ManifestProcessingFlags.LoadElements | ManifestProcessingFlags.WriteErrors | ManifestProcessingFlags.NullOnFirstError,
out found);
}
else if (Directory.Exists(rootedPath))
{
// If the path ends with a directory separator, remove it
if (rootedPath.EndsWith(Path.DirectorySeparatorChar))
{
rootedPath = Path.GetDirectoryName(rootedPath);
}
// Load the latest valid version if it is a multi-version module directory
foundModule = LoadUsingMultiVersionModuleBase(rootedPath,
ManifestProcessingFlags.LoadElements |
ManifestProcessingFlags.WriteErrors |
ManifestProcessingFlags.NullOnFirstError,
importModuleOptions, out found);
if (!found)
{
// If the path is a directory, double up the end of the string
// then try to load that using extensions...
rootedPath = Path.Combine(rootedPath, Path.GetFileName(rootedPath));
foundModule = LoadUsingExtensions(null, rootedPath, rootedPath, null, null, this.BasePrefix, /*SessionState*/ null,
importModuleOptions,
ManifestProcessingFlags.LoadElements | ManifestProcessingFlags.WriteErrors | ManifestProcessingFlags.NullOnFirstError,
out found);
}
}
}
}
else
{
// Check if module could be a snapin. This was the case for PowerShell version 2 engine modules.
if (InitialSessionState.IsEngineModule(name))
{
PSSnapInInfo snapin = ModuleCmdletBase.GetEngineSnapIn(Context, name);
// Return the command if we found a module
if (snapin != null)
{
// warn that this module already exists as a snapin
string warningMessage = string.Format(
CultureInfo.InvariantCulture,
Modules.ModuleLoadedAsASnapin,
snapin.Name);
WriteWarning(warningMessage);
found = true;
return foundModule;
}
}
// At this point, the name didn't resolve to an existing file or directory.
// It may still be rooted (relative or absolute). If it is, then we'll only use
// the extension search. If it's not rooted, use a path-based search.
if (IsRooted(name))
{
// If there is no extension, we'll have to search using the extensions
if (!string.IsNullOrEmpty(Path.GetExtension(name)))
{
foundModule = LoadModule(name, null, this.BasePrefix, null, ref importModuleOptions,
ManifestProcessingFlags.LoadElements | ManifestProcessingFlags.WriteErrors | ManifestProcessingFlags.NullOnFirstError,
out found);
}
else
{
foundModule = LoadUsingExtensions(null, name, name, null, null, this.BasePrefix, /*SessionState*/ null,
importModuleOptions,
ManifestProcessingFlags.LoadElements | ManifestProcessingFlags.WriteErrors | ManifestProcessingFlags.NullOnFirstError,
out found);
}
}
else
{
IEnumerable<string> modulePath = ModuleIntrinsics.GetModulePath(false, this.Context);
if (this.MinimumVersion == null && this.RequiredVersion == null && this.MaximumVersion == null)
{
this.AddToAppDomainLevelCache = true;
}
found = LoadUsingModulePath(found, modulePath, name, /* SessionState*/ null,
importModuleOptions,
ManifestProcessingFlags.LoadElements | ManifestProcessingFlags.WriteErrors | ManifestProcessingFlags.NullOnFirstError,
out foundModule);
}
}
if (!found)
{
ErrorRecord er = null;
string message = null;
if (BaseRequiredVersion != null)
{
message = StringUtil.Format(Modules.ModuleWithVersionNotFound, name, BaseRequiredVersion);
}
else if (BaseMinimumVersion != null && BaseMaximumVersion != null)
{
message = StringUtil.Format(Modules.MinimumVersionAndMaximumVersionNotFound, name, BaseMinimumVersion, BaseMaximumVersion);
}
else if (BaseMinimumVersion != null)
{
message = StringUtil.Format(Modules.ModuleWithVersionNotFound, name, BaseMinimumVersion);
}
else if (BaseMaximumVersion != null)
{
message = StringUtil.Format(Modules.MaximumVersionNotFound, name, BaseMaximumVersion);
}
if (BaseRequiredVersion != null || BaseMinimumVersion != null || BaseMaximumVersion != null)
{
FileNotFoundException fnf = new FileNotFoundException(message);
er = new ErrorRecord(fnf, "Modules_ModuleWithVersionNotFound",
ErrorCategory.ResourceUnavailable, name);
}
else
{
message = StringUtil.Format(Modules.ModuleNotFound, name);
FileNotFoundException fnf = new FileNotFoundException(message);
er = new ErrorRecord(fnf, "Modules_ModuleNotFound",
ErrorCategory.ResourceUnavailable, name);
}
WriteError(er);
}
return foundModule;
}
catch (PSInvalidOperationException e)
{
ErrorRecord er = new ErrorRecord(e.ErrorRecord, e);
WriteError(er);
}
return null;
}

Where we first check the path, then the folder, then the versions, then the extensions, etc.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comments here as above

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a new line.

@daxian-dbw daxian-dbw added the Breaking-Change breaking change that may affect users label Jun 25, 2018
@daxian-dbw daxian-dbw force-pushed the moduleRefactor branch 3 times, most recently from 4ed9b45 to cdff125 Compare June 28, 2018 21:44
@daxian-dbw daxian-dbw changed the title [WIP] Refactor module code related to 'Get-Module -ListAvailable' Refactor module code related to 'Get-Module -ListAvailable' Jun 28, 2018
@daxian-dbw daxian-dbw removed the Breaking-Change breaking change that may affect users label Jun 28, 2018
@daxian-dbw
Copy link
Member Author

daxian-dbw commented Jun 28, 2018

@rjmholt I have reverted the breaking change I made to the scenario of Get-Module <Path-to-directory> -ListAvailable (please see details in #7160). Now the changes introduce no change to the current behaviors. Can you please continue the review?

@daxian-dbw
Copy link
Member Author

daxian-dbw commented Jun 29, 2018

Added tests for different scenarios of Get-Module -ListAvailable and Get-Module -ListAvailable -All. All those tests pass in pwsh with or without changes in this PR.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How hard is the requirement on names being an array? I only ask because I notice this part of the code tends to have collection types in signatures vary a lot (some IEnumerable<string>, some string[], some List<string>) and we should probably work out a better rationale for what types we prefer.

If there is a particular performance implication from using string[] then that makes sense, but in general I think we should favour the principle of using an interface with only the properties required by each method; if we only want to go through entries sequentially, IEnumerable<T>, if we want to know how many there are up front, something like ICollection<T>, if we want indexing IList<T>.

I'm not making a particular suggestion here I suppose. I just have the impression that the various collection types used in the methods in the module cmdlets weren't used for any particular reason, and we should be more judicious about what collection types we use (and the semantics of each) given how intensively we handle them.

Copy link
Member Author

@daxian-dbw daxian-dbw Jun 29, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this method, it makes more sense to use List<string>, so the top-level caller doesn't need to cast the List to a string array. I have updated the code.

Copy link
Collaborator

@rjmholt rjmholt Jun 29, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment above I read as saying "we can skip analysis if all of functions/cmdlets/aliases to export are given".

But I would translate that as the condition:

!needToAnalyzeScriptModules && (sawExportedCmdlets && sawExportedFunctions && sawExportedAliases))

The expression here is equivalent to:

!needToAnalyzeScriptModules && !(sawExportedCmdlets && sawExportedFunctions && sawExportedAliases))

Which I think would be "analysis is not required and any of functions/cmdlets/aliases to export is missing".

Do the variable names mean something converse to how I read them?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment above I read as saying "we can skip analysis if all of functions/cmdlets/aliases to export are given".

That's correct.
That means if needToAnalyzeScriptModules == false when we reaching here, we might still change it to true if we didn't see one of "FunctionToExport", "CmdletToExport" and "AliasToExport".

One thing worth to note is the if block here is not doing further analysis, but checking if we really need to do further analysis -- it's changing the value of needToAnalyzeScriptModules to true.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh right! The comment is slightly unclear to me because I didn't realise that the if is really a check to see if we maybe do need to do further analysis.

I would have worded it more as "If any of 'FunctionsToExport', 'CmdletsToExport' or 'AliasesToExport' were not given, we must check to see if more analysis is needed". Not saying the wording should be that way, just submitting an alternate description

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suggestion. I like your comment better. Will update.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A much needed file!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully, more tests can be added gradually.

Copy link
Collaborator

@rjmholt rjmholt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, this looks good to me now!

@daxian-dbw daxian-dbw removed the Area-Maintainers-Documentation specific to documentation in this repo label Jul 1, 2018
@daxian-dbw daxian-dbw merged commit 0d9a3ac into PowerShell:master Jul 2, 2018
@daxian-dbw daxian-dbw deleted the moduleRefactor branch July 2, 2018 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants