-
Notifications
You must be signed in to change notification settings - Fork 8.1k
Refactor module code related to 'Get-Module -ListAvailable' #7145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
c4a82f7 to
ce51f0f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any chance we could turn the else here into a continue while we're at it? (I noticed foundModule only gets used after this)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The method GetModuleForRootedPaths is massively refactored to change the behavior of Get-Module <directory-path> -ListAvailable and Get-Module <directory-path> -ListAvailable -All. The new behvaior will make them the same as if running Get-Module -ListAvailable and Get-Module -ListAvailable -All when $env:PSModulePath == <directory-path>.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hooray for this and the one above!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we give the type here instead of the var? The ternary method call makes it less than obvious.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the nature of the change here? Why do we not need to do the duplicate checks now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The method GetModuleForRootedPaths is massively refactored. Please review again. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The formatter for Get-Module -ListAvailable does logic like this, and I think it would be very helpful as a property or method on PSModuleInfo in general. I know we've had to do the same logic in PowerShellEditorServices at some point.
I feel like we ought to factor it into a method. That and I'm just averse to such a large inline delegate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The OrderBy call was previous OrderBy(m => m.Name). However, simply ordering by name will cause the modules from the same parent folder to be divided by some modules in more nested sub-folders. I have moved this delegate to a method in ModuleUtils.cs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe String.IsNullOrEmpty is better here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method is removed now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I assume rather than throw this returns null or ""?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The method is removed now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method returns an empty string. I changed to use string.IsNullOrEmpty below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this method return a Version?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It returns List<Version>. But the method is removed now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The logic in this method looks very similar to the Import-Module local load logic:
PowerShell/src/System.Management.Automation/engine/Modules/ImportModuleCommand.cs
Lines 530 to 760 in bbb4f2e
| private PSModuleInfo ImportModule_LocallyViaName(ImportModuleOptions importModuleOptions, string name) | |
| { | |
| try | |
| { | |
| bool found = false; | |
| PSModuleInfo foundModule = null; | |
| string cachedPath = null; | |
| string rootedPath = null; | |
| // See if we can use the cached path for the file. If a version number has been specified, then | |
| // we won't look in the cache | |
| if (this.MinimumVersion == null && this.MaximumVersion == null && this.RequiredVersion == null && PSModuleInfo.UseAppDomainLevelModuleCache && !this.BaseForce) | |
| { | |
| // See if the name is in the appdomain-level module path name cache... | |
| cachedPath = PSModuleInfo.ResolveUsingAppDomainLevelModuleCache(name); | |
| } | |
| if (!string.IsNullOrEmpty(cachedPath)) | |
| { | |
| if (File.Exists(cachedPath)) | |
| { | |
| rootedPath = cachedPath; | |
| } | |
| else | |
| { | |
| PSModuleInfo.RemoveFromAppDomainLevelCache(name); | |
| } | |
| } | |
| if (rootedPath == null) | |
| { | |
| // Check for full-qualified paths - either absolute or relative | |
| rootedPath = ResolveRootedFilePath(name, this.Context); | |
| } | |
| bool alreadyLoaded = false; | |
| if (!String.IsNullOrEmpty(rootedPath)) | |
| { | |
| // TODO/FIXME: use IsModuleAlreadyLoaded to get consistent behavior | |
| // TODO/FIXME: (for example checking ModuleType != Manifest below seems incorrect - cdxml modules also declare their own version) | |
| // PSModuleInfo alreadyLoadedModule = null; | |
| // Context.Modules.ModuleTable.TryGetValue(rootedPath, out alreadyLoadedModule); | |
| // if (!BaseForce && IsModuleAlreadyLoaded(alreadyLoadedModule)) | |
| // If the module has already been loaded, just emit it and continue... | |
| PSModuleInfo module; | |
| if (!BaseForce && Context.Modules.ModuleTable.TryGetValue(rootedPath, out module)) | |
| { | |
| if (RequiredVersion == null | |
| || module.Version.Equals(RequiredVersion) | |
| || (BaseMinimumVersion == null && BaseMaximumVersion == null) | |
| || module.ModuleType != ModuleType.Manifest | |
| || (BaseMinimumVersion == null && BaseMaximumVersion != null && module.Version <= BaseMaximumVersion) | |
| || (BaseMinimumVersion != null && BaseMaximumVersion == null && module.Version >= BaseMinimumVersion) | |
| || (BaseMinimumVersion != null && BaseMaximumVersion != null && module.Version >= BaseMinimumVersion && module.Version <= BaseMaximumVersion)) | |
| { | |
| alreadyLoaded = true; | |
| AddModuleToModuleTables(this.Context, this.TargetSessionState.Internal, module); | |
| ImportModuleMembers(module, this.BasePrefix, importModuleOptions); | |
| if (BaseAsCustomObject) | |
| { | |
| if (module.ModuleType != ModuleType.Script) | |
| { | |
| string message = StringUtil.Format(Modules.CantUseAsCustomObjectWithBinaryModule, module.Path); | |
| InvalidOperationException invalidOp = new InvalidOperationException(message); | |
| ErrorRecord er = new ErrorRecord(invalidOp, "Modules_CantUseAsCustomObjectWithBinaryModule", | |
| ErrorCategory.PermissionDenied, null); | |
| WriteError(er); | |
| } | |
| else | |
| { | |
| WriteObject(module.AsCustomObject()); | |
| } | |
| } | |
| else if (BasePassThru) | |
| { | |
| WriteObject(module); | |
| } | |
| found = true; | |
| foundModule = module; | |
| } | |
| } | |
| if (!alreadyLoaded) | |
| { | |
| // If the path names a file, load that file... | |
| if (File.Exists(rootedPath)) | |
| { | |
| PSModuleInfo moduleToRemove; | |
| if (Context.Modules.ModuleTable.TryGetValue(rootedPath, out moduleToRemove)) | |
| { | |
| RemoveModule(moduleToRemove); | |
| } | |
| foundModule = LoadModule(rootedPath, null, this.BasePrefix, null, ref importModuleOptions, | |
| ManifestProcessingFlags.LoadElements | ManifestProcessingFlags.WriteErrors | ManifestProcessingFlags.NullOnFirstError, | |
| out found); | |
| } | |
| else if (Directory.Exists(rootedPath)) | |
| { | |
| // If the path ends with a directory separator, remove it | |
| if (rootedPath.EndsWith(Path.DirectorySeparatorChar)) | |
| { | |
| rootedPath = Path.GetDirectoryName(rootedPath); | |
| } | |
| // Load the latest valid version if it is a multi-version module directory | |
| foundModule = LoadUsingMultiVersionModuleBase(rootedPath, | |
| ManifestProcessingFlags.LoadElements | | |
| ManifestProcessingFlags.WriteErrors | | |
| ManifestProcessingFlags.NullOnFirstError, | |
| importModuleOptions, out found); | |
| if (!found) | |
| { | |
| // If the path is a directory, double up the end of the string | |
| // then try to load that using extensions... | |
| rootedPath = Path.Combine(rootedPath, Path.GetFileName(rootedPath)); | |
| foundModule = LoadUsingExtensions(null, rootedPath, rootedPath, null, null, this.BasePrefix, /*SessionState*/ null, | |
| importModuleOptions, | |
| ManifestProcessingFlags.LoadElements | ManifestProcessingFlags.WriteErrors | ManifestProcessingFlags.NullOnFirstError, | |
| out found); | |
| } | |
| } | |
| } | |
| } | |
| else | |
| { | |
| // Check if module could be a snapin. This was the case for PowerShell version 2 engine modules. | |
| if (InitialSessionState.IsEngineModule(name)) | |
| { | |
| PSSnapInInfo snapin = ModuleCmdletBase.GetEngineSnapIn(Context, name); | |
| // Return the command if we found a module | |
| if (snapin != null) | |
| { | |
| // warn that this module already exists as a snapin | |
| string warningMessage = string.Format( | |
| CultureInfo.InvariantCulture, | |
| Modules.ModuleLoadedAsASnapin, | |
| snapin.Name); | |
| WriteWarning(warningMessage); | |
| found = true; | |
| return foundModule; | |
| } | |
| } | |
| // At this point, the name didn't resolve to an existing file or directory. | |
| // It may still be rooted (relative or absolute). If it is, then we'll only use | |
| // the extension search. If it's not rooted, use a path-based search. | |
| if (IsRooted(name)) | |
| { | |
| // If there is no extension, we'll have to search using the extensions | |
| if (!string.IsNullOrEmpty(Path.GetExtension(name))) | |
| { | |
| foundModule = LoadModule(name, null, this.BasePrefix, null, ref importModuleOptions, | |
| ManifestProcessingFlags.LoadElements | ManifestProcessingFlags.WriteErrors | ManifestProcessingFlags.NullOnFirstError, | |
| out found); | |
| } | |
| else | |
| { | |
| foundModule = LoadUsingExtensions(null, name, name, null, null, this.BasePrefix, /*SessionState*/ null, | |
| importModuleOptions, | |
| ManifestProcessingFlags.LoadElements | ManifestProcessingFlags.WriteErrors | ManifestProcessingFlags.NullOnFirstError, | |
| out found); | |
| } | |
| } | |
| else | |
| { | |
| IEnumerable<string> modulePath = ModuleIntrinsics.GetModulePath(false, this.Context); | |
| if (this.MinimumVersion == null && this.RequiredVersion == null && this.MaximumVersion == null) | |
| { | |
| this.AddToAppDomainLevelCache = true; | |
| } | |
| found = LoadUsingModulePath(found, modulePath, name, /* SessionState*/ null, | |
| importModuleOptions, | |
| ManifestProcessingFlags.LoadElements | ManifestProcessingFlags.WriteErrors | ManifestProcessingFlags.NullOnFirstError, | |
| out foundModule); | |
| } | |
| } | |
| if (!found) | |
| { | |
| ErrorRecord er = null; | |
| string message = null; | |
| if (BaseRequiredVersion != null) | |
| { | |
| message = StringUtil.Format(Modules.ModuleWithVersionNotFound, name, BaseRequiredVersion); | |
| } | |
| else if (BaseMinimumVersion != null && BaseMaximumVersion != null) | |
| { | |
| message = StringUtil.Format(Modules.MinimumVersionAndMaximumVersionNotFound, name, BaseMinimumVersion, BaseMaximumVersion); | |
| } | |
| else if (BaseMinimumVersion != null) | |
| { | |
| message = StringUtil.Format(Modules.ModuleWithVersionNotFound, name, BaseMinimumVersion); | |
| } | |
| else if (BaseMaximumVersion != null) | |
| { | |
| message = StringUtil.Format(Modules.MaximumVersionNotFound, name, BaseMaximumVersion); | |
| } | |
| if (BaseRequiredVersion != null || BaseMinimumVersion != null || BaseMaximumVersion != null) | |
| { | |
| FileNotFoundException fnf = new FileNotFoundException(message); | |
| er = new ErrorRecord(fnf, "Modules_ModuleWithVersionNotFound", | |
| ErrorCategory.ResourceUnavailable, name); | |
| } | |
| else | |
| { | |
| message = StringUtil.Format(Modules.ModuleNotFound, name); | |
| FileNotFoundException fnf = new FileNotFoundException(message); | |
| er = new ErrorRecord(fnf, "Modules_ModuleNotFound", | |
| ErrorCategory.ResourceUnavailable, name); | |
| } | |
| WriteError(er); | |
| } | |
| return foundModule; | |
| } | |
| catch (PSInvalidOperationException e) | |
| { | |
| ErrorRecord er = new ErrorRecord(e.ErrorRecord, e); | |
| WriteError(er); | |
| } | |
| return null; | |
| } |
Where we first check the path, then the folder, then the versions, then the extensions, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same comments here as above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a new line.
9cf67ae to
a24fee9
Compare
4ed9b45 to
cdff125
Compare
|
Added tests for different scenarios of |
4c1c680 to
8663d78
Compare
8663d78 to
1363170
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How hard is the requirement on names being an array? I only ask because I notice this part of the code tends to have collection types in signatures vary a lot (some IEnumerable<string>, some string[], some List<string>) and we should probably work out a better rationale for what types we prefer.
If there is a particular performance implication from using string[] then that makes sense, but in general I think we should favour the principle of using an interface with only the properties required by each method; if we only want to go through entries sequentially, IEnumerable<T>, if we want to know how many there are up front, something like ICollection<T>, if we want indexing IList<T>.
I'm not making a particular suggestion here I suppose. I just have the impression that the various collection types used in the methods in the module cmdlets weren't used for any particular reason, and we should be more judicious about what collection types we use (and the semantics of each) given how intensively we handle them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For this method, it makes more sense to use List<string>, so the top-level caller doesn't need to cast the List to a string array. I have updated the code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment above I read as saying "we can skip analysis if all of functions/cmdlets/aliases to export are given".
But I would translate that as the condition:
!needToAnalyzeScriptModules && (sawExportedCmdlets && sawExportedFunctions && sawExportedAliases))The expression here is equivalent to:
!needToAnalyzeScriptModules && !(sawExportedCmdlets && sawExportedFunctions && sawExportedAliases))Which I think would be "analysis is not required and any of functions/cmdlets/aliases to export is missing".
Do the variable names mean something converse to how I read them?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment above I read as saying "we can skip analysis if all of functions/cmdlets/aliases to export are given".
That's correct.
That means if needToAnalyzeScriptModules == false when we reaching here, we might still change it to true if we didn't see one of "FunctionToExport", "CmdletToExport" and "AliasToExport".
One thing worth to note is the if block here is not doing further analysis, but checking if we really need to do further analysis -- it's changing the value of needToAnalyzeScriptModules to true.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh right! The comment is slightly unclear to me because I didn't realise that the if is really a check to see if we maybe do need to do further analysis.
I would have worded it more as "If any of 'FunctionsToExport', 'CmdletsToExport' or 'AliasesToExport' were not given, we must check to see if more analysis is needed". Not saying the wording should be that way, just submitting an alternate description
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good suggestion. I like your comment better. Will update.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A much needed file!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hopefully, more tests can be added gradually.
1363170 to
1ca7e12
Compare
rjmholt
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, this looks good to me now!
PR Summary
The major refactoring changes are:
ModuleIntrisic.cs, remove unneeded Windows-PowerShell-only code.ModuleUtils.csDirectory.GetDirectories(string path, string searchPattern, EnumerationOptions enumerationOptions)andDirectory.GetFiles(string path, string searchPattern, EnumerationOptions enumerationOptions)to enumerate files and sub-directories within a directory path.bool forcefromGetDefaultAvailableModuleFiles(bool force, bool isForAutoDiscovery, ExecutionContext context)GetModuleVersionsFromAbsolutePath. Add more comments and rename the method name.ModuleCmdletBase.cs, refactor the methodGetModuleForNonRootedPathstoGetModuleForNamesto simply its implementation.PSModuleInfo.csDeclared*Exportsfields together_detected*Exportsfields toDetected*Exportsto group them together. They are internal fields and used outsidePSModuleInfo.Pref improvement
There is some perf improvement after this refactoring change:
Get-Module -ListAvailable, there is about 36% speed improvement for 94 default modules.Get-Module -ListAvailable -All, there is about 14% speed improvement for totally 600 module files.Get-Module <name> -ListAvailable -List, there is over 17x speed improvement for finding 13 modules from 600 modules. This is because we now filter names using the module file before creating aPSMdouleInfoobject.Before
After
PR Checklist
.h,.cpp,.cs,.ps1and.psm1files have the correct copyright headerWIP:to the beginning of the title and remove the prefix when the PR is ready.[feature]if the change is significant or affects feature tests