[release/v7.6] Split TPN manifest and Component Governance manifest#26978
Conversation
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
daxian-dbw
added
the
CL-BuildPackaging
| @@ -0,0 +1,755 @@ | |||
| { | |||
| "Registrations": [ | |||
There was a problem hiding this comment.
This should not be back ported or needs to be regenerated for the branch.
There was a problem hiding this comment.
Can we backport this change and then regenerate for this branch? Or do we have to remove this file from this PR?
There was a problem hiding this comment.
I will run ".\tools\findMissingNotices.ps1 -ForceHarvestedOnly -fix" after merging this PR, to update this file.
There was a problem hiding this comment.
Pull request overview
Backport to release/v7.6 that reorganizes Component Governance manifests (split main vs TPN) and updates the ClearlyDefined compliance tooling to support cache persistence and harvested-version discovery.
Changes:
- Split
cgmanifest.jsonintotools/cgmanifest/main/cgmanifest.jsonandtools/cgmanifest/tpn/cgmanifest.json, updating pipeline/script references. - Extend ClearlyDefined tooling with cache import/export, search/version helpers, and revised caching TTL behavior.
- Enhance
findMissingNotices.ps1with an option to generate a harvested-only TPN manifest and revert non-harvested packages.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/packaging/packaging.psm1 | Updates CG manifest path used during global tool packaging. |
| tools/findMissingNotices.ps1 | Adds harvested-only TPN generation flow; updates manifest paths. |
| tools/clearlyDefined/src/ClearlyDefined/ClearlyDefined.psm1 | Adds cache/search helpers and changes harvest/caching behavior. |
| tools/clearlyDefined/Find-LastHarvestedVersion.ps1 | Adds helper for finding last harvested NuGet version (fallback to NuGet). |
| tools/clearlyDefined/ClearlyDefined.ps1 | Updates manifest path consumed by ClearlyDefined tooling. |
| tools/cgmanifest/main/cgmanifest.json | New location for main CG manifest content. |
| tools/cgmanifest/tpn/cgmanifest.json | New location for TPN-focused CG manifest content. |
| .vsts-ci/windows-arm64.yml | Updates PR path filters for new cgmanifest folder layout. |
| .vsts-ci/psresourceget-acr.yml | Updates PR path filters for new cgmanifest folder layout. |
| .vsts-ci/mac.yml | Updates PR path filters for new cgmanifest folder layout. |
| .vsts-ci/linux-internal.yml | Updates PR path filters for new cgmanifest folder layout. |
| .pipelines/templates/compliance/generateNotice.yml | Narrows component detection scan path to the TPN manifest directory. |
| Start-job -ScriptBlock { | ||
| Invoke-WebRequest -Method Post -Uri 'https://api.clearlydefined.io/harvest' -Body $using:body -ContentType 'application/json' -MaximumRetryCount $using:maxRetryCount -RetryIntervalSec $using:retryIntervalSec | ||
| } |
There was a problem hiding this comment.
Start-ClearlyDefinedHarvest now uses Start-Job to invoke the harvest request, but the job is never waited/received/removed. In non-interactive runs (CI pwsh -File), the PowerShell process can exit before jobs run, so harvest requests may never be sent; additionally this can create many orphaned jobs. Consider invoking the request synchronously, or explicitly Wait-Job/Receive-Job/Remove-Job (optionally with throttling) to ensure completion and avoid job buildup.
| # Parse the current version | ||
| try { | ||
| [System.Management.Automation.SemanticVersion]$currentSemVer = $CurrentVersion | ||
| } catch { | ||
| [Version]$currentSemVer = $CurrentVersion | ||
| } | ||
|
|
||
| # First try the ClearlyDefined search API (more efficient) | ||
| try { | ||
| Write-Verbose "Searching ClearlyDefined API for versions of $Name (sorted by release date)..." | ||
| # Get versions sorted by release date descending (newest first) for efficiency | ||
| $versions = Get-ClearlyDefinedPackageVersions -PackageName $Name | ||
|
|
||
| if ($versions -and $versions.Count -gt 0) { | ||
| # Results are already sorted by release date newest first | ||
| # Filter to versions <= current version | ||
| foreach ($versionInfo in $versions) { | ||
| try { | ||
| $versionObj = [System.Management.Automation.SemanticVersion]$versionInfo.Version | ||
| if ($versionObj -le $currentSemVer) { | ||
| # Check harvest status |
There was a problem hiding this comment.
Version parsing/comparison can break when $CurrentVersion isn't parseable as SemanticVersion (e.g., 4-part versions): $currentSemVer becomes a [Version], but later code compares it to a [SemanticVersion], which throws and is swallowed—causing the search loop to skip all candidates and return $null. Consider normalizing both sides to the same version type (or using the same fallback logic for both current and candidate versions) so non-semver package versions can still be handled reliably.
| # Build a lookup table of harvest status by package name + version | ||
| $harvestStatus = @{} | ||
| foreach ($item in $fullList) { | ||
| $key = "$($item.Name)|$($item.PackageVersion)" |
There was a problem hiding this comment.
The objects returned by Get-ClearlyDefinedData don't include .Name / .PackageVersion (they carry package identity under .coordinates.name and .coordinates.revision). As written, $key will be empty/incorrect for every item, so the harvest-status lookup will be wrong and all packages may be treated as non-harvested.
| $key = "$($item.Name)|$($item.PackageVersion)" | |
| $name = $item.coordinates.name | |
| $version = $item.coordinates.revision | |
| if (-not $name -or -not $version) { | |
| continue | |
| } | |
| $key = "$name|$version" |
| foreach ($item in $finalHarvestData) { | ||
| $matchingNewRegistration = $newRegistrations | Where-Object { | ||
| $_.Component.Nuget.Name -eq $item.Name -and | ||
| $_.Component.Nuget.Version -eq $item.PackageVersion | ||
| } |
There was a problem hiding this comment.
$finalHarvestData elements (from Get-ClearlyDefinedData) don't have .Name / .PackageVersion, so this match will never succeed and the TPN manifest update logic won't include any registrations. Use the package identity from .coordinates (or carry the name/version alongside the request) when correlating results back to $newRegistrations.
Backport of #26891 to release/v7.6
Triggered by @daxian-dbw on behalf of @TravisEz13
Original CL Label: CL-BuildPackaging
/cc @PowerShell/powershell-maintainers
Impact
REQUIRED: Choose either Tooling Impact or Customer Impact (or both). At least one checkbox must be selected.
Tooling Impact
Enhances ClearlyDefined compliance tooling with improved cache management, package version handling, and updated manifest file paths. Introduces cache persistence functions and new script for finding harvested NuGet package versions.
Customer Impact
Regression
REQUIRED: Check exactly one box.
This is not a regression.
Testing
Successfully backported and merged to v7.4 (#26955) and v7.5 (#26967). Verified through CI pipeline runs. Changes focus on compliance tooling cache management and manifest file organization.
Risk
REQUIRED: Check exactly one box.
Medium risk as it modifies ClearlyDefined compliance tooling and updates manifest file paths across multiple build and CI scripts. However, changes are well-tested through successful backports to v7.4 and v7.5. Focused on build-time operations with no runtime impact.