Skip to content

ROX-26089,ROX-26519: v4.VulnerabilityReport conversion#12798

Merged
Maddosaurus merged 25 commits intomasterfrom
mm/ROX-26089-convert
Oct 17, 2024
Merged

ROX-26089,ROX-26519: v4.VulnerabilityReport conversion#12798
Maddosaurus merged 25 commits intomasterfrom
mm/ROX-26089-convert

Conversation

@Maddosaurus
Copy link
Copy Markdown
Contributor

@Maddosaurus Maddosaurus commented Sep 24, 2024

Description

Introduces a converter to translate a v4.VulnerabilityReport into a storage.NodeScan.
This converter allows us to reuse existing storage and UI logic, as well as API endpoints.

User-facing documentation

  • CHANGELOG is updated OR update is not needed
  • documentation PR is created and is linked above OR is not needed

Testing and quality

  • the change is production ready: the change is GA or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • added unit tests
  • added e2e tests
  • added regression tests
  • added compatibility tests
  • modified existing tests

How I validated my change

  • Create an OS4 Cluster on Infra making sure to set Ensure an SSD StorageClass is the default StorageClass for the cluster to true
  • Deploy this image with ROX_SCANNER_V4=true
  • Wait for Matcher to finish initial sync (check logs of both matchers)
  • Set ROX_NODE_INDEX_ENABLED=true for the Collector DaemonSet
  • Recommendation: Lower Node Scan intervals by setting on the Collector DaemonSet
    • ROX_NODE_SCANNING_INTERVAL=45s and ROX_NODE_SCANNING_MAX_INITIAL_WAIT=10s
  • Set debug loglevel for Central
  • Observe the following log messages on Central:
    • Received Vulnerability Report with %d packages containing %d vulnerabilities
    • Successfully enriched node %s with %s report - found %d components
    • Discarding v2 NodeScan in favor of v4 NodeScan (If Scanner v2 is run in parallel)

The last mentioned log message will log out the Node name, the Scanner Version string, and the number of discovered components.
The scanner version is expected to be 4, and the number of components bigger than 0.

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Sep 24, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@rhacs-bot
Copy link
Copy Markdown
Contributor

rhacs-bot commented Sep 24, 2024

Images are ready for the commit at 449579b.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.6.x-773-g449579b2c6.

@codecov
Copy link
Copy Markdown

codecov bot commented Sep 24, 2024

Codecov Report

Attention: Patch coverage is 72.53521% with 39 lines in your changes missing coverage. Please review.

Project coverage is 48.26%. Comparing base (1678963) to head (449579b).

Files with missing lines Patch % Lines
central/graphql/resolvers/generated.go 10.52% 17 Missing ⚠️
pkg/scanners/scannerv4/nodescan_convert.go 89.21% 8 Missing and 3 partials ⚠️
.../sensor/service/pipeline/nodeinventory/pipeline.go 47.05% 6 Missing and 3 partials ⚠️
pkg/scanners/scannerv4/scannerv4.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #12798      +/-   ##
==========================================
+ Coverage   48.24%   48.26%   +0.02%     
==========================================
  Files        2452     2453       +1     
  Lines      176454   176585     +131     
==========================================
+ Hits        85123    85226     +103     
- Misses      84483    84507      +24     
- Partials     6848     6852       +4     
Flag Coverage Δ
go-unit-tests 48.26% <72.53%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Maddosaurus Maddosaurus force-pushed the mm/ROX-26089-convert branch 3 times, most recently from bfb6266 to f2fb397 Compare October 7, 2024 12:14
Copy link
Copy Markdown
Contributor

@jschnath jschnath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review since we just talked about this still being in draft. Didn't look at all the code yet.

@Maddosaurus Maddosaurus marked this pull request as ready for review October 7, 2024 12:57
@Maddosaurus Maddosaurus requested review from a team as code owners October 7, 2024 12:57
@Maddosaurus Maddosaurus force-pushed the mm/ROX-26089-convert branch from f2fb397 to 3cc5cb9 Compare October 8, 2024 14:38
@Maddosaurus Maddosaurus requested a review from jschnath October 9, 2024 09:24
Copy link
Copy Markdown
Contributor

@vikin91 vikin91 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good, but I am curious how you validated that additionally to the unit tests.

@Maddosaurus
Copy link
Copy Markdown
Contributor Author

Maddosaurus commented Oct 9, 2024

The code looks good, but I am curious how you validated that additionally to the unit tests.

@vikin91 I did not verify this just yet. The plan is to do that in the follow up, ROX-26519.
That issue deals with wiring this conversion function up and also tackling the potential concurrent update of v2 and v4 node scans arriving at the same time.
Once it is wired up, I can fully test whether this is producing correct translations

@Maddosaurus Maddosaurus requested a review from vikin91 October 9, 2024 12:09
@Maddosaurus
Copy link
Copy Markdown
Contributor Author

Side note: Looks like prow tests are skipping because of failing ARM builds, even though they are using x86 builds.

Copy link
Copy Markdown
Contributor

@vikin91 vikin91 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls check the style failing and my additional comments on using Len assertion.

@Maddosaurus Maddosaurus requested a review from vikin91 October 10, 2024 07:58
@vikin91
Copy link
Copy Markdown
Contributor

vikin91 commented Oct 10, 2024

@Maddosaurus as I said, the code looks good and I cannot see any issues there. But I need to know how did you make sure that it works.
Screenshot 2024-10-10 at 11 03 36
If this section is missing, I would need to deploy it myself and start playing with it and that would take extra time.

@Maddosaurus Maddosaurus force-pushed the mm/ROX-26089-convert branch 2 times, most recently from 0633b34 to 5f7808e Compare October 10, 2024 14:18
@Maddosaurus Maddosaurus changed the title ROX-26089: Introduce v4.VulnerabilityReport conversion ROX-26089, ROX-26519: Introduce v4.VulnerabilityReport conversion Oct 10, 2024
@Maddosaurus Maddosaurus changed the title ROX-26089, ROX-26519: Introduce v4.VulnerabilityReport conversion ROX-26089,ROX-26519: Introduce v4.VulnerabilityReport conversion Oct 10, 2024
Copy link
Copy Markdown
Contributor

@vikin91 vikin91 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still looks fine, thanks for testing this manually. I am leaving two minor cosmetic comments

@Maddosaurus Maddosaurus dismissed jschnath’s stale review October 14, 2024 09:24

Jan is OOO this week and cannot re-review

@Maddosaurus
Copy link
Copy Markdown
Contributor Author

/retest

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Oct 17, 2024

@Maddosaurus: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/ocp-4-17-compliance-e2e-tests 0633b34 link false /test ocp-4-17-compliance-e2e-tests
ci/prow/ocp-4-12-compliance-e2e-tests 0633b34 link false /test ocp-4-12-compliance-e2e-tests
ci/prow/ocp-4-17-ui-e2e-tests 449579b link false /test ocp-4-17-ui-e2e-tests
ci/prow/ocp-4-17-scanner-v4-tests 449579b link false /test ocp-4-17-scanner-v4-tests
ci/prow/ocp-4-12-scanner-v4-tests 449579b link false /test ocp-4-12-scanner-v4-tests
ci/prow/gke-ui-e2e-tests 449579b link false /test gke-ui-e2e-tests
ci/prow/ocp-4-12-ui-e2e-tests 449579b link false /test ocp-4-12-ui-e2e-tests
ci/prow/gke-scanner-v4-tests 449579b link false /test gke-scanner-v4-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@Maddosaurus Maddosaurus changed the title ROX-26089,ROX-26519: Introduce v4.VulnerabilityReport conversion ROX-26089,ROX-26519: v4.VulnerabilityReport conversion Oct 17, 2024
@Maddosaurus Maddosaurus merged commit ec9b5d1 into master Oct 17, 2024
@Maddosaurus Maddosaurus deleted the mm/ROX-26089-convert branch October 17, 2024 12:52
)

var (
rhcosOSImageRegexp = regexp.MustCompile(`(Red Hat Enterprise Linux) (CoreOS) ([\d])([\d]+)`)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: rhcosPattern

Also, according to my IDE, the [] are not needed

// Keep notes as they are for nodes other than RHCOS
return notes
}
fixedNotes := make([]storage.NodeScan_Note, 0)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

filtered := notes[:0]
for _, note := range notes {
	if note == storage.NodeScan_UNSUPPORTED {
		continue
	}
	filtered = append(filtered, note)
}
return filtered

if len(r) != 5 {
return ""
}
return fmt.Sprintf("rhcos:%s.%s", r[3], r[4])
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Red Hat Enterprise Linux CoreOS 41712345.94.2024 it gives me rhcos:4.1712345. Is this expected?


func toNodeScan(r *v4.VulnerabilityReport, osImageRef string) *storage.NodeScan {
// TODO(ROX-26593): Instead of fixing notes here, add RHCOS DistributionScanner to ClairCore
fixedNotes := fixNotes(toStorageNotes(r.Notes), osImageRef)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we can't determine the OS below, though, wouldn't we want the OS unsupported or unknown note?

func toOperatingSystem(ref string) string {
r := rhcosOSImageRegexp.FindStringSubmatch(ref)
if len(r) != 5 {
return ""
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do unknown for images. Why not do that here, too?


return &storage.NodeScan{
ScanTime: protocompat.TimestampNow(),
Components: toStorageComponents(r),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how much does this differ from components in convert.go? Can we just reuse that instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants