Skip to content

Auto Discovery failed to fetch Scan for removed pod with no scans made before. #2699

Closed

Description

@paraddise

🐞 Bug report

I deployed SCB Auto Discovery to kubernetes, annotated existing namespace with some pods in it. Then killed nginx pod and Auto Discovery service failed to fetch Scan and process another events.

Describe the bug

Secure Code Box auto discovery service tries to fetch ScheduledScan for pod with no scans made before.

Steps To Reproduce

Deploy auto-dicovery by instruction to namespace infra-securecodebox.
Annotate namespace that already has pods in it.

k annotate ns infra-securecodebox auto-discovery.securecodebox.io/enabled=true
k run -n infra-securecodebox --rm -it --image nginx:alpine3.17 nginx-test-scan --set config.containerAutoDiscovery.enabled=true

Expected behavior

Secure Code Box auto discovery service ignores deletion of pod with no scans.

System:

  • secureCodeBox: 4.9.0
  • Kubernetes Version: 1.30

Screenshots / Logs

Logs from auto-discovery container

2024-10-08T08:27:19Z    ERROR    controllers.ContainerScanController    Unable to fetch scan    {"name": "nginx-trivy-at-647c5c83418c19eef0cddc647b9899326e3081576390c4c", "error": "ScheduledScan.execution.securecodebox.io \"nginx-trivy-at-647c5c8
github.com/secureCodeBox/secureCodeBox/auto-discovery/kubernetes/controllers.(*ContainerScanReconciler).getOrphanedScanImageIDs
    /workspace/controllers/container_scan_controller.go:422
github.com/secureCodeBox/secureCodeBox/auto-discovery/kubernetes/controllers.(*ContainerScanReconciler).checkIfScansNeedToBeDeleted
    /workspace/controllers/container_scan_controller.go:406
github.com/secureCodeBox/secureCodeBox/auto-discovery/kubernetes/controllers.(*ContainerScanReconciler).Reconcile
    /workspace/controllers/container_scan_controller.go:84
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:222

Additional context

Activity

  1. changed the title [-]Auto Discovery failed to fetch ScheduledScans for removed pod with no scans made before.[/-] [+]Auto Discovery failed to fetch Scan for removed pod with no scans made before.[/+] on Oct 8, 2024
  2. paraddise commented on Oct 8, 2024

    @paraddise
    Author
  3. paraddise commented on Oct 8, 2024

    @paraddise
    Author

    Oh, I see getOrphanedScanImageIDs in case of error just returns empty array, so this must not affect other events

  4. paraddise commented on Oct 8, 2024

    @paraddise
    Author

    Btw if I deploy container to the same namespace as auto-discovery service it won't work. Just no action when creating container

  5. J12934 commented on Oct 9, 2024

    @J12934
    Member

    Hi thanks for the report, yeah that shouldn't log out such a error 👍

    Btw if I deploy container to the same namespace as auto-discovery service it won't work. Just no action when creating container

    Did you annotate the namespace for the auto-discovery with the annotation?
    When annotating a existing namespace with existing deployments it sometimes takes the auto-discovery a while to detect these services, this is a known problem unfortunatly. Haven't had the time yet to debug this fully. :/

  6. moved this from Backlog to Todo in secureCodeBoxon Oct 9, 2024
  7. paraddise commented on Oct 9, 2024

    @paraddise
    Author

    I definitely annotated namespace, because service log warning about non-existent scan type during direst deployment. After that I created scan type. Also service sees when pod is deleted and logs warning about non-existent Scan for this container.

  8. paraddise commented on Oct 9, 2024

    @paraddise
    Author

    But serviceDiscovery works in the same namespace, containerDiscovery - no. There are just no logs.

  9. moved this from Todo to In Progress in secureCodeBoxon Feb 23, 2025
  10. self-assigned this
    on Feb 23, 2025
  11. added a commit that references this issue on Mar 7, 2025
  12. BorisShek commented on Mar 23, 2025

    @BorisShek
    Contributor

    Hi @paraddise,

    While addressing your issue, I've removed unnecessary logs related to the deletion of pods without scans. Additionally, I've slightly updated the documentation page for ContainerAutoDiscovery for improved clarity (Link).

    Btw if I deploy container to the same namespace as auto-discovery service it won't work. Just no action when creating container

    I wasn't able to reproduce it. If the issue remains after updating to the latest release, please let us know. In any case, thank you for your contribution so far!

    But serviceDiscovery works in the same namespace, containerDiscovery - no. There are just no logs.

    I've added logs to ContainerAutoDiscovery about tracking newly discovered unscanned pods and the deletion of orphaned trivy scans, so now it should be more transparent (Commit).

  13. moved this from In Progress to Done in secureCodeBoxon Mar 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions