Skip to content

⚙️ Extend the git-repo-scanner with an activity filter #320

Closed

Description

@rfelber

Is your feature request related to a problem? Please describe.

As secureCodeBox user i am heavily using the git-repo-scanner in combination with the gitleaks scanner via cascadingRules. When scanning large Gitlab oder GitHub organisations with thousands of repos on a regular basis (daily, weekly) this leeds to thousands of independent gitleaks scans.

To save resources on the gitleaks side we extended gitleaks to be able to analyse all commits since a given timeframe (last 24h, last week,...): gitleaks/gitleaks#498

Problem is that the git-repo-scanner always returns all git repositories, even if they had no activity in the last timeframe. To safe even more resources it would be great to be able to configure an activity timeframe and use it as filter.

Describe the solution you'd like

Add the following configuration options to the git-repo-scanner:

--activity-since-duration= Return git repo findings with repo activity (e.g. commits) more recent than a specific date expresed by an duration (now + duration). A duration string is a possibly signed sequence of decimal numbers, each
                               with optional fraction and a unit suffix, such as '300ms', '-1.5h' or '2h45m'. Valid time units are 'ns', 'us' (or 'µs'), 'ms', 's', 'm', 'h'.
--activity-until-duration= Return git repo findings with repo activity (e.g. commits) older than a specific date expresed by an duration (now + duration). A duration string is a possibly signed sequence of decimal numbers, each with
                               optional fraction and a unit suffix, such as '300ms', '-1.5h' or '2h45m'. Valid time units are 'ns', 'us' (or 'µs'), 'ms', 's', 'm', 'h'.

Remark 🚧 : The concrete semantic of the timeframe definition can depend on the implementation details and might be slightly different. It is not important to implement it in exact that semantic.

Implementation Hints:

Example ScheduledScan Configuration

The following scheduled scan example will start a new git-repo-scanner scan every 24h and then cascades all the results with follow up gitleaks scans. The gitleaksscanner then analyses all the commits done within the last 24h timeframe.

apiVersion: "execution.securecodebox.io/v1"
kind: ScheduledScan
metadata:
  name: "scb-github-repos"
  labels:
    product: "secureCodeBox"
spec:
  interval: 24h
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  scanSpec:
    scanType: "git-repo-scanner"
    parameters:
      # configuration of the target git system
      - "--git-type"
      - "github"
      - "--organization"
      - "secureCodeBox"
      # Provide an access token from ENV Vars defined via secret
      - "--access-token"
      - "$(GITHUB_TOKEN)"
      # Filter findings - only return git repos with activity (commits) within a given timeframe 
      - "--activity-since-duration"
      - "24h"
    env:
      - name: GITHUB_TOKEN
        valueFrom:
          secretKeyRef:
            name: github-access-token
            key: token
    cascades:
      matchLabels:
        securecodebox.io/intensive: medium
        securecodebox.io/invasive: non-invasive
apiVersion: "cascading.securecodebox.io/v1"
kind: CascadingRule
metadata:
  name: "gitleaks-github-scan-public"
  labels:
    securecodebox.io/invasive: non-invasive
    securecodebox.io/intensive: medium
spec:
  matches:
    anyOf:
      - name: "GitHub Repo"
        attributes:
          visibility: public
  scanSpec:
    scanType: "gitleaks"
    parameters:
      - "--repo-url"
      - "{{{attributes.web_url}}}"
      # Apply all available rules
      - "--config-path"
      - "/home/config_all.toml"
      # Redact secrets from log messages and leaks
      - "--redact"
      # Only scan commits since the last 24h
      - "--commit-since-duration"
      - "24h"
      # Provide an access token from ENV Vars defined via secret
      - "--access-token"
      - "$(GITHUB_TOKEN)"
    env:
      - name: GITHUB_TOKEN
        valueFrom:
          secretKeyRef:
            name: github-access-token
            key: token

Describe alternatives you've considered

none

Additional context

none

Activity

  1. added
    enhancementNew feature or request
    scannerImplement or update a security scanner
    on Mar 9, 2021
  2. glermaidt commented on Mar 17, 2021

    @glermaidt

    Hi. I'm curious when this may be available? I too have thousands of repos, which puts a huge load on my k8s cluster. This is just the feature i'm looking for. :)

  3. rfelber commented on Mar 19, 2021

    @rfelber
    MemberAuthor

    Hi @glermaidt
    this feature is already implemented in the referenced PR #327 and in works in combination with the PR #312 already 👍🏻.
    There is still some code cleanup and refactoring work to do but i guess both PRs will make it into our next release within the next two weeks.

    You can already test both PRs for your setup as documented in the PR description. Feedback is always welcome :-)

  4. self-assigned this
    on Mar 19, 2021
  5. modified the milestones: v2.7.0, v2.6.0 on Mar 20, 2021
  6. added
    pythonIssues based on python implementations
    on Mar 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestpythonIssues based on python implementationsscannerImplement or update a security scanner

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions