Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
241 changes: 241 additions & 0 deletions docs/adr/adr_0009.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,241 @@
<!--
SPDX-FileCopyrightText: 2021 iteratec GmbH

SPDX-License-Identifier: Apache-2.0
-->

# ADR-0009: Architecture for pre-populating the file system of scanners

| <!-- --> | <!-- --> |
|----------------|----------|
| **Status**: | PROPOSED |
| **Date**: | 2021-10-07 |
| **Author(s)**: | Max Maass <max.maass@iteratec.com>, Jannik Hollenbach <Jannik.Hollenbach@iteratec.com> |

## Context

SecureCodeBox should be able to run scanners that require access to the source code of the target. This could include secret scanners like [Gitleaks][gitleaks] or SAST tools like [Semgrep][semgrep]. To achieve this and expose the feature to the user, we need to extend the syntax of the scan definitions and implement the pre-loading functionality.

The current scanners do not support dynamically loading data from a Git repository (except Gitleaks) or other sources. Thus, we need a solution for retrieving data from a source and exposing it to the scanner, without modifying the scanners or their images. In our view, the best way to achieve this is using [Kubernetes init containers][initc]. They can be used to retrieve data and place it in a volume they share with the scanner, and they can be trivially attached to any existing scanner container using built-in features of Kubernetes.


### Question 1: Implementing Different Pre-Population Data Sources

To be useful, the feature will need to support a number of different data sources. At a minimum, Git repositories and file downloads via curl should be supported, but further data sources may be desireable (e.g., SVN, curl+unzip, en- or decryption of data using openssl CLI, ...). Thus, the extensibility of the pre-populator architecture should be considered.

#### Option 1: Modular design using CRDs
For scanners, secureCodeBox uses a modular architecture where additional scanners can be added without modifying the controller. This allows people without knowledge of Go to contribute new scanners without worrying about implementation details of the operator, at the price of having to install the scanners individually via helm.

The pre-populators could be designed in a similar way: A general syntax in the ScanSpec defines which downloader should be started, and parameterizes it using arguments, similar to how the scanners are parameterized. It would use a general-purpose syntax (with the same YAML structure for all downloaders), which can be directly transformed into a Kubernetes specification, like it is done for the scan jobs. Power users can define their own pre-populator jobs, which should be a lot simpler than defining a scanner, as it does not require a parser, sidecar, etc.

A configuration may look something like this, using a fictional [semgrep][semgrep] scanner job (disregard the `mountPoint` for now, it will be discussed later in the ADR):

```yaml
apiVersion: "execution.securecodebox.io/v1"
kind: Scan
metadata:
name: "semgrep-juiceshop"
spec:
scanType: "semgrep"
parameters:
- "--config"
- "p/ci"
- "--config"
- "/etc/rules/semgrep-rules.yaml"
- "/etc/repo"
prepopulateFilesystemFrom:
- mountPoint: /etc/repo
prepopulateType: "git"
parameters:
- "--recursive"
- "https://github.com/bkimminich/juice-shop.git"
- "/etc/repo/juice-shop/"
- mountPoint: /etc/rules
prepopulateType: "curl"
parameters:
- "https://example.com/semgrep-rules.yaml"
- "--output"
- "/etc/rules/semgrep-rules.yaml"
```

##### Advantages
Comment thread
malexmave marked this conversation as resolved.
Resolved

- Modular and extensible
- No task-specific business logic inside the controller
- Easy to also support the feature for hooks
- User can specify the exact options they want the tool to use (e.g., git clone with or without submodules, curl with specific authentication headers, ...)
- Consistent architecture with how the scanners are designed

##### Disadvantages
Comment thread
malexmave marked this conversation as resolved.
Resolved

- Integration into the controller would require changes in several places (@J12934: Can you elaborate a bit on that?)
- Each pre-populator would have to be installed separately using helm
- Building a "combination" pre-loader like "curl+unzip" may require wrapper scripts, or splitting it into two preloaders (one that downloads and one that unzips).

#### Option 2: Implementation Inside the Operator

If we want to avoid adding another CRD and dealing with the overhead of managing the scanners that way (including the overhead of installing the preloaders etc.), the feature can also be implemented directly inside the operator. For this, the ScanSpec could be extended with specific fields for the different pre-populators (i.e., the Git downloader could explicitly ask for the repository, access key, etc., while the curl downloader would have other flags and options). The controller would then translate the spec into a Kubernetes job with the correct parameters and add it as an init container. Optionally, we could also let users specify their own custom jobs that use a container and script of their choice. This would allow them to run special scripts that are specific to their deployment and that would not be useful to send upstream to the project.

An example configuration could look like this:

```yaml
apiVersion: "execution.securecodebox.io/v1"
kind: Scan
metadata:
  name: "semgrep-juiceshop"
spec:
scanType: "semgrep"
parameters:
- "--config"
- "p/ci"
- "--config"
- "/etc/rules/semgrep-rules.yaml"
- "/etc/repo"
prepopulateFilesystemFrom:
- mountPoint: /etc/repo
git:
url: "https://github.com/bkimminich/juice-shop.git"
accessToken: "ABC123..."
recurseSubmodules: true
- mountPoint: /etc/rules
http:
url: "https://example.com/semgrep-rules.yaml"
- mountPoint: /etc/repo
custom:
image: "myuser/mypreparationimage"
parameters:
- "my"
- "custom"
- "parameters"
```

##### Advantages
Comment thread
malexmave marked this conversation as resolved.
Resolved

- No need for a new CRD or separate helm installs for all pre-populators
- More beginner-friendly way of specifying what the pre-populator jobs should do
- More approachable method for running custom pre-populator scripts (no need to place a number of YAML files in the right places inside the secureCodeBox directory structure)

##### Disadvantages
Comment thread
malexmave marked this conversation as resolved.
Resolved

- Requires implementing core functionality of the feature inside the controller instead of the YAML files
- Experts may prefer to parameterize their Git and curl jobs themselves instead of relying on our flags
- Much higher barrier of entry for contributors that want to send new pre-populator jobs upstream (requires knowledge of the controller)
- Allowing custom pre-populator images is a security concern for the infrastructure in a scenario where secureCodeBox is hosted by one company and used by another (i.e., the SaaS model).

#### Option 3: Expose the initContainer Config in the ScanSpec CRD

Another solution would be to simply allow the user to specify their own initContainer(s) in the ScanSpec CRD. This would allow them to create their own, arbitrary pre-populators, completely independently of how we support the feature. There is precedent for this as well: Scanners can define init containers in their `values.yaml` files. The documentation could provide configuration snippets for common tasks as a baseline for users to base their own configurations on.

An example config may look like this:

```yaml
apiVersion: "execution.securecodebox.io/v1"
kind: Scan
metadata:
  name: "semgrep-juiceshop"
spec:
scanType: "semgrep"
parameters:
- "--config"
- "p/ci"
- "--config"
- "/etc/rules/semgrep-rules.yaml"
- "/etc/repo"
volumes:
- name: repo
emptyDir: {}
- name: rules
emptyDir: {}
volumeMounts:
- name: repo
mountPath: "/etc/repo"
- name: rules
mountPath: "/etc/rules"
initContainers:
- name: git-clone
image: bitnami/git
command:
- git
- clone
- "--recursive"
- https://github.com/bkimminich/juice-shop.git
- /etc/repo
volumeMounts:
- name: repo
mountPath: "/etc/repo"
- name: curl-rules
image: busybox
command:
- curl
- "--output"
- "/etc/rules/semgrep-rules.yaml"
- "https://example.com/semgrep-rules.yaml"
volumeMounts:
- name: rules
mountPath: "/etc/rules"
```

##### Advantages
Comment thread
malexmave marked this conversation as resolved.
Resolved

- Maximum flexibility: Users can do anything that can be done with init containers
- No need for us to implement any feature-specific code, aside from one change to the ScanJob CRD

##### Disadvantages
Comment thread
malexmave marked this conversation as resolved.
Resolved

- Assumes the user is familiar with how to use init containers (although we can of course provide example configurations in the documentation)
- In a scenario where secureCodeBox is hosted by one company and used by another (i.e., the SaaS model), letting users specify their own init containers is a security concern for the infrastructure


### Question 2: Specifying Persisted Folders

Transferring data from the init container to the scanner requires [using volumes that are shared between the init container(s) and the scan job][initcvolumes]. These volumes can be defined in multiple ways.

#### Option 1: User-managed Kubernetes Volumes

The easiest solution is to allow the user to specify the volumes and volumeMounts themselves. This is already supported by the ScanSpec CRD, so it would just need to be added to the pre-populator CRD as well. Then, the user can specify exactly which container should see which volume under which path in the file system. This would be the obvious solution if we choose option 3 for question 1 (fully user-specified init containers), but can also work for option 1, and technically also with option 2, although it would likely not be a good fit.

##### Advantages
Comment thread
malexmave marked this conversation as resolved.
Resolved

- Full control for the user
- No need to add any custom code to manage volumes
- Validation is handled by Kubernetes

##### Disadvantages
Comment thread
malexmave marked this conversation as resolved.
Resolved

- Requires the user to know how to create and handle volumes (although we can provide code snippets in the documentation)
- In a scenario where secureCodeBox is hosted by one company and used by another (i.e., the SaaS model), letting users specify their own volume mounts is a security concern for the infrastructure (Denial of service through resource consumption, mounting host directories, ...). However, this is already possible right now.


#### Option 2: Limited User-Managed Volumes

A variation on option 1 would be to allow the user to specify the names of volumes, but not the type (i.e., we force it to be one of an allow-listed set of types: emptyDir, secret, ...). This would require slightly more validation code, but limit the attack surface from the volume mounts. However, this would likely be a separate ADR, as it would also impose new limitations on scans that do not use the pre-populate filesystem / init container feature. I will thus not consider it in greater detail here.


#### Option 3: secureCodeBox-Managed volumes

Finally, we can also abstract away the details of the Kubernetes volume definitions and simply have the user specify which folders they want to modify with their pre-populate job. An example of such a configuration is given in Option 1 and 2 of Question 1. In this case, the controller would be responsible for determining which volumes are needed, and creating and mounting them to the right containers. While this may seem a deceptively simple concepts, it hides a lot of complexity.

##### Advantages
Comment thread
malexmave marked this conversation as resolved.
Resolved

- Does not expose volume definitions to the user, thereby decreasing the attack surface towards malicious users in a SaaS setting
- Does not require the user to know how to define and mount Kubernetes volumes

##### Disadvantages
Comment thread
malexmave marked this conversation as resolved.
Resolved

- Requires a lot of edge case handling for the controller (what if one pre-populator wants to access /etc/repo and another /etc/repo/some/folder? Do both get a mount for /etc/repo? What if the folder /etc/repo/some/folder does not exist when the relevant job is started?)
- Has a high capacity for causing unintuitive and hard-to-debug issues for the user, who cannot benefit from Kubernetes documentation on volumes to understand and resolve them
- Reduces flexibility for power users


## Decision

To be determined.

## Consequences

To be determined.

[initc]: (https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
[initcvolumes]: (https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
[gitleaks]: (https://docs.securecodebox.io/docs/scanners/gitleaks)
[semgrep]: (https://github.com/returntocorp/semgrep)