Skip to content

Add containerd image verification to k8s and expand ecs-3 verifiers#4832

Open
ginglis13 wants to merge 3 commits into
bottlerocket-os:developfrom
ginglis13:image-verifiers
Open

Add containerd image verification to k8s and expand ecs-3 verifiers#4832
ginglis13 wants to merge 3 commits into
bottlerocket-os:developfrom
ginglis13:image-verifiers

Conversation

@ginglis13
Copy link
Copy Markdown
Contributor

@ginglis13 ginglis13 commented May 8, 2026 •
edited
Loading

Issue number:

Closes #4684

Description of changes:

  • Add image verification settings for k8s-1.33, k8s-1.34, and k8s-1.35 variants
  • Add image verification packages aws-signer-notation-plugin, digestion-image-verifier, notation-image-verifier, and bottlerocket-thar-be-image-verifiers, in k8s-1.33, k8s-1.34, and k8s-1.35 variants
  • Add digestion-image-verifier and thar-be-image-verifiers in ecs-3 variants
  • Add custom migration for image-verifier-plugins settings so that migration only applies to k8s variants

Pending #4827 to add these packages and settings for the new k8s-1.36 variants

Testing done:

k8s testing performed on:

{
  "os": {
    "arch": "x86_64",
    "build_id": "3adeb5d4",
    "pretty_name": "Bottlerocket OS 1.60.0 (aws-k8s-1.34)",
    "variant_id": "aws-k8s-1.34",
    "version_id": "1.60.0"
  }
}

ecs-3 testing performed on:

bash-5.2# apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "3adeb5d4",
    "pretty_name": "Bottlerocket OS 1.60.0 (aws-ecs-3)",
    "variant_id": "aws-ecs-3",
    "version_id": "1.60.0"
  }
}
  • notation verifier on k8s
Details

Trust policy that allows signed image public.ecr.aws/o7o0w6s5/public-alpine-clone:latest:

bash-5.2# apiclient get settings.image
{
  "settings": {
    "image-verifier-plugins": {
      "enabled": true,
      "notation": {
        "trustpolicy": "ewogICAidmVyc2lvbiI6IjEuMCIsCiAgICJ0cnVzdFBvbGljaWVzIjpbCiAgICAgIHsKICAgICAgICAgIm5hbWUiOiJhd3Mtc2lnbmVyLXRwIiwKICAgICAgICAgInJlZ2lzdHJ5U2NvcGVzIjpbCiAgICAgICAgICAgICIqIgogICAgICAgICBdLAogICAgICAgICAic2lnbmF0dXJlVmVyaWZpY2F0aW9uIjp7CiAgICAgICAgICAgICJsZXZlbCI6InN0cmljdCIsCgkJCSJvdmVycmlkZSI6IHsKICAgICAgICAgICAgICAicmV2b2NhdGlvbiI6ICJza2lwIgoJCQl9CiAgICAgICAgIH0sCiAgICAgICAgICJ0cnVzdFN0b3JlcyI6WwogICAgICAgICAgICAic2lnbmluZ0F1dGhvcml0eTphd3Mtc2lnbmVyLXRzIiwKICAgICAgICAgICAgInNpZ25pbmdBdXRob3JpdHk6YXdzLXVzLWdvdi1zaWduZXItdHMiCiAgICAgICAgIF0sCiAgICAgICAgICJ0cnVzdGVkSWRlbnRpdGllcyI6WwogICAgICAgICAgICAiYXJuOmF3czpzaWduZXI6dXMtd2VzdC0yOjQ1ODM1ODk2MjIyNDovc2lnbmluZy1wcm9maWxlcy9ub3RhdGlvbl90ZXN0IiwKICAgICAgICAgICAgImFybjphd3MtdXMtZ292OnNpZ25lcjp1cy1nb3Ytd2VzdC0xOjI3ODMwODIxMTQzMDovc2lnbmluZy1wcm9maWxlcy9ub3RhdGlvbl90ZXN0IgogICAgICAgICBdCiAgICAgIH0KICAgXQp9Cg=="
      }
    }
  }
}

Test 3 cases: 1/ pull the signed image (allow), 2/ pull an unsigned image (deny) 3/ pull an image with a different signature not explicitly allowed in the trustpolicy (deny):

bash-5.2# ctr i pull public.ecr.aws/o7o0w6s5/public-alpine-clone:latest
...
Pulling from OCI Registry (public.ecr.aws/o7o0w6s5/public-alpine-clone:latest)  elapsed: 2.3 s  total:  3.6 Mi  (1.6 MiB/

bash-5.2# ctr i pull docker.io/library/hello-world:latest
ctr: image verifier bindir blocked pull of docker.io/library/hello-world:latest with digest sha256:f9078146db2e05e794366b1bfe584a14ea6317f44027d10ef7dad65279026885 for reason: verifier notation-image-verifier rejected image (exit code 1): image verification failed: Error: signature verification failed: no signature is associated with "docker.io/library/hello-world@sha256:f9078146db2e05e794366b1bfe584a14ea6317f44027d10ef7dad65279026885", make sure the artifact was signed successfully

bash-5.2# ctr i pull public.ecr.aws/o7o0w6s5/public-nginx-clone:latest
ctr: image verifier bindir blocked pull of public.ecr.aws/o7o0w6s5/public-nginx-clone:latest with digest sha256:bd1578eec775d0b28fd7f664b182b7e1fb75f1dd09f92d865dababe8525dfe8b for reason: verifier notation-image-verifier rejected image (exit code 1): image verification failed: Error: signature verification failedfor all the signatures associated with public.ecr.aws/o7o0w6s5/public-nginx-clone@sha256:bd1578eec775d0b28fd7f664b182b7e1fb75f1dd09f92d865dababe8525dfe8b
  • digestion verifier on k8s
Details

Trust policy that allows digest of public.ecr.aws/docker/library/redis:7.4.8-alpine3.21 only:

bash-5.2# apiclient get settings.image
{
  "settings": {
    "image-verifier-plugins": {
      "digestion": {
        "trustpolicy": "ewogICJ2ZXJzaW9uIjogIjEuMCIsCiAgInRydXN0ZWREaWdlc3RzIjogWwogICAgInNoYTI1Njo3YWVjNzM0YjJiYjI5OGExZDc2OWZkODcyOWYxM2I4NTE0YTQxYmY5MGZjZGQxZjM4ZWM1MjI2N2ZiYWE4ZWU2IgogIF0KfQo="
      },
      "enabled": true
    }
  }
}

Test for allow/deny:

bash-5.2# ctr i pull public.ecr.aws/docker/library/redis:7.4.8-alpine3.21
...
Pulling from OCI Registry (public.ecr.aws/docker/library/redis:7.4.8-alpine3.21)        elapsed: 2.3 s  total:  16.5 M  (7.2 MiB/s)

bash-5.2# ctr i pull public.ecr.aws/docker/library/redis:6.2.20-alpine3.21
ctr: image verifier bindir blocked pull of public.ecr.aws/docker/library/redis:6.2.20-alpine3.21 with digest sha256:77697a75da9f94e9357b61fcaf8345f69e3d9d32e9d15032c8415c21263977dc for reason: verifier digestion-image-verifier rejected image (exit code 1): image verification failed: digest not in allowlist
  • custom verifier on k8s
Details

Used the Dockerfile and Go program below. The Go program checks for existence of a mock config file; if present, pass, if not, fail. The Go binary is written to /.bottlerocket/rootfs/opt/civ/bin/custom-verifier, which is the bin_dir configured for containerd image verifiers. Configured as a bootstrap container on k8s instance with user-data script below b64 encoded for the "allow" case. For deny case, adjusted the script to not include configuration file.

FROM golang:1.21-alpine AS builder

WORKDIR /app
COPY custom-civ/ .

RUN CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o custom-verifier .

FROM public.ecr.aws/bottlerocket/bottlerocket-bootstrap:v0.2.15

RUN mkdir -p /.bottlerocket/rootfs/opt/civ/bin /.bottlerocket/rootfs/var/lib
RUN echo "civ-test-config" > /.bottlerocket/rootfs/var/lib/civ-config

COPY --from=builder /app/custom-verifier /.bottlerocket/rootfs/opt/civ/bin/custom-verifier
RUN chmod +x /.bottlerocket/rootfs/opt/civ/bin/custom-verifier
#!/bin/bash
# Create directories and copy binary to host
mkdir -p /.bottlerocket/rootfs/opt/civ/bin /.bottlerocket/rootfs/var/lib
cp /usr/local/bin/custom-verifier /.bottlerocket/rootfs/opt/civ/bin/custom-verifier
chmod +x /.bottlerocket/rootfs/opt/civ/bin/custom-verifier

# Create mock configuration file
echo "civ-test-config" > /.bottlerocket/rootfs/var/lib/civ-config

echo "Bootstrap setup complete"
// The mock verifier

package main

import (
        "os"
)

func main() {
        if _, err := os.Stat("/var/lib/civ-config"); err != nil {
                os.Exit(1)
        }
        os.Exit(0)
}

On the instance:

bash-5.2# ls /opt/civ/bin
custom-verifier  digestion-image-verifier  notation-image-verifier

bash-5.2# ls /var/lib/civ-config
/var/lib/civ-config

bash-5.2# ctr i pull docker.io/library/hello-world:latest
...
Completed pull from OCI Registry (docker.io/library/hello-world:latest) elapsed: 3.6 s  total:  42.3 K  (11.7 KiB/s)

With script that does not write config file to /var/lib/civ-config, pull rejected by custom-verifier.

  • digestion verifier on ecs-3
Details

Same process as digestion verifier test on k8s with only public.ecr.aws/docker/library/redis:7.4.8-alpine3.21 in digest allowlist:

bash-5.2# ctr i pull public.ecr.aws/docker/library/redis:7.4.8-alpine3.21
...
Pulling from OCI Registry (public.ecr.aws/docker/library/redis:7.4.8-alpine3.21)        elapsed: 1.4 s  total:  16.5 M  (11.7 MiB/s)

bash-5.2# ctr i pull public.ecr.aws/docker/library/redis:6.2.20-alpine3.21
ctr: image verifier bindir blocked pull of public.ecr.aws/docker/library/redis:6.2.20-alpine3.21 with digest sha256:77697a75da9f94e9357b61fcaf8345f69e3d9d32e9d15032c8415c21263977dc for reason: verifier digestion-image-verifier rejected image (exit code 1): image verification failed: digest not in allowlist
  • custom verifier on ecs-3
Details Same userdata for image-verifier-plugins and boostrap-containers as the k8s testing above.
  • Settings migration up/down testing
Details

On ecs-3 variant. After 1.61.0 upgrade, my image verifier settings remain:

[ssm-user@control]$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "3adeb5d4-dirty",
    "pretty_name": "Bottlerocket OS 1.61.0 (aws-ecs-3)",
    "variant_id": "aws-ecs-3",
    "version_id": "1.61.0"
  }
}
[ssm-user@control]$ apiclient get settings.image
{
  "settings": {
    "image-verifier-plugins": {
      "enabled": true,
      "my-fake-verifier": {
        "trustpolicy": "ewogICJ2ZXJzaW9uIjogIjEuMCIsCiAgInRydXN0ZWREaWdlc3RzIjogWwogICAgInNoYTI1Njo3YWVjNzM0YjJiYjI5OGExZDc2OWZkODcyOWYxM2I4NTE0YTQxYmY5MGZjZGQxZjM4ZWM1MjI2N2ZiYWE4ZWU2IgogIF0KfQo="
      }
    }
  }
}

Downgrade:

bash-5.2# signpost rollback-to-inactive
bash-5.2# reboot

...

[ssm-user@control]$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "c1f9ba0c",
    "pretty_name": "Bottlerocket OS 1.60.0 (aws-ecs-3)",
    "variant_id": "aws-ecs-3",
    "version_id": "1.60.0"
  }
}
[ssm-user@control]$ apiclient get settings.image
{
  "settings": {
    "image-verifier-plugins": {
      "enabled": true,
      "my-custom": {
        "trustpolicy": "ewogICJ2ZXJzaW9uIjogIjEuMCIsCiAgInRydXN0ZWREaWdlc3RzIjogWwogICAgInNoYTI1Njo3YWVjNzM0YjJiYjI5OGExZDc2OWZkODcyOWYxM2I4NTE0YTQxYmY5MGZjZGQxZjM4ZWM1MjI2N2ZiYWE4ZWU2IgogIF0KfQo="
      }
    }
  }
}

Existing settings remain for ecs ✅

Now for k8s:

[ssm-user@control]$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "c1f9ba0c",
    "pretty_name": "Bottlerocket OS 1.60.0 (aws-k8s-1.34)",
    "variant_id": "aws-k8s-1.34",
    "version_id": "1.60.0"
  }
}

bash-5.2# apiclient get settings.image
{}

updog update -i 1.61.0 -r -n

[ssm-user@control]$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "3adeb5d4-dirty",
    "pretty_name": "Bottlerocket OS 1.61.0 (aws-k8s-1.34)",
    "variant_id": "aws-k8s-1.34",
    "version_id": "1.61.0"
  }
}

Downgrade:

bash-5.2# signpost rollback-to-inactive
bash-5.2# reboot

...

[ssm-user@control]$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "c1f9ba0c",
    "pretty_name": "Bottlerocket OS 1.60.0 (aws-k8s-1.34)",
    "variant_id": "aws-k8s-1.34",
    "version_id": "1.60.0"
  }
}
[ssm-user@control]$ apiclient get settings.image
{}
[ssm-user@control]$

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@ginglis13
sources: add image verifier settings to k8s variants
adfb433 
Signed-off-by: Gavin Inglis <giinglis@amazon.com>
@ginglis13
variants: add image verification packages to k8s and ecs-3
08ca0e8 
Add image verification packages aws-signer-notation-plugin,
digestion-image-verifier, notation-image-verifier, and
bottlerocket-thar-be-image-verifiers, in k8s-1.33, k8s-1.34, and
k8s-1.35 variants

Add digestion-image-verifier and bottlerocket-thar-be-image-verifiers in
ecs-3 variants

Signed-off-by: Gavin Inglis <giinglis@amazon.com>
@ginglis13
migrations: migrator for image-verifier-plugins on k8s
1c5ee7b 
Add migration for these new settings for k8s variants. no-op on ecs
variants as these settings have previously existed

Signed-off-by: Gavin Inglis <giinglis@amazon.com>
@ginglis13 ginglis13 marked this pull request as ready for review May 12, 2026 00:00
@ginglis13 ginglis13 mentioned this pull request May 12, 2026
Merged
@ginglis13 ginglis13 requested a review from piyush-jena May 13, 2026 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@KCSesh KCSesh Awaiting requested review from KCSesh

@arnaldo2792 arnaldo2792 Awaiting requested review from arnaldo2792

@vigh-m vigh-m Awaiting requested review from vigh-m

@piyush-jena piyush-jena Awaiting requested review from piyush-jena

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

support container image verification via containerd

1 participant

@ginglis13