pkglib/pkg: hash-dir, update-hashes subcommand, dep-tag propagation#4210
Draft
rucoder wants to merge 4 commits intolinuxkit:masterfrom
Draft
pkglib/pkg: hash-dir, update-hashes subcommand, dep-tag propagation#4210rucoder wants to merge 4 commits intolinuxkit:masterfrom
rucoder wants to merge 4 commits intolinuxkit:masterfrom
Conversation
contentHash previously ran git ls-files on the entire repo, causing extremely slow dirty-hash computation when large untracked directories exist anywhere in the repo (e.g. linux-firmware trees, build artifacts). Scope both the tracked and untracked file listings to the specific package path so contentHash only hashes files relevant to the package. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
7 tasks
ecb198f to
4fa150a
Compare
Add --hash-dir <path> persistent flag to `linuxkit pkg` (shared by
show-tag and build subcommands).
Write side (show-tag --hash-dir):
After computing a package tag, write a YAML manifest to
<hash-dir>/<pkgname>.hash with write-if-changed semantics: the file
mtime is only updated when the tag actually changes, so downstream make
DEPS_FORCE rules are not spuriously triggered.
Format:
tag: lfedge/eve-zfs:cbda1fe2...-2.4
build-yml: build-2.4.yml
Read side (pkg build --hash-dir and hash computation):
@lkt:pkg: and @lkt:pkgs: dep references are resolved by reading the
stored tag from <hash-dir>/<pkgname>.hash rather than calling
NewFromConfig recursively. This eliminates dependency cycles and
correctly propagates version-specific build variants (e.g. ZFS_VERSION
selecting build-2.4.yml) through downstream combined hashes.
When a dep's hash file is absent:
- --strict-deps: error ("run update-hashes first")
- default (lenient): fall back to NewFromConfig with build.yml
Hash propagation:
NewFromConfig now incorporates resolved @lkt: dep tags into the
combined package hash when --hash-dir is set. This means that when
pkg/zfs hash changes (e.g. due to ZFS_VERSION bump), downstream
packages (pillar, dom0-ztools, vtpm) also get new combined hashes,
giving full tag-based cache correctness without DEPS_FORCE as a crutch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
… build.yml requirement - Export PkgImageName and DockerfileARGNames for use in pkg_updatehashes.go - In @lkt:pkgs: wildcard resolution, parse the anchor package's Dockerfile to collect declared ARG names; skip any glob-matched package whose computed key is not declared — eliminates circular/oscillating combined hashes caused by packages referencing themselves or unrelated neighbours - Use hash manifest's stored build-yml (not DefaultPkgBuildYML) when resolving image names for versioned packages like pkg/zfs that have no build.yml - In resolveDepTag: check build.yml existence before NewFromConfig fallback; return "" (lenient) instead of erroring when no default build.yml exists — makes versioned packages first-class citizens without a build.yml symlink - Export ReadHashManifest for use outside pkglib Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…h file update-hashes: - New cobra subcommand 'linuxkit pkg update-hashes --hash-dir <dir> [path[:build-yml]]...' - Parses all package specs (with explicit build-yml for versioned packages) - Builds dep graph from @lkt:pkg:/pkgs: build args filtered by Dockerfile ARGs - Topological sort (Kahn's algorithm) ensures deps processed before consumers - Calls NewFromConfig per package in topo order; writes HashManifest with tag, build-yml, and deps: [{path, tag}] populated - --strict-deps flag errors when a dep package is missing from the list pkg build: - When --hash-dir is set and --build-yml was not explicitly provided, reads effective build-yml from the package's hash file automatically; simplifies EVE Makefile (no per-package --build-yml needed in eve-% rule) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
4fa150a to
516fd4e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Building EVE-OS (and similar multi-package Linuxkit projects) requires that:
pkg/zfs) propagates a new hash to all downstream consumers (pkg/pillar,pkg/dom0-ztools, etc.) so Docker cache tags change and images are rebuilt.linuxkit pkg build.contentHash(dirty-tree computation) must not be pathologically slow on repos with large untracked directories (e.g. linux-firmware, build artefacts).build.yml(onlybuild-2.4.yml,build-2.3.yml, …) must work without a symlink.Issues: #4180, related to #4195.
Solution (4 commits)
1.
pkglib: scope contentHash to package directorycontentHashpreviously rangit ls-fileson the entire repo. Scope it topkg/<name>/so it only hashes files relevant to the package being tagged.2.
pkglib: add --hash-dir flag; write YAML build manifests from show-tagAdd a
--hash-dir <path>persistent flag shared byshow-tagandbuild.Write side (
show-tag --hash-dir): after computing a tag, write a YAML manifest to<hash-dir>/<pkgname>.hashwith write-if-changed semantics (mtime only updates when the tag actually changes, so downstream make targets are not spuriously re-triggered):Read side (
pkg build --hash-dir):@lkt:pkg:and@lkt:pkgs:references are resolved from the stored hash files rather than by recursiveNewFromConfigcalls. This eliminates dependency cycles and correctly propagates version-specific build variants (e.g.ZFS_VERSIONselectingbuild-2.4.yml) through downstream combined hashes.pkg buildauto-detects build-yml: when--hash-diris set and--build-ymlwas not given,buildreads the effective build-yml from the package's hash file. No per-package--build-ymlflag needed in the calling Makefile.3.
pkglib: filter @lkt:pkgs: wildcard to Dockerfile-declared ARGs; relax build.yml requirementWhen resolving
@lkt:pkgs:../*wildcards, parse the anchor package'sDockerfileto collect declaredARGnames and skip any glob-matched package whose computed key is not declared. This eliminates oscillating hashes caused by packages referencing themselves or unrelated neighbours.Also: return
""(lenient) instead of erroring when a versioned package has no defaultbuild.yml— makesZFS-style packages first-class citizens.4.
pkg: add update-hashes subcommandNew subcommand:
linuxkit pkg update-hashes --hash-dir <dir> [path[:build-yml]]...build-ymlper package (pkg/zfs:build-2.4.yml) for versioned packages.@lkt:pkgs:build args, topologically sorts it (Kahn's algorithm), and callsshow-tag --hash-dirin dependency order so each package's hash file is ready before its consumers are processed.--strict-deps: error if a dep is missing from the provided list.This is the entry-point that a Makefile calls once per invocation to refresh all hash files before per-package
linuxkit pkg buildcalls.Usage (EVE Makefile pattern)
Testing
pkg/zfssource change →show-tag pkg/zfsnew hash →update-hashespropagates new hash topkg/pillar,pkg/dom0-ztools,pkg/vtpm.ZFS_VERSION=2.4 update-hashes→pkg/zfsgetsbuild-2.4.ymlrecorded in hash file →pkg build --hash-dirpicks it up automatically.@lkt:pkgs:../*patterns.contentHashcompletes in normal time on repos with large untracked trees.🤖 Generated with Claude Code