[stable-v2.2] cherry-pick docker and other upgrade from main branch #7551

marc-hb · 2023-05-02T21:00:38Z

95%, mostly clean cherry-picks from main branch

This avoids branching the Docker image and turns all checks green.

fredoh9 · 2023-05-02T21:16:15Z

there is no sof-cnl.ri generated. apl, jsl, tgl are built fine.

14:09:23
[2023-05-02 21:09:23 UTC][SOF-SH] Now time: 2023-05-02T14:09:22-07:00 | Device time: 2023-05-02T21:09:23+00:0014:09:24 
[2023-05-02 21:09:24 UTC][SOF-SH] ERROR: Failed to retrieve Firmware resource from [.../pr/sof/PR7551-6887/build6887/fw/xcc/sof-cnl.ri]

marc-hb · 2023-05-03T01:10:23Z

This is blocked by

[SKIP CI] sof_builder/Dockerfile: restore deleted toolchains #7552

+ manual rebuild and deploy.

kv2019i · 2023-05-15T08:18:25Z

#7552 now merged.

fredoh9 · 2023-05-16T19:08:06Z

I built new docker image using this PR. thesofproject/sof:20230515_ubuntu22.04. I found there are some missing commits.
Oh, #7552 is merged, then I need to re-build again with main.

marc-hb · 2023-05-16T21:18:38Z

I built new docker image using this PR.

This stable-2.2 PR does not affect the docker image. It's only waiting for a new docker image with #7552

This suddenly failed in for PR thesofproject#7551 in https://github.com/thesofproject/sof/actions/runs/4998109337/ jobs/8953192254 I have no idea why it never failed earlier (new yamllint version maybe?) but it's always been wrong and should have never been there anyway. Fixes commit 3d69a7f (".github: extend yamllint line-length to 100") Signed-off-by: Marc Herbert <marc.herbert@intel.com>

This is a release branch, we shouldn't be using any "latest" docker image. Signed-off-by: Marc Herbert <marc.herbert@intel.com>

Support downloads from within the container - notably cloning Zephyr. Similar to commit 424da2c ("zephyr/docker: pass http[s]_proxy variables to the container") on the main branch. Signed-off-by: Marc Herbert <marc.herbert@intel.com>

Dropping *.yml change from the original commit cause there is no west.yml in this branch. Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 3d69a7f)

This will make sure platforms without an open-source toolchain available are added to SUPPORTED_PLATFORMS and do not break the -a option Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 8a7a1ad)

This should get rid of most warnings in daily tests ``` Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/ 2022-09-22-github-actions-all-actions-will-begin-running-on-node16... Please update the following actions to use Node.js 16: actions/checkout@v2 ``` Example at https://github.com/thesofproject/sof/actions/runs/3597808171 v3 seems backward compatible. Upgrade only the most used instances for now (most used because of the `matrix` of platforms), upgrade everything in a few days if no issue is spotted. Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit f71eb15)

This upgrade was already performed for other jobs in commit f71eb15 (".github/workflows: upgrade actions/checkout@v2 -> v3") and everything went fine. Finish the job and get rid of the last warnings in the daily tests (example: https://github.com/thesofproject/sof/actions/runs/3709176785) stable-v2.2: dropped sof-docs action which does not exist in this branch. Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 2ea4bc0)

Search and replace checkout@v2 with checkout@v3. This finally gets rid of all warnings "Node.js 12 actions are deprecated". We've been using v3 in a few other places and never met any backwards compatibility issue. Fixed testbench.yml conflict, still embedded in pull-request.yml Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 65a68b7)

Fixes commit d09844a ("zephyr/docker-build.sh: match UID with 'adduser' instead of 'chgrp -R'") Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit dfc6b46)

sudo-cwd.sh was developed on the main branch for zephyr but now we want to re-use it for building topologies. Cherry-pick only that script, not the zephyr parts. Original commit message: Besides making things more obvious, the important functional change is that the user switch is now performed for _every_ invoked, command, not just for the build command. Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 027be98)

This script is now generic. This was not done earlier to be gentle on git blame. Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 0a4b1d6)

Fixes commit d09844a ("zephyr/docker-build.sh: match UID with 'adduser' instead of 'chgrp -R'") Also clarify comment and add reference to new sudo-cwd.sh script. Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit c28400b)

Support downloads from within the container. Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 424da2c)

Copying a file that does not exist obviously fails. This bug was found when trying to switch from the current "Developer Image" to the smaller "CI Image": https://github.com/zephyrproject-rtos/docker-image zephyr-build Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 75fa04d)

There is a UID mistmatch and file permission problem. sudo-cwd.sh will switch id every docker run command. Commit 80e9c34 ("scripts/docker-run.sh: run with sudo-cwd.sh") was reverted due to missing a toolchain. Signed-off-by: Fred Oh <fred.oh@linux.intel.com> (cherry picked from commit 745d4cc)

marc-hb · 2023-05-18T18:11:36Z

Only a couple suspend/resume failures in https://sof-ci.01.org/sofpr/PR7551/build8011/devicetest/index.html

checkpatch is wrong in https://github.com/thesofproject/sof/actions/runs/5008561941/jobs/8976565505?pr=7551

https://sof-ci.01.org/sof-pr-viewer/#/build/PR7551/build12013250 cannot work because quickbuild CI is simply not compatible with the stable-v2.2 branch which uses a very different manifest.

Everything else is green.

@wszypelt, @aborisovich can we please disable quickbuild on the stable-v2.2 branch? Unless someone is capable of reverting QB to what it was at v2.2 time but I'm really not holding my breath.

greg-intel · 2023-05-18T18:51:55Z

.github/workflows/zephyr.yml

      - name: build
        run: docker run -v "$(pwd)":/workdir
-             ghcr.io/zephyrproject-rtos/zephyr-build:latest
+             ghcr.io/zephyrproject-rtos/zephyr-build:v0.23.4


We are explicitly changing the behavior here? Changing from the "moving target" of "latest" to a hard-coded version? Does that mean someone has to make sure to keep this up-to-date?

Yes because this is a stable branch, see commit message.

greg-intel · 2023-05-18T18:53:04Z

scripts/docker-run.sh

+# ever goes wrong try replacing --user with the newer
+# scripts/sudo-cwd.sh script.
+test "$(id -u)" = 1000 ||
+  >&2 printf "Warning: this script should be run as user ID 1000 to match the container's account\n"


Out of curiosity, what changed the user from 1001 to 1000?

This is a bug fix, I got the ID wrong. See commit message.

greg-intel · 2023-05-18T18:56:42Z

zephyr/docker-build.sh

    # zephyr-build:/etc/sudoers: 'user' can do anything but... only as
    # root.
-    sudo sudo -u "$sof_user" "$0" "$@"
+    sudo sudo -u "$sof_user" http_proxy="$http_proxy"  https_proxy="$https_proxy" "$0" "$@"


I know this all works and has been tested, but where does $http_proxy and $https_proxy come from here?

This is a shell script so they come from your personal environment (and they're optional)

fredoh9

looks good for stable-v2.2

marc-hb

Thanks for the reviews!

marc-hb · 2023-05-18T23:00:24Z

.github/workflows/zephyr.yml

      - name: build
        run: docker run -v "$(pwd)":/workdir
-             ghcr.io/zephyrproject-rtos/zephyr-build:latest
+             ghcr.io/zephyrproject-rtos/zephyr-build:v0.23.4


Yes because this is a stable branch, see commit message.

marc-hb · 2023-05-18T23:00:59Z

scripts/docker-run.sh

+# ever goes wrong try replacing --user with the newer
+# scripts/sudo-cwd.sh script.
+test "$(id -u)" = 1000 ||
+  >&2 printf "Warning: this script should be run as user ID 1000 to match the container's account\n"


This is a bug fix, I got the ID wrong. See commit message.

marc-hb · 2023-05-18T23:01:55Z

zephyr/docker-build.sh

    # zephyr-build:/etc/sudoers: 'user' can do anything but... only as
    # root.
-    sudo sudo -u "$sof_user" "$0" "$@"
+    sudo sudo -u "$sof_user" http_proxy="$http_proxy"  https_proxy="$https_proxy" "$0" "$@"


This is a shell script so they come from your personal environment (and they're optional)

This comment was marked as outdated.

Sign in to view

marc-hb force-pushed the docker-2.2 branch 2 times, most recently from ad7bffc to fdd9aec Compare May 3, 2023 00:46

marc-hb mentioned this pull request May 12, 2023

[stable-v2.2] intel: ssp: drain RX fifo when starting #7601

Merged

marc-hb mentioned this pull request May 17, 2023

[SKIP SOF-TEST] .github/codestyle: drop spurious *.yml pattern #7632

Closed

marc-hb force-pushed the docker-2.2 branch from 788fab0 to 85fab53 Compare May 17, 2023 21:08

marc-hb added 3 commits May 17, 2023 14:28

.github/zephyr: hardcode docker image to v0.23.4 (ZSDK 0.14)

26d632a

This is a release branch, we shouldn't be using any "latest" docker image. Signed-off-by: Marc Herbert <marc.herbert@intel.com>

.github: extend yamllint line-length to 100

c02aa82

Dropping *.yml change from the original commit cause there is no west.yml in this branch. Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 3d69a7f)

marc-hb force-pushed the docker-2.2 branch from 85fab53 to d0d80eb Compare May 17, 2023 21:29

marc-hb and others added 11 commits May 17, 2023 15:41

.github: add new job that builds all DEFAULT_PLATFORMS

475f193

This will make sure platforms without an open-source toolchain available are added to SUPPORTED_PLATFORMS and do not break the -a option Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 8a7a1ad)

scripts/docker-run.sh: fix wrong "id -n" command -> "id -u"

be64732

Fixes commit d09844a ("zephyr/docker-build.sh: match UID with 'adduser' instead of 'chgrp -R'") Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit dfc6b46)

sudo-cwd.sh: drop the all 'sof_' prefixes and references

06eb7fc

This script is now generic. This was not done earlier to be gentle on git blame. Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 0a4b1d6)

zephyr/docker: pass http[s]_proxy variables to the container

6fadce3

Support downloads from within the container. Signed-off-by: Marc Herbert <marc.herbert@intel.com> (cherry picked from commit 424da2c)

marc-hb force-pushed the docker-2.2 branch from d0d80eb to 75f6ae2 Compare May 17, 2023 22:43

marc-hb marked this pull request as ready for review May 17, 2023 22:55

marc-hb requested review from aborisovich, cujomalainey, dbaluta and lgirdwood as code owners May 17, 2023 22:55

marc-hb changed the title ~~[stable-v2.2] cherry-pick docker upgrade from main branch~~ [stable-v2.2] cherry-pick docker and other upgrade from main branch May 17, 2023

cujomalainey approved these changes May 17, 2023

View reviewed changes

marc-hb requested review from aiChaoSONG, andyross, greg-intel, keqiaozhang, kv2019i, mengdonglin, mwasko and paulstelian97 May 18, 2023 18:14

greg-intel approved these changes May 18, 2023

View reviewed changes

fredoh9 approved these changes May 18, 2023

View reviewed changes

marc-hb commented May 18, 2023

View reviewed changes

kv2019i merged commit 4bf9059 into thesofproject:stable-v2.2 May 19, 2023

marc-hb deleted the docker-2.2 branch May 19, 2023 22:19

marc-hb mentioned this pull request May 23, 2023

[SKIP SOF-TEST] Remove old fuzzer #7675

Merged

[stable-v2.2] cherry-pick docker and other upgrade from main branch #7551

[stable-v2.2] cherry-pick docker and other upgrade from main branch #7551

Uh oh!

Conversation

marc-hb commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fredoh9 commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

marc-hb commented May 3, 2023

Uh oh!

kv2019i commented May 15, 2023

Uh oh!

fredoh9 commented May 16, 2023

Uh oh!

marc-hb commented May 16, 2023

Uh oh!

marc-hb commented May 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greg-intel May 18, 2023

Choose a reason for hiding this comment

Uh oh!

marc-hb May 18, 2023

Choose a reason for hiding this comment

Uh oh!

greg-intel May 18, 2023

Choose a reason for hiding this comment

Uh oh!

marc-hb May 18, 2023

Choose a reason for hiding this comment

Uh oh!

greg-intel May 18, 2023

Choose a reason for hiding this comment

Uh oh!

marc-hb May 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fredoh9 left a comment

Choose a reason for hiding this comment

Uh oh!

marc-hb left a comment

Choose a reason for hiding this comment

Uh oh!

marc-hb May 18, 2023

Choose a reason for hiding this comment

Uh oh!

marc-hb May 18, 2023

Choose a reason for hiding this comment

Uh oh!

marc-hb May 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

marc-hb commented May 2, 2023 •

edited

Loading

fredoh9 commented May 2, 2023 •

edited

Loading

marc-hb commented May 18, 2023 •

edited

Loading

marc-hb May 18, 2023 •

edited

Loading

marc-hb May 18, 2023 •

edited

Loading