Skip to content

Commit 219a9de

Browse files
pieqqpieqq
authored
Introduce refresh/revert tests for kernel/snapd/gadget snaps (New) (#805)
* Revamp the kernel_snap_test.py script into something more generic This commit builds upon Patrick Liu's kernel_snap_test.py script (see commit 3b73301) and does the following in order to make it more generic, so it can be used with not only the kernel snap, but also snapd and gadget snaps: - rename the script `snap_update_test.py` and replace references to "kernel" with something more generic - extract some utility functions from the SnapRefreshRevert class (these functions can be executed without being part of that class) - use revision numbers instead of channels to refresh the snap to; this is because we want to take another use case into account: refreshing to/reverting from the base revision, which is the revision of the snap that is present on the image at install time - add the `guess_snaps()` function to guess the names of the kernel, gadget and snapd snaps - add the `get_snap_base_rev()` function to retrieve the base revision of a given snap from the system Fix CHECKBOX-717 * Add unit tests for the snap_update_test.py script * Revamp kernel/gadget/snapd refresh/revert tests - Turn existing kernel-only jobs into generic ones, using the snap_revision_info resource job as a template for the naming - Add jobs to refresh to/revert from base revision - Separate the reboot section into its own job: because the noreturn/autorestart flags have a big impact on Checkbox, it's better to isolate these jobs to the smallest possible action (in this case, a reboot). This also helps with keeping stdout/stderr to avoid issue [1] #694 * Rework ubuntucore-automated Keep the kernel tests, but use the new jobs. * Add 3 test plans to test refresh/revert for each snaps (kernel/snapd/gadget) These 3 test plans are aimed at being used during the snap-update-verification process, where typically: - a new snap (kernel, gadget, snapd...) is released and put into the beta channel - this triggers a Checkbox run using a defined test plan on a set of defined devices Depending on the snap being updated, operator can select the right test plan for the job. I initially wanted to put everything into the same test plan and use `requires:` section of the jobs to skip the jobs that were not required (for instance, if testing a new gadget snap, the kernel snap will not have a new revision available, so related jobs should be skipped). However, I could not find a proper way to deal with this using the current requirement mechanism in Checkbox jobs. * Add requirements for the snap name and revisions to allow jobs to be skipped This requirement trick allows to specify the stable_rev/base_rev and original_installed_rev to check for the snap being tested. This allows to efficiently skip the job if said snap has the same source and destination revision. Thanks to this, these jobs are not marked as failed anymore, but instead skipped. * Use one test plan to test kernel/snapd/gadget snap refresh/revert Thanks to the previous commit, only one test plan can be used instead of 3 separate test plans. This way, if the test plan is called by the snap-update-verification process because there is a new gadget snap in beta channel, the jobs for the two other snaps (kernel and snapd) will be skipped instead of failing. * Move test data to its own directory * Add test for the snap_revert function * Import patch and mock_open directly * Fix test_snap_revert to work with Python 3.5 * Fix snap_update_test.py after PR feedback * Update related jobs according to feedback * Update unit tests * Few changes to the snap_update_test.py script to make it easier to test Namely: - Add a logger and use it instead of calling logging.xxx() directly - Decouple the logic that waits for a snap change from the verification logic (create a new `wait_for_snap_change()` method and modify `verify()`) - When error happens, put the message as a parameter of SystemExit() instead of logging it first - Pass arguments from command line to the `main()` function * Add error handling when the snap change result is "Error" If there is an error for current snap change, no need to wait for timeout. We log the information from this change and exit with an error. Also add unit tests to test this. * Flake 8 fixes * Fix tests to support Python 3.5 assert_called() was introduced in 3.6... 😭 * Removing kernel snap refresh/revert test from automated UC test plan This test plan is used by both QA and Cert. QA team faces issues when running the kernel refresh/revert snap tests on early builds because the kernel team may not have placed all the expected kernel snap updates in the right channels (they may use private branches for the tests, or share the same channels for different devices for hte same customer). All of this is resolved by the time the devices are sent to Cert lab, so the tests should be added to the SRU test plan instead. * Feedback from PR code review * Black * Remove unnecessary Snapd mock in `test_snap_refresh_different_revision` unit test
1 parent 5cd5a2e commit 219a9de

File tree

6 files changed

+1310
-290
lines changed

6 files changed

+1310
-290
lines changed

‎providers/base/bin/kernel_snap_test.py‎

Lines changed: 0 additions & 254 deletions
This file was deleted.

0 commit comments

Comments
 (0)