Skip to content

Conversation

@GabrielChenCC
Copy link
Contributor

@GabrielChenCC GabrielChenCC commented Nov 8, 2023

  1. The commit is for the alsa-loopback test case.
Actually, the alsa-loopback-automated test case does the same
thing with alsa-loopback. When we test iot project machines,
the alsa-loopback-automated needs to be tested. I guess we
could do the alsa-loopback-automated only in the future.
  1. This commit to modify the watchdog test cases.
The watchdog-automated's test items have included watchdog-maunal
so I remove the watchdog-manual test case;
  1. Modify the wwan related test cases in desktop pxus.
The mobilebroadband-cert-automated test case is only
part of the wwan test plan. We hope to replace
mobilebroadband-cert-automated with wwan-automated.

The relevant test plans we currently use include:
client-cert-desktop-22-04-automated
client-cert-desktop-20-04-automated
client-cert-desktop-18-04-automated
I have replaced mobilebroadband-cert-automated in them
with wwan-automated, and added after-sudpend-wwan-
automated to ensure that wwan is tested before and after
suspend.

In addition, for mobilebroadband-cert-manual in the Manual
test plan, when the SIM is inserted into the prototype in the
desktop, the system should automatically recognize wwan and
SIM card, and we do not need to perform manual testing. So I
deleted mobilebroadband-cert-manual in the relevant test
plans. At the same time, since wwan-automated already
contains wwan-manual cases, I deleted the wwan-manual in
these test plans:
client-cert-desktop-22-04-manual
client-cert-desktop-20-04-manual
client-cert-desktop-18-04-manual

Actually, the alsa-loopback-automated test case does the same
thing with alsa-loopback. When we test iot project machines,
the alsa-loopback-automated needs to be tested. I guess we
could do the alsa-loopback-automated only in the futuer.
The watchdog-automated's test items have included
watchdog-maunal,  so I remove the watchdog-manual
test case.
@Hook25
Copy link
Collaborator

Hook25 commented Nov 9, 2023

Please add a description to this, I have no idea what this is trying to do nor why it is removing those tests

@GabrielChenCC
Copy link
Contributor Author

Please add a description to this, I have no idea what this is trying to do nor why it is removing those tests

Hi Massimiliano, I updated my description. I am sorry for my bad.

Copy link
Collaborator

@pieqq pieqq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rarely see jobs being removed, but I agree with your analysis: these manual jobs have been superseded by automated jobs, so there is no point in running them!

I left a few remarks inline.

Also, I think it would be wise if other QA engineers review these changes. I will add them.

The mobilebroadband-cert-automated test case is only
part of the wwan test plan. We hope to replace
mobilebroadband-cert-automated with wwan-automated.

The relevant test plans we currently use include:
client-cert-desktop-22-04-automated
client-cert-desktop-20-04-automated
client-cert-desktop-18-04-automated

I have replaced mobilebroadband-cert-automated in them
with wwan-automated, and added after-sudpend-wwan-
automated to ensure that wwan is tested before and after
suspend.

In addition, for mobilebroadband-cert-manual in the Manual
test plan, when the SIM is inserted into the prototype in the
desktop, the system should automatically recognize wwan and
SIM card, and we do not need to perform manual testing. So I
deleted mobilebroadband-cert-manual in the relevant test
plans. At the same time, since wwan-automated already
contains wwan-manual cases, I deleted the wwan-manual in
these test plans:
client-cert-desktop-22-04-manual
client-cert-desktop-20-04-manual
client-cert-desktop-18-04-manual
@GabrielChenCC
Copy link
Contributor Author

Following Pierre's suggestion, I adjusted the relevant commits. Please review again. Thanks

Copy link
Collaborator

@zongminl zongminl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some changes would have to be done in wwan/scan-networks-manual, wwan/gsm-connection-interrupted-manual and their corresponding after-suspend jobs, for the reason they still depend on the removed job wwan/check-sim-present-manual

@codecov
Copy link

codecov bot commented Dec 12, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (219a9de) 35.25% compared to head (82bd889) 35.66%.
Report is 50 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #821      +/-   ##
==========================================
+ Coverage   35.25%   35.66%   +0.40%     
==========================================
  Files         302      302              
  Lines       34173    34233      +60     
  Branches     5907     5915       +8     
==========================================
+ Hits        12048    12208     +160     
+ Misses      21559    21459     -100     
  Partials      566      566              
Flag Coverage Δ
provider-base 5.29% <ø> (+1.12%) ⬆️
provider-certification-client 57.14% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pieqq
Copy link
Collaborator

pieqq commented Dec 22, 2023

/canonical/self-hosted-runners/run-workflows 82bd889

@pieqq pieqq changed the title Adjust the wwan, watchdog and alsa-loopback test cases. Adjust the wwan, watchdog and alsa-loopback test cases. (BugFix) Dec 22, 2023
@pieqq pieqq merged commit b48a2a2 into canonical:main Dec 22, 2023
LiaoU3 pushed a commit to LiaoU3/checkbox that referenced this pull request Jan 9, 2024
…onical#821)

* The commit is for the alsa-loopback test case delete.

Actually, the alsa-loopback-automated test case does the same
thing with alsa-loopback. When we test iot project machines,
the alsa-loopback-automated needs to be tested. I guess we
could do the alsa-loopback-automated only in the futuer.

* This commit to modify the watchdog test cases.

The watchdog-automated's test items have included
watchdog-maunal,  so I remove the watchdog-manual
test case.

* Modify the wwan related test cases.

The mobilebroadband-cert-automated test case is only
part of the wwan test plan. We hope to replace
mobilebroadband-cert-automated with wwan-automated.

The relevant test plans we currently use include:
client-cert-desktop-22-04-automated
client-cert-desktop-20-04-automated
client-cert-desktop-18-04-automated

I have replaced mobilebroadband-cert-automated in them
with wwan-automated, and added after-sudpend-wwan-
automated to ensure that wwan is tested before and after
suspend.

In addition, for mobilebroadband-cert-manual in the Manual
test plan, when the SIM is inserted into the prototype in the
desktop, the system should automatically recognize wwan and
SIM card, and we do not need to perform manual testing. So I
deleted mobilebroadband-cert-manual in the relevant test
plans. At the same time, since wwan-automated already
contains wwan-manual cases, I deleted the wwan-manual in
these test plans:
client-cert-desktop-22-04-manual
client-cert-desktop-20-04-manual
client-cert-desktop-18-04-manual

* Revert wwan test-plan.pxu. Keep the necessary dependence
of test cases.
LiaoU3 pushed a commit to LiaoU3/checkbox that referenced this pull request Mar 20, 2024
…onical#821)

* The commit is for the alsa-loopback test case delete.

Actually, the alsa-loopback-automated test case does the same
thing with alsa-loopback. When we test iot project machines,
the alsa-loopback-automated needs to be tested. I guess we
could do the alsa-loopback-automated only in the futuer.

* This commit to modify the watchdog test cases.

The watchdog-automated's test items have included
watchdog-maunal,  so I remove the watchdog-manual
test case.

* Modify the wwan related test cases.

The mobilebroadband-cert-automated test case is only
part of the wwan test plan. We hope to replace
mobilebroadband-cert-automated with wwan-automated.

The relevant test plans we currently use include:
client-cert-desktop-22-04-automated
client-cert-desktop-20-04-automated
client-cert-desktop-18-04-automated

I have replaced mobilebroadband-cert-automated in them
with wwan-automated, and added after-sudpend-wwan-
automated to ensure that wwan is tested before and after
suspend.

In addition, for mobilebroadband-cert-manual in the Manual
test plan, when the SIM is inserted into the prototype in the
desktop, the system should automatically recognize wwan and
SIM card, and we do not need to perform manual testing. So I
deleted mobilebroadband-cert-manual in the relevant test
plans. At the same time, since wwan-automated already
contains wwan-manual cases, I deleted the wwan-manual in
these test plans:
client-cert-desktop-22-04-manual
client-cert-desktop-20-04-manual
client-cert-desktop-18-04-manual

* Revert wwan test-plan.pxu. Keep the necessary dependence
of test cases.
binli pushed a commit to binli/checkbox that referenced this pull request Mar 22, 2024
…onical#821)

* The commit is for the alsa-loopback test case delete.

Actually, the alsa-loopback-automated test case does the same
thing with alsa-loopback. When we test iot project machines,
the alsa-loopback-automated needs to be tested. I guess we
could do the alsa-loopback-automated only in the futuer.

* This commit to modify the watchdog test cases.

The watchdog-automated's test items have included
watchdog-maunal,  so I remove the watchdog-manual
test case.

* Modify the wwan related test cases.

The mobilebroadband-cert-automated test case is only
part of the wwan test plan. We hope to replace
mobilebroadband-cert-automated with wwan-automated.

The relevant test plans we currently use include:
client-cert-desktop-22-04-automated
client-cert-desktop-20-04-automated
client-cert-desktop-18-04-automated

I have replaced mobilebroadband-cert-automated in them
with wwan-automated, and added after-sudpend-wwan-
automated to ensure that wwan is tested before and after
suspend.

In addition, for mobilebroadband-cert-manual in the Manual
test plan, when the SIM is inserted into the prototype in the
desktop, the system should automatically recognize wwan and
SIM card, and we do not need to perform manual testing. So I
deleted mobilebroadband-cert-manual in the relevant test
plans. At the same time, since wwan-automated already
contains wwan-manual cases, I deleted the wwan-manual in
these test plans:
client-cert-desktop-22-04-manual
client-cert-desktop-20-04-manual
client-cert-desktop-18-04-manual

* Revert wwan test-plan.pxu. Keep the necessary dependence
of test cases.
pieqq added a commit that referenced this pull request Sep 23, 2024
Following QA changes in #821¹ regarding the replacement of
mobilebroadband-cert-.* nested part for their wwan-.* counterparts, the
same is applied for:

- client-cert-desktop-24-04 (it was missed in #821 due to bad timing)
- client-cert-odm-desktop-2?-04 (to match what's being tested by the QA
team)

client-cert-odm-desktop-18-04 was not included due to the fact that
Ubuntu 18.04 has seen EOL and therefore its related test plan is not
actively maintained anymore.

¹ #821
pieqq added a commit that referenced this pull request Sep 24, 2024
Replace mobilebroadband-cert-automated with wwan-automated nested part,
as the wwan-automated test plan includes better coverage for
WWAN-related tests.

Add after-suspend-wwan-automated counterpart to make sure there are no
issues with WWAN after resuming from suspend.

This is a follow-up to work done by QA in the certification-client
desktop test plans (see #821)

Fix CER-2738
pieqq added a commit that referenced this pull request Sep 24, 2024
…(bugfix) (#1496)

* Switch from mobilebroadband to wwan nested parts for cert test plans

Following QA changes in #821¹ regarding the replacement of
mobilebroadband-cert-.* nested part for their wwan-.* counterparts, the
same is applied for:

- client-cert-desktop-24-04 (it was missed in #821 due to bad timing)
- client-cert-odm-desktop-2?-04 (to match what's being tested by the QA
team)

client-cert-odm-desktop-18-04 was not included due to the fact that
Ubuntu 18.04 has seen EOL and therefore its related test plan is not
actively maintained anymore.

¹ #821

* Add after-suspend-wwan-manual for homogeneity

* Remove superseded jobs from [after-suspend-]wwan-manual nested parts

wwan/detect-manual and wwan/check-sim-present-manual have an automated
version available. wwan/gsm-connection-interrupted-manual depends on the
manual versions of the detection and SIM verification jobs, and is
therefore removed as well, as we have an automated connection job.
pieqq added a commit that referenced this pull request Oct 1, 2024
Replace mobilebroadband-cert-automated with wwan-automated nested part,
as the wwan-automated test plan includes better coverage for
WWAN-related tests.

Add after-suspend-wwan-automated counterpart to make sure there are no
issues with WWAN after resuming from suspend.

This is a follow-up to work done by QA in the certification-client
desktop test plans (see #821)

Fix CER-2738
pieqq added a commit that referenced this pull request Oct 2, 2024
Use wwan-automated nested part in SRU test plan

Replace mobilebroadband-cert-automated with wwan-automated nested part,
as the wwan-automated test plan includes better coverage for
WWAN-related tests.

Add after-suspend-wwan-automated counterpart to make sure there are no
issues with WWAN after resuming from suspend.

This is a follow-up to work done by QA in the certification-client
desktop test plans (see #821)

Fix CER-2738
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants