Skip to content

TestNg Setup/Before errors should NOT be separate test cases in report #643

Closed

Description

@bgalamb

Describe the bug
Allure generates individual test case from testNG setup/before methods too if they break. Attached project to reproduce:
bug_reports.zip

To Reproduce
Steps to reproduce the behavior:

  1. unzip Attached project
  2. execute mvn clean test
  3. execute cd target and allure serve
  4. Check generated report. (I attached screenshot too)
  5. It contains one additional test "myBeforeClass" which is broken. This confuses testers because 1-the number of tests may be varying in the test class 2-it links a method which may be totally unknown for them (if its located in parent or static class).
  6. 3-Also It does not add too much value because the same stack can be found linked to the "real" test.

Expected behavior
I would NOT expect to see the setup/before hook's excpetion as independent tests in the report.
I suggest deleting the onConfigurationFailure() method from io/qameta/allure/testng/AllureTestNg.java.
Follow steps to verify:
11. Open attached project
12. comment or delete method onConfigurationFailure() in io/qameta/allure/testng/AllureTestNg.java
13. repeat steps 2,3
14. See that in this case only the real broken test is shown in the list. And we can also find the error starting from this.

BTW: the comment in code also indicates that configuration error should not be separate tests

Screenshots
If applicable, add screenshots to help explain your problem.
Screenshot 2021-08-30 at 14 35 30
Screenshot 2021-08-30 at 14 36 56
5b. if uploaded to allureEE it generated strange orphan tests.

Screenshot 2021-08-30 at 14 46 35

==expected images===

Screenshot 2021-08-30 at 14 33 35

Screenshot 2021-08-30 at 14 33 48

Desktop (please complete the following information):

  • OS: [MAC]
  • Browser [chrome]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Activity

  1. lilyushchenko commented on Jan 25, 2023

    @lilyushchenko

    Still actual, kind of the same issue raised here as well #623
    Is there a chance that there will be a fix?

  2. tomkeiev commented on Jun 26, 2023

    @tomkeiev

    seems it's going to be merged soon in #920

  3. added theissue type on Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions