[TestNG] Make "Display disabled tests in report" configurable

I'm submitting a ...

bug report
feature request
support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

Disabled tests (@Test(enabled = false)) appear as unknown in the report.

What is the expected behavior?

Make it configurable so there would be an option to allow disabled tests to appear in report or not.

What is the motivation / use case for changing the behavior?

We have a test suite which supports the same application with different features in multiple countries. Some countries do not have features that other countries have thus we're using Annotation Transformer to disable those tests for those countries.

It would be nice to not see those disabled tests in the report as they clutter the report and create a distraction.

Please tell us about your environment:

Allure version	2.8
Test framework	testng@6.14.3
Allure integration	-
Generate report using	allure-gradle@2.5

We are not updating to the newest versions because of that behavior and we would love to avoid forking and changing this behavior in forked Allure projects.

Not relevant to me anymore.

@MindaugasMateika

@MindaugasMateika I have faced the same issue as yours. Have you found another solution which does not involve code changes in allure? Thanks

@AutomatedOwl

@AutomatedOwl I've started using TestNG's method selector. Either way, I was using custom annotation with parameters to tell which tests should not be run during runtime. Something like this:

@Skip(shouldBeSkipped = someCondition)
@Test
fun test() {
...
}

Before I was using annotation transformer to add enabled = false to @Test annotation based on @Skip annotation's parameters. But now I exclude tests which should be skipped from the list of runnable tests.

This is my method selector:

class MethodSelector : IMethodSelector {
    override fun includeMethod(context: IMethodSelectorContext?, method: ITestNGMethod, isTestMethod: Boolean): Boolean {
        var shouldRunTest = true
        val testMethod = method.constructorOrMethod.method

        if (isTestMethod && testMethod.isAnnotationPresent(Skip::class.java)) {
            val skipParams = testMethod.getAnnotation(Skip::class.java)

            if (skipParams.shouldBeSkipped) {
                context!!.isStopped = true
                shouldRunTest = false
            }
        }
        return shouldRunTest
    }

    override fun setTestMethods(testMethods: List<ITestNGMethod>) {}
}

You can achieve the same thing with method interceptor but it does require TestNG 7.0-beta3 to work properly I think. Previous versions have a bug.

Hope this helps. :-)

is there a way to reopen this? i'd also like this behavior to avoid seeing skipped tests in the results without needing to add custom annotations.

sharing some details on why this is relevant in my environment and possibly in others.

we use the

@Test(enabled=false)

or

@Test(enabled = runTests)

to support a few scenarios

the test is an example test on how to do something, which shouldn't be run in test jobs but can easily be run by toggling the enabled setting manually in a class file
the tests in a class are only meant to be run manually after some environmental configuration has been completed. once it has, the user can toggle runTests and run the entire class in an IDE

before upgrading allure, these tests never appeared in any report. now they do - the purple in the image reflects 11 tests with enabled=false. that throws off the pass percentage number, which was 100% passing before upgrading allure.

if support is added for ignoring these tests completely in test results, we'd switch to that approach. if that support is not added, we'd likely switch all occurrences of the above patterns to

// @Test

we can do that, it's just not ideal, and i suspect we aren't the only ones that liked to set enabled = false and know they'd be ignored in allure reports.

I've reopened the issue. I hope someone will take a look at this and fix it :-)

@MindaugasMateika

@MindaugasMateika thanks!

@baev

@baev Is there any solution to the above mentioned scenario?

@MindaugasMateika

any plans to fix this one? at the moment the workarounds i've seen discussed in this thread are

commenting out @Test rather than doing @Test(enabled = false)
creating a custom annotation, like @Skip, as @MindaugasMateika did

i'd still prefer to use enabled = false, which used to work fine with allure, but if there are no plans to ever support this i'll likely switch everything over to // @Test

is this issue still open? no solutions yet?

Any chance of having this issue prioritized and implemented?
Unfortunately, it is affecting my reporting structure and therefore client is not willing to upgrade Allure version.
Note: the recommended approach here is not approved as good permanent solution.

I'd be happy to contribute and help implementing in case support could be provided :)

I'd be happy to contribute and help implementing in case support could be provided :)

👍 I'm ok with that. A few thoughts:

Introduce a new configuration POJO that goes to AllureTestNg class as a constructor parameter
For default constructor use io.qameta.allure.util.PropertiesUtils#loadAllureProperties to load properties and required options
New properties should be under allure.testng. namespace.

@baev

I'd be happy to contribute and help implementing in case support could be provided :)

👍 I'm ok with that. A few thoughts:

Introduce a new configuration POJO that goes to AllureTestNg class as a constructor parameter

For default constructor use io.qameta.allure.util.PropertiesUtils#loadAllureProperties to load properties and required options

New properties should be under allure.testng. namespace.

Cool! I will take a look at it.
@baev by any chance, do we have similar configurable implementation in place to be used as an example to speed-up things?

Thanks, mate.

@baev

@baev by any chance, do we have similar configurable implementation in place to be used as an example to speed-up things?

not sure. Simply add desired field to AllureTestNG, create a AllureTestNgConfig POJO with the same field, add a constructor to AllureTestNG that accepts AllureTestNgConfig, update default constructor so it load allure properties and init AllureTestNgConfig from property values and passes config to other constructor. That's should be it. And don't forget a test for that

@baev

@baev there was this #49 issue years ago fixed to display disabled tests as skipped.
Since we're changing this logic to make it configurable, is it fine if I refactor existing test by replacing it with this #369 here?

I might also replace Add tests with enabled=false to report as skipped #49.xml by [TestNG] Make "Display disabled tests in report" configurable #369.xml
Refer each other in Github to keep consistent tracking

@baev

@baev MR is ready. Please, review whenever you have some time.
All changes and suggestions are welcome - including conventions, standards, style etc...

feat[testng] make display of disabled tests in report configurable (f…

Navigation Menu

Repository navigation

[TestNG] Make "Display disabled tests in report" configurable #369

I'm submitting a ...

What is the current behavior?

What is the expected behavior?

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[TestNG] Make "Display disabled tests in report" configurable #369

Description

I'm submitting a ...

What is the current behavior?

What is the expected behavior?

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Activity

MindaugasMateika commented on Jun 9, 2019

AutomatedOwl commented on Jul 2, 2019

MindaugasMateika commented on Jul 2, 2019

snowymike commented on Feb 11, 2020

snowymike commented on Feb 12, 2020

MindaugasMateika commented on Feb 12, 2020

snowymike commented on Feb 12, 2020

prashanth56 commented on Apr 3, 2020

snowymike commented on Nov 4, 2021

Dilshat517 commented on Jan 4, 2022

vinipx commented on Sep 26, 2022

baev commented on Sep 27, 2022

vinipx commented on Sep 27, 2022

baev commented on Sep 27, 2022

vinipx commented on Sep 27, 2022

vinipx commented on Sep 27, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions