TestNG 7.5 cannot add attachments to test upon failure

Describe the bug
When using TestNG 7.5 with allure 2.17.2 & 2.17.3 trying to add attachments to tests once they failed does not work and generates an error.

To Reproduce
Steps to reproduce the behavior:

Have a project setup with selenium, testng and allure
Create a test listener for failure and save screenshot
Run a test that fails
See error

Expected behavior
The attachment should save and be added to the allure report - this can be done by going back to TestNG version 7.4.0

Screenshots

Desktop (please complete the following information):

OS: Windows 10
Browser: chrome
Version: 97

Additional context

testng-team/testng#2717

Actually looks like allure has a lot of things broken by TestNG 7.5

Example test listener class

package TestListeners;

import ProjectWide.BaseClass;
import ProjectWide.BaseClassHeadless;
import io.qameta.allure.Attachment;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class TestListener extends BaseClass implements ITestListener {

    private static String getTestMethodName(ITestResult iTestResult){
        return iTestResult.getMethod().getConstructorOrMethod().getName();
    }

    @Attachment(value = "Page Screenshot", type = "image/png")
    public byte[] saveScreenshot(WebDriver driver){
        return ((TakesScreenshot)driver).getScreenshotAs(OutputType.BYTES);
    }

    @Attachment(value = "{0}", type = "test/plain")
    public static String saveTextLog(String message){
        return message;
    }

    @Override
    public void onTestFailure(ITestResult iTestResult){
        //Get driver from Test
        iTestResult.getInstance();
        WebDriver driver;
        try{
            driver = getDriver();
        }
        catch(Exception ClassCastException){
            driver = BaseClassHeadless.getDriver();
        }

        if(driver != null){
            saveScreenshot(driver);
        }

        saveTextLog(getTestMethodName(iTestResult) + " failed");
    }

}

Example test which passed with TestNG 7.40 but not 7.5

package Tests;

import ProjectWide.BaseClass;
import TestListeners.TestListener;
import io.qameta.allure.Epic;
import io.qameta.allure.Feature;
import org.openqa.selenium.By;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners({TestListener.class})

@Epic("Utility Tests")
@Feature("Allure Failure Test")
public class AllureScreenshotTest extends BaseClass {

    @Test(description = "Generate a failure so that a screenshot appears on allure")
    public void testMethod(){
        driver.get().get("https://www.google.co.uk/");
        driver.get().findElement(By.id("nothing"));
    }

}

I have same problem with Allure 2.17.3 and TestNG 7.5.
Allure saves screenshot into allure-results folder, but doesn't attach it to generated report cause test-result.json file has empty attachment field.
Works fine with TestNG 7.4.0.

Have the same problem when was updated to TestNG 7.5,
i was debugging there is prepareAttachment method in AllureLifecycle, and after switch to TestNG 7.5 thereadContext.getCurrent(); returns Optional.empty

And one more thing, looks like Flaky tests are not marked as Flaky in Allure report

Are there any workarounds ?

@AlexChernookov

@AlexChernookov

I have found no workarounds, I had to downgrade the TestNG version to 7.4.0 to work with allure 2.17.3

For the flaky tests, they are working for me with 7.4.0 just by annotating tests with @flaky like this. MIght be broken using 7.5 only

 @Flaky
    @Severity(SeverityLevel.NORMAL)
    @Test(description = "Login using username and password")
    public void logIntoMyAccount() throws InterruptedException {
    }

@garyc0221

@garyc0221

So the best workaround now is to downgrade TestNg to 7.4.0. Hope that issue with 7.5 will be fixed in next releases

@garyc0221

Hello!

I ran into the same issue. I have came up with a solution which relies on Allure lifecycles instead of TestNG lifecycles. I would consider it better since I want to use an Allure feature and I don't want it to be that heavily dependent on TestNG.

I had to rework my listener so it is implementing Allure's TestLifecycleListener, and I had to register it with SPI, which is just a tiny bit more complicated than putting on annotation with TestNG.

I have created a short, basic and rough example repository based on @garyc0221 example code in the thread to show what it takes to use Allure lifecycle instead.

@biczomate

@biczomate This is amazing thanks, did not know allure had its own test listener. Works flawlessly.

@biczomate

@biczomate Amazing!
But the proper fix would be nice from Allure team.

But the proper fix would be nice from Allure team.

I'd say https://github.com/biczomate/allure-testng7.5-attachment-example is the way to go. Issues with listeners order are the main reason AllureLifecycleListener was created in the first place.

If the only way to get all the execution data is to use the same API that users will use to control tests execution (such as creating failure screenshots etc) and no way to change the ordering is provided, there aren't much we can do with it

Going to close this issue as https://github.com/biczomate/allure-testng7.5-attachment-example is the way to go. With Allure listener, you don't even need to define the listener within each test class, which is nice.

[#1] Registration and login tests

Navigation Menu

Repository navigation

TestNG 7.5 cannot add attachments to test upon failure #707

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TestNG 7.5 cannot add attachments to test upon failure #707

Description

Activity

garyc0221 commented on Jan 17, 2022

garyc0221 commented on Jan 17, 2022

vkhromenkov commented on Mar 1, 2022

AlexChernookov commented on Mar 2, 2022

garyc0221 commented on Mar 2, 2022

AlexChernookov commented on Mar 2, 2022

biczomate commented on Mar 12, 2022

garyc0221 commented on Mar 14, 2022

iRaunchy commented on Mar 30, 2022

baev commented on Mar 30, 2022

garyc0221 commented on Apr 1, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions