New testing lab for Tapestry Workshop @ SkillsMatter (London, Oct 5th)

I'm once again partnering with SkillsMatter to teach my full Tapestry workshop.

I've just finished up the materials for the new lab that covers testing of Tapestry components and applications. I'm quite pleased with how it came out. I'm really looking forward to this training. According to SkillsMatter, there's still room in the class for a couple of more students ... if you want to get started with Tapestry, or you want to get your Tapestry Ninja Skills going, this is the way to do it, fast!

The class will be taught at SkillsMatter's offices in London, from October 5th through the 8th.

TestNG and Selenium

I love working on client projects, because those help me really understand how Tapestry gets used, and the problems people are running in to. On site training is another good way to see where the theory meets (or misses) the reality.

In any case, I'm working for a couple of clients right now for whom testing is, rightfully, quite important. My normal approach is to write unit tests to test specific error cases (or other unusual cases), and then write integration tests to run through main use cases. I consider this a balanced approach, that recognizes that a lot of what Tapestry does is integration.

One of the reasons I like TestNG is that it seamlessly spans from unit tests to integration tests. All of Tapestry's internal tests (about 1500 individual tests) are written using TestNG, and Tapestry includes a base test case class for working with Selenium: AbstractIntegrationTestSuite. This class does some useful things:

• Launches your application using Jetty
• Launches a SeleniumServer (which drives a web browser that can exercise your application)
• Creates an instance of the Selenium client
• Implements all the methods of Selenium, redirecting each to the Selenium instance
• Adds additional error reporting around any Selenium client calls that fail

These are all useful things, but the class has gotten a little long in the tooth ... it has a couple of critical short-comings:

• It runs your application using Jetty 5 (bundled with SeleniumServer)
• It starts and stops the stack (Selenium, SeleniumServer, Jetty) around each class

For my current client, a couple of resources require JNDI, and so I'm using Jetty 7 to run the application (at least in development, and possibly in deployment as well). Fortunately, Jetty 5 uses the old org.mortbay.jetty packages, and Jetty 7 uses the new org.eclipse.jetty packages, so both versions of the server can co-exist within the same application.

The larger problem is that I didn't want a single titanic test case for my entire application; I wanted to break it up in other ways, by Tapestry page initially.

I could create additional subclasses of AbstractIntegrationTestSuite, but then the tests will spend a huge amount of time starting and stopping Firefox and friends. I really want that stuff to start just once.

What I've done is a bit of refactoring, by leveraging some features of TestNG that I hadn't previously used.

The part of AbstractIntegrationTestSuite responsible for starting and stopping the stack is broken out into its own class. This new class, SeleniumLauncher, is responsible for starting and stopping the stack around an entire TestNG test. In the TestNG terminology, a suite contains multiple tests, and a test contains test cases (found in individual classes, within scanned packages). The test case contains test and configuration methods.

Here's what I've come up with:

package com.myclient.itest;

import org.apache.tapestry5.test.ErrorReportingCommandProcessor;
import org.eclipse.jetty.server.Server;
import org.openqa.selenium.server.RemoteControlConfiguration;
import org.openqa.selenium.server.SeleniumServer;
import org.testng.ITestContext;
import org.testng.annotations.AfterTest;
import org.testng.annotations.BeforeTest;

import com.myclient.RunJetty;
import com.thoughtworks.selenium.CommandProcessor;
import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.HttpCommandProcessor;
import com.thoughtworks.selenium.Selenium;

public class SeleniumLauncher {

  public static final String SELENIUM_KEY = "myclient.selenium";

  public static final String BASE_URL_KEY = "myclient.base-url";

  public static final int JETTY_PORT = 9999;

  public static final String BROWSER_COMMAND = "*firefox";

  private Selenium selenium;

  private Server jettyServer;

  private SeleniumServer seleniumServer;

  /** Starts the SeleniumServer, the application, and the Selenium instance. */
  @BeforeTest(alwaysRun = true)
  public void setup(ITestContext context) throws Exception {

    jettyServer = RunJetty.start(JETTY_PORT);

    seleniumServer = new SeleniumServer();

    seleniumServer.start();

    String baseURL = String.format("http://localhost:%d/", JETTY_PORT);

    CommandProcessor cp = new HttpCommandProcessor("localhost",
        RemoteControlConfiguration.DEFAULT_PORT, BROWSER_COMMAND,
        baseURL);

    selenium = new DefaultSelenium(new ErrorReportingCommandProcessor(cp));

    selenium.start();

    context.setAttribute(SELENIUM_KEY, selenium);
    context.setAttribute(BASE_URL_KEY, baseURL);
  }

  /** Shuts everything down. */
  @AfterTest(alwaysRun = true)
  public void cleanup() throws Exception {
    if (selenium != null) {
      selenium.stop();
      selenium = null;
    }

    if (seleniumServer != null) {
      seleniumServer.stop();
      seleniumServer = null;
    }

    if (jettyServer != null) {
      jettyServer.stop();
      jettyServer = null;
    }
  }
}

Notice that we're using the @BeforeTest and @AfterTest annotations; that means any number of tests cases can execute using the same stack. The stack is only started once.

Also, notice how we're using the ITestContext to communicate information to the tests in the form of attributes. TestNG has a built in form of dependency injection; any method that needs the ITestContext can get it just by declaring a parameter of that type.

AbstractIntegrationTestSuite2 is the new base class for writing integration tests:

package com.myclient.itest;

import java.lang.reflect.Method;

import org.apache.tapestry5.test.AbstractIntegrationTestSuite;
import org.apache.tapestry5.test.RandomDataSource;
import org.testng.Assert;
import org.testng.ITestContext;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;

import com.mchange.util.AssertException;
import com.thoughtworks.selenium.Selenium;

public abstract class AbstractIntegrationTestSuite2 extends Assert implements
    Selenium {

  public static final String BROWSERBOT = "selenium.browserbot.getCurrentWindow()";

  public static final String SUBMIT = "//input[@type='submit']";

  /**
   * 15 seconds
   */
  public static final String PAGE_LOAD_TIMEOUT = "15000";

  private Selenium selenium;

  private String baseURL;

  protected String getBaseURL() {
    return baseURL;
  }

  @BeforeClass
  public void setup(ITestContext context) {
    selenium = (Selenium) context
        .getAttribute(SeleniumLauncher.SELENIUM_KEY);
    baseURL = (String) context.getAttribute(SeleniumLauncher.BASE_URL_KEY);
  }

  @AfterClass
  public void cleanup() {
    selenium = null;
    baseURL = null;
  }

  @BeforeMethod
  public void indicateTestMethodName(Method testMethod) {
    selenium.setContext(String.format("Running %s: %s", testMethod
        .getDeclaringClass().getSimpleName(), testMethod.getName()
        .replace("_", " ")));
  }

  /* Start of delegate methods */
  public void addCustomRequestHeader(String key, String value) {
    selenium.addCustomRequestHeader(key, value);
  }

  ...
}

Inside the @BeforeClass-annotated method, we receive the test context and extract the selenium instance and base URL put in there by SeleniumLauncher.

The last piece of the puzzle is the code that launches Jetty. Normally, I test my web applications using the Eclipse run-jetty-run plugin, but RJR doesn't support the "Jetty Plus" functionality, including JNDI. Thus I've created an application to run Jetty embedded:

package com.myclient;

import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.webapp.WebAppContext;

public class RunJetty {

  public static void main(String[] args) throws Exception {

    start().join();
  }

  public static Server start() throws Exception {
    return start(8080);
  }

  public static Server start(int port) throws Exception {
    Server server = new Server(port);

    WebAppContext webapp = new WebAppContext();
    webapp.setContextPath("/");
    webapp.setWar("src/main/webapp");

    // Note: Need jetty-plus and jetty-jndi on the classpath; otherwise
    // jetty-web.xml (where datasources are configured) will not be
    // read.

    server.setHandler(webapp);

    server.start();

    return server;
  }
}

This is all looking great. I expect to move this code into Tapestry 5.2 pretty soon. What I'm puzzling on is a couple of extra ideas:

• Better flexibility on starting up Jetty so that you can hook your own custom Jetty server configuration in.
• Ability to run multiple browser agents, so that a single test suite can execute against Internet Explorer, Firefox, Safari, etc. In many cases, the same test method might be invoked multiple times, to test against different browsers.

Anyway, this is just one of a number of very cool ideas I expect to roll into Tapestry 5.2 in the near future.

1819

Nope, that's not a date. 1819 is the current number of individual tests run every time a change is committed to Tapestry. Over 180 of those are Selenium-based integration tests. That's a lot of testing!

Selenium tests just started hanging? Don't Panic!

So, here I am, in the middle of some intense debugging related to the dreaded combination of threads, class loaders and deadlocks and suddenly ... my integration tests no longer run!

Ultimately, my changes were very modest (a little bit of extra synchronization against the context class loader), so what gives?

Well, if your tests are like mine, and run against Firefox, your issue might be that Selenium is unable to start up Firefox if an upgrade has occurred, especially if the Firefox upgrade is not compatible with your plugins.

The solution? Start Firefox manually, to get through the dialogs it presents. Then rerun your tests.

Ajax and Selenium: waitForCondition()

Selenium is a very useful tool but it can be very, very obtuse.

One challenge is dealing with Ajax; you might click on a button, but without a full page refresh, it's hard to know when to look for expected changes via Ajax and DHTML.

In the past, my test suites had short sleeps, a few hundred milliseconds. This makes them fail sporadically ... every once and a while on my MacBook Pro I'm doing so much other stuff while the tests run that the timing goes screwy.

You're then left with a difficult choice: sleep too short and the tests may fail. Sleep too long and your tests will always be slow.

Fortunately, there's a third option: Selenium's waitForCondition call. Of course, their documentation is worthless.

What it is supposed to do is evaluate a JavaScript snippet repeatedly, until the snippet returns true. However, it's tricky to get right. Like much in JavaScript, it's about context.

In my case, I wanted to wait for a client-side popup <div> to appear:

        type("amount", "abc");
        type("quantity", "abc");

        click(SUBMIT);

        waitForCondition("document.getElementById('amount:errorpopup')", "5000");

        assertText("//div[@id='amount:errorpopup']/span", "You must provide a numeric value for Amount.");
        assertText("//div[@id='quantity:errorpopup']/span", "Provide quantity as a number.");

JavaScript treats null as false, and getElementById() returns null if an element with the id does not exist.

I'm making the assumption that once one of two <div> elements appears, they both will. I then use some XPath to get the text inside the <span> inside each <div>, to make sure the correct message was displayed to the user.

But this code doesn't work.

The problem is that document isn't what you'd expect; I'm guessing that it's some other frame inside the browser (Selenium's UI and code executes in one frame, which runs the actual application inside the second frame).

The solution took some research and the sacrifice of a few small furry animals to obtain:

        waitForCondition("selenium.browserbot.getCurrentWindow().document.getElementById('amount:errorpopup')", "5000");

That works, and it works much faster than adding a Thread.sleep() in the middle of my code.