How to run concurrency unit test?

Unfortunately I don't believe you can definitively prove that your code is thread-safe by using run-time testing. You can throw as many threads as you like against it, and it may/may not pass depending on the scheduling.

Perhaps you should look at some static analysis tools, such as PMD, that can determine how you're using synchronisation and identify usage problems.

I would recommend using MultithreadedTC - Written by the concurrency master himself Bill Pugh (and Nat Ayewah). Quote from their overview:

MultithreadedTC is a framework for testing concurrent applications. It features a metronome that is used to provide fine control over the sequence of activities in multiple threads.

This framework allows you to deterministically test every thread interleaving in separate tests

You can only prove the presence of concurrent bugs, not their absence.

However you can write a specialized test runner that spawns several concurrent thread and then calls your @Test annotated methods.

In .NET, there are tools like TypeMock Racer or Microsoft CHESS that are designed specifically for unit testing concurrency. These tools not only find multithreading bugs like deadlocks, but also give you the set of thread interleaves that reproduce the errors.

I'd imagine there's something similar for the Java world.

Running concurrently can lead to unexpected results. For instance, I just discovered, that while my Test suite with 200 tests pass when executed one by one, it fails for concurrent execution, I dug that and it wasn't a thread safety problem, but a test depending on another one, which is a bad thing and I could solve the problem.

Mycila work on JUnit ConcurrentJunitRunner and ConcurrentSuite is very interesting. The article seems a little outdated compared to the latest GA release, in my examples I will show the updated usage.

Annotating a test class like the following will cause to execute test methods concurrently, with a concurrency level of 6:

import com.mycila.junit.concurrent.ConcurrentJunitRunner;
import com.mycila.junit.concurrent.Concurrency;

@RunWith(ConcurrentJunitRunner.class)
@Concurrency(6)
public final class ATest {
...

You can also run all the test classes concurrently:

import com.mycila.junit.concurrent.ConcurrentSuiteRunner;

@RunWith(ConcurrentSuiteRunner.class)
@Suite.SuiteClasses({ATest.class, ATest2.class, ATest3.class})
public class MySuite {
}

The Maven dependency is:

<dependency>
    <groupId>com.mycila</groupId>
    <artifactId>mycila-junit</artifactId>
    <version>1.4.ga</version>
</dependency> 

I am currently investigating how to run methods multiple times and concurrently with this package. It may be possible already, if anyone has an example let me know, below my homebrewed solution.

@Test
public final void runConcurrentMethod() throws InterruptedException {
    ExecutorService exec = Executors.newFixedThreadPool(16);
    for (int i = 0; i < 10000; i++) {
        exec.execute(new Runnable() {
             @Override
             public void run() {
                 concurrentMethod();
             }
        });
    }
    exec.shutdown();
    exec.awaitTermination(50, TimeUnit.SECONDS);
}

private void concurrentMethod() {
    //do and assert something
}

As others noted, is true that you can never be sure whether a concurrency bug would show up or not, but with ten of thousands, or hundred of thousands executions with a concurrency of, say 16, statistics is on your side.

Try looking at ActiveTestSuite which ships with JUnit. It can concurrently launch multiple JUnit tests:

public static Test suite()
{
    TestSuite suite = new ActiveTestSuite();
    suite.addTestSuite(PostMessageTest.class);
    suite.addTestSuite(PostMessageTest.class);
    suite.addTestSuite(PostMessageTest.class);
    suite.addTestSuite(PostMessageTest.class);
    suite.addTestSuite(PostMessageTest.class);
    return suite;
}

The above will run the same JUnit test class 5 times in paralell. If you wanted variation in your paralell tests, just create a different class.

In your example the postMessage() method is synchronized, so you won't actually see any concurrency effects from within a single VM but you might be able to evaluate the performance of the synchronized version.

You will need to run multiple copies of the test program at the same time in different VMs. You can use

If you can't get your test framework to do it you can launch some VMs your self. The process builder stuff is a pain with paths and whatnot, but here is the general sketch:

Process running[] = new Process[5];
for (int i = 0; i < 5; i++) {
 ProcessBuilder b = new ProcessBuilder("java -cp " + getCP() + " MyTestRunner");
 running[i] = b.start();
}

for(int i = 0; i < 5; i++) {
 running[i].waitFor();
}

I usually do something like this for simple threaded tests, like others have posted, testing is not a proof of correctness, but it usually shakes out silly bugs in practice. It helps to test for a long time under a variety of different conditions -- sometimes concurrency bugs take a while to manifest in a test.

public void testMesageBoard() {
 final MessageBoard b = new MessageBoard();

 int n = 5;
 Thread T[] = new Thread[n];
 for (int i = 0; i < n; i++) {
  T[i] = new Thread(new Runnable() {
   public void run() {
    for (int j = 0; j < maxIterations; j++) {
      Thread.sleep( random.nextInt(50) );
      b.postMessage(generateMessage(j));
      verifyContent(j); // put some assertions here
    }
   }
  });

  PerfTimer.start();
  for (Thread t : T) {
   t.start();
  }

  for (Thread t : T) {
   t.join();
  }
  PerfTimer.stop();
  log("took: " + PerfTimer.elapsed());
 }
}**strong text**

Testing concurrency bugs is impossible; you don't just have to validate input/output pairs, but you have to validate state in situations which may or may not occur during your tests. Unfortunately JUnit is not equipped to do this.

You can use the tempus-fugit library to run test methods in parallel and multiple times to simulate a load testing type environment. Although a previous comment points out that the post method is syncrhonised and so protected, associated members or methods may be involved which themselves are not protected, so its possible that a load / soak type test could catch these. I'd suggest you setup a fairly coarse grained / end-to-end like test to give you the best chance of catching any loop holes.

See JUnit integration section of the documentation.

BTW, I a developer on said project :)

TestNG has support for concurrency testing in Java. This article describes how it can be used and there is docs on the testng site.

Not sure if you can make the same test run at the same time though

The best approach here is to use system tests to blast your code with requests to see if it falls over. Then use the unit test to check the logical correctness. The way i would approach this is create a proxy for the asynchronous call and have it made synchronously under test.

However, if you wanted to do this at a level other than a full install and complete environment you could do this in junit by creating your object in a separate thread, then create lots of threads that fire requests to your object and block the main thread until it completes. This approach can make tests fail intermittently if you don't get it right.

You can also try HavaRunner. It runs tests in parallel by default.