Download as PDF, PPTX
Uploaded byWojciech Langiewicz
Mutation testing in Java
The document discusses mutation testing, a technique to evaluate test quality by automatically introducing faults ("mutants") into code and checking if tests can detect the faults. It introduces key concepts like mutants, mutation operators, and tools like PIT that make mutation testing practical. While previously limited by performance, modern tools like PIT can analyze large codebases in reasonable time. Mutation testing tests tests more rigorously than code coverage alone by checking tests can detect small code changes.
More Related Content
Similar to Mutation testing in Java
Mutation testing in Java
- 1. https://github.com/wlk/mutation-testing-demo Mutation Testing inJava Wojciech Langiewicz @ 33rd Degree 4 Charity 1
- 2. https://github.com/wlk/mutation-testing-demo How do youmake sure your tests are any good? 2
- 3. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo How do youmake sure your tests are any good? 3 ● Code Review ● TDD ● Code coverage ○ What does it really measure? ○ What does it prove? ○ Line/Statement/Branch coverage ○ Tests without assertions
- 4. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutation Testing 4 ● Techniqueto measure quality of tests ● Injects a fault into a system and uses our tests to find it ● Proposed in 1971 by Richard Lipton ● Competent programmer hypothesis ● Few new concepts: ○ Mutants ○ Mutation Operators ○ Generating Mutants ○ Mutation Coverage
- 5. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutation Testing -problems 5 ● Forgotten for many years (only some academic work) ○ Performance problems ○ Lack of tooling ● Performance problem: ○ Test suite takes 5 minutes to run ○ 500 classes, 10 tests per class, testing each class takes 0.6s ○ Naive: 10 mutants per class gives 10 * 5 * 500 ~ 70 hours ○ Fast: 10 * 0.6 * 500 = 50 minutes
- 6. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo How It Works Originalcode Mutant Run Tests 6 Tests Pass - mutation wasn’t found Tests Fail - mutation was found
- 7. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo How It Works if( i >= 0 ) { return "foo"; } else { return "bar"; } if ( i > 0 ) { return "foo"; } else { return "bar"; } Original code Mutant Run Tests 7 Tests Pass - mutation wasn’t found Tests Fail - mutation was found
- 8.
- 9. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutator: Conditionals BoundaryMutator 9 < <= > >= <= < >= > if (a < b) { // do something } if (a <= b) { // do something }
- 10. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutator: Negate ConditionalsMutator 10 == != <= >= < > != ++ > < >= <= if (a == b) { // do something } if (a != b) { // do something }
- 11. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutator: Void MethodCall 11 public void someVoidMethod(int i) { // does something } public int foo() { int i = 5; doSomething(i); return i; } public void someVoidMethod(int i) { // does something } public int foo() { int i = 5; // don’t do anything return i; }
- 12. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutator: Constructor Call 12 publicObject foo() { Object o = new Object(); return o; } public Object foo() { Object o = null; return o; }
- 13. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutators: Many More ●Replace constants ● Replace return values to defaults ● And many others 13
- 14. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Tooling 14 ● PIT -http://pitest.org/ ● Ruby: Mutant: https://github.com/mbj/mutant ● Popular in communities where testing (TDD) is already popular
- 15.
- 16. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo PIT Features 16 ● Bytecodemodifications (to avoid recompilation) ● Integrates easily with: ○ Java 6, 7, 8 ○ JUnit, TestNG ○ Eclipse, Intellij ○ Gradle, Maven, Ant ○ Sonar, Jenkins ○ Mocking frameworks ● For each mutation, it tries to minimize number of tests to run ● Allows to choose which mutators we want to use ● Doesn’t work with Scala ● Doesn’t store mutated code ● Generates simple HTML report, or XML report for other tools
- 17. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Performance Nowadays: 17 ● Notreally a problem on CI server when using modern tools (PIT) ● PIT can analyze only changed code (looking at your SCM) ● Practical tests: ○ Apache commons-math: ○ 177k lines of code ○ 109k lines of tests ○ 8 minutes to test ○ PIT takes 1:15h with 4 threads
- 18.
- 19. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Problems With MutationTesting 19 Mutants can be dangerous: if(false){ Runtime.getRuntime().exec("rm -rf /"); } Equivalent Mutants: int i = 2; if ( i >= 1 ) { return "foo"; } // is equivalent to int i = 2; if ( i > 1 ) { return "foo"; } Defensive programming: if(i > 0){ throw new IllegalArgumentException( “argument i must be positive” ); } return Math.sqrt(i);
- 20. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Summary 20 ● Mutation testingtests your tests ● Code coverage gives you false sense of security ● PIT is extremely easy to introduce into Java project
- 21.
- 22.