Sometimes, The Tests Suck

public class SampleSmallClassTest {

    @Test
    public void testStuff() {
        // The boss says we should write tests.
        // Doesn't seem that hard...

        assertTrue(true);
    }

}

So We Use Code Coverage

$ gradle test --tests *testStuff jacocoTestReport

It Makes Us Write Better Tests

@Test
public void testStuffWithCoverage() {
    // Need to get that coverage thing high...

    for(int a = -5; a <= 5; a ++) {
        for (int b = -5; b <= 5; b++) {
            SampleSmallClass.doSomeMath(a, b);
        }
    }
}

Which Give Better Metrics

$ gradle test --tests *testStuffWithCoverage jacocoTestReport

We Can Do Better Than That

1. Run the tests
2. Mess with the code
3. Run the tests again
4. If tests start failing, then the change is covered
5. If no test start failing, the change is not covered

The Idea Is Not New At All

• 1971: Richard Lipton suggests Mutation Analysis
• 1979: Timothy Budd implements the first FORTRAN framework
•  ···
• 2000: Jester ‐ the JUnit test tester
•  ···
•  ···

We Need Automated Tools

• Shouldn't take forever to run
• Should be easy to integrate with existing code
• Should support latest versions of stuff
• Shouldn't be abandoned

A Fancy Table

Let's Check It Out!

$ gradle test --tests *testStuffWithCoverage pitest

Some Very Advanced Business Logic

public static int doSomeMath(int a, int b) {
    long result = b;

    result += a;
    result -= b;
    result -= a;

    return (int) result;
}

Pit Has Messed With Your Code (1)

public static int doSomeMath(int a, int b) {
    long result = b;

    result -= a;
    result -= b;
    result -= a;

    return (int) result;
}

Pit Has Messed With Your Code (2)

public static int doSomeMath(int a, int b) {
    long result = b;

    result += a;
    result -= b;
    result += a;

    return (int) result;
}

Pit Has Messed With Your Code (3)

public static int doSomeMath(int a, int b) {
    long result = b;

    result += a;
    result -= b;
    result -= a;

    return ((int) result == 0 ? 1 : 0);
}

So We Need To Assert Something

@Test
public void testProperly() {
    final int expected = 0;

    for(int a = -5; a <= 5; a ++) {
        for(int b = -5; b <= 5; b ++) {
            assertEquals(expected, doSomeMath(a, b));
        }
    }
}

The Test Seems Fine Now

$ gradle test --tests *testProperly pitest

The Ultimate Guide To Mutation Analysis

1. Pick classes to mutate
2. Mutate the chosen classes
3. Run the tests against the mutants

Picking Mutation Targets

1. Compile the sources to bytecode
2. Run selected tests
3. Remember which instructions are executed by which tests

Building Up Your Team Of Mutants

What we want to have:

• Stability: a re-run produces identical results
• Recall: all the possible errors are made
• Precision: there are no false positives
• (aka Equivalent Mutations: mutations which result in no observable change in behavior)

Running The Analysis

1. Pick one mutant at a time
2. Inject it via Instrumentation API
3. Run the relevant tests until one fails
4. Watch out for:
   • Infinite loops
   • OOM and other failures
   • RuntimeExceptions

Pitest Grants You Confidence

A success story from TheLadders.com

• A project from scratch, Pitest used from the start
• Code and Mutation Coverage of about 95%
• Needed to refactor the core to use event sourcing
• So they just did it
• And had no issues!^*

Let's Test It On Commons Math

• Loads of complex mathematical algorithms
• 177K lines of source code
• About 5K tests in 109K lines of code

$ svn checkout http://svn.apache.org/repos/asf/comm [...]
$ cd commons-math3
$ patch commons-math3/pom.xml commons-math3-pom-patch.diff

Go pitest!

$ mvn clean test pitest:mutationCoverage
[...]
======================================================
- Timings
======================================================
> scan classpath : < 1 second
> coverage and dependency analysis : 27 minutes and 31 seconds
> build mutation tests : 7 seconds
> run mutation analysis : 3 hours, 47 minutes and 7 seconds
======================================================
> Total  : 4 hours, 14 minutes and 46 seconds

Throw In More Threads, Maybe?

$ mvn clean test pitest:mutationCoverage -Dpit.threads=2
[...]
======================================================
- Timings
======================================================
> scan classpath : < 1 second
> coverage and dependency analysis : 27 minutes and 28 seconds
> build mutation tests : 3 seconds
> run mutation analysis : 2 hours, 2 minutes and 1 seconds
======================================================
> Total  : 2 hours, 29 minutes and 33 seconds

Pitest Performance Over Time

Incremental analysis

Store the following info between runs:

• The hashes of all tested classes
• The hashes of all the tests
• Analysis results

Incremental analysis: Heuristics

Do not check a mutant, if in the previous run:

• It timed out, and the class has not changed
• It was killed, and neither the class nor the test have changed
• It survived, and there are no new/changed tests for it

Incremental Analysis: Overhead

$ mvn ... pitest:mutationCoverage -P pit-history -Dpit.threads=4
[...]
======================================================
- Timings
======================================================
> scan classpath : < 1 second
> coverage and dependency analysis : 7 minutes and 15 seconds
> build mutation tests : 2 seconds
> run mutation analysis : 55 minutes and 41 seconds
======================================================
> Total  : 1 hours, 2 minutes and 59 seconds

o.a.c.math3.linear.MatrixUtils

public static <T extends FieldElement<T>> FieldMatrix<T>
    createFieldIdentityMatrix(final Field<T> field, final int dimension) {
        final T zero = field.getZero();
        final T one  = field.getOne();
        final T[][] d = MathArrays.buildArray(field, dimension, dimension);
        for (int row = 0; row < dimension; row++) {
            final T[] dRow = d[row];
            Arrays.fill(dRow, zero);
            dRow[row] = one;
        }
        return new Array2DRowFieldMatrix<T>(field, d, false);
}

public static <T extends FieldElement<T>> FieldMatrix<T>
    createFieldIdentityMatrix(final Field<T> field, final int dimension) {
        final T zero = field.getZero();
        final T one  = field.getOne();
        final T[][] d = MathArrays.buildArray(field, dimension, dimension);
        for (int row = 0; row < dimension; row++) {
            final T[] dRow = d[row];
            dRow[row] = one;
        }
        return new Array2DRowFieldMatrix<T>(field, d, false);
}

200: removed call to java/util/Arrays::fill : SURVIVED

Incremental Analysis: Profit

$ mvn ... pitest:mutationCoverage -P pit-history -Dpit.threads=4
[...]
======================================================
- Timings
======================================================
> scan classpath : < 1 second
> coverage and dependency analysis : 7 minutes and 3 seconds
> build mutation tests : 1 minutes and 1 seconds
> run mutation analysis : 56 seconds
======================================================
> Total  : 9 minutes and 0 seconds

Pitest vs. Defensive Programming

public int evaluate(final int distance) {
    if(distance == 0) {
        throw new IllegalArgumentException("Distance should be non-zero");
    }

    int result = 0;
    if(distance > 0) {
        result += calculateDistanceAdjustment(distance);
    }

    return result;
}

Pitest vs. Dangerous Code

public class Minitrue {
    public static void rectify(String path) {
        ensurePathIsSecure(path);
        rmRf(path);
    }

    private static final void rmRf(String path) {
        String command = "$ rm -rf " + path;
        System.out.println("Executing: " + command);
        exec(command);
    }

    // ...
}

Pitest vs. Dangerous Code

@Test
public void testRectifySecurity() {
    boolean rejected = false;

    try {
        Minitrue.rectify("/");
    } catch(SecurityException e) {
        rejected = true;
    }

    Assert.assertTrue(rejected);
    Assert.assertFalse(Minitrue.isUnpath("/"));
}

What Could Possibly Go Wrong?

stderr  : PIT >> Running mutation  [...]
          mutator=VoidMethodCallMutator,
          description=removed call to Minitrue::ensurePathIsSecure

stderr  : PIT >> mutating method rectify
stderr  : PIT >> 2 relevant test for rectify
stderr  : PIT >> Running 1 units

stdout  : Executing: rm -rf /

See Also

• Official Guide: pitest.org/quickstart/
• Jenkins Plugin: github.com/edwardwe7/pitmutation-jenkins
• Gradle Sample: github.com/gvsmirnov/pitest-gradle-sample
• Maven Sample: github.com/gvsmirnov/pitest-sample
• Talk by Henry Coles goo.gl/AYhkm7
• Talk by Filip Van Laenen goo.gl/Cu8jUh