It is an established practice to keep you Unit Tests and your production code separate, in such a way that the compiled artifacts do not end up containing tests. It has recently drawn my attention, however, that it is usually not the case with benchmarks and performance tests. A possible reason is that it might be a little bit tricky. So here’s a sample github project of how it can be done in a more or less ideologically right way with JMH and gradle. This article goes into some further detail and explains why stuff is like it is.

What We Want to Have

While in some cases you can simply have benchmarks as a separate project, it is not quite the right solution in others. Especially when you have a multi-module project, and each of them has benchmarks.

Here are the basic requirements that should satisfy the people striving to get it right:

• The source classes should be kept in a separate source set from the production classes. Similar to how we put unit tests in src/test/, we place the benchmarks in the src/perf/ folder
• The dependencies that the benchmarks introduce should only relate to the benchmarks, not to the production code
• We should be able to get a single jar “with batteries” that can be effortlessly copied to any environment, enabling us to run the performance tests with a single command

Sounds reasonable, now let’s get cracking!

The Non-Tricky Part

Let’s create a folder named src/perf/java and place our benchmarks there. To make gradle recognize this as a new source set, we simply have to add the following line to the project definition:

sourceSets {
    perf
}

We now automatically have various configurations like perfCompile that we can use to specify dependencies like so:

dependencies {
    perfCompile project
    perfCompile 'org.openjdk.jmh:jmh-core:0.5.3'
    perfCompile 'org.openjdk.jmh:jmh-generator-annprocess:0.5.3'
}

The first dependency lets gradle know that the new source set requires the classes we have in src/main during the compilation, and also all the dependencies of the main project.

Finally, to pack the benchmarks and the production code into a jar, we need to create a corresponding task:

task perfJar(type: Jar, dependsOn: perfClasses) {
    from sourceSets.perf.output + sourceSets.main.output
}

Running gradle perfJar will now get us an archive in the build/libs folder, but it will not include the dependencies. To get the ease of running benchmarks on different environments with as little stuff to do as possible, we want the jar to include everything.

Enter Shadow

There are two gradle plugins that can pack the dependencies up into the target jar that I know of: One-jar and Shadow. The former does not unpack the jars during the build, but rather encloses a custom class loader that looks things up in the enclosed jars. Therefore, I prefer the latter, which was inspired by the Maven Shade Plugin, and simply unpacks all the jars, and then packs them back, all together now. We need to add it to the build script’s dependencies:

buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath 'com.github.jengelman.gradle.plugins:shadow:0.8'
    }
}

For clarity, we will create a separate gradle task called benchmarks, and invoke Shadow from within it:

task benchmarks(dependsOn: perfJar) {
    apply plugin: "shadow"

    shadow {
        classifier = "benchmarks"

        transformer(com.github.jengelman.gradle.plugins.shadow.transformers.ManifestResourceTransformer) {
            mainClass = "org.openjdk.jmh.Main"
        }
    }

    doLast {
        shadowJar.execute()
    }

}

Now, if we execute gradle benchmarks, we will get a jar named ${projectName}-${version}-benchmarks.jar in the build/distributions/ folder, which will contain all we need. Well, almost all we need. Unfortunately, Shadow does not currently support dependency inclusion for custom configurations. I have created a pull request that adds the required functionality, but it is not yet merged. When it is, we will be enabled to specify which configurations’ dependencies should be included:

shadow {
    //...
    includeDependenciesFor = ["runtime", "perfRuntime"]
}

If you don’t want to wait until the official artifact is out, you may want to fetch the unofficial 0.8.1 binary here:

buildscript {
    repositories {
        maven {
            name 'Shadow'
            url 'http://dl.bintray.com/content/gvsmirnov/gradle-plugins'
        }
        jcenter()
    }
    dependencies {
        classpath 'com.github.jengelman.gradle.plugins:shadow:0.8.1'
    }
}

Support in IDEs

Last but not least, we do not want the IDEs to go all red on us, complaining about unknown symbols and whatnot. Here’s what we can do for IDEA and Eclipse to make them understand us:

apply plugin: "idea"
idea {
    module {
        scopes.PROVIDED.plus  += configurations.perfCompile
        scopes.PROVIDED.minus += configurations.compile
    }
}

apply plugin: "eclipse"
eclipse {
    classpath {
        plusConfigurations += configurations.perfCompile
    }
}

We add PROVIDED.minus to IDEA, since it would otherwise make all the compile dependencies of the main project as provided, which would break the ability to run projects from within the IDE.

That’s all, folks!

Here’s the link to the sample project again, and this is how we use it:

$ gradle benchmarks
$ java -jar build/distributions/jmh-gradle-sample-0.0.1-benchmarks.jar