It is an established practice to keep you Unit Tests and your production code separate, in such a way that the compiled artifacts do not end up containing tests. It has recently drawn my attention, however, that it is usually not the case with benchmarks and performance tests. A possible reason is that it might be a little bit tricky. So here's a sample github project of how it can be done in a more or less ideologically right way with JMH and gradle. This article goes into some further detail and explains why stuff is like it is.