Something very similar applies to you when you are a vendor for database as a service. And if we again look at all the customers out there and we rank them in the descending order by their revenue, the cost that we have by providing DB-Software per customer determines the market that we can address. If you can lower the price of the cost of providing DB-Software per customer you can address a large market. And what we’ve been seen for the last 15 years is that, cloud providers such as Google, Telecom, IBM, Amazon, Microsoft, they have managed to really bring those costs down and to increase the market that they can address. Imagine you are a provider in this market. You have a narrow profit margins which means you want to utilize your ressources to the fullest, with available hardware you want to cater as many customers/tenants as possible. When I am such a provider, I have to make sure I am providing a good service. And the key metric here is the latencies.
So let’s capture the tail-latencies in DB systems benchmarking. And to do that we want to have trust in our results. And here we show that if we use a Java-based benchmark harness, the JVM/garbage collector can in fact make a noise and distort measured tail-latencies. Inside a JVM there’s this concept of Stop-the-World Pauses, for instant garbage collections. If the Stop-the-World event happens before or after you’ve taken your measurement, that’s fine, we are not going to count that towards the latencies that’ve been measured, but if this happens in between, Stop-the-World pause latencies will add this to our measured value.
Experimental Setup: A very popular benchmark harness is OLTP Bench, it is written in Java and it supports a range of DB systems such as Maria DB or PostgreSQL. It has also support from various Benchmarks. We chose TPC-C(analytical warehouse querys), YCSB(is more light-weighted compared to TPC-C) and NoOp(empty SQL statements, unrealistic benchamrk). Results that we are showing to you were on the Java Version 16 and we tried out 2 JVMs. And then we hand pick garbage collectors that are all geared towards producing low latencies in garbage collections.