Setup

System environment

Service workstation

Client workstation

Metric

Create 100.000 products

Load 100.000 products

Setup

Now that we have our implementations in place, we can start comparing them. We will begin by implementing some benchmarks to test the performance of both implementations.

Please do not confuse a benchmark with a load test. While the latter is intended to push a system or application to its limits, the former is intended to gather performance statistics for a specified set of conditions.

There are several applications available to perform load tests and benchmarks. For benchmarks, the Apache JMeter project is a good starting point.

It is quite easy to get something running. The documentation says: “For the real stuff, you should only use the command line application and use the GUI to create and test your benchmark.”

We’ll skip a lengthy introduction and tutorial for JMeter because you can find a lot about it within the documentation, and there are lots of tutorials online to help you understand it as well.

System environment

Service and testing software (Apache JMeter) were run on different workstations connected via 100 MBit/s network connection.

Service workstation

CPU: Core i5-9600K, 6 Cores, 3,7 GHz

RAM: 32 GB

HDD: 2x Samsung SSD 860 PRO 512GB, SATA

OS: FreeBSD 12 (HT disabled)

JDK: 11.0.4+11-2

DB: PostgreSQL 11.3

Client workstation

CPU: AMD Ryzen Threadripper 2950X

RAM: 32 GB

HDD: 2x Samsung SSD 970 PRO 512GB, M.2

OS: FreeBSD 12 (HT disabled)

JDK: 11.0.4+11-2

“Apache JMeter version 5.1.1 was used to run the benchmark, and 10 threads were used with a 10 seconds ramp up time for each benchmark.”

Metric

We’ll stick to using the average of the metrics across all three benchmark runs. The following abbreviations will be used in the tables and legends.

AVG: The average response time in milliseconds

MED: The median response time in milliseconds

90%: 90 percent of all requests were handled within the response time in milliseconds or less.

95%: 95 percent of all requests were handled within the response time in milliseconds or less.

99%: 99 percent of all requests were handled within the response time in milliseconds or less.

MIN: The minimum response time in milliseconds

MAX: The maximum response time in milliseconds

ERR: The error rate in percent

R/S: The number of requests per second that could be handled

MEM: The maximum amount of memory used by the service during the benchmark in MB

LD: The average system load on the service machine during the benchmark. 

Metrics Recap

Update 100.000 products

Bulk load all 100.000 products

Summary

Metrics Recap

To reiterate, we will be using the following metrics to benchmark our endpoints.

AVG: The average response time in milliseconds

MED: The median response time in milliseconds

90%: 90 percent of all requests were handled within the response time in milliseconds or less.

95%: 95 percent of all requests were handled within the response time in milliseconds or less.

99%: 99 percent of all requests were handled within the response time in milliseconds or less.

MIN: The minimum response time in milliseconds

MAX: The maximum response time in milliseconds

ERR: The error rate in percent

R/S: The number of requests per second that could be handled

MEM: The maximum amount of memory used by the service during the benchmark in MB

LD: The average system load on the service machine during the benchmark