First come, first recommended? Position bias in head-to-head brand recommendations by a frontier AI model

Eike Henrich¹ · Jimmy Park² · Tobias Peschke^3
1Independent   ²Opascope, New York   ³Loud & Lexis, Frankfurt am Main  ·  Whitepaper v1.0, October 2026

1 Introduction

Head-to-head prompts such as “It's between X and Y; which one should I choose?” occur late in purchase decisions and have become a standard way to measure brand visibility in AI search, as part of generative engine optimization (GEO). If the order of the names affects the answer, a measurement in one order mixes up brand preference with position.

Position bias, the tendency to favor an option because of where it appears, is documented in three settings. Language models judging other models' answers favor one position: GPT-4 kept its verdict in only 65.0% of swapped pairs (Zheng et al., 2023), swapping the order changed ChatGPT's verdict on 66 of 80 questions (Wang et al., 2024), and across 15 judges, verdicts flipped most often when the candidates were similar in quality (Shi et al., 2025). Lists of options are affected too: order shifts answers to multiple-choice and classification prompts (Pezeshkpour and Hruschka, 2024; Zheng et al., 2024; Wang et al., 2023) and the rankings of LLM recommenders (Hou et al., 2024; Bito et al., 2025). In similarity judgments, GPT-4 chose the first of two options in 63% of responses, and telling it the order was randomized did not help (Brucks and Toubia, 2025). Shopping decisions show it as well: some models favor the first of two product descriptions (Laurito et al., 2025), and AI shopping agents favor certain positions in a mock storefront (Allouah et al., 2025). People show comparable primacy effects (Miller and Krosnick, 1998; Mantonakis et al., 2009).

The closest design, Allouah et al. (2025), randomized the positions of real products in a mock storefront, but agents chose among eight listings inside a sandbox without web search. We are not aware of a study that tests a choice between two named real brands in a user-style question, asked in both orders with repeated sampling, on a frontier model with live web search.

2 Methods

2.1 Brand sets and prompts

The sample comprises 21 brand sets, most from Loud & Lexis or Opascope client projects; one is Loud & Lexis itself (Appendix B). Each set consists of a focus brand and, per product category, up to five competitors that gpt-5.6-luna identified from answers to competitor questions. The same model wrote a persona and a need for each category. This yields 97 categories, 482 matchups and 305 distinct brand pairs. All prompts followed one template (Appendix A): the persona states the need, names both brands and asks for a one-sentence verdict followed by four structured sections. The second order swaps the two brand names in both places they appear. Brand sets from German-speaking markets were prompted in German (7), the others in English (14).

2.2 Data collection and labeling

We collected 4,928 answers in complete vote pairs from gpt-5.6-luna through the OpenAI Responses API, with web search enabled and default settings, between 2026-10-08 and 2026-10-16. Every answer was an independent call, and each matchup's prompt was asked five times in each order (once per order for a few categories). A reading model (gpt-5.6-luna; gpt-4o-mini for 7 brand sets) read each verdict and returned the chosen brand or “none”, counting qualified choices (“leaning toward X”). A blind re-check of 100 random votes per reading model, using Claude Opus 5.5, agreed in 199 of 200 cases. One brand set (300 answers) was excluded because its generated personas were too vague for the model to decide (see Section 3.4). After removing the 10 vote pairs in which either answer had no verdict, 4,608 votes in 2,304 vote pairs remained.

2.3 Statistical analysis

Because every pair is tested in both orders, true brand strength cancels out: without a position effect, the first-named brand would be expected to win half of all votes. We tested the share of votes won by A with an exact binomial test (Clopper–Pearson intervals; Clopper and Pearson, 1934), changed vote pairs with an exact McNemar test (McNemar, 1947), and matchup majorities with sign tests.

To measure the size of the effect independently of brand strength, we used a Bradley–Terry model with an order term (Bradley and Terry, 1952; Agresti, 2013), fitted as a conditional logistic regression with matchups as strata:

logit P(brand i wins) = α_matchup + β [i = A].

α captures how strong the two brands in a matchup are; β is the extra pull of being named first. The odds ratio e^β is the factor by which naming a brand first multiplies its odds of winning; 1 would mean no position effect. For two brands the model rates as equal, the first-named brand then wins with probability logit⁻¹(β/2). Mantel–Haenszel (Mantel and Haenszel, 1959) and vote-pair estimates serve as cross-checks. Because the effect may differ between brand sets, we also report an interval from a cluster bootstrap that resamples whole brand sets (5,000 replicates; Cameron et al., 2008). The same bootstrap, resampling German and English sets separately, gives the interval for the ratio of the German and English odds ratios.

Whether the effect differs between brand sets was tested with a likelihood-ratio statistic comparing one odds ratio per brand set with a common odds ratio; its p-value comes from a permutation test that shuffles brand-set labels across matchups (2,000 draws). Differences by matchup closeness were tested with a likelihood-ratio test of one odds ratio per overall split against a single odds ratio, and differences across repetitions and collection days with χ² tests of the share won by A. Tests are two-sided.

3 Results

3.1 Votes and vote pairs

The first-named brand won 54.7% of all votes, more than the 50% expected without a position effect (Table 1). In most comparisons, the order did not change the winner: in 79.5% of vote pairs the same brand won in both orders (Figure 1). In the remaining 20.5%, the winner changed with the order, and in 72.9% of these cases it was the brand named first. Part of this is ordinary variation between runs. Across all combinations of two votes within a matchup, rather than only the matched vote pairs, votes asked in the same order named different winners in 15.6% of cases and votes asked in opposite orders in 20.0%.

3.2 Effect size

Being named first multiplied a brand's odds of winning by 2.98. For two brands the model rates as equal, this turns a 50 : 50 split into 63.3% vs 36.7%: the first-named brand wins nearly two-thirds of the time. The cross-checks and the brand-set bootstrap point the same way (Table 1).

3.3 Effect by matchup closeness

Order made no difference in the 230 of the 445 matchups with ten votes in which the same brand won all ten. In close matchups (6 : 4 or 5 : 5 overall), the first-named brand won 63.4% of votes. The odds advantage of being named first is no larger in close matchups; its effect on win shares is. Odds ratios fitted separately by closeness show no significant difference (p = 0.23), and the shares predicted from the single odds ratio match the observed ones (Figure 2). Studies of LLM judges report a similar pattern (Zheng et al., 2023; Wang et al., 2024; Shi et al., 2025).

3.4 Variation across brand sets and robustness

The first-named brand won the majority of changed vote pairs in 18 of the 21 brand sets (Figure 3). None of the other three shows a significant reverse effect (0 : 2, 17 : 18, 11 : 11; all CIs include 50%). The strength of the effect varied significantly between brand sets, from none to strong (permutation test, p = 0.002). The overall odds ratio of 2.98 is therefore an average across sets, not the effect to expect in any single one.

The effect appeared larger for German prompts than for English ones (odds ratio 4.39 vs 2.30). Once variation between brand sets is taken into account, however, this difference is uncertain: the bootstrap interval for the ratio of the two odds ratios (0.98–4.00) includes 1. Language is also confounded with industry, so we treat the language difference as exploratory.

The odds ratio barely changes when the Loud & Lexis brand set is left out (3.02) or the excluded set is included (3.09), and the effect did not vary significantly across repetitions or collection days (Appendix C).

4 Discussion

4.1 Implications for head-to-head measurement

Pooling both orders, as we did, cancels out the position effect. A measurement taken in only one order has no such protection. On average, it shifts a brand's win rate by 4.7 pp. The focus brands of this study show the gap clearly: they won 62.8% of their comparisons when named first and 53.4% when named second. In 16% of matchups (about one in six), the two orders named different majority winners, mostly in favor of the first-named brand (62 vs 15). The remedy is already established: ask in both orders (Wang et al., 2024; Brucks and Toubia, 2025) and count a win only if it holds in both (Zheng et al., 2023).

Instructions to ignore the order are not a practical remedy: buyers do not give them, and such instructions did not prevent position bias in LLM judges (Zheng et al., 2023) or shopping agents (Allouah et al., 2025), nor did telling a model that the order was randomized (Brucks and Toubia, 2025).

4.2 Prompt order and brand awareness

In real use, the buyer decides which brand comes first. We hypothesize that buyers tend to name first the brand that comes to mind first, i.e., the brand with higher top-of-mind awareness (Aaker, 1996; Keller, 1993), or the one they already lean toward. If this holds, the position effect turns brand awareness into an advantage inside AI answers: in close comparisons, the better-known brand would more often be named first and therefore more often be recommended. This would add to a direct advantage: with identical specifications, well-known brands won every recommendation in one study (Chu and Hou, 2026). Query logs or buyer surveys could test the naming-order link.

4.3 Limitations

• Other models may show weaker, stronger or reversed effects (Zheng et al., 2023; Wang et al., 2024; Allouah et al., 2025).
• The design measures the final answer and cannot separate effects of web retrieval from generation.
• The brand sets come mostly from client projects and are not a representative sample of the market.
• How buyers order brand names was not measured.

5 Conclusion

Position bias affects the brand recommendations of a frontier model with web search. Brands that the model clearly prefers win regardless of order. However, in close comparisons the first-named brand is favored: when the model has no preference between two brands, the first-named one wins 63.3% of comparisons.

For GEO practitioners, three rules follow:

• Ask in both orders and analyze them as a pair. Put every head-to-head question as “X or Y?” and as “Y or X?”
• Ask in each order several times. Answers vary even when nothing changes.
• Count a win only if it holds in both orders. Treat a result that reverses with the order as a tie. This matters most in close matchups, where a 60–70% win rate for the first-named brand, measured in one order, is consistent with an even split.

If buyers name the better-known brand first, as we hypothesize in Section 4.2, brand building outside AI search would also shape AI recommendations.

References

1. Aaker, D. A. (1996). Building strong brands. The Free Press.
2. Agresti, A. (2013). Categorical data analysis (3rd ed.). Wiley.
3. Allouah, A., Besbes, O., Figueroa, J. D., Kanoria, Y., & Kumar, A. (2025). What is your AI agent buying? Evaluation, biases, model dependence, & emerging implications for agentic e-commerce. Preprint, arXiv:2508.02630v3. https://arxiv.org/abs/2508.02630v3
4. Bito, E., Ren, Y., & He, E. (2025). Evaluating position bias in large language model recommendations. Preprint, arXiv:2508.02020. https://arxiv.org/abs/2508.02020
5. Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345. https://doi.org/10.2307/2334029
6. Brucks, M., & Toubia, O. (2025). Prompt architecture induces methodological artifacts in large language models. PLOS ONE, 20(4), e0319159. https://doi.org/10.1371/journal.pone.0319159
7. Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics, 90(3), 414–427. https://doi.org/10.1162/rest.90.3.414
8. Chu, X., & Hou, Y. (2026). Incumbent advantage: Brand bias and cognitive manipulation dynamics in LLM recommendation systems. Preprint, arXiv:2606.17443. https://arxiv.org/abs/2606.17443
9. Clopper, C. J., & Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4), 404–413. https://doi.org/10.1093/biomet/26.4.404
10. Hou, Y., Zhang, J., Lin, Z., Lu, H., Xie, R., McAuley, J., & Zhao, W. X. (2024). Large language models are zero-shot rankers for recommender systems. In Advances in Information Retrieval (ECIR 2024) (pp. 364–381). Springer. https://arxiv.org/abs/2305.08845
11. Keller, K. L. (1993). Conceptualizing, measuring, and managing customer-based brand equity. Journal of Marketing, 57(1), 1–22. https://doi.org/10.1177/002224299305700101
12. Laurito, W., Davis, B., Grietzer, P., Gavenčiak, T., Böhm, A., & Kulveit, J. (2025). AI–AI bias: Large language models favor communications generated by large language models. Proceedings of the National Academy of Sciences, 122(31), e2415697122. https://doi.org/10.1073/pnas.2415697122
13. Mantel, N., & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22(4), 719–748. https://doi.org/10.1093/jnci/22.4.719
14. Mantonakis, A., Rodero, P., Lesschaeve, I., & Hastie, R. (2009). Order in choice: Effects of serial position on preferences. Psychological Science, 20(11), 1309–1312. https://doi.org/10.1111/j.1467-9280.2009.02453.x
15. McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153–157. https://doi.org/10.1007/BF02295996
16. Miller, J. M., & Krosnick, J. A. (1998). The impact of candidate name order on election outcomes. Public Opinion Quarterly, 62(3), 291–330. https://doi.org/10.1086/297848
17. Pezeshkpour, P., & Hruschka, E. (2024). Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 2006–2017). https://doi.org/10.18653/v1/2024.findings-naacl.130
18. Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., & Vosoughi, S. (2025). Judging the judges: A systematic study of position bias in LLM-as-a-judge. In Proceedings of IJCNLP-AACL 2025 (Volume 1: Long Papers) (pp. 292–314). https://doi.org/10.18653/v1/2025.ijcnlp-long.18
19. Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., & Sui, Z. (2024). Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 9440–9450). https://doi.org/10.18653/v1/2024.acl-long.511
20. Wang, Y., Cai, Y., Chen, M., Liang, Y., & Hooi, B. (2023). Primacy effect of ChatGPT. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 108–115). https://doi.org/10.18653/v1/2023.emnlp-main.8
21. Zheng, C., Zhou, H., Meng, F., Zhou, J., & Huang, M. (2024). Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://openreview.net/forum?id=shr9PXz7T0
22. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (Datasets and Benchmarks Track). https://arxiv.org/abs/2306.05685

Appendix A Prompt template

English version; the German version is a translation with the same structure. In the second prompt of a vote pair, {brand 1} and {brand 2} swap places at both positions.

Appendix B Brand sets

Brand names are withheld. Changed: vote pairs with a changed winner, won by A : won by B.

Appendix C Robustness checks