The Agent Selection Index

The Agent Selection Index is the named, recurring measurement built on top of our general testing protocol. Where how we test agent selection is the reusable machinery, the Index is one specific product of that machinery: a controlled benchmark, re-run each quarter, of how much a given product-side change moves an agent's choice. It sits inside the wider strategy the handbook teaches in the agentic commerce optimization guide, turned into a repeatable measurement. This page publishes the method in full. It ships numbers-empty on purpose, and the results section stays that way until a run is generated in the lab and verified.

What the Index measures

The Index measures one thing: how often each agent selects a product or offer as we vary a single factor, holding the rest of the candidate set fixed. The unit is a selection rate (a proportion between 0 and 1) and, when we compare two conditions, the effect of the change on that rate. Both always carry a confidence interval. We define the underlying metric in full on agent win rate: definition and how to measure, and the choice an agent makes on agent selection; the Index is one disciplined way of quantifying that choice over time.

Be clear about what this is not. The Index is not a scrape of any platform's live results, and it is not an attempt to reverse-engineer how ChatGPT, Gemini, or Perplexity rank stores in production. We do not have that internal logic, and we do not pretend to. We measure the observable output (which product an agent picks under controlled conditions) and report it as exactly that. Anything we say about why a product won is inference, never a measured ranking rule.

Method: controlled agent simulations

The Index runs against synthetic control "stores": candidate sets we build, where every attribute is ours to set. Each experiment changes one factor and freezes the rest, so the difference in selection has one plausible cause.

• One factor at a time. A run isolates a single lever (for example a title phrasing, a structured data field, or an on-page position) and holds title, price, rating, availability, image, and every other attribute constant across conditions.
• Disclosed prompts, per-model sampling. The shopper prompts are pre-registered and published, and each condition is sampled over many independent trials per model, because one answer to one prompt is noise, not a rate.
• Position neutralized by design. Placement is randomized and counterbalanced across trials, so no candidate keeps a permanently good seat and position averages out instead of loading onto one item.

A single-factor run is exactly the shape of the description-rewrite experiment, which varies only a product's description and holds the rest fixed; the Index generalizes that pattern to a slate of factors, re-measured each quarter.

This design descends from a published, reproducible study. ACES audits agent decision-making by showing vision-language-model shopping agents a mock storefront in randomized trials and analyzing their picks with regression: a controlled simulation of agent choice, not a record of real-world purchases.ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17) That caveat travels with every result the Index produces: a simulated pick is not a sale. Position is the confound we most have to neutralize. ACES found that where a product sits on the page changes how often agents select it, and that this position bias differs from one model to another.ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17)

We build on the ACES harness rather than reinventing one. The ACES simulator is published under the MIT license, which permits reproducing and extending its code with attribution.Spec-factACES repository (MyCustomAI) Because a run changes exactly one factor while holding the rest of the candidate set fixed and washes out placement through randomization, we expect any measured shift in selection to be attributable to that factor rather than to price, rating, or position.Hypothesis (our analysis)

Models, versions, and cadence

Every run drives the official model APIs (Claude, OpenAI, Gemini) under their terms of use. We do not automate a consumer app or a logged-in shopping surface to produce Index figures. Each result is tagged with three things that make it re-checkable: which model, which model-version string (as the provider exposes it), and the date the run was collected.

The cadence is quarterly, and the reason is built into the subject. ACES reports that model updates can drastically reshuffle which products win, so any measurement is tied to a specific model version at a specific time.ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17) A quarterly index is therefore a series of dated snapshots, not a standing answer. An old edition is a historical record of how a named set of model versions behaved on a named date, and we present it that way rather than as current guidance. We expect selection to differ by model and to drift as models update, which is why the Index reports per-model, dated results instead of a single blended "AI recommendation" score.Hypothesis (our analysis)

Statistics

Selection is a proportion, so the Index reports it like one, with its uncertainty attached. The full statistical treatment lives in the research methodology; the Index applies it without change. In short:

• Intervals, not point estimates. Each selection rate ships with a confidence interval appropriate for proportions (for example a Wilson or Agresti-Coull interval), so a rate from a handful of trials is not read as the same claim as one from thousands.
• Differences with their own intervals. When we compare two conditions, we report the difference in selection rate and a confidence interval for that difference, because a gap that fits inside the error bars is not a finding.
• Pre-registration and multiple comparisons. The prompts, conditions, and primary comparison are fixed before a run, and when a quarter tests many factors or models at once we adjust for multiple comparisons, so we do not mistake noise across dozens of tests for a result.
• Effect size over p-values. We lead with how large a shift is and how wide its interval, not a bare significance stamp, because a significant but tiny move is not a lever worth chasing.

Limitations

We would rather you distrust the right things about this benchmark than trust the wrong ones.

• It is not a production ranking. The Index measures agent choice under controlled conditions we built. It is not what ChatGPT, Gemini, or Perplexity return to a live shopper, and it is not a proxy for your own sales.
• Simulation is not revenue. A selection shift in a controlled candidate set is not a conversion or a dollar figure, and we never translate one into the other.
• Every edition expires. Because model updates reshuffle outcomes, each figure is a snapshot of a named model version on a named date. An old edition is a historical record, not this quarter's guidance.
• A sample is not the market. A pre-registered set of factors, queries, and control stores cannot represent every shopper intent or every catalog. We report the scope tested and resist generalizing past it.
• We see outputs, not the ranking logic. We measure what an agent chose, not the internal rule behind it. Any explanation of why a product won is typed as a hypothesis, never as measured mechanism.

The index (results)

The methodology above is complete and reproducible: the prompts, the one-factor-at-a-time design, the model tagging, and the statistics are all published here now. Measured index values will appear below only once the author has generated a run in the lab and verified it. Until that verified data exists, no numbers are shown, per our integrity rules. Every figure that does land will be tagged with its model, model version, and collection date, and will carry a confidence interval.