How We Test Agent Selection

What we measure

The single number this protocol produces is your agent win rate: the share of agent shopping sessions, for a defined query, in which your product is the one the agent recommends or buys. It is a proportion between 0 and 1, measured over many sessions: not a search-style rank, not impressions, and not click-through. We define the metric and its honest edge cases in full on agent win rate: definition and how to measure →; this page is the measurement machinery behind that definition.

Win rate is the right unit because it maps directly onto the choice a shopping agent actually makes. When a shopper asks an agent for "a quiet desk fan under $60," the agent returns a small set of picks, often one. Over many such sessions, the fraction of times your product is in that set is a signal you can track, compare across engines, and try to move. That is the whole of what agent selection comes down to from a merchant's seat: not where you appear in a list a human scrolls, but how often the agent chooses you.

Prompt set and query sampling

A win rate is only meaningful relative to a fixed set of prompts. Ours is pre-registered before any measurement run, so the questions cannot be quietly reshaped to flatter a result.

• Query frame. We start from the real purchase intents a catalog competes for: the category, the constraints (price ceiling, size, use-case), and the phrasing a shopper actually types. Each frame becomes a small family of paraphrases, because agents respond to wording, and one phrasing is not the population.
• Prompt templates. For every query we hold a fixed template and vary only the shopper's words, so differences in win rate trace to the query, not to scaffolding we introduced.
• Sample size by target precision, not by a round number. We do not pick a prompt count for its looks. We choose the number of repetitions from the precision we need: enough independent sessions that the confidence interval around the win rate is narrower than the effect we care about detecting. Small effects need many more sessions than large ones; the sampling plan states that target width up front.

Engines and sessions

We run the same pre-registered prompt set against each shopping surface we cover (ChatGPT, Google's Gemini / AI Mode, and Perplexity) and treat each as its own measurement, never averaged into one "AI" number. The per-engine playbooks live at ChatGPT shopping →, Gemini shopping →, and Perplexity shopping →.

Each session is designed to be independent and reproducible:

• Fresh, uncontaminated sessions. Every prompt runs in a clean session with no carried-over memory, personalization, or prior conversation, so one answer cannot prime the next.
• Recorded in full. We capture the complete response, the products named, and the order they appear in (not just whether we "won"), so the raw record can be re-scored later without re-running.
• Stamped with engine, model version, and timestamp. Because agent behavior is not stationary, every session is tagged with which surface, which model version (where exposed), and when. ACES reports that model updates can drastically reshuffle which products win, so any measurement is tied to a specific model version at a specific time.ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17) That is why every result we publish carries a "last verified" date.

One evidence gap we state plainly: the academic study we adapt did not test Perplexity. Our live testing includes Perplexity anyway, because it is a real shopping surface, but the simulation-derived expectations below transfer to it only as hypotheses, not as findings.

Controlling for position and order

The hardest part of measuring selection is separating "the agent chose your product" from "the agent chose the slot your product happened to sit in." That confound is not hypothetical.

ACES found that where a product sits on the page changes how often agents pick it (for one model, moving a product from the bottom-right corner to the top row raised its selection rate roughly fivefold), and that this position bias differs from model to model.ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17) The ACES numbers come from a controlled simulation rather than live stores, but the mechanism is exactly the nuisance we have to neutralize.

So we do not let position ride along as a hidden variable:

• Randomize placement. Across sessions we randomize the order and, where the surface presents a grid, the grid position of the candidate set, so no product enjoys a permanent good seat.
• Counterbalance. Every product spends comparable time in strong and weak positions, so position averages out rather than loading onto one item.
• Change one thing at a time. When we test whether a product-side change (a title, a description, a schema field) moves win rate, everything else in the candidate set is held fixed, so the change is the only plausible cause of a difference.

Statistics and error bars

Win rate is a proportion, so we report it like one, with uncertainty attached, never as a bare percentage.

• Confidence intervals, not point estimates. Each win rate ships with a confidence interval appropriate for proportions (for example a Wilson or Agresti-Coull interval), so readers see the precision, not just the number. A win rate of "1 in 3" from 12 sessions and from 1,200 sessions are not the same claim, and the error bar is what says so.
• Differences with their own uncertainty. When we compare two conditions (engine A versus engine B, or a rewritten description versus the original), we report the difference in win rate and a confidence interval for that difference (bootstrapped over sessions), because a gap that fits inside the error bars is not a result.
• Pre-registration and multiple comparisons. The hypotheses and the primary comparison are fixed before data collection, and when we test many queries or engines at once we adjust for multiple comparisons, so we do not mistake noise across dozens of tests for a finding.
• Effect size over significance theater. We lead with how big a shift is and how wide its interval, not with a p-value alone, because a "significant" but tiny move is not a lever a merchant should chase.

Reproducing ACES versus testing live catalogs

This protocol runs on two tracks, and their trade-offs are opposite, which is exactly why we keep them separate and report each on its own terms.

Track 1: reproduce the simulation. We can re-run the audit inside the same kind of controlled environment the academic method uses. The ACES framework audits agent decisions by showing vision-language-model shopping agents a mock storefront in randomized trials and analyzing their picks with regression, a controlled simulation of agent choice, not a record of real-world purchases.ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17) The ACES simulator is released under the MIT license, which permits reproducing and extending its code with attribution.Spec-factACES repository (MyCustomAI) The strength of this track is clean causal control: every variable is ours to set. Its limit is realism: a mock storefront is not your catalog on a live engine, and a simulated pick is not a sale.

Track 2: test live catalogs. We also run the pre-registered prompts against the real ChatGPT, Gemini, and Perplexity shopping surfaces, scoring win rate on live responses. The strength here is realism: these are the agents shoppers actually use. Its limits are noise and non-stationarity: live engines personalize, rate-limit, and change under you, so live win rates carry wider error bars and shorter shelf lives than simulation results.

Neither track alone is the truth. Simulation says what can move a decision under control; live testing says what is moving on the surfaces that matter this month. The description-rewrite study, our first experiment to carry a specific product-side hypothesis through both tracks, is written up at do description rewrites shift AI recommendations? →.

Limitations

We would rather you distrust the right things than trust the wrong ones.

• Simulation is not sales. Track-1 results describe agent choices in a controlled sandbox, not revenue on your store. We never translate a simulated selection shift into a promised conversion or dollar figure.
• Live results expire. Because model updates reshuffle outcomes, a live win rate is a snapshot of one engine at one time. Every published result is dated, and an old result is a historical record, not current guidance.
• A sample is not the market. A pre-registered query set and a set of test catalogs cannot represent every shopper intent or every store. We report the scope we tested and resist generalizing past it.
• We see outputs, not the ranking. We measure what an agent chose, not the internal logic behind the choice. Explanations for why a product won are typed as hypotheses, never as measured mechanism.
• Correlation guardrails. Establishing that a specific product-side change caused a win-rate move needs a controlled A/B within this protocol, not a before-and-after on a shifting engine. Uncontrolled comparisons are reported as suggestive, not causal.

Results