Key takeaways
- The metric is agent win rate: the share of repeated, independent agent shopping sessions in which your product is the one recommended or bought, a proportion, not a rank.
- Method basis: this protocol adapts the ACES academic audit method, which is a controlled simulation of agent choice, not a record of real sales, so run the same idea against both a simulator and live engines and report the two separately.
- One query proves nothing: an engine's answer to the same prompt shifts run to run, so score win rate over many repetitions and report a confidence interval, never a single number.
- Control for the things that move selection for reasons unrelated to your product (mainly on-page position and option order) by randomizing and counterbalancing them.
- Record the engine, the model version, and the date beside every number you produce. An agent win rate without those three is not reproducible, and it stops being true the next time a model ships.
What to measure
The single number this protocol produces is your agent win rate: the share of agent shopping sessions, for a defined query, in which your product is the one the agent recommends or buys. It is a proportion between 0 and 1, measured over many sessions: not a search-style rank, not impressions, and not click-through. The metric and its honest edge cases are defined in full on agent win rate: definition and how to measure →; this page is the measurement machinery behind that definition.
Win rate is the right unit because it maps directly onto the choice a shopping agent actually makes. When a shopper asks an agent for "a quiet desk fan under $60," the agent returns a small set of picks, often one. Over many such sessions, the fraction of times your product is in that set is a signal you can track, compare across engines, and try to move. That is the whole of what agent selection comes down to from a merchant's seat: not where you appear in a list a human scrolls, but how often the agent chooses you.
Prompt set and query sampling
A win rate is only meaningful relative to a fixed set of prompts. Pre-register yours before any run, so the questions cannot be quietly reshaped to flatter a result.
- Query frame. Start from the real purchase intents your catalog competes for: the category, the constraints (price ceiling, size, use-case), and the phrasing a shopper types. Turn each frame into a small family of paraphrases, because agents respond to wording, and one phrasing is not the population.
- Prompt templates. For every query, hold a fixed template and vary only the shopper's words, so differences in win rate trace to the query, not to scaffolding you introduced.
- Sample size by target precision, not by a round number. Do not pick a prompt count for its looks. Choose the number of repetitions from the precision you need: enough independent sessions that the confidence interval around the win rate is narrower than the effect you care about detecting. Small effects need many more sessions than large ones, so state that target width in your sampling plan up front.
Engines and sessions
Run the same pre-registered prompt set against each shopping surface you cover (ChatGPT, Google's Gemini / AI Mode, and Perplexity) and treat each as its own measurement, never averaged into one "AI" number. The per-engine playbooks live at ChatGPT shopping →, Gemini shopping →, and Perplexity shopping →.
Design each session to be independent and reproducible:
- Fresh, uncontaminated sessions. Run every prompt in a clean session with no carried-over memory, personalization, or prior conversation, so one answer cannot prime the next.
- Recorded in full. Capture the complete response, the products named, and the order they appear in (not just whether you "won"), so the raw record can be re-scored later without re-running.
- Stamped with engine, model version, and timestamp. Because agent behavior is not stationary, tag every session with which surface, which model version (where exposed), and when. ACES reports that model updates can drastically reshuffle which products win, so any measurement is tied to a specific model version at a specific time.ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17) Any number without those three stops being reproducible the moment a model ships.
One evidence gap worth flagging: the academic study this protocol adapts did not test Perplexity. A live test can include Perplexity anyway, because it is a real shopping surface, but the simulation-derived expectations below transfer to it only as hypotheses, not as findings.
Controlling for position and order
The hardest part of measuring selection is separating "the agent chose your product" from "the agent chose the slot your product happened to sit in." That confound is not hypothetical.
ACES found that where a product sits on the page changes how often agents pick it (for one model, moving a product from the bottom-right corner to the top row raised its selection rate roughly fivefold), and that this position bias differs from model to model.ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17) The ACES numbers come from a controlled simulation rather than live stores, but the mechanism is exactly the nuisance you have to neutralize.
So do not let position ride along as a hidden variable:
- Randomize placement. Across sessions, randomize the order and, where the surface presents a grid, the grid position of the candidate set, so no product enjoys a permanent good seat.
- Counterbalance. Give every product comparable time in strong and weak positions, so position averages out rather than loading onto one item.
- Change one thing at a time. When you test whether a product-side change (a title, a description, a schema field) moves win rate, hold everything else in the candidate set fixed, so the change is the only plausible cause of a difference.
Statistics and error bars
Win rate is a proportion, so report it like one, with uncertainty attached, never as a bare percentage.
- Confidence intervals, not point estimates. Report each win rate with a confidence interval appropriate for proportions (for example a Wilson or Agresti-Coull interval), so a reader sees the precision, not just the number. A win rate of "1 in 3" from 12 sessions and from 1,200 sessions are not the same claim, and the error bar is what says so.
- Differences with their own uncertainty. When you compare two conditions (engine A versus engine B, or a rewritten description versus the original), report the difference in win rate and a confidence interval for that difference (bootstrapped over sessions), because a gap that fits inside the error bars is not a result.
- Pre-registration and multiple comparisons. Fix the hypotheses and the primary comparison before data collection, and when you test many queries or engines at once, adjust for multiple comparisons, so you do not mistake noise across dozens of tests for a finding.
- Effect size over significance theater. Lead with how big a shift is and how wide its interval, not with a p-value alone, because a "significant" but tiny move is not a lever a merchant should chase.
Reproducing ACES versus testing live catalogs
This protocol runs on two tracks with opposite trade-offs. Keep them separate and read each on its own terms.
Track 1: reproduce the simulation. You can re-run the audit inside the same kind of controlled environment the academic method uses. The ACES framework audits agent decisions by showing vision-language-model shopping agents a mock storefront in randomized trials and analyzing their picks with regression, a controlled simulation of agent choice, not a record of real-world purchases.ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17) The ACES simulator is released under the MIT license, which permits reproducing and extending its code with attribution.Spec-factACES repository (MyCustomAI) The strength of this track is clean causal control: every variable is yours to set. Its limit is realism: a mock storefront is not your catalog on a live engine, and a simulated pick is not a sale.
Track 2: test live catalogs. Run the pre-registered prompts against the real ChatGPT, Gemini, and Perplexity shopping surfaces, scoring win rate on live responses. The strength here is realism: these are the agents shoppers actually use. Its limits are noise and non-stationarity: live engines personalize, rate-limit, and change under you, so live win rates carry wider error bars and shorter shelf lives than simulation results.
Neither track alone is the truth. Simulation says what can move a decision under control; live testing says what is moving on the surfaces that matter this month. Running both is more work than most teams will do at first, and a disciplined live test alone still beats an undisciplined one.
Limitations
Distrust the right things rather than trust the wrong ones.
- Simulation is not sales. Track-1 results describe agent choices in a controlled sandbox, not revenue on your store. Never translate a simulated selection shift into a promised conversion or dollar figure.
- Live results expire. Because model updates reshuffle outcomes, a live win rate is a snapshot of one engine at one time. Date every result you record, and treat an old one as a historical record, not current guidance.
- A sample is not the market. A pre-registered query set and a set of test catalogs cannot represent every shopper intent or every store. Report the scope you tested and resist generalizing past it.
- You see outputs, not the ranking. You measure what an agent chose, not the internal logic behind the choice. Explanations for why a product won belong in a hypothesis, never in a measured mechanism.
- Correlation guardrails. Establishing that a specific product-side change caused a win-rate move needs a controlled A/B within this protocol, not a before-and-after on a shifting engine. Report uncontrolled comparisons as suggestive, not causal.