Key takeaways
- We run the lab to do two jobs no vendor or official protocol doc does: translate rigorous selection research into a merchant playbook, and track how each engine's choices change over time.
- Our academic anchor is the ACES study, which audits AI shopping agents by showing them a mock storefront in randomized trials; its simulator is MIT-licensed, so we can build on it and recommend it with attribution.
- Every claim on this site is typed (spec-fact, reported, or hypothesis), and our own experiments will only ever be typed 'measured' once real data exists in the repository.
- No numbers are invented. Until a data-collection run completes, every results area on this site reads 'Data collection in progress' rather than a placeholder figure.
- Agent behavior shifts with each model update, so anything we publish will be a dated snapshot, not a fixed rule.
Why we run our own tests
Most of what is written about "getting selected" by AI shopping agents is either an official eligibility doc (which tells you how to become listable, not how to win) or a vendor funnel that frames every problem as one its product solves. The rigorous, disinterested knowledge about why an agent prefers one product over another lives somewhere less convenient.
The strongest evidence on how agents actually choose sits in academic preprints, not in anything a merchant can act on directly; no vendor or official protocol doc translates that research into "what to change on your store." Our first job is that translation: cite the source, teach the implication, and label our own extrapolations plainly.Hypothesis (our analysis)The second job follows from a property of the research itself. Agent selection is not a fixed algorithm you can reverse-engineer once and be done.
Because selection behavior is model-dependent and can shift when a model is updated, a one-time audit goes stale quickly. So our second job is longitudinal: track how ChatGPT, Gemini, and Perplexity choose differently, and what changes from one model release to the next, rather than publishing a single "how AI ranks products" verdict and letting it rot.Hypothesis (our analysis)Together those two jobs are the practitioner layer this handbook occupies: the selection chapter explains the mechanics, the ACO guide frames the strategy, and the lab is where we test claims instead of asserting them.
The academic anchor: ACES
We do not start from scratch. Our methodology is grounded in a published, reproducible study.
ACES is a provider-agnostic framework for auditing how AI shopping agents make decisions: vision-language-model agents are shown a mock storefront in randomized trials, and their product selections are analyzed statistically. A central finding is that selection behavior is model-dependent and can reshuffle when a model is updated, which undermines any universal notion of a single "top" result across engines.ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17)Read that as a controlled simulation across a fixed set of models, not a record of real-world purchases, a distinction we carry into every citation. What ACES gives the lab is a rigorous, transparent starting point rather than a set of numbers to repeat.
The ACES simulator's code is published on GitHub under the MIT license.Spec-factACES simulator, github.com/mycustomai/ACES That license is why we can build on the harness, link it, and recommend it to readers with attribution instead of reinventing a storefront-simulation stack of our own.
Our principles: typed claims, open method, no fake numbers
The lab exists to be citable, and citability comes from discipline, not confidence. Three rules govern everything here.
- Every claim is typed. We label each factual statement as
spec-fact(in an official document, we link the primary source),reported(a third party found it, we name and date the source),hypothesis(our own inference, phrased as such), ormeasured(our own experiment, used only when real data exists). Nothing ships as an unqualified assertion. - The method is open before the results are. Each experiment publishes its full methodology (the prompts, the sampling plan, how we control for position and order, and the error bars) so anyone can reproduce it and check our arithmetic. The methodology is a page you can read today, well before any of our data lands.
- No number is invented. Until a collection run genuinely completes, every results area reads "Data collection in progress." We never fill a gap with a plausible-looking figure, and any illustrative example is explicitly labeled as illustration, not measurement.
The experiments
The following are live in methodology and collecting data. Each links to its own page, and each shows its results in an honest in-progress state until real data exists.
How we test agent selection (methodology)
The reusable protocol behind everything else: a repeatable way to measure your own agent win rate across ChatGPT, Gemini, and Perplexity, with the sampling plan and error bars spelled out. Read it in full on the methodology page.
Methodology results: data collection in progress
Do description rewrites shift AI recommendations?
A field test that takes a reported ACES result (that query-conditional description rewrites can move a product's recommendation share) and replicates it on live catalogs. The design, and what a "no change" outcome looks like, are on the description-rewrite experiment page.
Rewrite field-test results: data collection in progress
The Agent Selection Index
A quarterly, controlled benchmark of how much a given product-side change moves an agent's choice, re-measured each quarter across the official Claude, OpenAI, and Gemini model APIs. Its full methodology, the one-factor-at-a-time design, the model tagging, and the statistics, is published on the Agent Selection Index page; measured values appear there only after a run is generated in the lab and author-verified.
Agent Selection Index: data collection in progress
Data collection in progress
The lab is new and deliberately empty of results. That is the honest state, not a placeholder: publishing methodology first and numbers only when they are real is the whole point.