Key takeaways
- The premise is borrowed, not ours: the ACES simulation found that rewriting a product's description to fit the shopper's query shifted which product agents picked: on average a small move, but occasionally a decisive one.
- The effect was lopsided. In roughly two-thirds of the category-model pairs a one-shot rewrite changed nothing; almost all of the payoff lived in the remaining third.
- The paper's current version tests six models, the original three plus three newer successors (Claude Opus 4.5, GPT-5.1, Gemini 3.0 Pro Preview), and "office lamp" is the only product category with a positive, significant gain across all six, from +7.1 to +80.4 percentage points.
- Our field test replicates that finding on live catalogs across ChatGPT, Gemini, and Perplexity. That is a hypothesis about real stores, not a re-run of the simulation, and it has no results yet.
- We isolate the rewrite by changing only the description, holding title, price, rating, availability, image, and on-page position constant, then measuring agent win rate before versus after with the same prompts and sessions.
- Because most rewrites do nothing and a few do a lot, the honest output is a distribution: not a single 'rewrite = +X%' number. Until real data lands, results stay in a labelled data-collection state.
The hypothesis, borrowed from ACES
The most cited public evidence that changing product text moves an AI shopping agent's choice does not come from us: it comes from the ACES study of agent selection bias. That study included a "seller-side agent" step: an LLM acting on the seller's behalf rewrote each product's description to better fit the shopper's query, and the researchers measured whether the rewritten product then won more of the agents' picks.
Our experiment starts one step past that result. We take the ACES finding as a premise and ask the question a merchant actually has: does the same lever work on a live catalog? We expect a query-aligned description rewrite to move agent selection on real stores the way it moved selection inside the ACES simulation, but this is an inference from a controlled experiment to the open web, not a measured result, so it stays a hypothesis until our own data supports or refutes itHypothesis (our analysis). Everything below is built to test that one claim honestly, including the ways it could turn out to be wrong.
What ACES actually found
The premise rests on numbers reported by the ACES paper, none measured by us. Read them as evidence that text can move selection under controlled conditions: not as a conversion rate you can bank on your own catalog. The current version of the paper tests this on six models: the original three (Claude Sonnet 4, GPT-4.1, Gemini 2.5 Flash) plus three newer successors added in a later revision (Claude Opus 4.5, GPT-5.1, Gemini 3.0 Pro Preview).
In the ACES simulation, an agent rewriting a product's description to better fit the shopper's query raised that product's market share in the agents' picks by an average of +3.66 percentage points for Claude Sonnet 4 (standard error 1.33), +8.37 pp for GPT-4.1 (s.e. 1.31), and +14.79 pp for Gemini 2.5 Flash (s.e. 1.36), all statistically significantReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17). Using GPT-5.1 as the seller agent instead, the same rewrite raised market share by an average of +7.38 pp for Claude Opus 4.5 (s.e. 1.00, significant), +14.89 pp for GPT-5.1 (s.e. 1.05, significant), and +0.32 pp for Gemini 3.0 Pro Preview (s.e. 1.22, not significant)ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17). Five of the six averages are statistically significant, and even so, that is the first honest caveat: the size of the effect swings by more than 40x across models, from Gemini 3.0 Pro Preview's near-zero average to Gemini 2.5 Flash's +14.79 pp.
Those averages hide a lopsided distribution: across the six models, 67% of category-model pairs saw no statistically significant change, while 33% produced large gains, the effect is concentrated, not uniformReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17). And where it did land, the upside could be dramatic: named category outliers include +25.8 pp for an office lamp (Claude Sonnet 4); +7.1 pp for a fitness watch, +30.3 pp for an office lamp, and +16.6 pp for toothpaste (GPT-4.1); +6.5 pp for a mousepad, +52.0 pp for an office lamp, +29.1 pp for toothpaste, and +12.0 pp for a washing machine (Gemini 2.5 Flash); +41.0 pp for an office lamp (Claude Opus 4.5); and +80.4 pp for an office lamp (the single largest effect in the paper), +9.5 pp for a stapler, and +15.5 pp for toilet paper (GPT-5.1)ReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17). Not every large move was a gain: a stapler rewrite cost Claude Opus 4.5 -3.0 pp, and a mousepad rewrite cost Gemini 3.0 Pro Preview -12.5 ppReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17), a reminder that a query-aligned rewrite is not a one-way bet.
One category stands out from all the rest. "Office lamp" is the only product category with a positive, statistically significant gain across all six models tested, ranging from +7.1 pp to +80.4 ppReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17). Consistency across three model generations and two different seller agents is the closest thing in the paper to a universal result, and it is exactly why the caveats below still hold: one category behaving this consistently does not make the average rewrite a sure thing on the rest of a catalog.
Two caveats travel with every one of those figures. First, ACES is a controlled simulation: vision-language-model agents choosing from a mock storefront in randomized trials, not a record of real purchases on live stores. Second, the numbers are model-specific and were observed on a fixed set of six model versions (Claude Sonnet 4, GPT-4.1, and Gemini 2.5 Flash from the original test, plus Claude Opus 4.5, GPT-5.1, and Gemini 3.0 Pro Preview added in the current revision); agent behaviour reshuffles when those models update. Both are reasons to replicate rather than to quote the simulation as if it were your catalog's forecast. We re-checked the arXiv listing on 2026-07-08: v3 (2025-12-17) is still the current version of this actively revised paper, and every figure above was verified against it.
Our field-test design
The field test moves the same manipulation out of the mock storefront and onto real product pages, measured through the agents shoppers actually use. In outline:
- Engines. We query ChatGPT's shopping surface, Google's Gemini / AI Mode, and Perplexity, because ACES showed selection behaviour differs by model: a rewrite that helps on one may do nothing on another, so a single-engine result would be misleading.
- Query set. A fixed set of realistic shopper prompts per product category, written before we see any results and held constant across the before and after rounds.
- Unit of analysis. The product-model pair, matching how ACES reported its distribution, so our "share that moved" is comparable to its ~33%.
- Metric. We track agent win rate (how often a given product is recommended across repeated, independent sessions) before the rewrite and after it, and report the change with error bars rather than a single point estimate.
The full protocol (prompt wording, sampling depth, session hygiene, and how we compute error bars) lives in the research methodology → rather than being restated here, so every experiment on this site shares one reproducible measurement standard.
How we isolate the rewrite effect
An honest before/after test has to rule out the possibility that something other than the description caused the change. So the description is the only field we touch. Title, price, star rating, review count, availability, image, and the product's position in any list or grid are all held constant between the two rounds.
Position matters most as a confound. In the same ACES simulation, grid position alone was a powerful lever: for Claude Sonnet 4, moving a product from the worst corner to the top row raised its selection rate roughly fivefoldReportedACES, Allouah et al., arXiv:2508.02630 (2025-12-17). If we let position drift between rounds, we could easily mistake a position swing for a rewrite effect. So across repeated sessions we sample enough independent runs to average position out, and we compare like with like. Holding every non-text field fixed and varying only the description is what lets us attribute any measured change in win rate to the rewrite itself rather than to price, rating, or placement, provided the sampling is deep enough to wash out session-to-session noiseHypothesis (our analysis).
What "no change" looks like
The most likely single outcome for any one product is nothing happens, and the methodology has to treat that as a real result, not a failed test. ACES saw no significant change in roughly two-thirds of its pairs, so a field test that reports a clean "+X% across the board" would be more suspicious than reassuring: it would mean we had smoothed over exactly the lopsidedness the source finding is about.
That shapes how we will report. Rather than one headline percentage, the deliverable is a distribution: the share of product-model pairs that moved beyond the noise band, the size of the moves that did land, and how that split differs by engine. For a merchant, the practical read is that a description rewrite is a skewed bet: usually neutral, occasionally decisive, and impossible to call in advance for a specific product. The same skew is why we point readers toward testing rather than trusting a formula: see how the identical query-matching logic applies one field over, in product titles that AI agents match →.
Illustrative example
Results
The measurement is designed and reproducible; the data is not in yet. No numbers appear below until real results exist in our dataset, and any figures we do publish will be typed as measured only when they are backed by that data.
Data collection in progress
This experiment sits inside the wider AgentMint.net research lab → and feeds the catalog playbook it tests against: start from how AI shopping agents choose products → for the full set of signals a description competes within.