# The AgentMint.net Research Lab

> The AgentMint.net Research Lab runs independent, reproducible tests of what actually moves agent selection: methodology public, results labeled 'Data collection in progress' until real data exists.

<!-- Independent, reproducible tests of what moves agent selection: methodology public, results labeled 'Data collection in progress' until real. -->
<!-- Canonical: https://agentmint.net/research/ · By Abdalsalaam Halawa · Published 2026-07-07 -->
<!-- Evidence tags below are load-bearing: <Claim type="spec-fact|reported|hypothesis"> marks how each factual claim is sourced. -->

<KeyTakeaways
  items={[
    "We run the lab to do two jobs no vendor or official protocol doc does: translate rigorous selection research into a merchant playbook, and track how each engine's choices change over time.",
    "Our academic anchor is the ACES study, which audits AI shopping agents by showing them a mock storefront in randomized trials; its simulator is MIT-licensed, so we can build on it and recommend it with attribution.",
    "Every claim on this site is typed (spec-fact, reported, or hypothesis), and our own experiments will only ever be typed 'measured' once real data exists in the repository.",
    "No numbers are invented. Until a data-collection run completes, every results area on this site reads 'Data collection in progress' rather than a placeholder figure.",
    "Agent behavior shifts with each model update, so anything we publish will be a dated snapshot, not a fixed rule.",
  ]}
/>

## Why we run our own tests

Most of what is written about "getting selected" by AI shopping agents is either an official eligibility doc (which tells you how to become *listable*, not how to *win*) or a vendor funnel that frames every problem as one its product solves. The rigorous, disinterested knowledge about *why* an agent prefers one product over another lives somewhere less convenient.

<Claim type="hypothesis">The strongest evidence on how agents actually choose sits in academic preprints, not in anything a merchant can act on directly; no vendor or official protocol doc translates that research into "what to change on your store." Our first job is that translation: cite the source, teach the implication, and label our own extrapolations plainly.</Claim>

The second job follows from a property of the research itself. Agent selection is not a fixed algorithm you can reverse-engineer once and be done.

<Claim type="hypothesis">Because selection behavior is model-dependent and can shift when a model is updated, a one-time audit goes stale quickly. So our second job is longitudinal: track how ChatGPT, Gemini, and Perplexity choose differently, and what changes from one model release to the next, rather than publishing a single "how AI ranks products" verdict and letting it rot.</Claim>

Together those two jobs are the practitioner layer this handbook occupies: the [selection chapter](/how-ai-agents-choose-products/) explains the mechanics, the [ACO guide](/agentic-commerce-optimization/) frames the strategy, and the lab is where we test claims instead of asserting them.

## The academic anchor: ACES

We do not start from scratch. Our methodology is grounded in a published, reproducible study.

<Claim type="reported" source="ACES, Allouah et al., arXiv:2508.02630" date="2025-12-17" href="https://arxiv.org/abs/2508.02630">ACES is a provider-agnostic framework for auditing how AI shopping agents make decisions: vision-language-model agents are shown a mock storefront in randomized trials, and their product selections are analyzed statistically. A central finding is that selection behavior is model-dependent and can reshuffle when a model is updated, which undermines any universal notion of a single "top" result across engines.</Claim>

Read that as a controlled *simulation* across a fixed set of models, not a record of real-world purchases, a distinction we carry into every citation. What ACES gives the lab is a rigorous, transparent starting point rather than a set of numbers to repeat.

<Claim type="spec-fact" source="ACES simulator, github.com/mycustomai/ACES" href="https://github.com/mycustomai/ACES">The ACES simulator's code is published on GitHub under the MIT license.</Claim> That license is why we can build on the harness, link it, and recommend it to readers with attribution instead of reinventing a storefront-simulation stack of our own.

<Callout variant="note">
Agent behavior changes with every model update. Last verified: 2026-07-07. Any finding we publish will name the models and dates it was measured against, because a result that held last quarter may not hold after the next release.
</Callout>

## Our principles: typed claims, open method, no fake numbers

The lab exists to be *citable*, and citability comes from discipline, not confidence. Three rules govern everything here.

- **Every claim is typed.** We label each factual statement as `spec-fact` (in an official document, we link the primary source), `reported` (a third party found it, we name and date the source), `hypothesis` (our own inference, phrased as such), or `measured` (our own experiment, used *only* when real data exists). Nothing ships as an unqualified assertion.
- **The method is open before the results are.** Each experiment publishes its full methodology (the prompts, the sampling plan, how we control for position and order, and the error bars) so anyone can reproduce it and check our arithmetic. The [methodology](/research/methodology/) is a page you can read today, well before any of our data lands.
- **No number is invented.** Until a collection run genuinely completes, every results area reads "Data collection in progress." We never fill a gap with a plausible-looking figure, and any illustrative example is explicitly labeled as illustration, not measurement.

## The experiments

The following are live in methodology and collecting data. Each links to its own page, and each shows its results in an honest in-progress state until real data exists.

### How we test agent selection (methodology)

The reusable protocol behind everything else: a repeatable way to measure your own [agent win rate](/glossary/#agent-win-rate) across ChatGPT, Gemini, and Perplexity, with the sampling plan and error bars spelled out. Read it in full on the [methodology page](/research/methodology/).

<DataSlot
  state="collecting"
  methodologyHref="/research/methodology/"
  title="Methodology results: data collection in progress"
  description="The protocol is published and reproducible. Measured win-rate results will appear here once a collection run completes; no numbers are shown until then."
/>

### Do description rewrites shift AI recommendations?

A field test that takes a reported ACES result (that query-conditional description rewrites can move a product's recommendation share) and replicates it on live catalogs. The design, and what a "no change" outcome looks like, are on the [description-rewrite experiment page](/research/description-rewrite-experiment/).

<DataSlot
  state="collecting"
  methodologyHref="/research/description-rewrite-experiment/"
  title="Rewrite field-test results: data collection in progress"
  description="The experiment design is published. Measured effects on recommendation share will appear here once data lands; the premise is a reported finding, not our own measurement yet."
/>

### The Agent Selection Index

A quarterly, controlled benchmark of how much a given product-side change moves an agent's choice, re-measured each quarter across the official Claude, OpenAI, and Gemini model APIs. Its full methodology, the one-factor-at-a-time design, the model tagging, and the statistics, is published on the [Agent Selection Index page](/research/agent-selection-index/); measured values appear there only after a run is generated in the lab and author-verified.

<DataSlot
  state="collecting"
  methodologyHref="/research/agent-selection-index/"
  title="Agent Selection Index: data collection in progress"
  description="The Index methodology is published and reproducible. Measured, per-model index values appear on its page only after a run is generated in the lab and author-verified; no numbers are shown until then."
/>

## Data collection in progress

The lab is new and deliberately empty of results. That is the honest state, not a placeholder: publishing methodology first and numbers only when they are real is the whole point.

<DataSlot state="collecting" methodologyHref="/research/methodology/" />