Can AI shopping agents buy from online stores?

All 500 stores

Overall, about a third of stores stopped the agent

Of the 469 stores Claude could be scored on, 175 (37%) stopped it before payment and 272 (58%) let it reach payment; GPT and Gemini were stopped at the same rate (36%). When a store let it in, it got there 93% of the time (272 of 294). Most stops came at the front door: 129 of 175 on the homepage or first click, only 12 at checkout. How often a store stopped it depends on how popular the site is, so the 37% describes this mix of well-known stores, not every online store. The stores people visit most are the ones most likely to stop an agent.

Part 1 · Where does it break?

At the store. Shopify stores rarely stop it

When the store let Claude in, it reached payment 151 times out of 160. So the question is which stores stop it. Share of stores that stopped the agent before payment (a bot check, a block page, or a quiet failure such as an empty bag), grouped by how popular the site is. Among sites of similar popularity, Shopify stores rarely stop it and other platforms usually do (below 200,000 there are too few stores off Shopify to tell). It isn't just bot protection either: Shopify stores running a bot manager like Akamai or Cloudflare mostly let the agent in (3 of 29 stopped it), while stores elsewhere running one stopped it 146 times out of 191, and stores elsewhere with no bot manager detected still stopped it 21 times out of 60. A store counts as Shopify when its checkout runs on Shopify, including 11 stores with their own storefront in front of it.

Part 1 · Shopify's shortcuts for agents

Shopify's agent tools cut the work by half or more

Most Shopify stores now register WebMCP tools in the browser (search the catalog, add to cart, go to checkout): 97 of 124 Shopify stores had them, 1 of 126 stores elsewhere. Each AI shopped the same Shopify store twice: once by clicking, once using the tools. Median actions to reach payment, both runs successful.

16 to 20 stores per AI. Success held or rose with tools, and cost moved by a few cents. Shopify also publishes a server-side agent checkout, UCP, on 197 of 222 of its stores (8 of 278 elsewhere); the website runs above didn't use it. A script run through it on every store that publishes one built a checkout at 195 of 197 Shopify stores and 1 of 8 elsewhere, and every one stopped at payment: the store hands the shopper a link to finish, because paying takes a credential only an approved agent gets. Claude, GPT and Gemini shopping through it themselves reached payment at 40, 38 and 40 of 40 Shopify stores in a median of 5, 7 and 3 steps. Private connections like ChatGPT's checkout through ACP, developed by OpenAI and Stripe, don't run through the website and aren't measured here; no store advertises ACP to outside agents (0 of 500 serve its proposed discovery file).

Part 2 · Can the results be trusted?

The checks behind the numbers

• It's automated browsing, not the connection. 32 of 33 stores that blocked the automated browser loaded normally in an everyday Chrome on the same home connection. That check changes the profile, behaviour and automation at once, so it shows how the agent browses triggers the block, not which part.
• It repeats. 40 stores rerun a day later gave the same result, 40 of 40. Two weeks later, 236 of 241 Part 1 stores made the same call; the changes went both ways (DSW and Newegg started blocking, six stores including Patagonia and Samsung stopped).
• It isn't the network. 168 stores rerun over a phone hotspot, every store that blocked from home and as many that let the agent in: 165 made the same call, and the three changes went both ways.
• Outcomes come from the page, never the agent. Every run was reviewed against its screenshots and step log by AI reviewers working to written rules, and I hand-checked 592 of the 1,104 Claude, GPT and Gemini runs, including a random 150 and every borderline case; one label changed. It mattered: GPT and Gemini claimed to reach payment when they hadn't in 7% and 10% of their runs, usually because the address was half typed. 64 of the 500 Jev runs were also checked by hand, and all 64 labels stood.
• Popularity is measured, not guessed. Each store's rank on the Tranco list of the top 1 million websites, which anyone can check.

Part 3 · Does it depend on the AI?

No. Stores treat every AI almost exactly alike

Study 3 sent all three AIs to the same 250 new stores. Of the 201 stores all three could be scored on, 195 gave every AI the same result: 81 stopped all three, 114 stopped none. The six exceptions came from how far each agent got, not from a store treating one AI differently. On October 19 and 20 we sent GPT and Gemini to the original 250 stores too: 218 of 226 gave all three AIs the same result, so across both halves it is 413 of 427.

Paying

The routes built for agents stop at payment

At Shopify stores that offer UCP, the open checkout route for agents, the agent could set up a checkout almost every time (195 of 197), but each one stopped before payment and asked for a person or a card. As online stores work today, a store has no way to confirm that an agent is allowed to pay for the buyer.

What the agent pays for

Stores add charges nobody chose

When a paid add-on was on the page at payment, it ended up in the cart 29% of the time (47 of 162 runs): package protection, returns coverage, a protection plan or a membership (median $4.19, up to $150 a year). In 43 the store had ticked it in advance and the agent left it on; the agent added one itself only 4 times. None were required. Floyd, for example, adds a $150 a year membership to every cart by default and lets the buyer remove it; two different agents left it in. Agents did turn add-ons down 44 times, but not reliably: at 17 stores one agent left an add-on on and another did not.

Saying done

An agent saying "done" doesn't mean it's done

GPT and Gemini said they had reached payment when they hadn't in 7% and 10% of their runs (15 of 220 and 21 of 215), most often at a half-filled address form; Claude did so in 1% (3 of 225). Every result here is scored from the page, so these runs count as not reaching payment.

Part 4 · Does a cheap web-action model change the picture?

Stores treat it the same. The agent is the weak link

TypeSafe's Jev (jev-1.13.0), a small model built for web actions, ran the same 250 stores as Part 3, once on its own and once with Claude Haiku 4.5 taking any step where Jev's confidence fell below 0.4. Where Claude, GPT and Gemini all got the same result, Jev got it too at 181 of 191 stores (184 of 193 with Haiku). Almost every difference is Jev failing before it reached the page where the store stopped the others: it looped on the same click, left size pickers unset, or typed a garbled city. Jev alone failed on its own at 82 stores, against 13 for Claude. On the 140 stores that let both Jev runs in, Haiku stepping in raised payment from 70 to 85 (25 gained, 10 lost, paired p = 0.017) at about a third of Claude's cost per store.

Jev ran on October 11 from a Windows PC on the same home connection. Its runs were reviewed against screenshots by AI reviewers under the Part 3 rules; a second blind review of 126 gave the same label for 122. 64 of the 500 were then checked by hand, and every label stood.

What a session looks like

Three runs, step by step

Stitched from the screenshot saved at every step. Captions show what the agent did next.

A sample of the data

Twelve stores, three AIs

A small slice of the 500-store dataset: four Shopify stores every AI got through, four stores that walled every AI, two that failed silently for all three, and two where the AIs did differently. Every run was checked against its screenshots.

Download the sample (CSV)Request the full dataset

The full dataset (every run, every step, screenshots) is shared on request.