Visa's shopping test: AI agents got the purchase right about 57 percent of the time

Personal AI agents are being asked to shop, and a Visa experiment described by Fortune on Thursday found they often fail. Visa instructed 22 agents to buy three everyday items, a television, a laptop and a hair appliance. Fortune reported an overall success rate of about 57 percent. The results sit on a Visa site that shows 15 shopping trips for each agent.

Fortune said the agents built on newer models did better. An Aragon 4 version of Google's Gemini and a Wonder version of Meta's Muse each completed a purchase 82 percent of the time. A Claude agent was third at 76 percent. An OpenAI agent finished sixth at 64 percent. Fortune said the five most expensive models succeeded on average 67 percent of the time, and the cheapest five scored 50 percent. For the television, the shopper was given attributes such as a child's gaming console, a 30-day return window and Dolby Vision, and did not volunteer all of them. Success meant buying a set that matched. Many agents bought the wrong product or suggested nothing suitable.

Visa's global head of growth, Rubail Birwadker, told Fortune the industry is not yet at a point where people can trust an agent with the purchase, and that discovery is only one piece of agent shopping. Fortune also reported that Visa opened a site where people can try the agents used in the test, and that liability for a bad or fraudulent agent purchase is still unsettled.