ChatGPT’s product recommendations changed 80.2% when search was enabled, according to a study of 20,000 responses by Jeff Oxford, founder and CEO of Visibility Labs.
Oxford tested 1,000 product-recommendation prompts 10 times each with ChatGPT search enabled and 10 times with search disabled.
Only 19.8% of products recommended without search also appeared in recommendations generated with search enabled.
Search changed top picks. Even products ChatGPT recommended most often without search rarely carried over. Among products that appeared in 100% of search-disabled responses, only 15.8% also appeared when search was enabled.
- Oxford expected the most consistently recommended products to remain common when search was turned on. Instead, that group had the lowest overlap.
Source mentions tracked visibility. The study also examined whether products mentioned in ChatGPT’s cited sources appeared more often in its recommendations. Oxford reported a 0.4 Pearson correlation between cited-source mentions and recommendation frequency.
- The study measured recommendation frequency with a “Visibility Score,” defined as the percentage of runs in which a product appeared for a given prompt. Products mentioned more often in cited sources tended to have higher Visibility Scores.
- The analysis didn’t establish that cited-source mentions caused products to be recommended.
Search narrowed recommendations. ChatGPT responses with search enabled contained an average of 5.2 products, compared with 6.2 when search was disabled.
- Across 10 runs of each prompt, ChatGPT returned an average of 19 unique products per prompt with search enabled and 21.8 with search disabled.
Why we care. Search changed which products ChatGPT recommended, including products it named every time when web access was disabled. The findings suggest products appearing in cited sources may receive greater visibility when search is enabled, though the study does not determine whether cited-source visibility matters more than broader web visibility.
About the data. Oxford analyzed 1,000 product-recommendation prompts, running each 10 times with search enabled and 10 times with search disabled. Product names were standardized so naming variations counted as the same product. Because the study was observational, it did not establish a causal relationship between cited-source mentions and recommendation frequency.
Dig deeper. AI recommendation lists repeat less than 1% of the time: Study
Search Engine Land is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.
