Can AI predict what customers will do? ChatGPT, Claude and Gemini vs 11 live-market experiments
Key findings
- →In a Heatseeker study of 11 live-market ad experiments, ChatGPT, Claude and Gemini picked the winning customer need 4, 3 and 1 times out of 11 — 8 of 33 predictions.
- →All three models were wrong on 6 of the 11 tests, and chose the same wrong answer on 4 of them.
- →In the men's skincare test, all three models chose the angle that ranked last of eight on both click-through rate and Heatseeker's engagement score.
- →For reference, picking at random would match about 21% of the time, given 3 to 9 options per test.
The question
AI models are increasingly used to predict what customers want before a brand commits budget. Can they pick the customer need that actually wins with real buyers?
Method
- Experiments: 11 completed live advertising experiments run on Meta, across four categories: food delivery, wagering, men's skincare and buy-now-pay-later finance. Each tested several customer "jobs to be done" as ad angles for a specific audience. These were selected cases, not a random sample.
- Design: Retrospective. The models predicted after the experiments ran, without seeing results.
- Prompt: Each model received one identical prompt with every test's audience description and lettered ad angles, in the same order, and chose one option per test. Runs were on the same day in August 2026, in fresh chats without memory.
- Models: ChatGPT (Pro subscription, Instant tier), Claude Opus, Gemini 3.1 Pro.
- Winner: The top variant in Heatseeker's platform results for that test's audience. It is measured by Heatseeker's Buyer Engagement Score, which weights engagements, clicks, form opens and leads, with leads weighted most heavily.
- Scoring: 1 for matching the recorded winner, 0 otherwise. No partial credit.
Results
| Test | Category | Options | ChatGPT | Claude | Gemini |
|---|---|---|---|---|---|
| 1 | Food delivery | 4 | ✗ | ✗ | ✗ |
| 2 | Food delivery | 4 | ✗ | ✓ | ✗ |
| 3 | Food delivery | 3 | ✓ | ✓ | ✗ |
| 4 | Food delivery | 5 | ✗ | ✗ | ✗ |
| 5 | Food delivery | 5 | ✓ | ✗ | ✗ |
| 6 | Wagering | 4 | ✓ | ✓ | ✓ |
| 7 | Wagering | 4 | ✗ | ✗ | ✗ |
| 8 | Wagering | 4 | ✗ | ✗ | ✗ |
| 9 | Men's skincare | 8 | ✗ | ✗ | ✗ |
| 10 | Buy-now-pay-later | 9 | ✓ | ✗ | ✗ |
| 11 | Buy-now-pay-later | 9 | ✗ | ✗ | ✗ |
| Total | 4/11 (36%) | 3/11 (27%) | 1/11 (9%) |
Combined, the three models matched 8 of 33 recorded winners (24%), against roughly 21% expected from random picks.
What this means
Language models predict what sounds right, not what people do.
Treat every AI-generated customer insight as an assumption:
- Label your assumptions.
- Identify the ones that matter most.
- Test those against observable behavior.
- Match the evidence to the claim. Clicks, leads and purchases are different signals.
Context from earlier research
- A meta-analysis of 47 experimental tests found that changes in intention produced smaller changes in behavior (Webb & Sheeran, 2006).
- In political survey research, 48% of regression coefficients from ChatGPT-generated responses differed significantly from their human-survey counterparts (Bisbee et al., Political Analysis).
Limitations
- Small, selected sample.
- Retrospective design.
- Some winners are measured within a demographic subgroup.
- Winners are determined by Heatseeker's platform score, and confidence levels vary between tests.
- The prompt described audiences and angles, not finished ads.
- Results describe this protocol, not every use of AI in marketing.
Frequently asked questions
Can ChatGPT predict which ad message will win?
In this study, ChatGPT picked the winning customer need in 4 of 11 live-market experiments. Claude picked 3 and Gemini 1. Across all three, 8 of 33 picks matched the recorded winner.
How accurate are AI models at predicting customer behavior?
Across 33 predictions, the models matched the recorded winner 24% of the time, close to the roughly 21% expected from random picks. All three were wrong together on 6 of 11 tests.
What is the difference between a language model and a behavioral model?
A language model learns from what people write and say. A behavioral model learns from what people actually do, such as clicks, sign-ups and purchases. That's why stated preferences and AI predictions can diverge from real choices.
Why do AI models agree on the wrong answer?
The three models chose the same wrong option on 4 of 11 tests. Agreement between models is not evidence. They are trained on similar language, so they can share the same blind spots.
How should marketers use AI predictions?
As hypotheses, not answers. Identify the assumptions that matter most and test them against real buyer behavior before committing budget.
How was this study run?
Three models each received the same prompt describing 11 completed Meta ad experiments: the audiences and the ad angles tested, but not the results. Each picked one winner per test, and picks were scored against Heatseeker's recorded results.
How to cite
O'Keeffe, K. (2026). Can AI predict what customers will do? ChatGPT, Claude and Gemini vs 11 live-market experiments. Heatseeker. https://www.heatseeker.ai/research/can-ai-predict-customer-behavior
Coverage
- Forbes Technology Council: A Language Model Is Not A Behavioral Model · Oct 8, 2026Kate O'Keeffe on why AI-generated customer predictions need behavioral evidence, drawing on a Heatseeker study of ChatGPT, Claude and Gemini against 11 live-market ad experiments.
Data
The results table above as a CSV: 1 for a pick that matched the recorded winner, 0 otherwise. Licensed CC BY 4.0.
Download CSV →