A good demo is a magic trick: the hard parts happen offstage. Here are the three questions I ask before anyone shares their screen — because the answers, not the demo, predict what you’ll actually live with.
1. “What does this look like on my worst data?”
Every demo runs on the cleanest possible input. Ask to run it on a real, messy sample of yours — live, in the call. Watch whether the confidence holds when the data gets ugly, because your data is always ugly.
2. “Where does a human stay in the loop, and why there?”
If the answer is “nowhere,” they’re selling you risk. If it’s “everywhere,” they’re selling you a very expensive intern. The interesting answer names specific decision points and explains the reasoning.
3. “How would I know if this quietly got worse?”
Models drift, data shifts, prompts rot. A serious vendor has an answer about monitoring and evals. A demo-stage vendor changes the subject.
The demo shows you the best day. These questions show you the median one.