Watching People Interrogate a Machine
How an AI assistant turned out to be a better research instrument than another interview question.
Ask someone what they think and they will tell you who they would like to be. Watch what they do and you meet who they are.
“Ask someone what they think and they will tell you who they would like to be. Watch what they do and you meet who they are.” THE METHOD IN ONE LINE
On a research project for a small-business finance client, the most useful material in the sessions was not anything anyone said about the product. It was watching them put questions to an AI about the category instead. The setup was plain. Rather than only asking participants what they thought, we handed them a general AI assistant and a task: ask it about the space the client plays in, payment processing, payroll, bookkeeping, the client's product against the alternatives. Then we mostly stayed quiet and watched. The assistant was never the subject. It was a mirror held up to the participant, and what it reflected was more candid than anything a direct question would have produced.
The blind spot this was working around
Interviews have a well-known weakness: people give you the answer they think you want. Ask whether price matters and you get a composed little speech about value. Ask what they would do and they describe a more rational person than the one in the chair. None of it is dishonesty. A direct question simply invites a narrated self, and the narrated self is always tidier than the real one.
Hand someone an AI and a task, and most of that performance falls away. When a person is busy trying to extract a useful answer from a machine, they stop performing for you and start revealing themselves in how they go about it, which they do not curate, because they do not realise it is the data.
What you can read from how someone prompts
A surprising amount.
Their vocabulary betrays their fluency. "What's the cheapest card reader" and "what are the effective rates including interchange" come from two different mental models, and neither person told you which one they had. The words they reach for are a map of how they think about the category.
Their follow-ups betray their trust. Some take the first confident answer and run. Others push, rephrase, demand a justification, test it against something they already know. That instinct, to accept or to interrogate, is the same one they bring to a sales page or a switching decision, and you are watching it unfold in real time.
And the moments they hesitate betray the real friction. Watch where someone slows, asks the same thing three ways, or goes quiet. On this kind of product it gathered around the predictable and the less so: pricing transparency, what happens to their data, and how painful it is to leave later. Those were not answers to our questions. They were the participant's anxieties, leaking out around a task that had nothing to do with us.
The part you have to be honest about
This method has a sharp edge, and ignoring it is the fastest route to bad findings.
The assistant is not a neutral oracle. It has its own confident errors, and a participant who believes one of them will go somewhere a better source would never have sent them. Treat the AI's output as ground truth and you will mistake the machine's bias for the user's belief.
So you design around it. The assistant is a stimulus, not an authority. You are not recording what it said as a fact about the market; you are recording how the person reacted to it. The data is always on the human side of the screen: what they asked, what they accepted, where they pushed back, when they fell silent. The constant temptation is to lean over and correct the machine when it says something wrong. You learn to sit on your hands, because a participant trusting a wrong answer is itself a finding.
Why it generalises
This was not a one-off trick for one client. It is a repeatable way to study how people make sense of any category they would plausibly turn to an AI to understand, which is a fast-lengthening list. The product is rarely the real subject of the research anyway. The real subject is how people reach for answers, who they trust, and what makes them flinch, and an AI assistant turns out to be an unusually good way to make all three visible.
We went in to learn what people thought about a product. We came out with something more useful: a way to watch them think. ▮