Staying in Character: Whether a Simulated Buyer Holds Across a Run

The other research areas on this site establish that a simulated buyer produces signal worth acting on: that the differences between buyer types are real, that the reasoning trace is richer than the verdict, that the model behind the buyer changes the result. All of that assumes something the research had not directly tested. It assumes the buyer stays the same buyer for the length of the run.

Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning, from researchers at UC Berkeley, the University of Washington, and Google DeepMind, is the first work to measure that directly and put numbers on how often it fails.

Three checks, and why there have to be three

The researchers score a simulated conversation three ways, because character can break in three different places.

• Against the brief. Every reply is compared to the persona description the model was given at the start. This catches the patient who was written as depressed and is cheerful by turn four, or the high school student who starts reasoning like a graduate researcher.
• Against the history. Every reply is compared to everything the persona said earlier in the same conversation. This catches flat contradictions.
• Against the beliefs. The persona is asked the same questions at several points in the conversation, and the answers are compared. This catches the character who changed without ever contradicting a sentence.

A language model does the scoring, which invites the obvious objection, so the authors tested the scorer against people. Thirty annotators rated dialogue snippets on a six-point scale. The automatic scores matched human ratings 76.73% of the time; the humans matched each other 69.16% of the time. On the stricter measure that corrects for chance agreement, the gap is wider still: 0.400 for the automatic judge against people, 0.063 for people against each other. The check is more stable than the judgment it was built to imitate.

The result that matters: fluent is not faithful

Across three models and three settings, roughly 39,000 lines of dialogue, one pattern held. Staying locally coherent was easy. Staying the same person was not.

Llama-8B-Instruct in open conversation scored 0.992 against its own earlier lines and 0.619 against its brief. Gemma-2-2B-IT in the education setting scored 0.928 against its earlier lines and 0.511 against its brief. In both cases the transcript is clean and the character has quietly moved.

Surface-level coherence can mask belief inconsistencies.

That is the authors' phrasing. It is the finding with the sharpest consequence for anyone reading simulation output, because it describes a failure with no visible symptom. A transcript can be smooth, on topic, and free of contradictions while the person in it has been replaced. Reading the output more carefully does not help. The drift is not in the prose.

Length is where it shows

Conversations were run at four lengths: 10, 20, 40, and 60 turns. The comparison between training methods is the useful part. Ordinary supervised fine-tuning performed well at short lengths and degraded as conversations grew. Only the model trained on scores computed over whole conversations held steady or improved out to 60 turns.

The practical reading is that a short excerpt proves nothing. If a simulated buyer is shown holding character for five turns, the failure mode has not been tested yet. It appears at the lengths where a buyer would actually work through a page, a comparison, and a decision.

Consistency can be trained

Using the consistency score as a reward across whole conversations, the researchers raised persona consistency by 58.5% in open conversation, 37.6% in the mental health setting, and 20.6% in education, against the untrained baseline. Scoring the full run rather than the individual reply is the part that carries the result, because drift is never visible in a single line.

This matters beyond the specific numbers. It establishes that character drift is an engineering problem with a known direction of fix, rather than a limit of how language models work.

What this does not establish

The paper studies three settings: a patient talking to a therapist, a student being taught, and two people in open conversation looking for common ground. None of them is commerce. Extending the result to a buyer reading a product page is an inference, and it should be named as one. The same is true of several areas on this page, which build on negotiation and on choices between money now and money later, but that does not make the gap smaller here.

The authors are also direct about their own limits. Consistency is not the same as realism: real people change their minds, and a persona trained never to move is wrong in a different way. The framework measures a single conversation, so it says nothing about a persona holding across separate sessions. The human study used thirty raters. And the training run optimized only the first of the three checks, not all three together.

The question this puts to buyer simulation

The finding transfers as a question rather than a fix, and it is a question worth putting to any buyer simulation, this one included. If a simulated buyer's findings are only as good as the buyer who produced them, what evidence is there that the buyer held?

Three things follow directly from the paper, and none of them require agreeing with its methods. Test at the length a real evaluation would take, not at the length that demos well. Treat a clean transcript as evidence of fluency and nothing else, because the paper separates the two directly. And ask the persona the same question early and late, since that check is cheap and it catches the drift that reads fine.

This sits next to simulation validity, which asks whether findings track real outcomes, and human heterogeneity in agentic markets, which establishes that how an agent is specified drives most of what it does. This area adds the part neither covers: specification decides who the buyer is at the start, and consistency decides whether that holds to the end.