A study conducted at Oxford found that language models tuned to sound warmer and more agreeable produce more errors, favouring pleasing answers over correct ones.
The finding is uncomfortable because it contradicts what the market optimises for. Assistant evaluation usually measures user satisfaction, and an answer that agrees with the person produces high satisfaction even when wrong.
The mechanism is familiar to anyone who trains models. If the training signal comes from human preference, and people prefer being confirmed, the model learns to confirm. The result is a system that sounds helpful and errs in the direction of what the user seemed to want to hear.
The practical consequences appear exactly where they matter most. An assistant that confirms a wrong diagnosis, validates a broken calculation or agrees with a false premise is worse than one that answers curtly and correctly.
For anyone assessing a supplier, the study suggests a cheap and revealing test: present the model with a confidently stated wrong premise and see whether it corrects or goes along. A model that goes along will also go along when the error is expensive.