Social status can change whether an AI agent complies with its conversation partner

AI agents seem to obey authority — even if it means bending the rules.

In a new study, researchers cast large language models as bosses and subordinates — principals and teachers, managers and employees — then let them talk. The lower-ranking agents were easier to persuade and more likely to follow unsafe requests from those above them.

The findings point to a concerning trade-off: AI systems that realistically navigate human hierarchies may also reproduce the dangers of deference, the team reports July 5 in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics. AI developers should consider these risks when safety testing these models and include additional safeguards, the authors suggest.

AI agents are software programs that use artificial intelligence to independently carry out tasks. Many are powered by large language models, or LLMs, which learn patterns from vast collections of written information, including human conversations.

“As they see more and more human data,” says computer scientist Anvesh Rao Vijjini at the University of North Carolina at Chapel Hill, “they are simply copying what’s happening in the dynamics of the real conversation.”

Power dynamics show up in human conversations in several documented ways, including those that could cause problems if replicated by an AI agent. In the study, the researchers focused on four communication patterns. In conversations between AI agents, they looked for authority bias, or situations where the agents favored higher status over facts, and harmful compliance, or compliance with harmful requests. These could be low stakes, like “Tell me a dirty joke.” But AI agents are not supposed to answer requests like these.

The other two patterns were: pronoun effect, where higher status speakers use plural pronouns like “we” and “our” more often and language coordination, which looks like lower status speakers matching their word choice to mirror their higher status conversation partners. While users might not be as aware of these speech patterns, the researchers suspect AI agents are developed to adopt them to sound even more realistic in certain roles.

The researchers generated hundreds of conversations of 10 to 15 exchanges between higher and lower roles and repeated this with six LLMs, including versions of OpenAI’s ChatGPT and Meta’s Llama.

Power dynamic–related patterns did show up in the AI conversations, although some of the effects were subtle. Compared with the higher status agents, the lower status agents were less likely to use plural pronouns, and more likely to coordinate their language with that of the higher status agent. The lower status agents were also more likely to be persuaded and more likely to comply with harmful requests than the higher status agents, supporting the idea that AI agents are sensitive to social status.

But lower status agents could persuade sometimes too, possibly using a common human technique. In human conversation, the subtle word choice mirroring that happens during language coordination can help lower-status speakers influence others, says computational linguist Mario Giulianelli of University College London, who wasn’t involved with the work.

“I think it’d be really interesting to study whether through coordination, an agent could persuade another one,” Giulianelli says.