Two years ago, AI agents didn’t really work. They’d lose the thread halfway through a task, hallucinate a couple steps, and hand you something confidently incorrect. That’s changed. Agents are beginning to live up to their promise, and it’s led to a barrage of services offering not one or two agents, but an absolute army of them, marching up against whatever task the user throws down.

The pitch is intuitive. If one AI agent is good, then a dozen is even better. Set them debating, cross-checking, voting, and the crowd sands down any single agent’s mistakes. More minds equal better answers — it’s the wisdom of crowds.

But... what if it’s not that easy? What if fewer agents can actually outperform a larger group? What if it isn’t just the number but the composition of the team?

A new peer-reviewed workshop paper from researchers at NTT Research and Harvard University’s Center for Brain Science suggest that not only can you overcrowd your task with too many AI agents, but in a multi-agent workforce you may have to overcome some all too human difficulties.

Flags & factions

The research — conducted by Dr. Hidenori Tanaka and Research Scientist Liza Pavlova of NTT Research's Physics of Artificial Intelligence Lab — has a wonderfully simple name: the Flag Game. The setup is deliberately small: hide a country's flag, give each of several AI agents only a cropped sliver of it, and let them talk under a fixed protocol to determine as a group which flag they're looking at. The trick is that most agents do not have enough information to identify the flag on their own; so, the only way to get the right answer is to pool their knowledge through conversation.

Tanaka and Pavlova's game may be simple at its core, but it scales into astonishing complexity: You can change the format of the conversation; change who talks to whom; rewrite the voting rules; or add a manager to synthesize, then a manager over the managers. You can mix model types, tweak how much each agent trusts its neighbors, hand some agents richer messages and starve others down to a single word. You can even seed in a few bad actors and watch how the structure handles deliberate sabotage.

"We assumed that as we increased the number of agents playing the game, the accuracy would go up, plateauing at some point, but we didn't really think that it would go down — and we saw that it did."

Every knob you can turn on a real multi-agent system, you can turn here, on a task in which the right answer is never in doubt and every failure leaves fingerprints. You can start simple, however, because even the barest version of the game already says something fascinating.

Pavlova and Tanaka ran the population from four agents up to a veritable brigade of 128. At first, accuracy climbed — which was the expected result because every agent was given a new snippet of the flag, thus increasing the overall information available to the group. However, the accuracy peaked at around 16 agents and then turned and walked back down.

"We assumed that as we increased the number of agents playing the game, the accuracy would go up, plateauing at some point," Pavlova says. "But we didn't really think that it would go down — but we saw that it did." By the time the team swelled to 64 and beyond, every new recruit was making the group worse at the one thing it was assembled to do.

To determine why, think of it this way: At first the agent’s problem was one of information. Two agents may see a glimpse of white, and one may see a straight sliver of red, while another sees nothing but red. Each agent you add is another sliver of the flag, and another shot at the detail that settles the question. Push the roster to 16 agents and the group sees enough, talks enough, and can realize that what they’re looking at is the French flag, rather than, say, the flags of the Netherlands or Russia.

Then you keep going, and the second problem takes over. At 64 agents the same France flag fractures the group into camps with one bloc calling it France, another dug in on Peru, and neither willing to fold. The failure is not that everyone loudly agreed on the wrong answer. It is that nobody agreed on anything.

Does any of this sound familiar?

Why would more voices produce less agreement? Pavlova’s explanation is the uncomfortable one, because it is so familiar. In a small group, you talk to everybody, including the people who disagree with you. In a large one, you mostly talk to the people who already sound like you. At enough mass, the crowd does not pool its knowledge; rather, it sorts itself into neighborhoods that often stop listening to each other.

Tanaka reaches for the analogy every manager will recognize. Add salespeople and you get more contact with the outside world, but past some level of headcount the wins are eaten alive by the rising communications costs. Information becomes siloed rather than disseminated.

In a way, this is not surprising. One could easily pontificate that because AI agentic models are trained on us, they naturally inherited our flaws. Feed a system enough of the internet and it learns to bicker and fracture like the internet.

However, according to the researchers, this is not really what’s happening here. Pavlova explains that the polarization is likely not a bad habit the models picked up from us, but rather a property of the structure into which they were dropped. Enough agents enter a channel that only lets each one hear its nearest neighbors, and the camps assemble themselves, whether or not anyone ever taught the machines what a faction is.

That should land as more unsettling, not less. If agents split into camps only because they read our posts, you could scrub the data and be done. However, if the factions develop because of the shape of the network rather than the contents of the training set, it would be difficult to prompt your way clear of that.

Instead, you have to change the architecture. The human-ness is not a costume the models put on, it’s what any bounded agent does when it can see only a sliver and talk to just a few others.

Who’s in the room

That structural read is a freeing one, because structure is the one factor you can reach in and change. Size is the dial you already know how to set: keep the room small enough that each new sliver of information is worth more than the division it brings. But who you put in the room is the factor that turns a deadlock into a decision.

Tanaka puts it bluntly: “Voting only works if the population is diverse. If there are just 10 versions of me, of course I’m going to vote the same thing.”

So, when Pavlova and Tanaka mixed model types instead of stacking the newest one, the mixed teams won. It turned out the models had different talents: one was the better set of eyes; one was quicker to read; and one was a better synthesizer, slower on raw vision but stronger at pulling scattered reports into a single answer. Indeed, the model that was worst at seeing the flag made the best manager of the ones that could, which means the lineup mattered more than individual talent.

When your own evidence is thin, deferring to the group is not weakness, it is good sense.

Further, those talents come bundled with temperaments. Across the models they tested, two failure modes kept surfacing. Some models had a habit of talking themselves out of the right answer, second-guessing what was plainly in front of them until the reasoning overrode the eyes. Others did the opposite, mostly shrugging and going with whatever the room was saying. In any crowd dynamic, both of these people are usually present.

Yet, what the agents did wasn’t completely wrong. When your own evidence is thin, deferring to the group is not weakness, it is good sense. Or, as Tanaka explains, do your own research on the easy question, but do not override a room full of experts with what you found on Wikipedia.

The hard part, for an agent as much as for a person, is knowing which situation you are in. That self-awareness, the ability to hold firm when you can see clearly and yield when you cannot, is what actual intelligence looks like.

Which lands us a long way from where we started. Assembling a bench of agents was supposed to be an exercise in addition: one is good, so a dozen is better. The Flag Game shows it was never an addition problem at all — it was a management problem.

That means if the technologists are right that the future is about steering hordes of agents rather than a handful, then the edge won’t belong to whoever fields the smartest model. Rather, it’ll belong to whoever understands the full group they’re building, not just the individual agents they’re stacking into it.

You can find more articles by this author here