Agents, orchestration and use cases

Ask a language model to assess a 170,000-word novel and you get a polite summary. The prose is "strong and polyphonic", the characters "interesting", the ending "powerful". None of it can be used for anything. The fault is not the model's but the question's: in one call, the model produces the average of everything a book review usually contains.

For assessing my own manuscript I built a tool that asks differently. The model is called many times, each time in a different role, and the roles are set to talk to each other. I call it the reading circle. The same structure works for any long body of material whose quality cannot be checked by a test; a manuscript is the example here only because there are measurements for it.

This article first covers what multi-agent systems generally mean and why such a thing would work at all. Then comes the structure itself: how many members a circle should have, whether they should be the same model, what the circle produced, and which parts of its output not to trust.

First, the vocabulary

In multi-agent talk, the same words mean different things in different mouths. These six will see you to the end of this article.

• Agent. A language model in a loop: it gets a task, chooses a step, looks at the result and corrects. One agent is one loop and not a personality, even if it is given a name and an occupation.
• Multi-agent system. Several such loops wired together so that one's output is another's input.
• Orchestration. The part that decides who runs, in what order, on what input, and when to stop. In the circle described here the orchestrator is ordinary code: a loop and a conditional. An orchestrator can also be a model that decides the division of work itself. The difference is worth knowing, because decisions made in code are repeatable and decisions made by a model are not.
• Role prompt. An instruction telling the model from what point of view it reads. A role gives the model no new information; it narrows what the model attends to.
• Aggregation. The stage where many answers are compressed into one: a vote, an average, a merge, or a separate summarising call.
• LLM as a judge. A model that assesses another's output against stated criteria. The circle in this article is an LLM judge, multiplied.

One misconception is worth correcting immediately, because it is the most common in the whole subject. Agents do not think together and share nothing of themselves. Every call is separate, and what travels from one member to another is only what the orchestrator moves there. In this circle that is text, but more generally it can be structured data, images, tool results or shared memory. What they have in common is that the system does the moving, not the agent. The meeting is staged. An "AI team" is a figure of speech, not a description.

Four ways to put agents together

There are in practice four structures, and they solve different problems.

The difference is in direction. Chains and supervisors produce something, and there the work is split into parts. Panels and debates assess something, and there the same work is done several times from different angles. Use the first kind when you know what needs doing, the second kind when you do not know what is wrong.

The reading circle is a panel with one debate round and a vote added. It writes nothing; it produces a list.

Why a panel works, when it works

The idea is older than language models. In machine learning it is called an ensemble: many mediocre models together are right more often than any of them alone. The same result is known on the voting side from the 18th century as Condorcet's jury theorem.

Both rest on two ideal conditions. In practice partial satisfaction still helps, but the further you are from the conditions, the less agreement tells you:

1. Every member has to be better than guessing. If a member is wrong on average, growing the group makes the result worse rather than better.
2. The members' errors have to be independent. If everyone errs in the same way, the majority errs just as surely as an individual, but looks more convincing.

The first condition is usually met by a language model: it reads more carefully than at random. The second is the one broken here, and I return to it in its own section. For that reason the panel's output is read as overlap rather than as votes: the signal is not an opinion but the fact that different angles arrive at the same place.

A second reason to collect several assessments is a language model's tendency to be sycophantic. Ask a model to assess a text and it produces an assessment fitting what it takes the asker to want. The truth value of a single assessment is therefore hard to infer. Sycophancy does not disappear because there are several assessments, but it points in different directions once each member has been told something different about what they value.

It is also worth noting what a panel does not do. It produces no new information. Everything the circle says comes from the same model that could have said it in one call. The structure does not add information; it selects and weights it, and that is the whole benefit.

This is also a different job from proofreading. A proofreading agent works at line level: grammar, awkward constructions, sentences that do not parse. A panel works at the level of the work: does the opening lie about what kind of novel this is, does the second act sag, at what point would a tired evening reader stop, and does any of this survive being read aloud.

The line-up is prompts, not people

The circle has five members and a chair. Each is one prompt describing a background and what this particular reader notices.

The roles are deliberately concrete. "Be a critical reader" produces bland generalities, whereas an occupation, a number of years and a bias of their own force the model to choose what it attends to and what it does not. There is nothing magical here. A role is a constraint, and a constraint is the cheapest way to get genuinely different answers out of the same model. Other ways are different material for each member, a different question, and a different model.

The line-up is also where the tool is fitted to its purpose. The audiobook narrator is there because the manuscript is on its way to speech synthesis. For another purpose the list would hold different occupations.

Five is not a magic number

Derive the member count from the material: how many genuinely different lenses does it have? The test is simple. If you cannot name one thing that only this role sees, the role is a duplicate of another.

A duplicate is not merely redundant but harmful, and the reason is the vote. The circle ranks proposals by support, so the line-up is a thumb on the scale. If a circle has three roles reading for structure and one reading for language, structural problems win the vote because of the line-up rather than because of the text. Choosing the line-up therefore decides what carries weight in the result.

A practical range, based on my own runs rather than on any studied optimum:

• Fewer than three members. The vote cannot tell one reader's matter of taste from a shared observation. Two members is a conversation, not a measurement.
• Three to seven. The working range for most material. Three is enough for a proposal: the customer, the lawyer and the implementer. Four to six suits a broad code change: maintainer, security, newcomer, performance.
• More than seven. The cost grows directly with the member count, because every member goes through every round, and the chair's merging work grows faster than the benefit. A large circle's minutes start to resemble an average, which is exactly what you set out to escape.

Five happened to suit the book. That does not mean it suits a proposal or a requirements specification.

One model or several?

The circle is built so that all members are the same model in different prompts. That is simple and cheap, and it is the structure's biggest weakness.

Here the ensemble's second condition is broken. The whole idea rests on overlap telling you something about the material, but with one model part of the overlap tells you about the model. The same training data produces the same way of reading and the same blind spots, so the members' errors are correlated rather than independent. Agreement then looks stronger than it is.

Models from different suppliers break exactly this. When two models from different families arrive at the same observation, the common explanation is more likely to be in the text. Mixing is not worth doing among the members, though, and the reason is measurement: if the psychologist runs on a different model from the editor, the cause of a disagreement can no longer be isolated. Was it the role or the model? A mixed-model circle loses the one thing the roles were defined for.

A more sensible order, cheapest first:

1. Switch the chair to a different model. The summary is where one model's tendencies get amplified, and the check costs one call.
2. Run the whole circle twice on different models. Comparability among the members survives, because one run is internally consistent. Keep the findings that came through both runs. The cost doubles. Two models are not fully independent, since they were trained partly on the same material from the same web, but the difference is clearly larger than between two prompts on one model.
3. Run a third model alongside only on the disputes. An unresolved disagreement is cheap to hand to an outside model when the whole body of material does not have to be read again.

Mixing has a mundane price too. Every model follows a requested output structure slightly its own way, and in a multi-stage meeting one deviant field breaks the next round. On top of that, every supplier brings its own key, its own price list and its own pace of retiring models.

One measurement argues for the trouble. I ran the same circle on the same model over two language editions, and the top finding was different. If merely changing language moves the attention elsewhere, changing the model moves it more.

The meeting runs in four rounds

1. READING       5 parallel calls
   each reads alone
        |
2. DISCUSSION    5 parallel calls
   the others' notes, not one's own
        |
3. PROPOSALS     5 parallel calls
   at most 4 each + "do not touch this"
        |
4. VOTE          5 parallel calls
   everyone scores everything 1-5
        |
   CHAIR            1 call
   minutes from the whole record

Round 1, reading. A member writes a one-sentence overall assessment, a few paragraphs of reading experience through their own lens, a couple of strengths and a few concerns, each tied to a precise place, and one question for the others. Nobody sees anyone else's text at this stage. That is deliberate: if members read each other immediately, the first answer to finish would steer the rest, and overlap would mean only imitation.

Round 2, discussion. Now a member gets everyone else's notes but not their own. That too is deliberate, because on seeing their own notes the model repeats itself instead of answering the others. It addresses people by name, says what it supports, what it will not swallow, and what nobody in the room has said yet.

Round 3, proposals. At most four concrete changes per member: problem, change, benefit, effort, risk. In addition, each names what they would defend to the last. These "do not touch this" answers are as valuable as the proposals, because they are the only point at which the circle is asked to defend the work against itself. Without it, an assessing agent always produces a list of corrections, because that is what it was asked for.

Round 4, vote. Everyone scores every proposal from 1 to 5, including their own, on the instruction "5 = do this first, 1 = this would damage the work". The average and the number of supporters set the order. Only this round separates one member's matter of taste from what the whole circle agrees on.

A round's five calls run in parallel, so the meeting takes roughly as long as its slowest member. One member failing does not bring the meeting down; their result is simply missing and the others carry on.

The material is not fed in raw

A 170,000-word novel is not worth reading five times over four rounds. That would be slow, expensive and unnecessary, because most of the questions concern structure rather than individual sentences. So the material is prepared once.

• Summaries. Each chapter gets a reading summary: what happens, to whom, what is planted, a few verbatim lines, a note on technique, and where the chapter ends. The summaries are cached to disk, keyed by a hash of the chapter's text. Edit one chapter and only that chapter is read again.
• Samples. A few chapters go through whole, so that members assess real prose rather than a summary of it. Three by default, evenly spaced, skipping front and back matter, because a preface is not where a novel's prose lives.
• An honesty instruction. Members are told which chapters they have in full, and are required to say out loud when a claim rests on a summary.

That last point is the most important and the most easily forgotten. A claim about prose made on the basis of a summary is a claim about the summary. Without the instruction the model does not draw that distinction itself; it writes about sentence rhythm just as convincingly whether or not it has seen a single sentence.

The result is a reading pack of about 34,000 words per member. That is a fraction of the book and still sufficient for what the circle is being asked.

The chair's three rules

The chair gets the whole record: the notes, the discussion, the proposals with their scores and the voters' comments. The chair does not defend a lens of their own but presents what the circle actually said. Three rules keep the minutes usable.

1. Attribute claims by name. The reader sees who said what and can weigh it against what that role is capable of seeing at all.
2. Do not average the disagreement away. A dispute is reported as a dispute, and an unresolved dispute as unresolved. Otherwise the summary is always more agreeable than the discussion it was made from.
3. Merge overlapping proposals. This rule was added only after the first run. Five members proposed the same cut, and the minutes listed it as ten separate items. Five readers asking for the same cut is one item with five supporters, not five items.

The third rule is a good example of where most of the time goes in multi-agent systems. The individual calls work immediately. Combining the results in a way that does not lie about quantities is a separate problem, and often the more laborious one.

What the circle found

I ran the circle on my own manuscript twice, separately on the English and the Finnish edition. The Finnish circle reads the Finnish book, not a translation of the English discussion.

The circles arrived at different things. In the English meeting all five independently proposed cutting the same circling narrative habit, and it scored an average of 5.0 with everyone's support. The Finnish circle's top item was something else entirely: marking the first-person narrator's sources of knowledge in the passages where the narration follows characters beyond what the narrator could know. The English circle did not notice this at all.

Then I checked the findings against the text before changing anything. Measurement altered three of them:

• One mannerism was real and, moreover, a phenomenon of the back half: 84 occurrences, and almost none in the first half of the book. The circle was right but could not locate it.
• Four members treated a recurring feature as a structural problem. Across the whole book there were 17 instances. The measurement does not support the claim.
• The Finnish circle's top finding was half wrong. The suspect passage was already framed, but the frame ran out partway and the narration continued without a source. The fix was two sentences, not the structural rebuild that was proposed.

This is the heart of the whole approach. The circle produces hypotheses, not decisions. The hypotheses are checked against the material before anything is touched, and the check is best done with ordinary search and counting rather than another model call.

What it costs

For a book of 30 chapters and 169,000 words, one meeting is about 51 model calls: 30 chapter summaries, five members over four rounds, and one chair. It takes a little over 20 minutes. A second run on the same book is 21 calls, because the summaries are cached and only changed chapters are read again.

The structure of the cost is instructive here and holds for any multi-agent system. The bill is not one call but the number of calls multiplied by how much text each call sees. A panel costs its member count more than a single assessment, and that is exactly why the size of the reading pack matters. Caching the summaries is not an optimisation but the thing that makes repeated runs possible at all.

Where this fails

The circle is a set of prompts, not a set of people. If the members are the same model, and the model does not recognise some phenomenon, no member recognises it. Agreement still looks strong. The structure protects against random nonsense, not against a systematic blind spot, and only another model helps with that.

It is good at what a careful reader notices on a second reading, and bad at what a first reader feels once. Surprise, boredom and being moved are one-off experiences, and they cannot be recovered by reading summaries.

Quotations have to be checked. Members are required to quote the text and to say when they are relying on a summary, but the requirement does not remove hallucination. Check before you cut.

Weigh the vote rather than the volume. A proposal one member loves and four score at two is a matter of taste, not a defect. The scored table in the minutes is the most honest view in the whole document, and it is worth reading before the elegant summary.

Which structure to use for what

Nothing above is specific to fiction. The structure is general: long material, several roles, reading alone before discussion, a bounded number of proposals, a vote and a merging summary. The use case decides which of the four shapes is right.

That last row is the one most often forgotten. A multi-agent system is a structure for a problem one well-instructed agent does not solve. If the task is clear and checkable, extra agents bring only cost and new places for the chain to break.

Assessment (panels and debates) is worth reaching for only once three conditions hold:

1. The material is too long to assess at a glance.
2. Quality is a matter of interpretation, meaning the right answer cannot be checked by a test.
3. The cost of an error is greater than the cost of a run.

If the answer is in a test or a measurement, run the test. A panel does not replace measurement; it is a way of working out what would be worth measuring.

In one sentence

A multi-agent system does not make the model wiser but makes the orchestration visible: it settles who reads what, in what order, and how disagreement is handled, and that is where a usable result comes from.