LLM annotation panel

Overview

Annotating creative thinking in conversations is slow and subjective work for people. This project tests whether a panel of large language models can do it consistently, and whether the models agree more when each one can see how the others annotated before.

The panel has three members: Llama 3.3 70B (served through Groq), GPT-4.1 and Gemini 2.5 Pro, each accessed through its provider's API. The research question is narrow: does showing a model its peers' earlier annotations as in-context examples change how much the panel agrees, compared with the same models annotating the same text alone?

We worked on it as a team: other members built the model interfaces, the preprocessing, the agreement metric, the plots and the statistical analysis. My part was the experiment loop, which runs the panel over the iterations and both conditions and routes the peer annotations between models, including running both conditions on the same transcript in every iteration. I also wrote most of the prompts.

Data and codebook

The data are 232 meeting transcripts from the QMSum dataset, covering academic, committee and product meetings. Speech-recognition markup such as vocal-sound and disfluency tags is removed, spelled-out acronyms are joined back together, and each meeting is truncated to its first 100 speaker turns, written as one speaker: text line per turn.

The codebook defines eight types of creative thinking, each with a definition and an example utterance:

Method

QMSum meetings -> cleaned transcripts (first 100 turns)
  -> per iteration: one unseen transcript
       baseline:  codebook + transcript                        -> 3 models -> JSON spans
       feedback:  codebook + transcript + peers' previous spans -> 3 models -> JSON spans
  -> locate spans in transcript -> pooled pairwise F1 per iteration and condition
  -> mean over 5 runs -> Shapiro-Wilk, paired t-test, Kendall's tau

Prompting

Each model receives an expert-annotator instruction, the codebook and the transcript. It must return pure JSON with no surrounding prose or markdown: a list of objects, each holding a passage copied verbatim from the transcript and one creative type from the codebook. The prompt includes one example object and asks for passages that match the transcript exactly. Asking for verbatim passages rather than paraphrases or turn numbers is what makes the output machine-checkable: every annotation can be located in the source text and compared position by position with the other models' annotations.

In the feedback condition, the prompt is extended with the other two models' annotations of the previous transcript, appended after the transcript as two JSON blocks labelled as peer annotations of the previous sample. The peer annotations come only from the other two models, never from the model itself, and from the previous transcript rather than the current one, so higher agreement cannot come from copying answers for the text being annotated. They are in the exact JSON format the model must produce, so they also serve as few-shot demonstrations of the output schema.

Experiment loop

Each run shuffles the transcript pool once and walks through it, drawing one unseen transcript per iteration, for about forty iterations. Within an iteration, the baseline condition and then the feedback condition are run on the same transcript, with every panel member called in turn.

Both conditions annotate the same transcript in every iteration, and only feedback-condition answers are carried forward as peer examples. The baseline condition therefore forms an independent thread that never sees peer output, and the two conditions differ only in what is in the prompt. This makes the comparison paired at the level of the transcript: the difficulty of a particular meeting affects both conditions equally and cancels out in the difference. The first iteration of each run has no previous annotations, so it runs the baseline condition only, and its answers seed the peer examples for the second iteration.

Five independent runs were carried out, each with its own random order of transcripts.

Measuring agreement

There is no reference annotation, so the measure is agreement between annotators rather than accuracy against a gold standard. Each annotated passage is then located in the transcript by exact word-sequence matching, which turns it into a span of word positions. Passages that cannot be found verbatim are left out of the scoring.

Two annotations agree when they carry the same label and their spans share at least one word, with each annotation matched only once. This partial-overlap criterion lets annotations of the same idea with slightly different boundaries count as agreement. For each pair of models, one is treated as the reference and the other as the prediction. Since F1 is symmetric in false positives and false negatives, the choice of reference does not matter. True positives, false positives and false negatives are summed over all three pairs before computing F1, giving one micro-averaged pairwise F1 per iteration and condition. Pooling the counts weights each pair by how much it annotated, rather than letting a pair with very few annotations dominate an average of per-pair scores.

Statistical analysis

Scores are averaged per iteration over the five runs, and only iterations with a score in both conditions are compared, which gives 39 paired observations. Normality of the per-iteration scores is checked with Shapiro-Wilk before applying a paired t-test to the two conditions. Kendall's tau between iteration number and F1, computed separately for each condition, tests whether agreement drifts as the panel accumulates feedback. As a rank correlation, it detects any monotonic trend without assuming that the change is linear.

Results

Seeing their peers' annotations made the models agree more (paired t-test, t(38) = 4.20, p < 0.001), and the gain held in four of the five runs. Agreement showed no trend over the iterations in either condition, so the effect comes from the examples in the prompt rather than from a gradual drift.

Key takeaways

• Including peers' earlier annotations in the prompt increased agreement between the models, and the effect did not accumulate over iterations.
• Requiring verbatim passages makes LLM output verifiable against the source text.
• Agreement does not imply accuracy. A small human-verified set remains necessary before such a panel is applied at scale.