Adding confidence scores with system one models

In an earlier post I wrote about our index cards - thin paper, show-through from the next card, handwritten corrections in pen - and how the model's explanations of its decisions had almost nothing to do with the decisions it made. We ended on needing a confidence score from outside the model, or a deterministic check, to flag cards for an expert - something to keep the work moving while accurately recording where the pipeline would fail, so expert-in-the-loop checks could be made.

Since then, system one models have hit the scene, which adds a lot of interesting possibilities for a combination of deterministic code and more deterministic model outputs, including confidence scores. Here’s some of the things we’ve been trying.

Asking what the model can see, not what it thinks

The first change was to stop asking the model whether it was confident. Instead, the prompt now asks three things about the mark itself:

• Does the mark add or change a digit?
• Does it add anything you'd have to read?
• Is it unclear which typed text it applies to?

If any answer is yes, the card is flagged for a cataloguer to check. These are questions about what's on the card, not about the model's inner state, and it turns out it can answer those. The flag went from never firing at all to catching 9 to 13 of the 14 cards a cataloguer had had to fix, depending on the run.

It then started seeing corrections everywhere, mistaking typed underlining and show-through for pen. Cutting the corrections part of the prompt by almost half fixed that, which is a whole other post about prompts getting worse the more you add to them.

Counting where it goes wrong

Of the 23 cards where the model said it had applied a correction, 10 were wrong: 7 where it never actually applied it, 2 where show-through was taken for a correction, and 1 misread. At least 6 of the 18 cards it flagged for checking had applied the change anyway. Only 1 of the 155 cards it said had no corrections had missed one.

So the model is good at noticing that a card has been corrected, and not reliable at saying what it did about it.

Checks that don't need a model

Some of this can be caught with plain code:

• If the notes say a correction was made but the field still has the typed text, something's wrong. This simple text comparison caught 6 of the 7 cards that said "applied" but weren't.
• If we run the same cards several times and the runs disagree on how many entries a card has, or on a manuscript number, the card is wrong about 80% of the time. Disagreement on names and epithets turned out to be noise. A shoutout to William Mattingly at Yale for writing up this technique on his blog from the work they’ve been doing at Yale. It’s a deceptively simple and powerful method.
• Different models format folios and dates slightly differently, so we tidy those before comparing. About 15 lines of code took the folio check from 46% of flags being real errors to 83%.

What's left after tidying are real misreads: 997 for 10997, "trial" for "memorial", 1726 for 1826. That's also why we didn't ask a second language model (or decision model) to judge whether two readings were "broadly the same" - it would wave through exactly the errors we want to catch.

Decision models

There's a newer kind of model, sometimes called a "System One" model (after Kahneman), that doesn't write text at all. You give it a state - here, the card image and what our model extracted - and a set of yes/no questions, and it gives back a probability for each. There's no explanation to be unfaithful to, just a number.

We tried one of these before for subject indexing, and it didn't help. It could tell whether a subject was relevant to a book, but not whether it was the most specific one a cataloguer would choose.

The index cards are a better fit, because the questions can be asked against the image. Clef, from Cloudflare, is the first openly licensed decision model that can see, and it runs on the same local machine as everything else. For each card we ask things like "does this card show manuscript number 11764?" or "is there a handwritten correction on this card?". It takes about 4 seconds a card.

Over our 98 gold-standard cards:

• If we sort the cards by how doubtful Clef is, 9 of the 10 it's most doubtful about are cards our model got wrong.
• The questions about entries, manuscript numbers, descriptions and folios work well. The questions about names and epithets were no use at first; rewording them to point at the typed entry, "with nothing left out", made them useful.
• On corrections, Clef didn't do better than the model's own flag. But used as a veto, it removed all 4 of the model's false correction flags, at the cost of 2 real ones.
• Clef and the run-to-run disagreement check catch different cards. Together they flagged 9 cards, and 8 of them were wrong.

An interesting catch: on one card the edge of the paper hides the start of the name, so it reads "TON (William)". Our model had filled in ALSTON from the filing label. Clef flagged it, because ALSTON isn't what the card shows. The model was arguably right, but that's exactly the kind of decision a cataloguer should make, not the model.

Here's what Clef was asked about that card (trimmed to three of its ten questions):

body = {
    "state": (
        "A catalogue index card for manuscripts. "
        "Answer about the front card only; faint "
        "reversed text showing through from other "
        "cards does not count."
    ),
    "images": ["data:image/jpeg;base64,..."],
    "questions": {
        "heading": {
            "type": "noul",
            "instructions": (
                "The typed entry in the middle of the "
                "front card starts with a name in "
                "capitals. Is that whole name exactly "
                "'ALSTON (William)', with nothing "
                "left out?"
            ),
        },
        "ms_no": {
            "type": "noul",
            "instructions": (
                "Does the front card give manuscript "
                "number 17563?"
            ),
        },
        "corrections": {
            "type": "noul",
            "instructions": (
                "Does the front card carry a handwritten "
                "correction, addition or crossing-out "
                "on the typed text?"
            ),
        },
    },
}

And what came back, the probability of "yes" for each:

{
  "heading": 0.09,
  "ms_no": 0.87,
  "corrections": 0.03
}

A note on the tech

Clef runs as a quantised GGUF (Q6_K, about 23 GB) on llama.cpp's /v1/systemone endpoint, which needed a newer llama.cpp build than we'd been pinned to. It scores all the questions for a card in one pass, so it needs the whole input - image and questions - in a single batch. Our usual model server doesn't pass that endpoint through yet, so the checking script talks to llama.cpp directly.

What's next

• GLiDE, from Fastino, is built on the same base model as Clef and currently tops a benchmark for this kind of model. We'll ask it the same questions on the same cards and compare. It doesn't run on our machine as easily, but the index cards are openly licensed, so we can use the hosted version for this test. Its licence is more restrictive than Clef's, so that needs reading before we build anything on it.
• Clairic, also from Cloudflare, extracts fields and gives a confidence score for each one, which is what our model can't do. An early preview garbled too much text to replace our model, so we're waiting for the open release. Then the idea is to use the three together: our model extracts, Clairic adds field-level confidence, and Clef checks against the image.
• Kev, an openly licensed decision model, came out top of eighteen decision models William Mattingly compared on linking names in Europeana records to Wikidata, with decider-35b-a3b a close second. Neither can see images, so they won't replace Clef on the cards. But they could check our model's extracted text against itself, and they could run on other material. We'll try both.