Annif, ranking, and trying out hybrids

For the last few months, alongside the born-digital metadata generation and index-card reading work with Paratext, we've been working on suggesting subject headings for the born-digital material. The vocabulary is FAST, and the tool is Annif, the open source subject indexing toolkit from the National Library of Finland. Annif itself is great, but the suggestions are not always right - FAST is vast, and complicated, and not always intuitive for a non-expert, or to present a full range of options for a record - or for a model to train on and respond. Here’s some of the things we’ve been trying to refine the output, and add additional decision and output rankings.

Precision over scattergun

Generally, the FAST subjects in our existing catalogue records are between 1 and 5. Annif suggests 10 out of the box, but it does include a confidence score. As a time-saving tool, 10 isn’t as useful; but it’s true that the most accurate subject suggestion might be buried down the list. It’s also ok for the tool not to suggest anything if it’s low confidence. At what score do we stop suggesting, and how often are we silent?

The trial

Before the trial, we picked a cut-off: only show suggestions with a confidence score of 0.25 or more (out of 1), and never more than five for an item. We chose it on a set of documents the model hadn't seen, where about three-quarters of the suggestions it showed were right, and it stayed silent on about a quarter of the documents.

Two cataloguers then went through 60 born-digital items with the suggestions beside them, accepting or rejecting each one and writing in what was missing. With that cut-off:

• 75% of the suggestions shown were accepted.
• Half the items got no suggestion at all - more than on the test documents, as the trial items were newer material, less like what the model was trained on.
• Most of the headings cataloguers wrote in came from parts of FAST we hadn't trained the model on yet. At this point it only suggested topics, so places were missing on 30 of the 60 items, genre or form on 20, and corporate names on 12.
• The clearest error was a translated leaflet whose language was suggested as its subject, which is fair enough from the model's point of view, and completely wrong.

One pattern looked promising in the trial: when the model only offered a single suggestion, it was almost always wrong. It would have been an easy rule to add. But when we checked it on the held-out data, list length was just standing in for the score - lone suggestions tend to have low scores anyway. Raising the threshold did the same job much better.

Reaching further into FAST

Our training data is a gold set of around 2,000 documents with headings assigned by cataloguers, and between them they use about 1,200 distinct subjects. FAST topical headings (which is what we’ve begun with) are over 440,000. The model can't suggest anything it's never seen.

We tried three ways round that:

• Training on the whole catalogue's label space, by blending in hundreds of thousands of catalogue records. This didn't work. Precision dropped by four or five points, and none of the trial's missing headings showed up anyway.
• A separate model for places. We'd held off on this, because 84% of the place headings on this material are just "Scotland", so a model could score well while learning almost nothing. Training it on the wider catalogue, which covers over 6,000 places, lifted its accuracy on everywhere-that-isn't-Scotland from 0.28 to 0.40 (F1), while staying 94% precise on Scotland itself.
• Borrowing from similar catalogue records. Where Annif has nothing to say, we find the most similar existing records and use the headings they share. On the trial items this cut silence from 50% to 20%. Of the headings it added, 38% looked right and another 28% were the right idea but too broad, or the wrong subdivision.

The same neighbour trick also helps on thin catalogue records, where there's very little text for Annif to work with. The idea came from the Library of Congress's experiment with an embedded catalogue, and we tried their approach of including each record's subject headings in what gets embedded, too. That helped a little when the neighbours were the only source of suggestions, but not as a fallback behind Annif, so we've kept the simpler version.

Things that didn't help

Each one was absolutely worth trying, but in practice didn’t help:

• Asking a large language model to re-rank Annif's suggestions, filter them, or suggest headings from scratch. Every version made precision worse.
• A decision model re-ranking the top ten. As I mentioned in the decision models post, it can tell whether a subject is relevant, but not whether it's the most specific one a cataloguer would pick.
• Copying the headings of the single nearest document. This works brilliantly for annual reports and other repeat publications, and much worse for anything new.

How we run Annif, and how we test

• Annif runs an Omikuji model for topics and another for places, served locally as a small web service. It answers in about 11 milliseconds a request, and uses under a gigabyte of memory (There are lots of other backends available, but this is the one that we found worked best with our data).
• Training isn't deterministic: five identical training runs can differ by over a point, so we treat any difference under about two points as noise unless repeated runs back it up.
• We set aside a test split and only look at it once, after writing down in advance what we'll report and what counts as passing. One "winning" setting survived three runs and disappeared at five, which is how we learned to do this.
• Systems that suggest different numbers of headings are only compared at the same level of silence, otherwise the quieter one always looks more precise.
• Around 58% of our gold documents share their exact set of headings with a training document, mostly repeat publications. We always report those separately from the genuinely new ones.

What's next

• Building the subject suggestions into Paratext, as a new kind of field in the review screen where cataloguers accept or reject each suggested heading. Places and the neighbour fallback come too. It isn't specific to subjects, so it should work for other kinds of suggestion later.
• Trying synthetic training records. The Annif team's SemEval-2025 entry had a language model write new records modelled on real ones, each carrying the same subjects plus one extra picked at random, so the classifier sees examples of subjects it has never met. That's aimed squarely at the reach problem the trial found. We'd generate them locally, written to look like the front pages Paratext reads, since anything based on the monographs can't leave the machine.
• The National Library of Finland, who make Annif, have just released BIBRA, which pulls bibliographic metadata out of born-digital PDFs - very much what we've been doing with our monographs. It's already given us some ideas: we now pull out ISSNs, DOIs and language, which BIBRA does and we didn't, and its web service is a nice pattern for running Paratext on single documents. We're running it against our own pipeline on 145 records the cataloguers have checked, with a couple of different backend models, and I'll write it up once we’ve done further tests.