Embedding Search Meets Archive RAG

The previous embedding note focused on multilingual benchmark datasets: semantic similarity, paraphrase retrieval, cross-lingual alignment, and retrieval benchmarks in Estonian, Finnish, Latvian, and Russian. That work was useful for choosing candidate embedding models, but it left one practical archive question partly unanswered:

What happens when the retrieval task looks more like real RAG search over archive text?

In this follow-up we tested the strongest embedding candidates on manually checked Estonian HTR/OCR text split into page-level chunks. The result is clearer than the model ranking alone: dense embeddings are valuable, but dense-only retrieval is not enough for archival search. The strongest practical direction is a hybrid architecture that combines semantic vectors with lexical, structured, and query-aware search.

What changed since the last post

The earlier benchmark compared embedding models on general multilingual evaluation datasets. Since then, we built a more archive-like retrieval dataset from four manually checked Estonian text files. Each file represents a document, and page breaks were preserved so that retrieval could be evaluated at page level.

The corpus contains:

The synthetic query set deliberately mixes different retrieval behaviours:

This matters because archive users do not only ask broad semantic questions. They also search for people, places, dates, identifiers, page ranges, case numbers, and specific terms. A realistic retrieval test needs to include both semantic and exact lookup behaviour.

Dense embedding results

We evaluated nine local Hugging Face embedding models selected from the benchmark tables. The best dense model on the new RAG-style dataset was Octen/Octen-Embedding-8B.

This does not overturn the earlier operational recommendation. Octen/Octen-Embedding-4B remains very attractive as a default because it is close to 8B in retrieval quality and materially faster. On this page-level RAG dataset:

The practical reading is: use Octen-Embedding-8B when pure dense quality matters most, but use Octen-Embedding-4B when speed, index size, and operating cost matter.

Where dense retrieval works

Dense embeddings performed best on semantic paraphrase queries. These are the queries closest to what embedding search is designed for.

For Octen/Octen-Embedding-8B, the semantic-only slice scored:

One successful example was:

miks kadusid eestlaste sidemed Eestiga pärast piiri tekkimist

The correct page was ranked first. The top neighbouring results were also sensible: pages about the Eesti-Läti border, Alolinna eestlased, and disrupted contact with Estonia.

This is the strong case for dense retrieval. It can connect user phrasing to meaning even when the exact wording differs.

Where dense retrieval fails

The weak cases were equally important. Dense-only retrieval performed poorly on exact dates, numbers, and some topic labels.

For Octen/Octen-Embedding-8B:

One failure was:

Leia lehekülg, kus esineb number 75-100.

The correct page was ranked 2,432nd out of 2,451 pages. Dense embeddings treated the query as a vague semantic request about a number, not as an exact value that should be matched literally.

This is not a surprising failure. Embedding models are not reliable exact-match engines. Archive search, however, often depends on exact values: dates, names, identifiers, reference numbers, page ranges, and institutional terms.

Lexical and hybrid retrieval

We then compared dense retrieval with lexical and hybrid retrieval.

The lexical baselines used TF-IDF word n-grams and TF-IDF character n-grams. The hybrid retriever combined dense scores from Octen/Octen-Embedding-8B with lexical scores after per-query normalization:

hybrid_score = alpha * dense_score + (1 - alpha) * lexical_score

The best overall result came from character n-gram TF-IDF with a small dense component:

The improvement is too large to ignore. The best hybrid setup more than doubled Hit@1 and MRR@10 compared with dense-only retrieval.

The character n-gram result is especially relevant for archive material. It is more tolerant of OCR/HTR noise, spelling variation, and inflectional endings than strict word matching.

What this means architecturally

The main conclusion is not simply “use TF-IDF instead of embeddings”. The conclusion is that archive RAG should not be dense-only.

A practical retrieval architecture should use several complementary signals:

user query
  -> query analysis
     -> semantic embedding
     -> lexical terms
     -> entities
     -> dates
     -> numbers and identifiers
     -> topics / keyphrases

document pages
  -> dense vectors
  -> raw text index
  -> lemmatized text index
  -> character n-gram index
  -> entity/date/number fields

retrieval
  -> dense candidates
  -> lexical candidates
  -> structured exact-match candidates
  -> fused or reranked result list

This matters because different query types need different treatment.

In other words, the embedding model should be one part of the retrieval system, not the whole retrieval system.

Query-side analysis is the next step

The current hybrid experiment used simple score fusion. The next version should also analyze the query itself.

For example:

Leia lehekülg, kus esineb number 75-100.

A query-aware system should extract 75-100 as a number/range and strongly boost or require pages containing that value. It should not leave that decision to dense embeddings.

Similarly:

Mis toimus 10. detsembril 1934?

The system should normalize the date, search a parsed-date field, and still use dense retrieval for the semantic context around “what happened”.

The same applies to names and organisations. NER should run on both documents and queries. This would allow the retriever to combine “find pages about this concept” with “and this person or institution must be present”.

Lemmatisation should be added

Estonian morphology makes lemmatisation important for the lexical side of retrieval. Dense models often handle inflectional variation reasonably well, but lexical retrieval needs help.

For example, these forms should be connectable:

valitsus
valitsuse
valitsusele
valitsuses
valitsuselt

The safer architecture is not to replace raw text with lemmatised text, but to index both:

Annif already uses simplemma(et) in the EMS setup, so a lightweight first prototype can use the same lemmatisation route for Estonian lexical retrieval.

Updated recommendation

The earlier embedding benchmark remains useful for selecting dense vector models. The new RAG-style test changes the deployment recommendation:

1. Use Octen/Octen-Embedding-4B as the default dense model when speed, storage, and operating cost matter.
2. Use Octen/Octen-Embedding-8B when dense-only ranking quality is the priority and the higher cost is acceptable.
3. Do not use dense retrieval alone for archive RAG.
4. Build the production retriever as a hybrid system with lexical, character n-gram, lemmatised, entity, date, and number-aware retrieval.
5. Run NER and date/number parsing on the query as well as on the documents.

The most important learning is that embedding models are good at meaning, not at every form of retrieval. Archival access needs meaning, but it also needs exact evidence. A useful system has to support both.

What to test next

The next evaluation step should compare:

• dense-only retrieval
• lexical BM25 or TF-IDF retrieval
• character n-gram retrieval
• lemmatised lexical retrieval
• query-aware entity/date/number retrieval
• hybrid fusion
• reranking over the top candidates

The current synthetic dataset is useful for development, but the next validation step should include real user-style search questions or partner-provided retrieval examples. That will show how much of the hybrid gain transfers from synthetic queries to actual archive access workflows.