North Sami NLP Grammar

Finite state and Constraint Grammar based analysers, proofing tools and other resources

giellalt/lang-sme
giellalt/lang-sme

North Sámi speller accuracy review — path to 95% top‑1 / 99% top‑5

Date: 2026‑08‑20 (outcomes addendum 2026‑08‑21) Scope: lang-sme desktop speller (se.zhfst) suggestion quality Author: Divvun tooling review (automated analysis + build-system survey)

Outcomes (addendum, 2026‑08‑21)

The programme below was executed over two days. Results on the corrected corpus (10,679 pairs; three wrong expected forms fixed, one real-word pair and one non-typo pair removed, five mojibake lines repaired):

Metric (n‑best 100) Before After Production config
Top‑1 80.98% 83.3% 83.2%
Top‑5 93.94% 96.0% 94.8%
Correct offered at all 97.04% 98.5% 95.3%
No suggestions 18 0 —
Typo accepted as correct 11 0 —

Landed: the LM maxweight raise (10→50, recommendation A); the missing confusion pairs ŋ↔n, i↔y, s↔ŧ and chunk rewrites incl. the nj↔ŋ digraph family (B, C); lexicon curation of the false accepts and missing forms (E); the +Cmp penalty confirmed already optimal (G). In divvunspell (affects all Giella languages): the n‑best recall bug fixed, the search converted to best‑first with exact state deduplication — 29% faster mean, 7× better worst case, identical results. Recommendation F was implemented not as pair enumeration but as extract-errorth-strings.py: 5,238 stem-level alternation rules derived from the lexicon’s +Err/Orth* entries into the strings component — beating the enumerated-pairs approach on every metric and correcting misspelled stems inside dynamic compounds no pair list could contain (make errorth-regen reproduces the rules byte-for-byte). Known cost: se.zhfst grows to ~131 MB (from 56 MB); reduction options (rule-support thresholds, non-determinised error models at a search-time cost) are understood if size matters downstream.

Sign‑off: full make check green under both the C++ 3.17.1 toolchain and the Divvun HFST (Rust) toolchain, identical test sets. Remaining distance to 95/99 is ranking-dominated; the per-category tracking recommended below is the tool for the next iteration.

Summary

Measured against the full 10,682‑pair tools/spellcheckers/test/typos.tsv gold set, the current speller places the correct suggestion first for 80.98% of typos and in the top five for 93.94%. The two dominant, fixable causes are:

  1. Lexical frequency barely participates in ranking. The corpus language model compresses the entire frequency spectrum of the language into 8 weight units, while a single edit in the error model costs 5–59. Rare wrong-lemma neighbours routinely outrank frequent correct words, and exact weight ties are broken alphabetically.
  2. The error model cannot reach 316 of the corrections at all, 255 of which are ≥3 character edits away — beyond its 2‑edit budget. Most of these are single contiguous stem or compound alternations (e.g. kultuvra‑ ↔ kultur‑) that targeted string patterns can cover without enlarging the edit budget.

With the recommended changes, ~90% top‑1 and ~97% top‑5 are high-confidence outcomes; 95% / 99% is reachable but requires the full programme, including pattern mining for stem alternations and exploiting the lexicon’s +Err/Orth knowledge. The corpus itself is clean (ceiling 99.94%), so the targets are not data-limited.

Methodology

Current results

Metric Count Rate
Correct suggestion first (top‑1) 8,650 / 10,682 80.98%
Correct in top 5 10,035 93.94%
Correct anywhere in top 100 10,366 97.04%
Correct never offered 316 2.96%
Typo accepted as correct (false negative) 11 0.10%
Correct word rejected (false accept) 0 0%

Gap to targets: 95% top‑1 needs +1,498 promotions to first place; 99% top‑5 needs +541 into the top five.

Anatomy of the 2,032 top‑1 misses

Bucket Count Diagnosis
Correct at position 1–4 1,385 Ranking failure: a wrong one-edit neighbour outranks the right correction
— of which exact weight ties 139 Tie-break is alphabetical; the winner sorts before the expected word in all 139
— weight gap ≤ 5 units 507 Flippable with modest cost tuning
— weight gap > 10 units 588 Need category-level repricing or frequency separation
Correct at position 5–99 331 87% are overpriced by >20 units
Correct absent at n=100 316 310 are accepted by the lexicon but never generated; only 4 words are genuinely missing
— of the absent: edit distance ≥3 255 Beyond the 2‑edit model. 180 are one contiguous ≤6‑char mid-word chunk (stem/compound alternation)

Confusion analysis across all misses (top pairs, input→expected): a→á 216, á→a 138, consonant gemination/degemination (the del/ins t, k, l mass, several hundred combined), o→u 79, e→i 73, u→o 67, d→đ 49, s→š 43, j→ŋ 43, i→e 48.

The wrong corrections that win are dominated by wrong-lemma a↔á flips (140), spurious vowel insertions/deletions, and ii↔i — i.e. the cheap edits applied to the wrong word.

Root causes

1. The frequency model is crushed (ranking)

tools/spellcheckers/weights/config.json sets maxweight: 10. The resulting unigram LM (built by giella-core/scripts/corpus2unigramlm.py over weights/spellercorpus.raw.txt, 3.1M tokens / 166k types) maps the most frequent word in the language (ja, weight 1.98) and a hapax (9.54) into a spread of 8 units, with out-of-corpus words at 10.0. Suggestion ranking is frequency + tag penalties + error-model cost, and error-model costs run 5–59 with +Cmp at 45 — so frequency is nearly irrelevant. Symptoms:

2. Edit costs contradict the observed error distribution (ranking)

tools/spellcheckers/editdist.default-new.regex (the hand-written model; it shadows the older generated path) prices:

The result: for a typo whose real fix is gemination or a vowel-pair substitution, some other word one cheap á-flip away wins. This is the single largest miss category.

3. The 2‑edit budget cannot reach 316 corrections (coverage)

Of the 316 never-offered corrections, 310 are in the lexicon. 255 are ≥3 raw edits from the typo — but 180 of those 255 differ by one contiguous mid-word chunk of ≤6 characters: compound-form alternations (kultuvradoalut → kulturdoalut), stem alternations, colon-inflection forms (Ab:a → Ab). These are systematic morphology, not random noise, and are addressable as single weighted string-rewrite patterns. Raising EDIT_DISTANCE to 3 is not recommended: the error model is already 245 MB with 1.27 s worst-case lookups.

4. divvunspell n‑best pruning leaks recall (client)

For 9 of the 43 words that hit the n=100 suggestion cap, the correct word does surface at -n 2000 — one at true post-reweight rank 12. The n‑best cut is applied against pre-reweight raw weights, so genuinely good candidates are pruned before reordering. This also explains previously observed n‑best-sensitivity between equivalent archives.

5. Small lexicon issues

# Change Where Effort Expected effect
A Raise LM maxweight from 10 to ~30–50 (sweep empirically) tools/spellcheckers/weights/config.json 1 line + acceptor rebuild (~5 min) per trial Largest single lever: restores frequency separation across all ranking buckets; fixes alphabetical ties as a side effect
B Recalibrate edit costs against the confusion matrix: gemination classes cheaper; add o↔u, e↔i, a↔e/i (~10–15), ŋ↔n, nj↔ŋ (~5–10), i↔y; keep generic fallbacks as-is tools/spellcheckers/editdist.default-new.regex Iterative; errmodel-only rebuilds are minutes Attacks the 507 near-miss ranking failures and much of the deep bucket
C Add chunk-rewrite patterns for the systematic stem/compound alternations and colon-inflection paradigms strings.default.regex, final_strings.default.txt Pattern mining, iterative The only viable route to 99% top‑5: covers the 255 distance‑3+ absents at one edit each
D Fix n‑best pruning to cut on post-reweight weights divvunspell Client-side fix Recovers up to ~43 absents, some at true rank ≤31; benefits all Giella languages
E Lexicon curation: add the 4 missing forms; restrict kultuvra‑ in compound-left position; review the 11 false accepts src/fst/ lexc sources Small Removes the false negatives; ~15 words
F Compile the lexicon’s +Err/Orth* pairs (incl. the dedicated Err/Orth-a-á class) into an error-model component instead of hard-deleting them via remove-error-strings.regex build system + src/fst/ Medium project Injects curated normative-error knowledge; closes part of the final gap
G Tune +Cmp +45 (per compound boundary) once A lands weights/tags.reweight 1 line per trial Currently suppresses legitimate compound suggestions; must be re-balanced against the new LM range

Housekeeping found along the way: accents.default.regex and its composition rule (Makefile.mod-desktop-hfst.am:105‑122) are dead code — the target is never consumed; strings.default.txt, initial_letters.*, DEFAULT_WEIGHT, USE_SWAPS, CORPUS_SIZE are all inert in the desktop configuration; clean_weight_corpus.sh output is built but never read (the LM tokenises the raw corpus itself). These should be deleted or deliberately revived to prevent future tuning-by-dead-knob.

Explicit non-recommendation: do not seed words.default.txt from typos.tsv pairs. That is training on the evaluation set; it would inflate the reported metric without improving real-world accuracy. Whole-word replacement entries should come from independent corpora (e.g. SIKOR error annotations).

Feasibility

Reproduction

# Build (Divvun HFST toolchain first in PATH):
./configure --without-forrest --disable-syntax --enable-spellers \
            --with-backend-format=openfst-tropical
make -j4

# Evaluate:
divvunspell accuracy -c cfg-n100.json -o report.json \
    tools/spellcheckers/test/typos.tsv tools/spellcheckers/se.zhfst
# cfg-n100.json = production config with "n-best": 100, "beam": null
Last updated: Edit on GitHub

Sitemap

On this page