Test diary

Lexical coverage

(testing on different conversion versions of the nt)

Number of words (standing in lang-tkl):

cat misc/nt_tkl.txt |hfst-tokenise -m tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst |wc -l
cat misc/nt.tkl.txt |tr -d "[0-9]"|hfst-tokenise -m tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst |wc -l

Number of unknown words:

cat misc/nt_tkl.txt |hfst-tokenise -mgW tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst |grep ' ?'|wc -l
cat misc/nt.tkl.txt |tr -d "[0-9]"|hfst-tokenise -mgW tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst |grep " ?"|wc -l

Lexical coverage nt

Coverage, NT:

241115: 1-(34193/351080) = 0.9026
241115: 1-(30018/350323) = 0.9143
241117: 1-(17098/350323) = 0.9512
241201: 1-(14505/320561) = 0.9548
241209: 1-(11008/320540) = 0.9656

Coverage, book collection:

        All words                      Not words with initial capital
241210: 1-(6868/62281) = 0.8897        1-(4549/44442) = 0.8976

Disambiguation

Assuming the disambiguation is correct:

Number of words in the test corpus:

cat misc/nt2.txt |hfst-tokenise tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst |wc -l

Number of readings in the test corpus:

cat misc/nt2.txt |hfst-tokenise -gmW tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst |vislcg3 -g src/cg3/disambiguator.cg3|grep -v '^[":]'|wc -l

Number of readings per word:

Without disambiguation, the number of readings is 1.835 per word (601788/327949)

With disambiguation:

241202: 408241/327949 = 1.24483
241202: 391579/327949 = 1.19402
241209: 383837/327928 = 1.17049