GiellaLT provides an infrastructure for rule-based language technology aimed at minority and indigenous languages, and streamlines building anything from keyboards to speech technology. Read more about Why. See also How to get started and our Privacy document.
The script lookup2cg reformats the lookup output so that it can be
interpreted as input to CG, the tool vislcg3 input. lookup2cg is a
perl script, and as all other scripts, it is located in the gt/script
directory.
The input to the script is the output of lookup. The command to
produce the input is, e.g:
$ echo "Dán" | lookup -flags mbTT -utf8 ~/main/gt/sme/bin/sme.fst
0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
Dán dát+Pron+Dem+Sg+Acc
Dán dát+Pron+Dem+Sg+Gen
The lookup gives a list of available analyses for a given word form. The
output of the lookup2cg script is input to vislcg which
requires a format where the analyzed word form comes before the
analyses. The analysis lines will have to consist of a base form in “”
followed by the morphological tags.
"<Dán>"
"dát" Pron Dem Sg Acc
"dát" Pron Dem Sg Gen
The script reads one cohort at the time, and reorganizes the different parts of the analysis. In addition to the basic processing, lookup2cg has some special functions: It constructs the base form of an analyzed compound by comparing the analyses and the original input form. The compounds are rated according to the compounding points and only the analyses with least number of the compounding points are preserved. In addition to compound processing, the derivational tags which are not taken into account in CG are marked in the analysis with asterix (*). These special funcitons of the lookup2cg are discussed in detail in the following sections.
The input to CG consists of the analyzed word form followed by a list of possible analyses. Each analysis contains a base form and the morphological tags. The compounds are problematic in this respect; in the lookup output, the analysis of a compound expression contains also the complete analyses of its parts. For example,
$ echo "bohccobiergobuktagiid" | lookup -flags mbTT -utf8 ~/main/gt/sme/bin/sme.fst
0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
bohccobiergobuktagiid boazu+N+SgGenCmp+Cmp#buvtta+N+Sg+Acc
bohccobiergobuktagiid boazu+N+SgGenCmp+Cmp#buvtta+N+Sg+Gen
However, in CG, only the tags of the last compounding word are examined, and the analyses of the compounding parts are redundant information. The intermediate tags may thus be removed. On the other hand, the base form of the compound as a whole is not available, but has to be constructed in lookup2cg.
The problematic part is identifying the compound boundary. Just taking the first part from the analysis will not do, as there may be changes of 3 kinds: The final vowel (á, i, u) may have been weakened to (a, e, o), as for dállodoall_o_ekonomiija; there may be consonant gradation in the form (as when ‘alimus#riekti#duopmu’ becomes ‘alimusrievttiduomuin’) with a kt:vtt change; and the compound form may be shortened (and eventually changed), as when ‘geahččat + vuohki’ becomes ‘geahččanvuogi’.
The base form is constructed basically by taking the analyzed word form, in this case “bohccobuktaga” and replacing the last word “buktaga” by its base form “buvtta”. The output of the lookup2cg is then:
"<bohccobuktaga>"
"bohcco#buvtta" N Sg Gen
"bohcco#buvtta" N Sg Acc
The compound boundary is not marked in the input word “bohccobuktaga” but it has to be searched. The search is done by seraching the first letters of the base form with the input word form. First, the first 4 letters of the base form are searched from the input word; then 3, 2, and, as a last resort, 1. In the previous example, the matching string consisted of two letters: “bu”.
This method is a source of a number of errors, since it is common that a string of two letters occurs several times in the compound, not to mention a string of only one letter. For example the compound “sierravuoigatvuođaid” has among others the following reading:
sierravuoigatvuođaid sierra+A+Attr+Cmp#vuoigat+A+Der/vuohta+N+Pl+Acc
Now when the base form “vuohta” is searched from the analyzed form “sierravuoigatvuođaid” by first comparing the first 4 letters of “vuohta” namely “vuoh”. This string cannot be found. Then first 3 letters are searched: “vuo”. That string occurs two times in the input word and basically there is no way to determine which one is correct. There is a heuristic rule involved that selects the latter occurence and replaces the word “vuođaid” vith “vuohta”. The resulting base form is thus:
"<sierravuoigatvuođaid>"
"sierravuoigatvuohta" N Pl Acc
(This is not completely true, since the derivational tags (here A) are not removed in the lookup2cg, but if they were.) The word form “sierravuoigatvuođaid” has another analysis as well, namely one which does not involve the derivational tag:
sierravuoigatvuođaid sierra+A+Attr+Cmp#vuoigatvuohta+N+Pl+Acc
Now the base form of the last part is “vuoigatvuohta”. The first 4 letters “vuoi” are found from the analyzed form. Importantly, the first 3 letters would not sufffice to determine the correct word boundary, since the string “vuo” occurs two times in the analyzed word form. If the heuristic that selects the latter string were used, the wrong word form would be produced: “sierravuoigatvuoigatvuohta”. Consider the word “sealgeetniin” which has among others the following readings:
sealgeetniin sealgi+N+SgNomCmp+Cmp#eadni+N+Sg+Com
sealgeetniin sealgi+N+SgNomCmp+Cmp#eadni+N+Pl+Loc
The strings that are to be searched from the form “sealgeetniin” are the following “eadn”, “ead”, “ea” and “e”, from the base form of the last part “eadni”. By chance, the string “ea” is found from the input word, but not in the correct place. The correct string to search would have been the last “e” in the input word form. The procedure wrongly generates:
"<sealgeetniin>"
"s#eadni" N Pl Loc
"s#eadni" N Sg Com
This bug seems to be solved (?) also without lexicalising the word:
"<vealgeetniin>"
"vealge#eadni" Hum N Pl Loc
"vealge#eadni" Hum N Sg Com
Clearly, the basic string-comparison operations are not satisfactory method for producing base forms for compounds. The alternative would be to start using generative lexicon in finding out the base form, this is not implemented, mainly due to practical reasons.
Note that dropping the analyses of the compound parts also make it possible to get rid of “ambiguities” like the following:
rámmaeaktu rámma+N+SgNomCmp+Cmp#eaktu+N+Sg+Nom
rámmaeaktu rámma+N+SgGenCmp+Cmp#eaktu+N+Sg+Nom
And to produce only one one analysis for CG:
"<rámmaeaktu>"
"rámma#eaktu" N Sg Nom
The compounds are rated according to (Fred) Karlsson’s law: “In a compound word analysis, the analysis with the fewest word boundaries is the correct one.” Only the compounds with the fewest word boundaries are preserved.
For example, the following input to the lookup2cg:
$ echo "bohccobiergobuktagiid" | lookup -flags mbTT -utf8 ~/main/gt/sme/bin/sme.fst
0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
bohccobiergobuktagiid boazu+Ani+N+SgGenCmp+Cmp#biergu+N+SgNomCmp+Cmp#buvtta+N+Pl+Gen
bohccobiergobuktagiid boazu+Ani+N+SgGenCmp+Cmp#biergu+N+SgNomCmp+Cmp#buvtta+N+Pl+Acc
bohccobiergobuktagiid boazu+Ani+N+SgGenCmp+Cmp#biergobuvtta+N+Pl+Gen
bohccobiergobuktagiid boazu+Ani+N+SgGenCmp+Cmp#biergobuvtta+N+Pl+Acc
bohccobiergobuktagiid bohccobiergu+N+SgNomCmp+Cmp#buvtta+N+Pl+Gen
bohccobiergobuktagiid bohccobiergu+N+SgNomCmp+Cmp#buvtta+N+Pl+Acc
The compounds are rated straight after they arrive to the lookup2cg. Only the readings with the fewest compounding points are subject to further processing, in this example the lines:
"bohccobiergo#buvtta" N Pl Acc
"bohccobiergo#buvtta" N Pl Gen
"bohcco#biergobuvtta" N Pl Gen
"bohcco#biergobuvtta" N Pl Acc
Since the input to the parser is a human-readable dictionary, many derivations are present already in the dictionary. Due to the dynamic derivation component, they come out with a double or even multiple analysis, as the analysis with the derivational affix added in the parsing process is given as well. thus, we have “ambiguities” like the following:
$ echo "mearkkašupmi" | lookup -flags mbTT -utf8 ~/main/gt/sme/bin/sme.fst
0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
mearkkašupmi mearkkašit+V+TV+Der/PassL+V+Der/upmi+N+Sg+Nom
mearkkašupmi mearkkašupmi+N+Sg+Nom
$ echo "ealáhusheiveheapmi" | lookup -flags mbTT -utf8 ~/main/gt/sme/bin/sme.fst
0%>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
ealáhusheiveheapmi ealáhus+N+SgNomCmp+Cmp#heivet+V+IV+Der/h+V+Der/NomAct+N+Sg+Nom
ealáhusheiveheapmi ealáhus+N+SgNomCmp+Cmp#heivehit+V+TV+Der/NomAct+N+Sg+Nom
ealáhusheiveheapmi ealáhus+N+SgNomCmp+Cmp#heiveheapmi+N+Sg+Nom
ealáhusheiveheapmi ealáhusheiveheapmi+N+Sg+Nom
Here is the list of the derivational tags for North Sámi:
+Der/adda +Der/ahtti +Der/alla +Der/asti +Der/d +Der/NomAct +Der/eamoš +Der/eapmi +Der/g +Der/geahtes +Der/h
+Der/heapmi +Der/hudda +Der/huhtti +Der/huvva +Der/j +Der/l +Der/laš
Der/+meahttun +Der/muš +Der/n +Der/š +Der/st +Der/stuvva +Der/us +Der/vuohta +Der/goahti +Der/lágan
+Der/Dimin +Der/PassL +Der/PassS
The derivational tags are associated with at least a POS tag (N, V, Adv, A). The POS tags are marked with asterisk (*) to distinguish them from the POS tag of the compound. Thus the output of lookup2cg for is the following:
"<mearkkašupmi>"
"mearkkašit" V* TV Der/PassL V* Der/upmi N Sg Nom
"mearkkašupmi" N Sg Nom
"<ealáhusjurddašeapmi>"
"ealáhus#jurddašit" V* TV Der/NomAct N Sg Nom
The marking of the tags and constructing the base form of a word with derivational suffixes has to be reconsidered. More of derivational suffixes is presented in the following chapter. The improvements listed there are not implemented in lookup2cg.
The goal is to feed only syntactically relevant information to the disambiguator. So, in the analysis of “bargiin”, the correct analysis is that it is Sg Com of “bargi”. Since this word is lexicalised, it is found as a noun in the lexicon.
"bargiin" S:1995
"bargat" V* TV Der/NomAg N Sg Com
"bargi" N NomAg Sg Com
What we want is thus to treat all Actor nouns as if they were found in the lexicon in the first place. The problem is then to reverse the morphological process, and find the stem.
"lohkamat" S:631, 631, 631
"lohkat" V* TV Der/NomAct N Pl Nom
"lohkan" N Pl Nom
These ones do not induce consonant gradation in the stem:
These ones do:
For the non-gradating verb-to-noun suffixes, remove the V label preceeding the N.
"" S:1708
"čuovvut" V* TV Der/l V* Der/NomAct N Sg Nom
"čuovvulit" V* TV Der/NomAct N Sg Nom
""
"vuodjat" V* IV Der/d V* Der/NomAct N Sg Acc
"vuodjat" V* IV Der/d V* Der/NomAct N Sg Gen
""
"jorgalit" V* TV Der/ahtti V* TV Der/NomAct N Sg Ill
"jorgalahttit" V* TV Der/NomAct N Sg Ill
"" S:662
"mearridit" V Der/NomAg N Pl Ill
For the gradating suffixes, we should think more before doing anything.
"" S:636, 1479
"lassi" N* Der/heapmi A Sg Nom