Udmurt NLP Grammar

Finite state and Constraint Grammar based analysers, proofing tools and other resources

View the project on GitHub giellalt/lang-udm

Udmurt language model documentation

All doc-comment documentation in one large file.


src-cg3-functions.cg3.md

These sets model noun phrases (NPs). The idea is to first define whatever can occur in front of the head of the NP, and thereafter negate that with the expression WORD - premodifiers.

The set NOT-NPMOD is used to find barriers between NPs. Typical usage: … (*1 N BARRIER NPT-NPMOD) … meaning: Scan to the first noun, ignoring anything that can be part of the noun phrase of that noun (i.e., “scan to the next NP head”)

These were the set types.

HABITIVE MAPPING

sma object

SUBJ MAPPING - leftovers

OBJ MAPPING - leftovers

HNOUN MAPPING


This (part of) documentation was generated from src/cg3/functions.cg3


src-fst-morphology-affixes-adjectives-old.lexc.md

UDMURT Adjective inflection

Udmurt adjectives inflect for:

Determinative notes

When an adjective occurs in determinative, it must agree for number and case. However, when determinative co-occurs with certain cases, these cases have alternate forms.

The cases with alternate forms are:

Note that these alternative suffixes have palatal and non-palatal versions

Nom. бадӟым-ез гурт бадӟым-ъёс-ыз гурт-ъёс
Iness. бадӟым-а-з гурт-ын бадӟым-ъёс-а-з гурт-ъёс-ын
Illat. бадӟым-а-з гурт-э бадӟым-ъёс-а-з гурт-ъёс-ы
Elat. бадӟым-ысьты-з гурт-ысь бадӟым-ъёс-ысьты-з гурт-ъёс-ысь
Nom. пичи-ез бакча пичи-ос-ыз бакча-ос
Iness. пичи-я-з бакча-ын пичи-ос-а-з бакча-ос-ын
Illat. пичи-я-з бакча-е пичи-ос-а-з бакча-ос-ы
Elat. пичи-ысьты-з бакча-ысь пичи-ос-ысьты-з бакча-ос-ысь

TODO: ordering of suffixes sometimes differs? Compare this with possession morph.

бадӟым-ъёс-ысьты-з
ADJ - PL - CASE - DET
vs
бадӟым-ъёс-ыз-лэн
ADJ - PL - DET - CASE

Plurality in Determinative Note that there are two options for how plural attributive adjectives may appear. One option is the aforementioned, another option is for plural to appear twice.

In this situation, one of the plurals has a different form: These examples are from Udmurtin kielioppi:

Absolutive congruence

Case Yks. Mon.
Nom. бадӟым гурт бадӟым(-есь) гурт-ъёс
Iness. бадӟым гурт-ын бадӟым(-есь) гурт-ъёс-ын
Illat. бадӟым гурт-э бадӟым(-есь) гурт-ъёс-ы
Elat. бадӟым гурт-ысь бадӟым(-есь) гурт-ъёс-ысь

Determinative Singular ~ Plural | Nom. | бадӟым-ез гурт | бадӟым-ъёс-ыз гурт-ъёс | Iness. | бадӟым-а-з гурт-ын | бадӟым-ъёс-а-з гурт-ъёс-ын | Illat. | бадӟым-а-з гурт-э | бадӟым-ъёс-а-з гурт-ъёс-ы | Elat. | бадӟым-ысьты-з гурт-ысь | бадӟым-ъёс-ысьты-з гурт-ъёс-ысь

TODO: what is the difference?

Determinative Plural with congruence

Case Abs. Det.
Nom. бадӟым-есь гурт-ъёс бадӟым-есь-ёс-ыз гурт-ъёс
Iness. бадӟым-есь гурт-ъёс-ын бадӟым-есь-ёс-а-з гурт-ъёс-ын
Illat. бадӟым-есь гурт-ъёс-ы бадӟым-есь-ёс-а-з гурт-ъёс-ы
Elat. бадӟым-есь гурт-ъёс-ысь бадӟым-есь-ёс-ысьты-з гурт-ъёс-ысь

words with this contlex have not been examined

Cases here all lead to a determinative suffix

Cases here all lead to a determinative suffix


This (part of) documentation was generated from src/fst/morphology/affixes/adjectives-old.lexc


src-fst-morphology-affixes-adjectives.lexc.md

UDMURT Adjective inflection

Udmurt adjectives inflect for:

Determinative notes

When an adjective occurs in determinative, it must agree for number and case. However, when determinative co-occurs with certain cases, these cases have alternate forms.

The cases with alternate forms are:

Note that these alternative suffixes have palatal and non-palatal versions

Nom. бадӟым-ез гурт бадӟым-ъёс-ыз гурт-ъёс
Iness. бадӟым-а-з гурт-ын бадӟым-ъёс-а-з гурт-ъёс-ын
Illat. бадӟым-а-з гурт-э бадӟым-ъёс-а-з гурт-ъёс-ы
Elat. бадӟым-ысьты-з гурт-ысь бадӟым-ъёс-ысьты-з гурт-ъёс-ысь
Nom. пичи-ез бакча пичи-ос-ыз бакча-ос
Iness. пичи-я-з бакча-ын пичи-ос-а-з бакча-ос-ын
Illat. пичи-я-з бакча-е пичи-ос-а-з бакча-ос-ы
Elat. пичи-ысьты-з бакча-ысь пичи-ос-ысьты-з бакча-ос-ысь

TODO: ordering of suffixes sometimes differs? Compare this with possession morph.

бадӟым-ъёс-ысьты-з
ADJ - PL - CASE - DET
vs
бадӟым-ъёс-ыз-лэн
ADJ - PL - DET - CASE

Plurality in Determinative Note that there are two options for how plural attributive adjectives may appear. One option is the aforementioned, another option is for plural to appear twice.

In this situation, one of the plurals has a different form: These examples are from Udmurtin kielioppi:

Absolutive congruence

Case Yks. Mon.
Nom. бадӟым гурт бадӟым(-есь) гурт-ъёс
Iness. бадӟым гурт-ын бадӟым(-есь) гурт-ъёс-ын
Illat. бадӟым гурт-э бадӟым(-есь) гурт-ъёс-ы
Elat. бадӟым гурт-ысь бадӟым(-есь) гурт-ъёс-ысь

Determinative Singular ~ Plural | Nom. | бадӟым-ез гурт | бадӟым-ъёс-ыз гурт-ъёс | Iness. | бадӟым-а-з гурт-ын | бадӟым-ъёс-а-з гурт-ъёс-ын | Illat. | бадӟым-а-з гурт-э | бадӟым-ъёс-а-з гурт-ъёс-ы | Elat. | бадӟым-ысьты-з гурт-ысь | бадӟым-ъёс-ысьты-з гурт-ъёс-ысь

TODO: what is the difference?

Determinative Plural with congruence

Case Abs. Det.
Nom. бадӟым-есь гурт-ъёс бадӟым-есь-ёс-ыз гурт-ъёс
Iness. бадӟым-есь гурт-ъёс-ын бадӟым-есь-ёс-а-з гурт-ъёс-ын
Illat. бадӟым-есь гурт-ъёс-ы бадӟым-есь-ёс-а-з гурт-ъёс-ы
Elat. бадӟым-есь гурт-ъёс-ысь бадӟым-есь-ёс-ысьты-з гурт-ъёс-ысь

words with this contlex have not been examined


This (part of) documentation was generated from src/fst/morphology/affixes/adjectives.lexc


src-fst-morphology-affixes-adverbs.lexc.md

Adverbs

Udmurt (Votyak) .

The adverbs have a minimal morphology, just the tag +Adv.

LEXICON ADV_ Undeveloped contlexes


This (part of) documentation was generated from src/fst/morphology/affixes/adverbs.lexc


src-fst-morphology-affixes-nouns-old.lexc.md

UDMURT Noun inflection

Nouns in Udmurt inflects for case, number, and possession. Following are some notes for things that need to be done so far, and also an overview of nominal inflection types.

Most Udmurt nouns (the vast majority) do not require changes to the stem as a result of inflection. These nouns however, for FST purposes, fall into some subcategories:

Some inflectional endings thus may have two alternatives:

Case Non-yod Yod
Nominative plural ос ёс
Accusative singular эз ез
Instrumental singular эн ен
Illative singular э е
Prolative singular этӥ етӥ

Other cases do not vary depending on these environments.

Note that depending on the palatal/nonpalatal status of the last segment, the yodified vowel series may be preceded by a <ъ>. According to references though, the palatal marker <ь> must remain even when followed by this series. Udmurt also has a unique way of handling hiatus relationships, where the non-yod series follows vowels, and the yod series follows consonants.

Also note that though stems may not change, orthographically, there may need to be a hard sign inserted to maintain the non-palatal pronunciation of certain consonant segments.

Some examples:

-V stems

NomSg NomPl IneSg GenSg Advl Sg
бачка бачкаос бакчаын бакчалэн бачкая
дунне дуннеос дуннеын дуннелэн дуннея
ты тыос тыын тылэн тыя

-C stems

NomSg NomPl IneSg GenSg Advl Sg
гурт гуртъёс гуртын гуртлэн гуртъя
тэль тельёс тэлъын тэлълэн тэлья
вить витьёс витьын витьлэн витья

Diphthongs

NomSg NomPl IneSg GenSg Advl Sg
уй уйёс уйын уйлэн уйя

Russian loans ending in -ия/-ие

Russian loanwords ending on these segments delete the final yodified vowel in all cases, except for the singular adverbial and nominative singular.

Examples:

Nominative Singular конференци–0–я собрани–0–е
Nominative Plural конференци–0–ос собрани–0–ос
Inessive Singular конференци–0–ын собрани–0–ын
Genitive Singular конференци–0–лэн собрани–0–лэн
Adverbial Singular конференци–я–я собрани–е–я

Note also that Russian stress is preserved in Nominative Singular, but Udmurt stress takes over for inflected forms, and shifts to the final vowel.

This applies to all sorts of Russian words, also including placenames ending in -я, excepting biblical place names, and Russian personal names.

TODO: implement in propernouns

But:

There are also some subtypes, where this deletion does not occur even though the word ends in -я:

Place names based on names of old tribes

See affixes/propernouns.lexc, but quickly:

These typically end on -я, –га, –ка, –ра, –ла. When inflecting in some cases, they take an infix which is optional. The optional infix causes short versions of case suffixes, otherwise the words take the long version.

корка / куала

The words корка, куала have some forms similar to the tribe/place names, and some cases behave differently: Inessive, Elative have the short consonant-only suffix, but unlike the tribe/place names, there is no -ла- infix.

TODO: write tests

“Old” Finno-ugric stems (-м-, -г-, -т-)

TODO: add these to lexicon, and add morphology. Consider marking in morphology in a way that NDS can present: син син(м)- TODO: write tests TODO: how to mark the inflectional stems in the lexicon for display in dictionaries, similar to smenob?

<l>muohta</l>
<lc>muohttaga</lc>

Some particularly old words have since experienced some amount of deletion and simplification of clusters. Deleted segments reappear in some forms (inflectional, derivational), but their appearance isn’t completely regular, in that some variation exists, and sometimes the “short” stem without this segment appears where the “long” stem usually would.

Usually, these will also be marked in dictionaries. син син(м)-.

When the possessive is applied to nominative versions of these forms, the possessive will have the vowel -ы- instead of -э- or -е-. Also, the ‘old’ segments appear.

Examples:

-м–

син син–тэм синм–ысь сынм–ы
silmä sokea (elat.) (px. y. 1. p.)
визь визь–тэк визьм–о  
järki (abess.) viisas  
куйнь куйнь–лэсь куйньм–ой  
kolme (abl.) 3:n päivän kuluessa  
  куйнь–етй ~ куйньм–етӥ  
  kolmas kolmas  
ин   инм–ын  
taivas   (iness.)  

-к–

кус кус–лы куск–ын куск–ы
lantio (dat.) (iness.) (Px Sg1)
vyötärö      
улеп улеп–лы улепк–ын  
elävä (dat.) (instr.)  
nюлэс нюлэс–лы нюлэск–ын  
metsä (dat.) (iness.)  
вис вис–лы виск–ын  
tauko (dat.) (iness.)  
väli      

-т–

кык кык–намы кыкт–о
kaksi me kaksin kaksoset
    kaksospari
  кык–етӥ кыкт–ой
  toinen kahden päivän kuluessa
    кыкт–етӥ
      toinen

Vowel deletion stems

Some stems also delete vowels in specific cases, generally when the stem is attached to a vowel-initial suffix. Alternatively, the vowels may remain, but a epenthetic consonant may appear: -к-

Optional deletion

сэрег серг–ын  
nurkka (iness.)  
kulma      
сюлэм сюлм–ыд сюлэм–ыд
sydän (Px Sg2) (Px Sg2)

Deletion or epenthesis

вадес вадьс-ын вадеск–ын
aika, hetki (iness.) (iness.)
paikka, palsta      
пыдэс пыдс–ын пидэск–ын
pohja (iness.) (iness.)
удрэс удрс–ын удрэск–ын
vierus (iness.) (iness.)

NB: palatal quality implied by the yodified vowel in the first example is preserved via a soft sign.

Suffix ordering with possessive construction

The possessive construction results in different case ordering:

N-Num-Poss-Case:

N-Num-Case-Poss:

Note that since the adjective ‘determinative’ is essentially the 3rd person singular possessive, the above ordering also follows for adjective inflection. Also note, some cases have atlernate forms for when tehy come before the possessive.

Cases with alternate forms:

Examples:

Genetive:

Px 1pSg Px 2pSg Px 3pSg
бакча-е-лэн бакча-ед-лэн бакча-ез-лэн
бакча-ос-ы-лэн бакча-ос-ыд-лэн бакча-ос-ыз-лэн

Inessive:

Px 1pSg Px 2pSg Px 3pSg
бакча-я-м бакча-я-д бакча-я-з
бакча-ос-а-м бакча-ос-а-д бакча-ос-а-з

Elative:

Px 1pSg Px 2pSg Px 3pSg
бакча-ысьты-м бакча-ысьты-д бакча-ысьты-з
бакча-ос-ысьты-м бакча-ос-ысьты-д бакча-ос-ысьты-з

Noun Lexica

Words from lexicon N_ are mostly covered. These are from the main class of nouns described in more depth above.

TODO: +Sg: POSTPX ;

Words from Russian in N_RUS_JA end on –ия, but our lexicon entries set the stem as ending in –и, because the final -я appears only in some situations. TODO: possessives

Words from Russian in N_RUS_JE end on –ие, but our lexicon entries set the stem as ending in –и, because the final -е appears only in some situations. TODO: possessives

CASESUF_JE controls the occurrence of the original -е in the stem, which occurs in Nominative and Adverbial cases. Also note that cases with a yodified possibility in their initial vowel use this yodified version.

CASESUF_JA controls the occurrence of the original -я in the stem, which occurs in Nominative and Adverbial cases. Also note that cases with a yodified possibility in their initial vowel use this yodified version.

CASESUF_KORKA controls the output of case suffixes in the корка/куала type which tends to take short case suffixes when they are available.

Words from Tatar in N_TAT preserve the original stem я in all cases.

À la Jaska


This (part of) documentation was generated from src/fst/morphology/affixes/nouns-old.lexc


src-fst-morphology-affixes-nouns.lexc.md

UDMURT Noun inflection

Nouns in Udmurt inflects for case, number, and possession. Following are some notes for things that need to be done so far, and also an overview of nominal inflection types.

Most Udmurt nouns (the vast majority) do not require changes to the stem as a result of inflection. These nouns however, for FST purposes, fall into some subcategories:

Some inflectional endings thus may have two alternatives:

Case Non-yod Yod
Nominative plural ос ёс
Accusative singular эз ез
Instrumental singular эн ен
Illative singular э е
Prolative singular этӥ етӥ

Other cases do not vary depending on these environments.

Note that depending on the palatal/nonpalatal status of the last segment, the yodified vowel series may be preceded by a <ъ>. According to references though, the palatal marker <ь> must remain even when followed by this series. Udmurt also has a unique way of handling hiatus relationships, where the non-yod series follows vowels, and the yod series follows consonants.

Also note that though stems may not change, orthographically, there may need to be a hard sign inserted to maintain the non-palatal pronunciation of certain consonant segments.

Some examples:

-V stems

NomSg NomPl IneSg GenSg Advl Sg
бакча бакчаос бакчаын бакчалэн бакчая
дунне дуннеос дуннеын дуннелэн дуннея
ты тыос тыын тылэн тыя

-C stems

NomSg NomPl IneSg GenSg Advl Sg
гурт гуртъёс гуртын гуртлэн гуртъя
тэль тэльёс тэльын тэльлэн тэлья
вить витьёс витьын витьлэн витья

Diphthongs

NomSg NomPl IneSg GenSg Advl Sg
уй уйёс уйын уйлэн уйя

Russian loans ending in -ия/-ие

Russian loanwords ending on these segments delete the final yodified vowel in all cases, except for the singular adverbial and nominative singular.

Examples:

Nominative Singular конференци–0–я собрани–0–е
Nominative Plural конференци–0–ос собрани–0–ос
Inessive Singular конференци–0–ын собрани–0–ын
Genitive Singular конференци–0–лэн собрани–0–лэн
Adverbial Singular конференци–я–я собрани–е–я

Note also that Russian stress is preserved in Nominative Singular, but Udmurt stress takes over for inflected forms, and shifts to the final vowel.

This applies to all sorts of Russian words, also including placenames ending in -я, excepting biblical place names, and Russian personal names.

TODO: implement in propernouns

But:

There are also some subtypes, where this deletion does not occur even though the word ends in -я:

Place names based on names of old tribes

See affixes/propernouns.lexc, but quickly:

These typically end on -я, –га, –ка, –ра, –ла. When inflecting in some cases, they take an infix which is optional. The optional infix causes short versions of case suffixes, otherwise the words take the long version.

корка / куала

The words корка, куала have some forms similar to the tribe/place names, and some cases behave differently: Inessive, Elative have the short consonant-only suffix, but unlike the tribe/place names, there is no -ла- infix.

TODO: write tests

“Old” Finno-ugric stems (-м-, -г-, -т-)

TODO: add these to lexicon, and add morphology. Consider marking in morphology in a way that NDS can present: син син(м)- TODO: write tests TODO: how to mark the inflectional stems in the lexicon for display in dictionaries, similar to smenob?

<l>muohta</l>
<lc>muohttaga</lc>

Some particularly old words have since experienced some amount of deletion and simplification of clusters. Deleted segments reappear in some forms (inflectional, derivational), but their appearance isn’t completely regular, in that some variation exists, and sometimes the “short” stem without this segment appears where the “long” stem usually would.

Usually, these will also be marked in dictionaries. син син(м)-.

When the possessive is applied to nominative versions of these forms, the possessive will have the vowel -ы- instead of -э- or -е-. Also, the ‘old’ segments appear.

Examples:

-м–

син син–тэм синм–ысь сынм–ы
silmä sokea (elat.) (px. y. 1. p.)
визь визь–тэк визьм–о  
järki (abess.) viisas  
куйнь куйнь–лэсь куйньм–ой  
kolme (abl.) 3:n päivän kuluessa  
  куйнь–етй ~ куйньм–етӥ  
  kolmas kolmas  
ин   инм–ын  
taivas   (iness.)  

-к–

кус кус–лы куск–ын куск–ы
lantio (dat.) (iness.) (Px Sg1)
vyötärö      
улеп улеп–лы улепк–ын  
elävä (dat.) (instr.)  
nюлэс нюлэс–лы нюлэск–ын  
metsä (dat.) (iness.)  
вис вис–лы виск–ын  
tauko (dat.) (iness.)  
väli      

-т–

кык кык–намы кыкт–о
kaksi me kaksin kaksoset
    kaksospari
  кык–етӥ кыкт–ой
  toinen kahden päivän kuluessa
    кыкт–етӥ
      toinen

Vowel deletion stems

Some stems also delete vowels in specific cases, generally when the stem is attached to a vowel-initial suffix. Alternatively, the vowels may remain, but a epenthetic consonant may appear: -к-

Optional deletion

сэрег серг–ын  
nurkka (iness.)  
kulma      
сюлэм сюлм–ыд сюлэм–ыд
sydän (Px Sg2) (Px Sg2)

Deletion or epenthesis

вадес вадьс-ын вадеск–ын
aika, hetki (iness.) (iness.)
paikka, palsta      
пыдэс пыдс–ын пидэск–ын
pohja (iness.) (iness.)
урдэс урдс–ын урдэск–ын
vierus (iness.) (iness.)

NB: palatal quality implied by the yodified vowel in the first example is preserved via a soft sign.

Suffix ordering with possessive construction

The possessive construction results in different case ordering:

N-Num-Poss-Case:

N-Num-Case-Poss:

Note that since the adjective ‘determinative’ is essentially the 3rd person singular possessive, the above ordering also follows for adjective inflection. Also note, some cases have atlernate forms for when tehy come before the possessive.

Cases with alternate forms:

Examples:

Genetive:

Px 1pSg Px 2pSg Px 3pSg
бакча-е-лэн бакча-ед-лэн бакча-ез-лэн
бакча-ос-ы-лэн бакча-ос-ыд-лэн бакча-ос-ыз-лэн

Inessive:

Px 1pSg Px 2pSg Px 3pSg
бакча-я-м бакча-я-д бакча-я-з
бакча-ос-а-м бакча-ос-а-д бакча-ос-а-з

Elative:

Px 1pSg Px 2pSg Px 3pSg
бакча-ысьты-м бакча-ысьты-д бакча-ысьты-з
бакча-ос-ысьты-м бакча-ос-ысьты-д бакча-ос-ысьты-з

Noun Lexica Words from lexicon N_Y are characterized by the onset vowel ы instead of э ки, кыл, ныл, ним

Words from lexicon N_ are mostly covered. These are from the main class of nouns described in more depth above.

this nouns, as well as коала, have special +N+Sg+Ine endings in “н” TODO: +N+Sg+Ine endings in “н”

Words from Russian in N_RUS_JA end in –ия, our lexicon entries set the stem as ending in –ия, but there is a trigger that causes the final “я” to disappear in some situations.

Words from Russian in N_RUS_JE end in –ие, our lexicon entries set the stem as ending in –ие, but there is a trigger that causes the final “е” to disappear in some situations.

Words from Tatar in N_TAT preserve the original stem я in all cases.

according to kpv temporary 2019-11-07

À la Jaska


This (part of) documentation was generated from src/fst/morphology/affixes/nouns.lexc


src-fst-morphology-affixes-pronouns.lexc.md

UDMURT Pronoun inflection

Udmurt adjectives inflect for:

Determinative notes

When an adjective occurs in determinative, it must agree for number and case. However, when determinative co-occurs with certain cases, these cases have alternate forms.

The cases with alternate forms are:

Note that these alternative suffixes have palatal and non-palatal versions

Nom. бадӟым-ез гурт бадӟым-ъёс-ыз гурт-ъёс
Iness. бадӟым-а-з гурт-ын бадӟым-ъёс-а-з гурт-ъёс-ын
Illat. бадӟым-а-з гурт-э бадӟым-ъёс-а-з гурт-ъёс-ы
Elat. бадӟым-ысьты-з гурт-ысь бадӟым-ъёс-ысьты-з гурт-ъёс-ысь
Nom. пичи-ез бакча пичи-ос-ыз бакча-ос
Iness. пичи-я-з бакча-ын пичи-ос-а-з бакча-ос-ын
Illat. пичи-я-з бакча-е пичи-ос-а-з бакча-ос-ы
Elat. пичи-ысьты-з бакча-ысь пичи-ос-ысьты-з бакча-ос-ысь

TODO: ordering of suffixes sometimes differs? Compare this with possession morph.

words with this contlex have not been examined

мон:м


This (part of) documentation was generated from src/fst/morphology/affixes/pronouns.lexc


src-fst-morphology-affixes-propernouns.lexc.md

Udmurt Proper noun inflection

Place names based on names of old tribes

These words typically end on -я, –га, –ка, –ра, –ла. When inflecting in some cases, they take an infix which is optional.

The following may be exhaustive, but unsure yet.

Infix is optional in some cases

Case Infix No infix
Iness. Салья-ла-н Салья-ын
Illat. Салья-ла Салья-е
Elat. Салья-ла-сь Салья-ысь
Egress. Салья-ла-сен Салья-ысен

The following cases do not have the infix at all

Case No infix
Terminat. Салья-озь
Prol. Салья-тӥ
Appr. Салья-лань

So far the PROP (underline) lexicon is a dummy pexicon that just gives +N+Prop, and then redirects to NOUNSUF_01.

Words from lexicon PROP_HEIMONNIMET are “old” tribe and place names, which have an optional infix for some cases, while some do not.

In HEIMO_SUFFIXES_LA, we append the cases with their post-ла realizations. Examples: | Case | Infix | No infix

Iness. Салья-ла-н Салья-ын
Illat. Салья-ла Салья-е
Elat. Салья-ла-сь Салья-ысь
Egress. Салья-ла-сен Салья-ысен

Note also that the infix results in a ‘short’ case affix.

In HEIMO_SUFFIXES_NIL, the word gets cases with no infix.

 Case wordform
Terminat. Салья-озь
Prol. Салья-тӥ
Appr. Салья-лань

Russian type Surnames

Preparing for the template urj-Cyrl Beginning 2012-11-15

Абдеев:Абдеев

Багрий:Багр

Аморский:Аморск

These are vowel-final stems They have previously received +Sem/Fem tags

Should this be limited to +Sg? 2015-09-06

Вили:Вил

Андрей:Андре

Ending 2012-11-15

FEMALE NAMES FROM TEMPLATE


This (part of) documentation was generated from src/fst/morphology/affixes/propernouns.lexc


src-fst-morphology-affixes-symbols.lexc.md

Symbol affixes


This (part of) documentation was generated from src/fst/morphology/affixes/symbols.lexc


src-fst-morphology-affixes-verbs-old.lexc.md

Udmurt verb inflection Udmurt verbs are mainly split into two types:

Conjugation I verbs also may have a ‘short’ stem, which subtracts the final ы. The rule is not necessarily as simple as suffixes that begin on consonants require a vowel-ending stem, because there are some exceptions. Following is a list of things that use the short stem (so far incomplete, but feel free to add to it)

And the long stem:

There are some orthographic things to consider, and they show up in the way verbs are inflected. The palatizing series [d z l n s t] is represented in two ways: vowel letters following, or by the palatizing symbol.

возьыны .. возе .. возисько бертыны .. бертэ .. бертӥсько

LEXICON V_ Verb type is still to be determined.

Typical V_MONO entry is the long stem, ending on <ы> берты, возьы

бертыны:берт V_MONO “mennä kotiin” ;

Lexicon V_MONO

Lexicon V_UZA

возь > .. возе .. возисько берт > .. бертэ .. бертӥсько

Lexicon OPRS NB: here we use non-palatal vowels, and shift them to palatal variants in phonology

Lexicon APRS

Lexicon FUTPAST

Lexicon FUTPASTSG

Lexicon FUTPASTPL

Lexicon FUTPASTPLSHORT

Lexicon FUT-PERSON

Lexicon PRTI-PERSON

Lexicon PRTII-PERSON

Lexicon COND

Lexicon IMPRT

Lexicon OPT

Lexicon NONFINITES

Mutual à la Jaska мыныны:мыны Lexicon V_МЫНЫНЫ

Lexicon APRSjaska

Lexicon FUT-PERSONjaska

Lexicon PRTI-PERSONjaska

Lexicon PRTII-PERSONjaska

Lexicon CONDjaska

Lexicon IMPRTjaska

Lexicon OPTjaska

Lexicon NONFINITESjaska


This (part of) documentation was generated from src/fst/morphology/affixes/verbs-old.lexc


src-fst-morphology-affixes-verbs.lexc.md

Udmurt verb inflection Udmurt verbs are mainly split into two types:

Conjugation I verbs also may have a ‘short’ stem, which subtracts the final ы. The rule is not necessarily as simple as suffixes that begin on consonants require a vowel-ending stem, because there are some exceptions. Following is a list of things that use the short stem (so far incomplete, but feel free to add to it)

And the long stem:

There are some orthographic things to consider, and they show up in the way verbs are inflected. The palatizing series [[d z l n s t] is represented in two ways: vowel letters following, or by the palatizing symbol.

возьыны .. возе .. возисько бертыны .. бертэ .. бертӥсько

LEXICON V_ Verb type is still to be determined.

Typical V_MONO entry is the long stem, ending on <ы> берты, возьы

бертыны:берт V_MONO “mennä kotiin” ;

Lexicon V_MONO мыныны:мыны

words that do not occur alone Russian verbs are often followed by the Udmurt карыны OR карон

Lexicon V_UZA

Mutual à la Jaska мыныны:мыны Lexicon V_МЫНЫНЫ

Lexicon APRS

Lexicon FUT-PERSON

Lexicon PRTI-PERSON

Lexicon PRTII-PERSON

Lexicon COND

Lexicon IMPRT

Lexicon OPT

Lexicon NONFINITES


This (part of) documentation was generated from src/fst/morphology/affixes/verbs.lexc


src-fst-morphology-phonology.twolc.md

Udmurt twol file

This file documents the phonology.twolc file

Alphabet, Sets and Definitions

Letters of the alphabet

Archiphonemes for vowels

%^@:0 This apparently causes :о +V+Ind+PrtII:%>еме%>%^@ +V+Ind+Fut:%>о%>%^@ +V+Ind+PrtI:%>и%>%^@

Triggers

Boundary symbols

Sets

Definitions

Rules

DEPALATALISATION

Depalatalize и

Depalatalize е

**Depalatalized %{еэ%} **

Depalatalize и

Depalatalize е3:е

Depalatalize е3:э

Depalatalize е3:е

Depalatalize Ы3:ы

Depalatalize and raise е3:ы

Depalatalize and raise ё:о

%^А3:а

%^О2:о

Palatal %^О2:ё

Palatal э:е

Palatal е

Palatal и

Palatal а

Palatal и:й

VOWEL LOSS

Is this really necessary **Stem vowel loss ы:0 LEFT **

Stem vowel loss Ы2:0 RIGHT

я:0

е:0

е:ь

э:0

Palatal е:0

Palatal и:0

%^О2:0

й:0

ь:0

ъ:0

ZERO TO CONSONANT

%^Л3:л

CONSONANTS

Devoicing д:т

Devoicing з:с

Surface consonant before vowel %^К3:к

Surface consonant before vowel %^М3:м

Disallow %^NotRusJa after cons or other vowels

Disallow %^RusJa after cons or other vowels

Disallow %^RusJaErr after cons or other vowels


This (part of) documentation was generated from src/fst/morphology/phonology.twolc


src-fst-morphology-root.lexc.md

Morphology INTRODUCTION TO MORPHOLOGICAL ANALYSER OF UNDEFINED LANGUAGE.

Definitions for Multichar_Symbols

Analysis symbols The morphological analyses of wordforms for the UDMURT language are presented in this system in terms of the following symbols. (It is highly suggested to follow existing standards when adding new tags).

The parts of speech are further split up into:

The Usage extents are marked using following tags:

The nominals are inflected in the following Number

The nominals are inflected in the following Case TODO: Document case names!

The possession is marked as such:

The comparative forms are:

Numerals are classified under:

Verb moods are: +Cond Conditional +Imprt Imperative +Ind Indicative +Opt Optional

Verb tenses are: +Fut Future +Prs Present +PrtI Preterite One +PrtII Preterite Two

Verb personal forms are:

Other verb forms are: (TODO: Document the tags)

Special symbols are classified with:

The verbs are syntactically split according to transitivity:

Special multiword units are analysed with:

Question and Focus particles:

Tags distinguishing different versions of the same lemma (before POS)

Semantics are classified with

Derivations are classified under the morphophonetic form of the suffix, the source and target part-of-speech.

Dialectical variation

Morphophonology To represent phonologic variations in word forms we use the following symbols in the lexicon files:

%^@ This apparently causes :о +V+Ind+PrtII:%>еме%>%^@ +V+Ind+Fut:%>о%>%^@ +V+Ind+PrtI:%>и%>%^@

And following triggers to control variation

Symbols that need to be escaped on the lower side (towards twolc):

Flag diacritics

We have manually optimised the structure of our lexicon using following flag diacritics to restrict morhpological combinatorics - only allow compounds with verbs if the verb is further derived into a noun again: | @P.NeedNoun.ON@ | (Dis)allow compounds with verbs unless nominalised | @D.NeedNoun.ON@ | (Dis)allow compounds with verbs unless nominalised | @C.NeedNoun@ | (Dis)allow compounds with verbs unless nominalised

For languages that allow compounding, the following flag diacritics are needed to control position-based compounding restrictions for nominals. Their use is handled automatically if combined with +CmpN/xxx tags. If not used, they will do no harm. | @P.CmpFrst.FALSE@ | Require that words tagged as such only appear first | @D.CmpPref.TRUE@ | Block such words from entering ENDLEX | @P.CmpPref.FALSE@ | Block these words from making further compounds | @D.CmpLast.TRUE@ | Block such words from entering R | @D.CmpNone.TRUE@ | Combines with the next tag to prohibit compounding | @U.CmpNone.FALSE@ | Combines with the prev tag to prohibit compounding | @P.CmpOnly.TRUE@ | Sets a flag to indicate that the word has passed R | @D.CmpOnly.FALSE@ | Disallow words coming directly from root.

Use the following flag diacritics to control downcasing of derived proper nouns (e.g. Finnish Pariisi -> pariisilainen). See e.g. North Sámi for how to use these flags. There exists a ready-made regex that will do the actual down-casing given the proper use of these flags. | @U.Cap.Obl@ | Allowing downcasing of derived names: deatnulasj. | @U.Cap.Opt@ | Allowing downcasing of derived names: deatnulasj.

The word forms in Udmurt language start from the lexeme roots of basic word classes, or optionally from prefixes:

ABBREVIATIONS these still need development 2015-11-03

AdjTag ; AdvTag ; AdjTag ; AdvTag ; DescrTag ; AdjTag ; AdvTag ; PcleTag ; AdjTag ; NumTag ; AdjTag ; NTag ; AdjTag ; NTag ; AdvTag ; AdjTag ; PrcTag ; NTag ; AdvTag ; AdjTag ; AdvTag ; AdjTag ; NTag ; AdvTag ; AdjTag ; PronTag ; AdvTag ; NTag ; AdvTag ; NTag ; PoTag ; AdvTag ; PoTag ; AdvTag ; PronTag ; AdvTag ; PronTag ; NumTag ; DescrTag ; AdjTag ; DescrTag ; AdvTag ; NTag ; DescrTag ; NTag ; DescrTag ; NTag ; AdjTag ; GerTag ; ConjTag ; GerTag ; PoTag ; AdjTag ; NumTag ; NTag ; AdjTag ; PcleTag ; NTag ; AdvTag ; NTag ; AdvTag ; DescrTag ; NTag ; DescrTag ; NTag ; NumTag ; NTag ; PrcTag ; PcleTag ; NTag ; PcleTag ; NTag ; PoTag ; NTag ; VTag ; NTag ; VTag ; PronTag ; PcleTag ; AdjTag ; AdvTag ; PoTag ; PcleTag ; PoTag ; AdvTag ; PronTag ; PronTag ; NumTag ; PronTag ; NumTag ; AdvTag ; PronTag ; PcleTag ;


This (part of) documentation was generated from src/fst/morphology/root.lexc


src-fst-morphology-stems-adjectives_newwords.lexc.md

This is where new words are added as lexc entries before they are added to Verdd

ADD ADJECTIVES BELOW


This (part of) documentation was generated from src/fst/morphology/stems/adjectives_newwords.lexc


src-fst-morphology-stems-exceptions.lexc.md

Exceptions are quite strange word-forms. the ones that do not fit anywhere else. This file contains all enumerated word forms that cannot reasonably be created from lexical data by regular inflection. Usually there should be next to none exceptions, it’s always better to have a paradigm that covers only one or few words than an exception since these will not work nicely with e.g. compounding scheme or possibly many end applications.

This is a temporary work around words that do not occur alone Russian verbs are often followed by the Udmurt карыны OR карон

Proper names from Bible


This (part of) documentation was generated from src/fst/morphology/stems/exceptions.lexc


src-fst-morphology-stems-nouns_newwords.lexc.md

This is where new words are added as lexc entries before they are added to the xml source files. апра+N:апра N_ “(eng) /(fin) /(hun) /(rus) “ ;

ADD NOUNS BELOW


This (part of) documentation was generated from src/fst/morphology/stems/nouns_newwords.lexc


src-fst-morphology-stems-numerals.lexc.md

Numerals Numerals in UDMURT language are numbers.


This (part of) documentation was generated from src/fst/morphology/stems/numerals.lexc


src-fst-morphology-stems-prefixes.lexc.md

Prefixes Prefixes in the Udmurt language are bound to beginning of other words.


This (part of) documentation was generated from src/fst/morphology/stems/prefixes.lexc


src-fst-morphology-stems-verbs_newwords.lexc.md

This is where new words are added as lexc entries before they are added to Verdd

ADD VERBS BELOW


This (part of) documentation was generated from src/fst/morphology/stems/verbs_newwords.lexc


src-fst-phonetics-txt2ipa.xfscript.md

retroflex plosive, voiceless t ʈ 0288, 648 ( = ASCII 096) retroflex plosive, voiced d ɖ 0256, 598 labiodental nasal F ɱ 0271, 625 retroflex nasal n ɳ 0273, 627 palatal nasal J ɲ 0272, 626 velar nasal N ŋ 014B, 331 uvular nasal N\ ɴ 0274, 628

bilabial trill B\ ʙ 0299, 665 uvular trill R\ ʀ 0280, 640 alveolar tap 4 ɾ 027E, 638 retroflex flap r ɽ 027D, 637 bilabial fricative, voiceless p\ ɸ 0278, 632 bilabial fricative, voiced B β 03B2, 946 dental fricative, voiceless T θ 03B8, 952 dental fricative, voiced D ð 00F0, 240 postalveolar fricative, voiceless S ʃ 0283, 643 postalveolar fricative, voiced Z ʒ 0292, 658 retroflex fricative, voiceless s ʂ 0282, 642 retroflex fricative, voiced z` ʐ 0290, 656 palatal fricative, voiceless C ç 00E7, 231 palatal fricative, voiced j\ ʝ 029D, 669 velar fricative, voiced G ɣ 0263, 611 uvular fricative, voiceless X χ 03C7, 967 uvular fricative, voiced R ʁ 0281, 641 pharyngeal fricative, voiceless X\ ħ 0127, 295 pharyngeal fricative, voiced ?\ ʕ 0295, 661 glottal fricative, voiced h\ ɦ 0266, 614

alveolar lateral fricative, vl. K alveolar lateral fricative, vd. K\

labiodental approximant P (or v) alveolar approximant r\ retroflex approximant r` velar approximant M\

retroflex lateral approximant l` palatal lateral approximant L velar lateral approximant L
Clicks

bilabial O\ (O = capital letter) dental |
(post)alveolar !\ palatoalveolar =\ alveolar lateral ||
Ejectives, implosives

ejective > e.g. ejective p p> implosive < e.g. implosive b b< Vowels

close back unrounded M close central unrounded 1 close central rounded } lax i I lax y Y lax u U

close-mid front rounded 2 close-mid central unrounded @\ close-mid central rounded 8 close-mid back unrounded 7

schwa ə @

open-mid front unrounded E open-mid front rounded 9 open-mid central unrounded 3 open-mid central rounded 3\ open-mid back unrounded V open-mid back rounded O

ash (ae digraph) { open schwa (turned a) 6

open front rounded & open back unrounded A open back rounded Q Other symbols

voiceless labial-velar fricative W voiced labial-palatal approx. H voiceless epiglottal fricative H\ voiced epiglottal fricative <\ epiglottal plosive >\

alveolo-palatal fricative, vl. s\ alveolo-palatal fricative, voiced z\ alveolar lateral flap l\ simultaneous S and x x\ tie bar _ Suprasegmentals

primary stress “ secondary stress % long : half-long :\ extra-short _X linking mark -
Tones and word accents

level extra high _T level high _H level mid _M level low _L level extra low _B downstep ! upstep ^ (caret, circumflex)

contour, rising contour, falling _F contour, high rising _H_T contour, low rising _B_L

contour, rising-falling _R_F (NB Instead of being written as diacritics with _, all prosodic marks can alternatively be placed in a separate tier, set off by < >, as recommended for the next two symbols.) global rise global fall Diacritics

voiceless 0 (0 = figure), e.g. n_0 voiced _v aspirated _h more rounded _O (O = letter) less rounded _c advanced _+ retracted _- centralized _” syllabic = (or _=) e.g. n= (or n=) non-syllabic _^ rhoticity `

breathy voiced _t creaky voiced _k linguolabial _N labialized _w palatalized ‘ (or _j) e.g. t’ (or t_j) velarized _G pharyngealized _?\

dental d apical _a laminal _m nasalized ~ (or _~) e.g. A~ (or A~) nasal release _n lateral release _l no audible release _}

velarized or pharyngealized _e velarized l, alternatively 5 raised _r lowered _o advanced tongue root _A retracted tongue root _q


This (part of) documentation was generated from src/fst/phonetics/txt2ipa.xfscript


src-fst-transcriptions-transcriptor-abbrevs2text.lexc.md

We describe here how abbreviations are in Udmurt are read out, e.g. for text-to-speech systems.

For example:


This (part of) documentation was generated from src/fst/transcriptions/transcriptor-abbrevs2text.lexc


src-fst-transcriptions-transcriptor-numbers-digit2text.lexc.md

:одӥг% 1MILJON ;

This is for the numerals 20-69


This (part of) documentation was generated from src/fst/transcriptions/transcriptor-numbers-digit2text.lexc


tools-grammarcheckers-grammarchecker.cg3.md

U D M U R T G R A M M A R C H E C K E R

DELIMITERS

TAGS AND SETS

Tags

This section lists all the tags inherited from the fst, and used as tags in the syntactic analysis. The next section, Sets, contains sets defined on the basis of the tags listed here, those set names are not visible in the output.

Beginning and end of sentence

BOS EOS

Parts of speech tags

N A Adv V Pron CS CC CC-CS Po Pr Pcle Num Interj ABBR ACR CLB LEFT RIGHT WEB PPUNCT PUNCT

COMMA ¶

Tags for POS sub-categories

Pers Dem Interr Indef Recipr Refl Rel Coll NomAg Prop Allegro Arab Romertall

Tags for morphosyntactic properties

Nom Acc Gen Ill Loc Com Ess Ess Sg Du Pl Cmp/SplitR Cmp/SgNom Cmp/SgGen Cmp/SgGen PxSg1 PxSg2 PxSg3 PxDu1 PxDu2 PxDu3 PxPl1 PxPl2 PxPl3 Px

Comp Superl Attr Ord Qst IV TV Prt Prs Ind Pot Cond Imprt ImprtII Sg1 Sg2 Sg3 Du1 Du2 Du3 Pl1 Pl2 Pl3 Inf ConNeg Neg PrfPrc VGen PrsPrc Ger Sup Actio VAbess

Err/Orth

Semantic tags

Sem/Act Sem/Ani Sem/Atr Sem/Body Sem/Clth Sem/Domain Sem/Feat-phys Sem/Fem Sem/Group Sem/Lang Sem/Mal Sem/Measr Sem/Money Sem/Obj Sem/Obj-el Sem/Org Sem/Perc-emo Sem/Plc Sem/Sign Sem/State-sick Sem/Sur Sem/Time Sem/Txt

HUMAN

PROP-ATTR PROP-SUR

TIME-N-SET

Syntactic tags

@+FAUXV @+FMAINV @-FAUXV @-FMAINV @-FSUBJ> @-F<OBJ @-FOBJ> @-FSPRED<OBJ @-F<ADVL @-FADVL> @-F<SPRED @-F<OPRED @-FSPRED> @-FOPRED> @>ADVL @ADVL< @<ADVL @ADVL> @ADVL @HAB> @<HAB @>N @Interj @N< @>A @P< @>P @HNOUN @INTERJ @>Num @Pron< @>Pron @Num< @OBJ @<OBJ @OBJ> @OPRED @<OPRED @OPRED> @PCLE @COMP-CS< @SPRED @<SPRED @SPRED> @SUBJ @<SUBJ @SUBJ> SUBJ SPRED OPRED @PPRED @APP @APP-N< @APP-Pron< @APP>Pron @APP-Num< @APP-ADVL< @VOC @CVP @CNP OBJ

-OTHERS SYN-V @X ## Sets containing sets of lists and tags This part of the file lists a large number of sets based partly upon the tags defined above, and partly upon lexemes drawn from the lexicon. See the sourcefile itself to inspect the sets, what follows here is an overview of the set types. ### Sets for Single-word sets INITIAL ### Sets for word or not WORD NOT-COMMA ### Case sets ADLVCASE CASE-AGREEMENT CASE NOT-NOM NOT-GEN NOT-ACC ### Verb sets NOT-V ### Sets for finiteness and mood REAL-NEG MOOD-V NOT-PRFPRC ### Sets for person SG1-V SG2-V SG3-V DU1-V DU2-V DU3-V PL1-V PL2-V PL3-V ### Pronoun sets ### Adjectival sets and their complements ### Adverbial sets and their complements ### Sets of elements with common syntactic behaviour ### NP sets defined according to their morphosyntactic features ### The PRE-NP-HEAD family of sets These sets model noun phrases (NPs). The idea is to first define whatever can occur in front of the head of the NP, and thereafter negate that with the expression **WORD - premodifiers**. ### Border sets and their complements ### Grammarchecker sets * * * This (part of) documentation was generated from [tools/grammarcheckers/grammarchecker.cg3](https://github.com/giellalt/lang-udm/blob/main/tools/grammarcheckers/grammarchecker.cg3) --- # tools-tokenisers-tokeniser-disamb-gt-desc.pmscript.md # Tokeniser for udm Usage: ``` $ make $ echo "ja, ja" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa boasttu olmmoš, man mielde lahtuid." | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "márffibiillagáffe" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` Pmatch documentation: <https://github.com/hfst/hfst/wiki/HfstPmatch> Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words: * Punct contains ASCII punctuation marks * The symbol after m-dash is soft-hyphen `U+00AD` * The symbol following {•} is byte-order-mark / zero-width no-break space `U+FEFF`. Whitespace contains ASCII white space and the List contains some unicode white space characters * En Quad U+2000 to Zero-Width Joiner U+200d' * Narrow No-Break Space U+202F * Medium Mathematical Space U+205F * Word joiner U+2060 Apart from what's in our morphology, there are 1. unknown word-like forms, and 2. unmatched strings We want to give 1) a match, but let 2) be treated specially by `hfst-tokenise -a` Unknowns are made of: * lower-case ASCII * upper-case ASCII * select extended latin symbols * cyrillics ASCII digits * select symbols * Combining diacritics as individual symbols, * various symbols from Private area (probably Microsoft), so far: * U+F0B7 for "x in box" ## Unknown handling Unknowns are tagged ?? and treated specially with `hfst-tokenise` hfst-tokenise --giella-cg will treat such empty analyses as unknowns, and remove empty analyses from other readings. Empty readings are also legal in CG, they get a default baseform equal to the wordform, but no tag to check, so it's safer to let hfst-tokenise handle them. Finally we mark as a token any sequence making up a: * known word in context * unknown (OOV) token in context * sequence of word and punctuation * URL in context * * * This (part of) documentation was generated from [tools/tokenisers/tokeniser-disamb-gt-desc.pmscript](https://github.com/giellalt/lang-udm/blob/main/tools/tokenisers/tokeniser-disamb-gt-desc.pmscript) --- # tools-tokenisers-tokeniser-gramcheck-gt-desc.pmscript.md # Grammar checker tokenisation for udm Requires a recent version of HFST (3.10.0 / git revision>=3aecdbc) Then just: ``` $ make $ echo "ja, ja" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` More usage examples: ``` $ echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa boasttu olmmoš, man mielde lahtuid." | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "márffibiillagáffe" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` Pmatch documentation: <https://github.com/hfst/hfst/wiki/HfstPmatch> Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words: * Punct contains ASCII punctuation marks * The symbol after m-dash is soft-hyphen `U+00AD` * The symbol following {•} is byte-order-mark / zero-width no-break space `U+FEFF`. Whitespace contains ASCII white space and the List contains some unicode white space characters * En Quad U+2000 to Zero-Width Joiner U+200d' * Narrow No-Break Space U+202F * Medium Mathematical Space U+205F * Word joiner U+2060 Apart from what's in our morphology, there are 1) unknown word-like forms, and 2) unmatched strings We want to give 1) a match, but let 2) be treated specially by hfst-tokenise -a * select extended latin symbols * select symbols * various symbols from Private area (probably Microsoft), so far: * U+F0B7 for "x in box" TODO: Could use something like this, but built-in's don't include šžđčŋ: Simply give an empty reading when something is unknown: hfst-tokenise --giella-cg will treat such empty analyses as unknowns, and remove empty analyses from other readings. Empty readings are also legal in CG, they get a default baseform equal to the wordform, but no tag to check, so it's safer to let hfst-tokenise handle them. Finally we mark as a token any sequence making up a: * known word in context * unknown (OOV) token in context * sequence of word and punctuation * URL in context * * * This (part of) documentation was generated from [tools/tokenisers/tokeniser-gramcheck-gt-desc.pmscript](https://github.com/giellalt/lang-udm/blob/main/tools/tokenisers/tokeniser-gramcheck-gt-desc.pmscript) --- # tools-tokenisers-tokeniser-tts-cggt-desc.pmscript.md # TTS tokenisation for smj Requires a recent version of HFST (3.10.0 / git revision>=3aecdbc) Then just: ```sh make echo "ja, ja" \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` More usage examples: ```sh echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa \ boasttu olmmoš, man mielde lahtuid." \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst echo "márffibiillagáffe" \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` Pmatch documentation: <https://kitwiki.csc.fi/twiki/bin/view/KitWiki/HfstPmatch> Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words: * Punct contains ASCII punctuation marks * The symbol after m-dash is soft-hyphen `U+00AD` * The symbol following {•} is byte-order-mark / zero-width no-break space `U+FEFF`. Whitespace contains ASCII white space and the List contains some unicode white space characters * En Quad U+2000 to Zero-Width Joiner U+200d' * Narrow No-Break Space U+202F * Medium Mathematical Space U+205F * Word joiner U+2060 Apart from what's in our morphology, there are 1) unknown word-like forms, and 2) unmatched strings We want to give 1) a match, but let 2) be treated specially by hfst-tokenise -a * select extended latin symbols * select symbols * various symbols from Private area (probably Microsoft), so far: * U+F0B7 for "x in box" TODO: Could use something like this, but built-in's don't include šžđčŋ: Simply give an empty reading when something is unknown: hfst-tokenise --giella-cg will treat such empty analyses as unknowns, and remove empty analyses from other readings. Empty readings are also legal in CG, they get a default baseform equal to the wordform, but no tag to check, so it's safer to let hfst-tokenise handle them. Needs hfst-tokenise to output things differently depending on the tag they get * * * This (part of) documentation was generated from [tools/tokenisers/tokeniser-tts-cggt-desc.pmscript](https://github.com/giellalt/lang-udm/blob/main/tools/tokenisers/tokeniser-tts-cggt-desc.pmscript)