Estonian NLP Grammar

Finite state and Constraint Grammar based analysers, proofing tools and other resources

View the project on GitHub giellalt/lang-est-x-utee

Estonian (Tartu Ülikool) language model documentation

All doc-comment documentation in one large file.


src-cg3-functions.cg3.md

These sets model noun phrases (NPs). The idea is to first define whatever can occur in front of the head of the NP, and thereafter negate that with the expression WORD - premodifiers.

The set NOT-NPMOD is used to find barriers between NPs. Typical usage: … (*1 N BARRIER NPT-NPMOD) … meaning: Scan to the first noun, ignoring anything that can be part of the noun phrase of that noun (i.e., “scan to the next NP head”)

These were the set types.

HABITIVE MAPPING

sma object

SUBJ MAPPING - leftovers

OBJ MAPPING - leftovers

HNOUN MAPPING


This (part of) documentation was generated from src/cg3/functions.cg3


src-fst-morphology-affixes-exceptional_declinations.lexc.md

inflectional types and analogue rules for nouns, adjectives and other declinables VVS stands for Ü. Viks “Väike vormisõnastik” for reference purposes


This (part of) documentation was generated from src/fst/morphology/affixes/exceptional_declinations.lexc


src-fst-morphology-affixes-gi.lexc.md

stressing affix gi/ki can be appended to almost any word form

liide gi/ki this ensures that downcasing derivation (helveetslane) works


This (part of) documentation was generated from src/fst/morphology/affixes/gi.lexc


src-fst-morphology-affixes-regular_declinations.lexc.md

flag diacritics can be used for remembering the case

flag diacritics are used for remembering the case for compounding; not all the cases must be distinguished


This (part of) documentation was generated from src/fst/morphology/affixes/regular_declinations.lexc


src-fst-morphology-affixes-verbs.lexc.md

Verb inflection

LEXICON SAAMA is a lexicon for saama-type words. ! VVS 37 saama, jääma


This (part of) documentation was generated from src/fst/morphology/affixes/verbs.lexc


src-fst-morphology-phonology.twolc.md

Phonology

Placeholders

Sami GT convention

Triggers

Usage tag. It may relate to an individual word in the lexicon, or to a set of inflectional forms of some inflectional type, i.e. its sub-paradigm. It never surfaces. It is used to pair the surface form with the usage tag of the lexical representation.

Special surface side symbols, used in rule contexts

Apostrophe is used for separating inflectional affix from a foreign word lemma

Morphophonemes

If the sound change is unproductive and difficult to relate to its immediate context, we use capital letters with numbers to denote them. In stems, they typically result from diachronic processes. In affixes, they are typically related to the declination or conjugation and the form of the stem they attach to.

Short stops

Orthographic convention: after voiceless (e.g. s or h, or k p t), gbd is written as kpt e.g. õhk-õhu, vask-vase K1 also in: uks, jooksma P1 also in: laps T1 also in: jätma, katma, kütma, matma, võtma, mõtlema, ütlema

Short stops in stem illatives for words that do not have grade alternation. They surface (or appear as extra long) in strong grade, expressed by stem illative only.

Unstressed syllable vowels disappear…

A stem vowel in inflectional forms of ne/s words, to make them formally similar in inflection

Ad hoc stem vowels for ne/s words

Few words…

j surfacing and changing

4 words have h-illative: sohu, suhu, öhe, pähe

only hea and pea

6 words have õ in indicative past

A handful of words…

Verb affix lexicons are simpler if we introduce these:

Stem vowels for verbs of some inflectional types

Verb affixes have k-g and t-d-0 alternations:

Imperative mood affixes gu/ku, ge/kem etc

Infinitive affixes ta/da/a, and gerund affixes tes/des/es

Impersonal voice affixes tud/tud, takse/dakse etc

To form past indicative forms and make them pronouncable

Sometimes the choice of an allomorph or allophone is related to the frequency of the word.

For plural partitive, the form is generated either with sg vowel + sid or plural vowel + 0 So we must allow stem vowels for singular and plural to appear and disappear in certain conditions.

Singular stem vowel tag in lexicon

Plural stem vowel tag in lexicon

Inflectional affixes having the same grammatical meaning: Pl Par endings sid/0, id/sid, Sg Ill endings sse/0. Their choice depends on triggers in the lexicon, have to be defined un-naturally letter by letter,

If the sound change is productive and/or very regularly determined by context (e.g. by morpheme border), we do not use special symbols to denote the changing phonemes

ne, s ending words have similar paradigms; only sg nom is different

-le/-el stem alternations also use e:0, in addition to 0:e (sip0lema-sipel0da)

high vowel lowering in certain contexts

1.1. plural partitive: -sid vs stem vowel change


This (part of) documentation was generated from src/fst/morphology/phonology.twolc


src-fst-morphology-root.lexc.md

Morphology

Multicharacter symbols

Sõnaliik / Part of speech

Genitiivatribuut pole eraldi kategooria / No special tag for genitive attribute : angoora+N+Sg+Gen

Alamkategooriad / Sub-pos categories

Võrdlusastmed / Comparation

Sõnamuutus / Inflection

Käändsõna / Noun

Arv / number

Kääne / Case

Verb

Verbil on finiitsed (pöördelised) ja infiniitsed (nn käänduvad) vormid. Pöördelistel vormidel on kategooriad: tegumood, aeg, kõneviis, isik, arv, kõneliik (tegumood ja aeg on ka mõnedel infiniitsetel vormidel).

Verbs have finite (conjugable) and infinite (“declinable”) forms. The finite forms have categories: voice, tense, mood, person, number, negation (some infinite forms also have voice and tense).

Pöördelised vormid / Conjugable forms

Tegumood / Voice

Aeg / Tense

Lihtminevik = minevik / Past imperfect = past

Täisminevik / Past perfect: olema (pres) + nud/tud/dud (olen teinud)

Enneminevik / Past pluperfect: olema (impf) + nud/tud/dud (olin teinud)

Kõneviis / Mood

Pööre ja arv / Person and number

Kõneliik / Polarity

Infiniitsed (nn käänduvad) verbivormid / Infinite (“declinable”) verb forms

Kategooriate võimalikud kombinatsioonid / Possible combinations of categories

allikas / source : Heiki-Jaan Kaalep. Eesti verbi vormistik. Keel ja kirjandus 1/2015 lk 1-15.

The categories are given in the order in which the allomorphs (if they can be distinguished) that represent them are attached to the word stem (note that the treatment of allomorphs is sloppy here). The justification is that the categories are not equal, but form an hierarchy: those closer to the word end tend to be more optional, more often non-specified.

  1. voice: personal vs. impersonal (0-morph vs. t/d(aiu)), eg. elaks vs. elataks, elav vs. elatav
  2. tense: present vs. past (0-morph vs. s/si/nu), e.g. elan vs. elasin, elaks vs. elanuks
  3. mood: indicative vs. conditional vs imperative vs quotative (0-morph vs. ks vs. k/g(ue) vs. vat)
  4. person+number: notice that in personal present and past conditional, imperative and quotative moods one form is underspecified for person+number
  5. negation: affirmative vs. negative. It manifests itself via lexical means: it is either present in an exceptional wordform (some forms of olema) or gets adhered to a form, normally used in affirmative, from an immediately preceding word ei, ega or ära (e.g. ei elaks). The only case when negation has a dedicated form of its own is impersonal present indicative negative (e.g. ei elata).

The forms of negative words pole and ära are included in order to capture really all the combinations; also note that for olema, there are some constellations of categories that are the same for a wordform beginning with ole- and pole-

—- Personal finite forms —-

\

NB! ei ole = pole

\

\

\

\

\

\

NB! ei olnud = polnud

\


elanud: elama+V+Pers+Prt+Imprt: 1.11.2016 ei genereerita ega tunta ära / not recognized nor generated as of Nov 1, 2016

\

—- Personal infinite forms —-

(on, oli, …) + V+Pers+Prt+Prc = some analytical personal form

—- Impersonal finite forms —-

NB! ei olda = polda

\

\

\

NB! ei oldud = poldud

—- Impersonal infinite forms —-

(on, oli, …) + V+Impers+Prt+Prc = some analytical personal form

—- Infinite forms with no voice category —-

Exceptional cases:

personal present (Prs not implemented????), 3 words: kuulukse, tunnukse, näikse

Negation verb

Analytical forms (olen elanud, olin elanud, oleksin elanud, ei olnud elanud, ei olnuks elanud etc) are not treated here…

Partikkel -gi/-ki / Particle -gi/-ki

Paralleelvormide erinevat kasutussagedust iseloomustavad / Usage info for parallel forms (either correct according to the norm, or incorrect)

Oletatav analüüs / Guessed analysis

Tuletus / Derivation

Eesliited on harvad, aga ikkagi … / Prefixes are seldom used, but still…

Järelliited / Suffixes

V –> N

V –> A

N –> A

A –> Adv

A –> A

N –> N

N Prop –> N

A –> N, V –> N

N –> Adv

N –> N, A –> A

Num –> N

Muud / Other tags

Copied from Sami root.lexc

Flag diacritics

Use the following flag diacritics to control downcasing of derived proper nouns (e.g. Finnish Pariisi -> pariisilainen). See e.g. North Sámi for how to use these flags. There exists a ready-made regex that will do the actual down-casing given the proper use of these flags. | @U.Cap.Obl@ | Allowing downcasing of derived names: deatnulasj. | @U.Cap.Opt@ | Allowing downcasing of derived names: deatnulasj.

These are used for limiting the number of components in a compound word (the compound transducer is cyclic, but in reality there is a limit to the length of words) | @D.Part@ | No part of a compound should have been encountered yet | @P.Part.One@ | Indicate that this could be the first part of a compound | @R.Part.One@ | Require that the first part has been encountered; if a lemma has it, it means that the lemma cannot be part2 or part3 of a compound | @D.Part.One@ | Require that the first part has NOT been encountered | @P.Part.Two@ | Indicate that this could be the second part of a compound | @R.Part.Two@ | Require that the second part has been encountered | @P.Part.Three@ | Indicate that this could be the third part of a compound | @P.Part.Bad@ | Indicate that this cannot be a part of a compound | @R.Part.Bad@ | Require that the first part has been encountered; if a lemma has it, it means that the lemma cannot be part2

POS is used for filtering derivations and compounds | @R.POS.NumCardCompound@ | compound numeral (viis+sada - five hundred)

Case is used for filtering derivations and compounds

Remember there has been some derivation from A or N; for filtering compounds derivation from V is called paradigmatic and does not result in Der; just a new POS…

Remember the stem type; for filtering compounds

A special condition that is used for filtering derivations and compounds

Tokeniser

Guesser

Symbols that need to be escaped on the lower side (towards twolc):

Sami GT convention

Legitimate strings that are not words: numbers, acronyms, …

Lexicons

For modelling compounds, the simplex word fst is concatenated with itself. For this, Kleene star operation is used, i.e. fst is concatenated zero to any number of times. For the lookup process, this creates a possibility of infinitely many passes through the fst, thus allowing infinitely long words. For limiting and controlling the passes, flag diacritics are used. Lookup process remembers which paths it has taken, and counts the passes. For remembering, it sets up flags on the path:

  1. Flag the first 3 passes through the simplex fst (thus forbidding any more passes) (the lexicon contains conventionalised compounds, thus the max number of compound components is larger than three; however, no Estonian word with more than five components has been found yet)
  2. In every pass, flag the POS and inflection, to be used in the following pass Different paths may correspond to the same surface string, e.g. mootoriõlilik = (mootori+õli)+lik and (mootori)+(õli+lik)

Lexicon-based passes

strictly simplex word; cannot be a part of a compound a simplex word, or the first part of a compound

Guesser assumes that there is only one pass, and that only the final part is important (out-of-vocabulary simplex words are treated elsewhere)

— end guesser

lexicon-based

strictly simplex words; cannot be a part of a compound


This (part of) documentation was generated from src/fst/morphology/root.lexc


src-fst-morphology-stems-acronyms.lexc.md

Acronyms


This (part of) documentation was generated from src/fst/morphology/stems/acronyms.lexc


src-fst-phonetics-txt2ipa.xfscript.md

retroflex plosive, voiceless t ʈ 0288, 648 ( = ASCII 096) retroflex plosive, voiced d ɖ 0256, 598 labiodental nasal F ɱ 0271, 625 retroflex nasal n ɳ 0273, 627 palatal nasal J ɲ 0272, 626 velar nasal N ŋ 014B, 331 uvular nasal N\ ɴ 0274, 628

bilabial trill B\ ʙ 0299, 665 uvular trill R\ ʀ 0280, 640 alveolar tap 4 ɾ 027E, 638 retroflex flap r ɽ 027D, 637 bilabial fricative, voiceless p\ ɸ 0278, 632 bilabial fricative, voiced B β 03B2, 946 dental fricative, voiceless T θ 03B8, 952 dental fricative, voiced D ð 00F0, 240 postalveolar fricative, voiceless S ʃ 0283, 643 postalveolar fricative, voiced Z ʒ 0292, 658 retroflex fricative, voiceless s ʂ 0282, 642 retroflex fricative, voiced z` ʐ 0290, 656 palatal fricative, voiceless C ç 00E7, 231 palatal fricative, voiced j\ ʝ 029D, 669 velar fricative, voiced G ɣ 0263, 611 uvular fricative, voiceless X χ 03C7, 967 uvular fricative, voiced R ʁ 0281, 641 pharyngeal fricative, voiceless X\ ħ 0127, 295 pharyngeal fricative, voiced ?\ ʕ 0295, 661 glottal fricative, voiced h\ ɦ 0266, 614

alveolar lateral fricative, vl. K alveolar lateral fricative, vd. K\

labiodental approximant P (or v) alveolar approximant r\ retroflex approximant r` velar approximant M\

retroflex lateral approximant l` palatal lateral approximant L velar lateral approximant L
Clicks

bilabial O\ (O = capital letter) dental |
(post)alveolar !\ palatoalveolar =\ alveolar lateral ||
Ejectives, implosives

ejective > e.g. ejective p p> implosive < e.g. implosive b b< Vowels

close back unrounded M close central unrounded 1 close central rounded } lax i I lax y Y lax u U

close-mid front rounded 2 close-mid central unrounded @\ close-mid central rounded 8 close-mid back unrounded 7

schwa ə @

open-mid front unrounded E open-mid front rounded 9 open-mid central unrounded 3 open-mid central rounded 3\ open-mid back unrounded V open-mid back rounded O

ash (ae digraph) { open schwa (turned a) 6

open front rounded & open back unrounded A open back rounded Q Other symbols

voiceless labial-velar fricative W voiced labial-palatal approx. H voiceless epiglottal fricative H\ voiced epiglottal fricative <\ epiglottal plosive >\

alveolo-palatal fricative, vl. s\ alveolo-palatal fricative, voiced z\ alveolar lateral flap l\ simultaneous S and x x\ tie bar _ Suprasegmentals

primary stress “ secondary stress % long : half-long :\ extra-short _X linking mark -
Tones and word accents

level extra high _T level high _H level mid _M level low _L level extra low _B downstep ! upstep ^ (caret, circumflex)

contour, rising contour, falling _F contour, high rising _H_T contour, low rising _B_L

contour, rising-falling _R_F (NB Instead of being written as diacritics with _, all prosodic marks can alternatively be placed in a separate tier, set off by < >, as recommended for the next two symbols.) global rise global fall Diacritics

voiceless 0 (0 = figure), e.g. n_0 voiced _v aspirated _h more rounded _O (O = letter) less rounded _c advanced _+ retracted _- centralized _” syllabic = (or _=) e.g. n= (or n=) non-syllabic _^ rhoticity `

breathy voiced _t creaky voiced _k linguolabial _N labialized _w palatalized ‘ (or _j) e.g. t’ (or t_j) velarized _G pharyngealized _?\

dental d apical _a laminal _m nasalized ~ (or _~) e.g. A~ (or A~) nasal release _n lateral release _l no audible release _}

velarized or pharyngealized _e velarized l, alternatively 5 raised _r lowered _o advanced tongue root _A retracted tongue root _q


This (part of) documentation was generated from src/fst/phonetics/txt2ipa.xfscript


src-fst-transcriptions-transcriptor-abbrevs2text.lexc.md

We describe here how abbreviations are in Estonian are read out, e.g. for text-to-speech systems.

For example:


This (part of) documentation was generated from src/fst/transcriptions/transcriptor-abbrevs2text.lexc


src-fst-transcriptions-transcriptor-numbers-digit2text.lexc.md

:kümmend+Use/NG LOHKAIM ; Analyzed but not generated

% komma% :, Root ; % tjuohkkis% :%. Root ; % kolon% :%: Root ; % sárggis% :%- Root ; % násti% :%* Root ;


This (part of) documentation was generated from src/fst/transcriptions/transcriptor-numbers-digit2text.lexc


tools-grammarcheckers-grammarchecker.cg3.md

ESTONIAN GRAMMAR CHECKER

DELIMITERS

TAGS AND SETS

Tags

This section lists all the tags inherited from the fst, and used as tags in the syntactic analysis. The next section, Sets, contains sets defined on the basis of the tags listed here, those set names are not visible in the output.

Beginning and end of sentence

BOS EOS

Parts of speech tags

N A Adv V Pron CS CC CC-CS Po Pr Pcle Num Interj ABBR ACR CLB LEFT RIGHT WEB PPUNCT PUNCT

COMMA ¶

Tags for POS sub-categories

Pers Dem Interr Indef Recipr Refl Rel Coll NomAg Prop Allegro Arab Romertall

Tags for morphosyntactic properties

Nom Gen Ill Com Ess Ess

Sg Du Pl Cmp/SplitR Cmp/SgNom Cmp/SgGen Cmp/SgGen

Comp Superl Attr Ord Qst IV TV Prt Prs Ind Pot Cond Imprt

Sg1 Sg2 Sg3 Pl1 Pl2 Pl3

Inf ConNeg Neg PrfPrc PrsPrc Ger Sup

Err/Orth

Semantic tags

Sem/Act Sem/Ani Sem/Atr Sem/Body Sem/Clth Sem/Domain Sem/Feat-phys Sem/Fem Sem/Group Sem/Lang Sem/Mal Sem/Measr Sem/Money Sem/Obj Sem/Obj-el Sem/Org Sem/Perc-emo Sem/Plc Sem/Sign Sem/State-sick Sem/Sur Sem/Time Sem/Txt

HUMAN

PROP-ATTR PROP-SUR

TIME-N-SET

Syntactic tags

@+FAUXV @+FMAINV @-FAUXV @-FMAINV @-FSUBJ> @-F<OBJ @-FOBJ> @-FSPRED<OBJ @-F<ADVL @-FADVL> @-F<SPRED @-F<OPRED @-FSPRED> @-FOPRED> @>ADVL @ADVL< @<ADVL @ADVL> @ADVL @HAB> @<HAB @>N @Interj @N< @>A @P< @>P @HNOUN @INTERJ @>Num @Pron< @>Pron @Num< @OBJ @<OBJ @OBJ> @OPRED @<OPRED @OPRED> @PCLE @COMP-CS< @SPRED @<SPRED @SPRED> @SUBJ @<SUBJ @SUBJ> SUBJ SPRED OPRED @PPRED @APP @APP-N< @APP-Pron< @APP>Pron @APP-Num< @APP-ADVL< @VOC @CVP @CNP OBJ

-OTHERS SYN-V @X ## Sets containing sets of lists and tags This part of the file lists a large number of sets based partly upon the tags defined above, and partly upon lexemes drawn from the lexicon. See the sourcefile itself to inspect the sets, what follows here is an overview of the set types. ### Sets for Single-word sets INITIAL ### Sets for word or not WORD NOT-COMMA ### Case sets ADLVCASE CASE-AGREEMENT CASE NOT-NOM NOT-GEN ### Verb sets NOT-V ### Sets for finiteness and mood REAL-NEG MOOD-V NOT-PRFPRC ### Sets for person SG1-V SG2-V SG3-V DU1-V DU2-V DU3-V PL1-V PL2-V PL3-V ### Pronoun sets ### Adjectival sets and their complements ### Adverbial sets and their complements ### Sets of elements with common syntactic behaviour ### NP sets defined according to their morphosyntactic features ### The PRE-NP-HEAD family of sets These sets model noun phrases (NPs). The idea is to first define whatever can occur in front of the head of the NP, and thereafter negate that with the expression **WORD - premodifiers**. ### Border sets and their complements Grammarchecker rules begin here ### Grammarchecker sets ## Grammarchecker rules ### Speller rules ### Agreement rules ### Negation verb rules ### Postposition rules ### Punctuation rules * * * This (part of) documentation was generated from [tools/grammarcheckers/grammarchecker.cg3](https://github.com/giellalt/lang-est-x-utee/blob/main/tools/grammarcheckers/grammarchecker.cg3) --- # tools-tokenisers-tokeniser-disamb-gt-desc.pmscript.md # Tokeniser for est Usage: ``` $ make $ echo "ja, ja" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa boasttu olmmoš, man mielde lahtuid." | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "márffibiillagáffe" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` Pmatch documentation: <https://github.com/hfst/hfst/wiki/HfstPmatch> Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words: * Punct contains ASCII punctuation marks * The symbol after m-dash is soft-hyphen `U+00AD` * The symbol following {•} is byte-order-mark / zero-width no-break space `U+FEFF`. Whitespace contains ASCII white space and the List contains some unicode white space characters * En Quad U+2000 to Zero-Width Joiner U+200d' * Narrow No-Break Space U+202F * Medium Mathematical Space U+205F * Word joiner U+2060 Apart from what's in our morphology, there are 1. unknown word-like forms, and 2. unmatched strings We want to give 1) a match, but let 2) be treated specially by `hfst-tokenise -a` Unknowns are made of: * lower-case ASCII * upper-case ASCII * select extended latin symbols ASCII digits * select symbols * Combining diacritics as individual symbols, * various symbols from Private area (probably Microsoft), so far: * U+F0B7 for "x in box" ## Unknown handling Unknowns are tagged ?? and treated specially with `hfst-tokenise` hfst-tokenise --giella-cg will treat such empty analyses as unknowns, and remove empty analyses from other readings. Empty readings are also legal in CG, they get a default baseform equal to the wordform, but no tag to check, so it's safer to let hfst-tokenise handle them. Finally we mark as a token any sequence making up a: * known word in context * unknown (OOV) token in context * sequence of word and punctuation * URL in context * * * This (part of) documentation was generated from [tools/tokenisers/tokeniser-disamb-gt-desc.pmscript](https://github.com/giellalt/lang-est-x-utee/blob/main/tools/tokenisers/tokeniser-disamb-gt-desc.pmscript) --- # tools-tokenisers-tokeniser-gramcheck-gt-desc.pmscript.md # Grammar checker tokenisation for est Requires a recent version of HFST (3.10.0 / git revision>=3aecdbc) Then just: ``` $ make $ echo "ja, ja" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` More usage examples: ``` $ echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa boasttu olmmoš, man mielde lahtuid." | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "márffibiillagáffe" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` Pmatch documentation: <https://github.com/hfst/hfst/wiki/HfstPmatch> Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words: * Punct contains ASCII punctuation marks * The symbol after m-dash is soft-hyphen `U+00AD` * The symbol following {•} is byte-order-mark / zero-width no-break space `U+FEFF`. Whitespace contains ASCII white space and the List contains some unicode white space characters * En Quad U+2000 to Zero-Width Joiner U+200d' * Narrow No-Break Space U+202F * Medium Mathematical Space U+205F * Word joiner U+2060 Apart from what's in our morphology, there are 1) unknown word-like forms, and 2) unmatched strings We want to give 1) a match, but let 2) be treated specially by hfst-tokenise -a * select extended latin symbols * select symbols * various symbols from Private area (probably Microsoft), so far: * U+F0B7 for "x in box" TODO: Could use something like this, but built-in's don't include šžđčŋ: Simply give an empty reading when something is unknown: hfst-tokenise --giella-cg will treat such empty analyses as unknowns, and remove empty analyses from other readings. Empty readings are also legal in CG, they get a default baseform equal to the wordform, but no tag to check, so it's safer to let hfst-tokenise handle them. Finally we mark as a token any sequence making up a: * known word in context * unknown (OOV) token in context * sequence of word and punctuation * URL in context * * * This (part of) documentation was generated from [tools/tokenisers/tokeniser-gramcheck-gt-desc.pmscript](https://github.com/giellalt/lang-est-x-utee/blob/main/tools/tokenisers/tokeniser-gramcheck-gt-desc.pmscript) --- # tools-tokenisers-tokeniser-tts-cggt-desc.pmscript.md # TTS tokenisation for smj Requires a recent version of HFST (3.10.0 / git revision>=3aecdbc) Then just: ```sh make echo "ja, ja" \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` More usage examples: ```sh echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa \ boasttu olmmoš, man mielde lahtuid." \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst echo "márffibiillagáffe" \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` Pmatch documentation: <https://kitwiki.csc.fi/twiki/bin/view/KitWiki/HfstPmatch> Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words: * Punct contains ASCII punctuation marks * The symbol after m-dash is soft-hyphen `U+00AD` * The symbol following {•} is byte-order-mark / zero-width no-break space `U+FEFF`. Whitespace contains ASCII white space and the List contains some unicode white space characters * En Quad U+2000 to Zero-Width Joiner U+200d' * Narrow No-Break Space U+202F * Medium Mathematical Space U+205F * Word joiner U+2060 Apart from what's in our morphology, there are 1) unknown word-like forms, and 2) unmatched strings We want to give 1) a match, but let 2) be treated specially by hfst-tokenise -a * select extended latin symbols * select symbols * various symbols from Private area (probably Microsoft), so far: * U+F0B7 for "x in box" TODO: Could use something like this, but built-in's don't include šžđčŋ: Simply give an empty reading when something is unknown: hfst-tokenise --giella-cg will treat such empty analyses as unknowns, and remove empty analyses from other readings. Empty readings are also legal in CG, they get a default baseform equal to the wordform, but no tag to check, so it's safer to let hfst-tokenise handle them. Needs hfst-tokenise to output things differently depending on the tag they get * * * This (part of) documentation was generated from [tools/tokenisers/tokeniser-tts-cggt-desc.pmscript](https://github.com/giellalt/lang-est-x-utee/blob/main/tools/tokenisers/tokeniser-tts-cggt-desc.pmscript)