Komi-Permyak NLP Grammar

Finite state and Constraint Grammar based analysers, proofing tools and other resources

View the project on GitHub giellalt/lang-koi

Page Content

Komi-Permyak language model documentation

All doc-comment documentation in one large file.

src-cg3-disambiguator.cg3.md

Komi-Permyak disambiguator

Delimiters

Sentence delimiters are: “<.>” “<!>” “<?>” “<…>” “<¶>” sent

Tags and sets

Beginning and end of sentence

BOS EOS

Miscellanous

CmpTest Err Err вӧлі Sg3

Parts of speech tags

N V A Adv CC CS Inter Pron Num Pcle Clt Po Adp just in case Dem Det Tot Coll Qnt Prop

Derivation tags

Ex/A (former adj) Ex/N Ex/Num Ex/V Ex/WORD DerTag AspDerTag

Verbal categories

Prs Fut Fut1 Imprt Prt1 Prt2 Prf PrfIpf HstPrf PluPrf HstPluPrf Ind Imp Cond Opt

Sg1 Sg2 …

Nominal categories Sg Pl Nom Gen Abl Dat Com Cns …

Verb sets

VNEG (all Neg verbs)

VFIN

ASKI (tomorrow set)

NOT-PRL (have no homograph Prolative pairs set)

Rule section

This (part of) documentation was generated from src/cg3/disambiguator.cg3

src-cg3-functions.cg3.md

These sets model noun phrases (NPs). The idea is to first define whatever can occur in front of the head of the NP, and thereafter negate that with the expression WORD - premodifiers.

The set NOT-NPMOD is used to find barriers between NPs. Typical usage: … (*1 N BARRIER NPT-NPMOD) … meaning: Scan to the first noun, ignoring anything that can be part of the noun phrase of that noun (i.e., “scan to the next NP head”)

These were the set types.

HABITIVE MAPPING

sma object

SUBJ MAPPING - leftovers

OBJ MAPPING - leftovers

HNOUN MAPPING

This (part of) documentation was generated from src/cg3/functions.cg3

src-fst-morphology-affixes-adjectives.lexc.md

Adjective inflection

Komi (Permyak) adjectives compare.

Continuation lexicon has been assigned according to content

This (part of) documentation was generated from src/fst/morphology/affixes/adjectives.lexc

src-fst-morphology-affixes-adpositions.lexc.md

Postposition inflection

Komi postpositions inflect for direction.

Prep lexica

Russian на

Postp lexica

This contlex allows for relational word which, otherwise, are open to extensive declension

аддза, бӧрті, бокиті, боксянь, дырйи, йитӧдын, кузя, ног, ньылыд, паныдӧн, пӧлӧн, пыдди, пыр, понда, ради, уліті, выліті, вывті, вомас, вомӧн пӧвст

аддза, бӧрті, бокиті, боксянь, дырйи, йитӧдын, кузя, ног, ньылыд, паныдӧн, пӧлӧн, пыдди, пыр, понда, ради, уліті, выліті, вывті, вомас, вомӧн пӧвст

This (part of) documentation was generated from src/fst/morphology/affixes/adpositions.lexc

src-fst-morphology-affixes-adverbs.lexc.md

Adverb inflection

Komi adverbs inflect for direction.

This (part of) documentation was generated from src/fst/morphology/affixes/adverbs.lexc

src-fst-morphology-affixes-conjunctors.lexc.md

Conjunctors

Komi conjunctors

This (part of) documentation was generated from src/fst/morphology/affixes/conjunctors.lexc

src-fst-morphology-affixes-nouns.lexc.md

Noun morphological lexica !

Basic nouns.

The lexicon for basic nouns is N_

This should be phased out 2013-05-07

subsequent Cns vs Vow

Inflectional lexica

All nouns follow one contlex “N_” to begin with here is simply a list of all variant with no more variants beyond:

SG1

SG2

SG3

PL1

PL2 PL3

SG1 SG2 SG3 PL1 PL2 PL3

SG1 SG2 SG3 PL1 PL2 PL3

SG1

SG2

SG3

PL1

PL2 PL3

Case followed by possible clitic OR WORDEND

+Der/A+Adv:%>а K ;

This (part of) documentation was generated from src/fst/morphology/affixes/nouns.lexc

src-fst-morphology-affixes-numerals.lexc.md

Numeral morphological lexica

LEXICON NUM-APPR ! 2011-11-03 This will need work

Inflectional lexica

All nouns follow one contlex “Noun1” to begin with here is simply a list of all variant with no more variants beyond:

Arabic numerals

This (part of) documentation was generated from src/fst/morphology/affixes/numerals.lexc

src-fst-morphology-affixes-particles.lexc.md

Particles

Komi Particles

This (part of) documentation was generated from src/fst/morphology/affixes/particles.lexc

src-fst-morphology-affixes-prefixes.lexc.md

Prefixes Prefixes in the Komi-Permyak language are bound to the beginning of other words.

This (part of) documentation was generated from src/fst/morphology/affixes/prefixes.lexc

src-fst-morphology-affixes-pronouns.lexc.md

Pronominal morphology

Closed class personal pronouns

ми мийӧ The 1st and 2nd persons have Oblique case stem strategies that differ from the 3rd person: ті тійӧ nämä ovat aivan eri asioita? сы сійӧ tosin joskus

Tagged in the src/morphology/stems/pronouns.xml file

This (part of) documentation was generated from src/fst/morphology/affixes/pronouns.lexc

src-fst-morphology-affixes-propernouns.lexc.md

Proper noun inflection

Komi proper nouns inflect in the same cases as regular nouns.

Russian type Surnames

Preparing for the template urj-Cyrl Beginning 2012-11-15

Абдеев:Абдеев

Багрий:Багр

Аморский:Аморск

These are vowel-final stems They have previously received +Sem/Fem tags

Should this be limited to +Sg? 2015-09-06

Вили:Вил

Андрей:Андре

Ending 2012-11-15

FEMALE NAMES FROM TEMPLATE

This (part of) documentation was generated from src/fst/morphology/affixes/propernouns.lexc

src-fst-morphology-affixes-symbols.lexc.md

Symbol affixes

This (part of) documentation was generated from src/fst/morphology/affixes/symbols.lexc

src-fst-morphology-affixes-verbs.lexc.md

Verbal morphology

Closed class verbs

Open class verbs

Some Flag diacritic lines are with regexes, other with aligned zeros. We want to migrate to regexes < … > , for readability reasons (sic!)

Begin work with TV and IV

Verb conjugation

Differs from kpv 2016-09-15

Derivation

This is fed by LEXICON V_SHUNY, and therefore certain corrections must be made 2012-01-18

овсьыны пусьыштлывлыны босьтчыштлывлыны

verb-to-noun

вевттьысьыны

бертласьны

This (part of) documentation was generated from src/fst/morphology/affixes/verbs.lexc

src-fst-morphology-phonology.twolc.md

=================================== ! The Komi-Permyak morphophonological/twolc rules file =================================== !

This file documents the phonology.twolc file

Alphabet, Sets and Definitions

Letters of the alphabet

Archiphonemes for vowels

Triggers

Boundary symbols

Diacritics

Sets

Vowel

Palatal Vowel Cns-initial vowels

All non-vowels, consonants and hard and soft signs

All non-vowels with exception of soft sign

All consonants with hard A

All consonants followed by Cyrillic и Why is т2 here 2015-10-01

Letters

Dummy

Definitions

Rules

Double stem-final consonant

Double stem-final consonant л:в

%^1Cns to zero

э:е

The famous L/V

Paragogic consonants mDeletion

Paragogic consonants tDeletion

jDeletion after vowel

j to soft sign after r

j to hard sign after consonant

l deletion

d deletion

Vowel Palatalisation а 2 я, о 2 ё, у 2 ю

%{иі%} 2 і

%{иі%} 2 и

%{ая%} 2 а

%{ая%} 2 я

Soft Sign Deletion

Hard Sign Deletion

Hard Sign Palatalization

No triple letters

IClitic

Disallow soft retention when not сь or зь

This (part of) documentation was generated from src/fst/morphology/phonology.twolc

src-fst-morphology-root.lexc.md

INTRODUCTION TO MORPHOLOGICAL ANALYSER OF Komi-Permyak LANGUAGE.

Definitions for Multichar_Symbols

Analysis symbols

The morphological analyses of wordforms for the Komi-Permyak language are presented in this system in terms of the following symbols. (It is highly suggested to follow existing standards when adding new tags).

The parts-of-speech tags are:

The parts of speech are further split up into: Adjectives

Adverbs These are ideophonic descriptors used to modify the verb вырк ливтясь “flit and it flew off”

Interjections +Conative Used for calling animals, for example

Nouns proper used with paired nouns collective nouns

Postpositions

Pronouns

Quantifiers (numerals)

Quantifiers and Numerals are classified under:

Nominals are inflected for Number and Case

Number

singular plural

Case

accusative -ӧс ablative case -лісь approximative -лань approximative egressive -ланьсянь approximative elative -ланьысь approximative illative -ланьӧ approximative inessive -ланьын approximative prolative -ланьӧт approximative terminative -ланьӧдз approximative terminative -ланьви carative -тӧг consecultative -ла Comitative -кӧт Comparative case form -ся

transitive THIS should be removed 2019-11-01

used when possessive marking occurs before case used when case occurs before possessive marking

this will be used for marking complement forms such as кыкнан in collective numerals, сы , which can only appear with a following NP. DO WE NEED THIS 2019-10-22? jaska Niko

Symbols that need to be escaped on the lower side (towards twolc):

The possession is marked as such: ODD

The comparative forms are:

Numerals are classified under:

Verb moods are:

Question and Focus particles:

Tags distinguishing different versions of the same lemma (before POS)

The Usage extents are marked using following tags:

Where do these come from source

Semantics are classified with

Derivations are classified under the morphophonetic form of the suffix, the source and target part-of-speech.

2012-09-11 Perhaps this is only syntactic

Tags for Ethymological Origin marking. This has initially used used with proper nouns

Morphophonology To represent phonologic variations in word forms we use the following symbols in the lexicon files:

And following triggers to control variation

Flag diacritics

We have manually optimised the structure of our lexicon using following flag diacritics to restrict morhpological combinatorics - only allow compounds with verbs if the verb is further derived into a noun again: | @P.NeedNoun.ON@ | (Dis)allow compounds with verbs unless nominalised | @D.NeedNoun.ON@ | (Dis)allow compounds with verbs unless nominalised | @C.NeedNoun@ | (Dis)allow compounds with verbs unless nominalised

For languages that allow compounding, the following flag diacritics are needed to control position-based compounding restrictions for nominals. Their use is handled automatically if combined with +CmpN/xxx tags. If not used, they will do no harm. | @P.CmpFrst.FALSE@ | Require that words tagged as such only appear first | @D.CmpPref.TRUE@ | Block such words from entering ENDLEX | @P.CmpPref.FALSE@ | Block these words from making further compounds | @D.CmpLast.TRUE@ | Block such words from entering R | @D.CmpNone.TRUE@ | Combines with the next tag to prohibit compounding | @U.CmpNone.FALSE@ | Combines with the prev tag to prohibit compounding | @P.CmpOnly.TRUE@ | Sets a flag to indicate that the word has passed R | @D.CmpOnly.FALSE@ | Disallow words coming directly from root.

Use the following flag diacritics to control downcasing of derived proper nouns (e.g. Finnish Pariisi -> pariisilainen). See e.g. North Sámi for how to use these flags. There exists a ready-made regex that will do the actual down-casing given the proper use of these flags. | @U.Cap.Obl@ | Allowing downcasing of derived names: deatnulasj. | @U.Cap.Opt@ | Allowing downcasing of derived names: deatnulasj.

FLAGS USED WITH COLLECTIVE NOUNS

Removal

@U.Cap.Obl@ Allowing downcasing of derived names: deatnulasj
@U.Cap.Opt@ Allowing downcasing of derived names: deatnulasj

The word forms in the KOMI-PERMYAK language start from the lexeme roots of basic

word classes, or optionally from prefixes:

Incoming from akusanat Pron_koi2x ;

Incoming for conversion from kpv

This (part of) documentation was generated from src/fst/morphology/root.lexc

src-fst-morphology-stems-adjectives.lexc.md

Adjectives Adjectives in Komi-Permyak language describe things.

This (part of) documentation was generated from src/fst/morphology/stems/adjectives.lexc

src-fst-morphology-stems-adjectives_newwords.lexc.md

This is where new words from kpv are initially shown

This (part of) documentation was generated from src/fst/morphology/stems/adjectives_newwords.lexc

src-fst-morphology-stems-adverbs_newwords.lexc.md

This is where new words from kpv are initially shown

This (part of) documentation was generated from src/fst/morphology/stems/adverbs_newwords.lexc

src-fst-morphology-stems-exceptions.lexc.md

Exceptions are quite strange word-forms. the ones that do not fit anywhere else. This file contains all enumerated word forms that cannot reasonably be created from lexical data by regular inflection. Usually there should be next to none exceptions, it’s always better to have a paradigm that covers only one or few words than an exception since these will not work nicely with e.g. compounding scheme or possibly many end applications.

The pair verb овны-вывны conjugates in more forms than are attested for the single verb вывны:

Russian-type

This (part of) documentation was generated from src/fst/morphology/stems/exceptions.lexc

src-fst-morphology-stems-kom-rus-propernouns_newwords.lexc.md

This is where new words are added as lexc entries before they are added to the xml source files. Абъячой+N+Prop+Sem/Plc:Абъячой PROP_ “(eng) fish/(fin) /(rus)” ;

ADD NOUNS BELOW

This (part of) documentation was generated from src/fst/morphology/stems/kom-rus-propernouns_newwords.lexc

src-fst-morphology-stems-nouns.lexc.md

Nouns Nouns in the Komi Permyak language are things.

This (part of) documentation was generated from src/fst/morphology/stems/nouns.lexc

src-fst-morphology-stems-nouns_newwords.lexc.md

This is where new words from kpv are initially shown

2016-07-14

This (part of) documentation was generated from src/fst/morphology/stems/nouns_newwords.lexc

src-fst-morphology-stems-numerals.lexc.md

Numerals Numerals in the KOMI-PERMYAK language are numbers.

This (part of) documentation was generated from src/fst/morphology/stems/numerals.lexc

src-fst-morphology-stems-numerals_newwords.lexc.md

This is where new words from kpv are initially shown

This (part of) documentation was generated from src/fst/morphology/stems/numerals_newwords.lexc

src-fst-morphology-stems-pronouns.lexc.md

Pronouns Pronouns in KOMI PERMYAK language are references to things.

This (part of) documentation was generated from src/fst/morphology/stems/pronouns.lexc

src-fst-morphology-stems-pronouns_newwords.lexc.md

This is where new words from kpv are initially shown

This (part of) documentation was generated from src/fst/morphology/stems/pronouns_newwords.lexc

src-fst-morphology-stems-verbs_newwords.lexc.md

This is where new words from kpv are initially shown

These are koi verbs

This (part of) documentation was generated from src/fst/morphology/stems/verbs_newwords.lexc

src-fst-phonetics-txt2ipa.xfscript.md

retroflex plosive, voiceless t ʈ 0288, 648 ( = ASCII 096) retroflex plosive, voiced d ɖ 0256, 598 labiodental nasal F ɱ 0271, 625 retroflex nasal n ɳ 0273, 627 palatal nasal J ɲ 0272, 626 velar nasal N ŋ 014B, 331 uvular nasal N\ ɴ 0274, 628

bilabial trill B\ ʙ 0299, 665 uvular trill R\ ʀ 0280, 640 alveolar tap 4 ɾ 027E, 638 retroflex flap r ɽ 027D, 637 bilabial fricative, voiceless p\ ɸ 0278, 632 bilabial fricative, voiced B β 03B2, 946 dental fricative, voiceless T θ 03B8, 952 dental fricative, voiced D ð 00F0, 240 postalveolar fricative, voiceless S ʃ 0283, 643 postalveolar fricative, voiced Z ʒ 0292, 658 retroflex fricative, voiceless s ʂ 0282, 642 retroflex fricative, voiced z` ʐ 0290, 656 palatal fricative, voiceless C ç 00E7, 231 palatal fricative, voiced j\ ʝ 029D, 669 velar fricative, voiced G ɣ 0263, 611 uvular fricative, voiceless X χ 03C7, 967 uvular fricative, voiced R ʁ 0281, 641 pharyngeal fricative, voiceless X\ ħ 0127, 295 pharyngeal fricative, voiced ?\ ʕ 0295, 661 glottal fricative, voiced h\ ɦ 0266, 614

alveolar lateral fricative, vl. K alveolar lateral fricative, vd. K\

labiodental approximant P (or v) alveolar approximant r\ retroflex approximant r` velar approximant M\

retroflex lateral approximant l` palatal lateral approximant L velar lateral approximant L
Clicks

bilabial O\ (O = capital letter) dental |
(post)alveolar !\ palatoalveolar =\ alveolar lateral ||
Ejectives, implosives

ejective > e.g. ejective p p> implosive < e.g. implosive b b< Vowels

close back unrounded M close central unrounded 1 close central rounded } lax i I lax y Y lax u U

close-mid front rounded 2 close-mid central unrounded @\ close-mid central rounded 8 close-mid back unrounded 7

schwa ə @

open-mid front unrounded E open-mid front rounded 9 open-mid central unrounded 3 open-mid central rounded 3\ open-mid back unrounded V open-mid back rounded O

ash (ae digraph) { open schwa (turned a) 6

open front rounded & open back unrounded A open back rounded Q Other symbols

voiceless labial-velar fricative W voiced labial-palatal approx. H voiceless epiglottal fricative H\ voiced epiglottal fricative <\ epiglottal plosive >\

alveolo-palatal fricative, vl. s\ alveolo-palatal fricative, voiced z\ alveolar lateral flap l\ simultaneous S and x x\ tie bar _ Suprasegmentals

primary stress “ secondary stress % long : half-long :\ extra-short _X linking mark -
Tones and word accents

level extra high _T level high _H level mid _M level low _L level extra low _B downstep ! upstep ^ (caret, circumflex)

contour, rising contour, falling _F contour, high rising _H_T contour, low rising _B_L

contour, rising-falling _R_F (NB Instead of being written as diacritics with _, all prosodic marks can alternatively be placed in a separate tier, set off by < >, as recommended for the next two symbols.) global rise global fall Diacritics

voiceless 0 (0 = figure), e.g. n_0 voiced _v aspirated _h more rounded _O (O = letter) less rounded _c advanced _+ retracted _- centralized _” syllabic = (or _=) e.g. n= (or n=) non-syllabic _^ rhoticity `

breathy voiced _t creaky voiced _k linguolabial _N labialized _w palatalized ‘ (or _j) e.g. t’ (or t_j) velarized _G pharyngealized _?\

dental d apical _a laminal _m nasalized ~ (or _~) e.g. A~ (or A~) nasal release _n lateral release _l no audible release _}

velarized or pharyngealized _e velarized l, alternatively 5 raised _r lowered _o advanced tongue root _A retracted tongue root _q

This (part of) documentation was generated from src/fst/phonetics/txt2ipa.xfscript

src-fst-transcriptions-transcriptor-abbrevs2text.lexc.md

We describe here how abbreviations are in Komi-Permyak are read out, e.g. for text-to-speech systems.

For example:

This (part of) documentation was generated from src/fst/transcriptions/transcriptor-abbrevs2text.lexc

src-fst-transcriptions-transcriptor-symbols2text.lexc.md

This file contains mappings from abbreviations and some acronyms to full forms for text-to-speech purposes. This is a supplement to the analyser; the analyser must tag the strings as +ABBR or similar for the transcriptions to work. The resulting full form must be lemmas known to the analyser, for further processing.

We describe here how abbreviations in Komi-Permyak are read out, for text-to-speech systems.

The file contains:

This (part of) documentation was generated from src/fst/transcriptions/transcriptor-symbols2text.lexc

tools-grammarcheckers-grammarchecker.cg3.md

[ L A N G U A G E ] G R A M M A R C H E C K E R

DELIMITERS

TAGS AND SETS

Tags

This section lists all the tags inherited from the fst, and used as tags in the syntactic analysis. The next section, Sets, contains sets defined on the basis of the tags listed here, those set names are not visible in the output.

Beginning and end of sentence

BOS EOS

Parts of speech tags

N A Adv V Pron CS CC CC-CS Po Pr Pcle Num Interj ABBR ACR CLB LEFT RIGHT WEB PPUNCT PUNCT

COMMA ¶

Tags for POS sub-categories

Pers Dem Interr Indef Recipr Refl Rel Coll NomAg Prop Allegro Arab Romertall

Tags for morphosyntactic properties

Nom Acc Gen Ill Loc Com Ess Ess Sg Du Pl Cmp/SplitR Cmp/SgNom Cmp/SgGen Cmp/SgGen PxSg1 PxSg2 PxSg3 PxDu1 PxDu2 PxDu3 PxPl1 PxPl2 PxPl3 Px

Comp Superl Attr Ord Qst IV TV Prt Prs Ind Pot Cond Imprt ImprtII Sg1 Sg2 Sg3 Du1 Du2 Du3 Pl1 Pl2 Pl3 Inf ConNeg Neg PrfPrc VGen PrsPrc Ger Sup Actio VAbess

Err/Orth

Semantic tags

Sem/Act Sem/Ani Sem/Atr Sem/Body Sem/Clth Sem/Domain Sem/Feat-phys Sem/Fem Sem/Group Sem/Lang Sem/Mal Sem/Measr Sem/Money Sem/Obj Sem/Obj-el Sem/Org Sem/Perc-emo Sem/Plc Sem/Sign Sem/State-sick Sem/Sur Sem/Time Sem/Txt

HUMAN

PROP-ATTR PROP-SUR

TIME-N-SET

Syntactic tags

@+FAUXV @+FMAINV @-FAUXV @-FMAINV @-FSUBJ> @-F<OBJ @-FOBJ> @-FSPRED<OBJ @-F<ADVL @-FADVL> @-F<SPRED @-F<OPRED @-FSPRED> @-FOPRED> @>ADVL @ADVL< @<ADVL @ADVL> @ADVL @HAB> @<HAB @>N @Interj @N< @>A @P< @>P @HNOUN @INTERJ @>Num @Pron< @>Pron @Num< @OBJ @<OBJ @OBJ> @OPRED @<OPRED @OPRED> @PCLE @COMP-CS< @SPRED @<SPRED @SPRED> @SUBJ @<SUBJ @SUBJ> SUBJ SPRED OPRED @PPRED @APP @APP-N< @APP-Pron< @APP>Pron @APP-Num< @APP-ADVL< @VOC @CVP @CNP OBJ

-OTHERS SYN-V @X ### Sets containing sets of lists and tags This part of the file lists a large number of sets based partly upon the tags defined above, and partly upon lexemes drawn from the lexicon. See the sourcefile itself to inspect the sets, what follows here is an overview of the set types. #### Sets for Single-word sets INITIAL #### Sets for word or not WORD NOT-COMMA #### Case sets ADLVCASE CASE-AGREEMENT CASE NOT-NOM NOT-GEN NOT-ACC #### Verb sets NOT-V #### Sets for finiteness and mood REAL-NEG MOOD-V NOT-PRFPRC #### Sets for person SG1-V SG2-V SG3-V DU1-V DU2-V DU3-V PL1-V PL2-V PL3-V #### Pronoun sets #### Adjectival sets and their complements #### Adverbial sets and their complements #### Sets of elements with common syntactic behaviour #### NP sets defined according to their morphosyntactic features #### The PRE-NP-HEAD family of sets These sets model noun phrases (NPs). The idea is to first define whatever can occur in front of the head of the NP, and thereafter negate that with the expression **WORD - premodifiers**. #### Border sets and their complements #### Grammarchecker sets * * * This (part of) documentation was generated from [tools/grammarcheckers/grammarchecker.cg3](https://github.com/giellalt/lang-koi/blob/main/tools/grammarcheckers/grammarchecker.cg3) --- ## tools-tokenisers-tokeniser-disamb-gt-desc.pmscript.md ## Tokeniser for koi Usage: ``` $ make $ echo "ja, ja" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa boasttu olmmoš, man mielde lahtuid." | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "márffibiillagáffe" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` Pmatch documentation: <https://github.com/hfst/hfst/wiki/HfstPmatch> Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words: * Punct contains ASCII punctuation marks * The symbol after m-dash is soft-hyphen `U+00AD` * The symbol following {•} is byte-order-mark / zero-width no-break space `U+FEFF`. Whitespace contains ASCII white space and the List contains some unicode white space characters * En Quad U+2000 to Zero-Width Joiner U+200d' * Narrow No-Break Space U+202F * Medium Mathematical Space U+205F * Word joiner U+2060 Apart from what's in our morphology, there are 1. unknown word-like forms, and 2. unmatched strings We want to give 1) a match, but let 2) be treated specially by `hfst-tokenise -a` Unknowns are made of: * lower-case ASCII * upper-case ASCII * lower-case cyrillic * upper-case cyrillic * select extended latin symbols * extended cyrillic ASCII digits * select symbols * Combining diacritics as individual symbols, * various symbols from Private area (probably Microsoft), so far: * U+F0B7 for "x in box" ### Unknown handling Unknowns are tagged ?? and treated specially with `hfst-tokenise` hfst-tokenise --giella-cg will treat such empty analyses as unknowns, and remove empty analyses from other readings. Empty readings are also legal in CG, they get a default baseform equal to the wordform, but no tag to check, so it's safer to let hfst-tokenise handle them. Finally we mark as a token any sequence making up a: * known word in context * unknown (OOV) token in context * sequence of word and punctuation * URL in context * * * This (part of) documentation was generated from [tools/tokenisers/tokeniser-disamb-gt-desc.pmscript](https://github.com/giellalt/lang-koi/blob/main/tools/tokenisers/tokeniser-disamb-gt-desc.pmscript) --- ## tools-tokenisers-tokeniser-gramcheck-gt-desc.pmscript.md ## Grammar checker tokenisation for koi Requires a recent version of HFST (3.10.0 / git revision>=3aecdbc) Then just: ``` $ make $ echo "ja, ja" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` More usage examples: ``` $ echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa boasttu olmmoš, man mielde lahtuid." | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst $ echo "márffibiillagáffe" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` Pmatch documentation: <https://github.com/hfst/hfst/wiki/HfstPmatch> Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words: * Punct contains ASCII punctuation marks * The symbol after m-dash is soft-hyphen `U+00AD` * The symbol following {•} is byte-order-mark / zero-width no-break space `U+FEFF`. Whitespace contains ASCII white space and the List contains some unicode white space characters * En Quad U+2000 to Zero-Width Joiner U+200d' * Narrow No-Break Space U+202F * Medium Mathematical Space U+205F * Word joiner U+2060 Apart from what's in our morphology, there are 1) unknown word-like forms, and 2) unmatched strings We want to give 1) a match, but let 2) be treated specially by hfst-tokenise -a * select extended latin symbols * select symbols * various symbols from Private area (probably Microsoft), so far: * U+F0B7 for "x in box" TODO: Could use something like this, but built-in's don't include šžđčŋ: Simply give an empty reading when something is unknown: hfst-tokenise --giella-cg will treat such empty analyses as unknowns, and remove empty analyses from other readings. Empty readings are also legal in CG, they get a default baseform equal to the wordform, but no tag to check, so it's safer to let hfst-tokenise handle them. Finally we mark as a token any sequence making up a: * known word in context * unknown (OOV) token in context * sequence of word and punctuation * URL in context * * * This (part of) documentation was generated from [tools/tokenisers/tokeniser-gramcheck-gt-desc.pmscript](https://github.com/giellalt/lang-koi/blob/main/tools/tokenisers/tokeniser-gramcheck-gt-desc.pmscript) --- ## tools-tokenisers-tokeniser-tts-cggt-desc.pmscript.md ## TTS tokenisation for smj Requires a recent version of HFST (3.10.0 / git revision>=3aecdbc) Then just: ```sh make echo "ja, ja" \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` More usage examples: ```sh echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa \ boasttu olmmoš, man mielde lahtuid." \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst echo "márffibiillagáffe" \ | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst ``` Pmatch documentation: <https://kitwiki.csc.fi/twiki/bin/view/KitWiki/HfstPmatch> Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words: * Punct contains ASCII punctuation marks * The symbol after m-dash is soft-hyphen `U+00AD` * The symbol following {•} is byte-order-mark / zero-width no-break space `U+FEFF`. Whitespace contains ASCII white space and the List contains some unicode white space characters * En Quad U+2000 to Zero-Width Joiner U+200d' * Narrow No-Break Space U+202F * Medium Mathematical Space U+205F * Word joiner U+2060 Apart from what's in our morphology, there are 1) unknown word-like forms, and 2) unmatched strings We want to give 1) a match, but let 2) be treated specially by hfst-tokenise -a * select extended latin symbols * select symbols * various symbols from Private area (probably Microsoft), so far: * U+F0B7 for "x in box" TODO: Could use something like this, but built-in's don't include šžđčŋ: Simply give an empty reading when something is unknown: hfst-tokenise --giella-cg will treat such empty analyses as unknowns, and remove empty analyses from other readings. Empty readings are also legal in CG, they get a default baseform equal to the wordform, but no tag to check, so it's safer to let hfst-tokenise handle them. Needs hfst-tokenise to output things differently depending on the tag they get * * * This (part of) documentation was generated from [tools/tokenisers/tokeniser-tts-cggt-desc.pmscript](https://github.com/giellalt/lang-koi/blob/main/tools/tokenisers/tokeniser-tts-cggt-desc.pmscript)

Sitemap