INTRODUCTION TO THE MORPHOLOGICAL ANALYSER FOR Tokelauan .

Definitions for Multichar_Symbols

Composed letters

• a2 e2 i2 o2 u2 these are for first part of lexical aa, ee, ii, oo, uu

Analysis symbols

The morphological analyses of wordforms for the Tokelauan language are presented in this system in terms of the following symbols. (It is highly suggested to follow existing standards when adding new tags).

The parts-of-speech are:

• +N +A +Adv +V
• +Pron +CS +CC +Adp +Po +Pr +Interj +Pcle +Num
• +Incl +Excl
• +Rom

The parts of speech are further split up into:

• +Prop +Pers +Dem +Interr +Refl +Recipr +Rel +Indef +Poss
• +Def +Det

• +Prox +Med +Dist = proximal, medial, distal (near speaker, hearer, neiter)

• +Post +Pre = postmodifiers, premodifiers

The Usage extents are marked using following tags:

• +Err/Orth
• +Use/-Spell
• +Use/TTS – only retained in the HFST Text-To-Speech disambiguation tokeniser
• +Use/-TTS – never retained in the HFST Text-To-Speech disambiguation tokeniser

The nominals are inflected in the following Number

• +Sg +Du +Pl
• +Nom +Gen +Acc
• +Loc for locative nouns

The possession is marked as such: Numerals are classified under: Verb moods are: Other verb forms are

• +Symbol = independent symbols in the text stream, like £, €, © Special symbols are classified with: The verbs are syntactically split according to transitivity: Special multiword units are analysed with: Non-dictionary words can be recognised with:

For reduplication ^R ^E ^R1 ^R2 ^E1 ^E2 ^3S ^T ^A ^[ ^] cons, vow, for 2- and 3-syll

+Dir directional

Question and Focus particles: +Qst +Foc

+Prf +Ipf +Fut Aspect, perhaps

• +Emph
• +Dem-Dist
• +Du1
• +Du2
• +Du3
• +Pl1
• +Pl2
• +Pl3
• +Sg1
• +Sg2
• +Sg3
• +Use/GC

Semantics are classified with

• +Sem/Date
• +Sem/Measr

• +Sem/Time-clock

• +Sem/Ani
• +Sem/Build
• +Sem/Clth
• +Sem/Edu
• +Sem/Fem
• +Sem/Group
• +Sem/Hum
• +Sem/Mal
• +Sem/Measr
• +Sem/Obj
• +Sem/Org
• +Sem/Plant
• +Sem/Plc
• +Sem/Route
• +Sem/Sur
• +Sem/Time
• +Sem/Txt
• +Sem/Veh
• +Sem/Wthr

These tags are used somewhere:

Derivations are classified under the morphophonetic form of the suffix, the source and target part-of-speech.

Morphophonology To represent phonologic variations in word forms we use the following symbols in the lexicon files:

And following triggers to control variation

Flag diacritics

We have manually optimised the structure of our lexicon using following flag diacritics to restrict morhpological combinatorics - only allow compounds with verbs if the verb is further derived into a noun again: | @P.NeedNoun.ON@ | (Dis)allow compounds with verbs unless nominalised | @D.NeedNoun.ON@ | (Dis)allow compounds with verbs unless nominalised | @C.NeedNoun@ | (Dis)allow compounds with verbs unless nominalised

For languages that allow compounding, the following flag diacritics are needed to control position-based compounding restrictions for nominals. Their use is handled automatically if combined with +CmpN/xxx tags. If not used, they will do no harm. | @P.CmpFrst.FALSE@ | Require that words tagged as such only appear first | @D.CmpPref.TRUE@ | Block such words from entering ENDLEX | @P.CmpPref.FALSE@ | Block these words from making further compounds | @D.CmpLast.TRUE@ | Block such words from entering R | @D.CmpNone.TRUE@ | Combines with the next tag to prohibit compounding | @U.CmpNone.FALSE@ | Combines with the prev tag to prohibit compounding | @P.CmpOnly.TRUE@ | Sets a flag to indicate that the word has passed R | @D.CmpOnly.FALSE@ | Disallow words coming directly from root.

Use the following flag diacritics to control downcasing of derived proper nouns (e.g. Finnish Pariisi -> pariisilainen). See e.g. North Sámi for how to use these flags. There exists a ready-made regex that will do the actual down-casing given the proper use of these flags. | @U.Cap.Obl@ | Allowing downcasing of derived names: deatnulasj. | @U.Cap.Opt@ | Allowing downcasing of derived names: deatnulasj.

Use the following flag diacritics to control tag placement for prefixing | @U.verb.sg@ | Tag moving from prefixing to +Sg | @U.verb.pl@ | Tag moving from prefixing to +Pl | @U.noun.sg@ | Tag moving from prefixing to +Sg | @U.noun.pl@ | Tag moving from prefixing to +Pl | @C.ErrOrth@ | a pematch tag to remove

Flag diacritic	Explanation
@P.Pmatch.Loc@	Used on multi-token analyses; tell hfst-tokenise/pmatch where in the form/analysis the token should be split. Used e.g. in bijladagi to split bijla from dagi, or after abbreviations with full stops before the full stop, to allow an alternate +CLB analysis of it in case of a sentence final abbreviation. NB! This will give a faulty lemma for the abbreviation, as it will not include the full stop. This can lead to other issues, but presently we have no other solution if we want to keep the full stopp as a separate token. We could leave a full stop at the end of the abbreviation lemma as well (but not on the input side - we only have one full stop in the input). That must be tested, it could work, but then requires special attention when generating suggestions in e.g. grammar checkers - it should not generate two full stops.
@P.Pmatch.Backtrack@	Used on single-token analyses; tell hfst-tokenise/pmatch to backtrack by reanalysing the substrings before and after this point in the form (to find combinations of shorter analyses that would otherwise be missed)

Flag diacritic	Explanation
@D.ErrOrth.ON@	To be written
@R.ErrOrth.ON@	To be written
@C.ErrOrth@	To be written
@P.ErrOrth.ON@	To be written

Flag diacritic	Explanation
@U.number.one@	Flag used to give arabic numerals in smj different cases ;
@U.number.two@	Flag used to give arabic numerals in smj different cases ;
@U.number.three@	Flag used to give arabic numerals in smj different cases ;
@U.number.four@	Flag used to give arabic numerals in smj different cases ;
@U.number.five@	Flag used to give arabic numerals in smj different cases ;
@U.number.six@	Flag used to give arabic numerals in smj different cases ;
@U.number.seven@	Flag used to give arabic numerals in smj different cases ;
@U.number.eight@	Flag used to give arabic numerals in smj different cases ;
@U.number.nine@	Flag used to give arabic numerals in smj different cases ;
@U.number.zero@	Flag used to give arabic numerals in smj different cases ;

@P.number.one@	Flag used to give arabic numerals in smj different cases ;
@P.number.two@	Flag used to give arabic numerals in smj different cases ;
@P.number.three@	Flag used to give arabic numerals in smj different cases ;
@P.number.four@	Flag used to give arabic numerals in smj different cases ;
@P.number.five@	Flag used to give arabic numerals in smj different cases ;
@P.number.six@	Flag used to give arabic numerals in smj different cases ;
@P.number.seven@	Flag used to give arabic numerals in smj different cases ;
@P.number.eight@	Flag used to give arabic numerals in smj different cases ;
@P.number.nine@	Flag used to give arabic numerals in smj different cases ;
@P.number.ten@	Flag used to give arabic numerals in smj different cases ;
@P.number.one@	Flag used to give arabic numerals in smj different cases ;
@P.number.two@	Flag used to give arabic numerals in smj different cases ;
@P.number.three@	Flag used to give arabic numerals in smj different cases ;
@P.number.four@	Flag used to give arabic numerals in smj different cases ;
@P.number.five@	Flag used to give arabic numerals in smj different cases ;
@P.number.six@	Flag used to give arabic numerals in smj different cases ;
@P.number.seven@	Flag used to give arabic numerals in smj different cases ;
@P.number.eight@	Flag used to give arabic numerals in smj different cases ;
@P.number.nine@	Flag used to give arabic numerals in smj different cases ;
@P.number.ten@	Flag used to give arabic numerals in smj different cases ;
@P.number.one@	Flag used to give arabic numerals in smj different cases ;
@P.number.two@	Flag used to give arabic numerals in smj different cases ;
@P.number.three@	Flag used to give arabic numerals in smj different cases ;
@P.number.four@	Flag used to give arabic numerals in smj different cases ;
@P.number.five@	Flag used to give arabic numerals in smj different cases ;
@P.number.six@	Flag used to give arabic numerals in smj different cases ;
@P.number.seven@	Flag used to give arabic numerals in smj different cases ;
@P.number.eight@	Flag used to give arabic numerals in smj different cases ;
@P.number.nine@	Flag used to give arabic numerals in smj different cases ;
@P.number.ten@	Flag used to give arabic numerals in smj different cases ;

• @@ODE@ The word forms in Tokelauan language start from the lexeme roots of basic word classes, or optionally from prefixes:

---

This (part of) documentation was generated from src/fst/morphology/root.lexc