Old French language model documentation
All doc-comment documentation in one large file.
src-cg3-functions.cg3.md
- 
    Sets for POS sub-categories 
- 
    Sets for Semantic tags 
- 
    Sets for Morphosyntactic properties 
- 
    Sets for verbs 
- 
    V is all readings with a V tag in them, REAL-V should be the ones without an N tag following the V. 
 The REAL-V set thus awaits a fix to the preprocess V … N bug.
- 
    The set COPULAS is for predicative constructions 
- 
    NP sets defined according to their morphosyntactic features 
- 
    The PRE-NP-HEAD family of sets 
These sets model noun phrases (NPs). The idea is to first define whatever can occur in front of the head of the NP, and thereafter negate that with the expression WORD - premodifiers.
The set NOT-NPMOD is used to find barriers between NPs. Typical usage: … (*1 N BARRIER NPT-NPMOD) … meaning: Scan to the first noun, ignoring anything that can be part of the noun phrase of that noun (i.e., “scan to the next NP head”)
- 
    Miscellaneous sets 
- 
    Border sets and their complements 
- 
    Syntactic sets 
These were the set types.
HABITIVE MAPPING
- 
    hab1 
- 
    hab2 
- 
    hab3 ( @ADVL>) for hab-actor and hab-case; if leat to the right, and Nom to the right of leat. Lots of restrictions. 
- 
    habNomLeft 
- 
    hab4 
- 
    hab6 
- 
    hab7 
- hab8 This is not HAB
- 
    hab5 This is not HAB 
- 
    habDain ( @ADVL>) for (Pron Dem Pl Loc) if leat followed by Nom to the right 
- 
    habGen ( @<ADVL) hab for Gen; if Gen is located in the end of the sentence and Nom is sentence initial 
- 
    spred<obj (@SPRED<OBJ) for Acc; the object of an SPRPED. Not to be mistaken with OPRED. If SPRED is to the left, and copulas is to the left of it. Nom or Hab are found sentence initially. 
- 
    Hab<spred (@<SPRED) for Nom; if copulas, goallut or jápmit is FMAINV and habitive or human Loc is found to the left. OR: if Ill or @Pron< followed by HAB are found to the left. 
- 
    Hab>Advlcase<spred ( @<SUBJ) for Nom; it allows adverbials with Ill/Loc/Com/Ess to be found inbetween HAB and . 
- 
    Nom>Advlcase<spred ( @<SUBJ) for Nom; it allows adverbials with Ill/Loc/Com/Ess to be found inbetween Nom and @<SUBJ. 
- 
    <spred ( @<SUBJ) for Nom; if copulas to the left, and some kind of adverb, N Loc, time related word or Po to the left of it. OR: if Ill or @Pron< to the left, followed by copulas and the before mentioned to the left of copulas. 
- 
    <spred ( @<SUBJ) for Nom, but not for Pers. To the left boahtit or heaŋgát as MAINV, and futher to the left is some kind of place related word, or time related word 
- 
    <spredQst1 ( @<SUBJ) for Nom in a typically question sentence; if A) Hab, some kind of place word, Po or Nom to the left, and Qst followed by copulas to the left. B) same as a, only the Qst-pcle is attached to copulas. C) Qst to the left, with copulas to its left, but not if two Nom:s are found somewhere to the right. D) copulas to the left, and BOS to the left. E) Loc or Ill to the left, and Loc or Hab to the left of this, Qst and copulas to the left. F) Num @>N to the left, Hab, some kind of place word, Po or Nom to the left, and Qst followed by copulas to the left. NOTE) for all these rules; human, Loc or Sem/Plc not allowed to the right. 
- 
    <spredQst2 (@<SPRED) for Nom; in a typically question sentence; differs from <spredQst1 by not beeing as restricted to the right. Though you are not allowed to be Pers or human. 
- 
    Nom<spredQst (@<SPRED) for Nom; in a typically question sentence. Differs from <spredQst2 by letting Nom be found between SPRED and copulas 
- 
    <spred (@<SPRED) for A Nom or N Nom if; the subject Nom is on the same side of copulas as you: on the right side of copulas 
- 
    <spredVeara (@<SPRED) for veara + Nom; if genitive immediately to the right, and intransitive mainverb to the right of genitive 
- 
    leftCop<spred (@<SPRED) for Nom; if copulas is the main verb to the left, and there is no Ess found to the left of cop (note that Loc is allowed between target and cop). OR: if you are Coll or Sem/Group with copulas to your left. 
- 
    <spredLocEXPERIMENT (@<SPRED) for material Loc; if you are to the right of copulas, and the Nom to the left of copulas is not a hab-actor 
- 
    NumTime (@<SPRED) for A Nom 
- 
    <spredSg (@<SPRED) for Sg Nom 
- 
    <spredPg (@<SPRED) for Pl Nom 
- 
    <spred (@<SPRED) for Nom; if copulas to the left, and Nom or sentence boundary to the left of copulas. First one to the right is EOS. 
- 
    <spred (@<SPRED) for N Ess 
- 
    spredEss> (@SPRED>) for N Ess; if copulas to the right of you, and if an NP with nom-case first one to your left. 
- 
    HABSpredSg> (@SPRED>) for Nom; if habitive first one to the left, followed by copulas. 
- 
    GalleSpred> (@SPRED>) for Num Nom; if sentence initial 
- 
    spredSgMII> (@SPRED>) 
- 
    r492> (@SPRED>) for Interr Gen; consisting only of negations. You are not allowed to be MII. You are not allowed to have an adjective or noun to yor right. You are not allowed to have a verb to your right; the exception beeing an aux. 
- 
    AdjSpredSg> (@SPRED>) for A Sg Nom; if copulas to the right, but not if A or @<SPRED are found to the right of copulas 
- 
    SpredSg>Hab (@SPRED>) for Nom; if you are sentence initial, copulas is located to the right, and there is a habitive to the right of copulas 
- 
    Spred>SubjInf (@SPRED>) for Nom; if copulas to the right, and the subject of copulas is an Inf to the right 
- 
    spredCoord (@<SPRED) coordination for Nom; only if there already is a SPRED to the left of CNP. Not if there is some kind of comparison involved. 
- 
    subj>Sgnr1 (@SUBJ>) for Nom Sg, including Indef Nom if; VFIN + Sg3 or Pl3 to the right (VFIN not allowed to the left) 
- subj>Du (@SUBJ>) for dual nominatives, including Coll Nom. VFIN + Du3 to the right.
- 
    subj>Pl (@SUBJ>) for plural nominatives, including Coll and Sem/Group. VFIN + Pl3 to the right. 
- 
    subj>Pl (@SUBJ>) for plural nominatives 
- 
    subj>Sgnr2 (@SUBJ>) for Nom Sg; if VFIN + Sg3 to the right. 
- 
    <subjSg (@<SUBJ) for Nom Sg; if VFIN Sg3 or Du2 to the left (no HAB allowed to the left). 
- 
    f<advl (@-F<ADVL) for infinite adverbials 
- 
    f<advl (@-F<ADVL) for infinite adverbials 
- 
    s-boundary=advl> (@ADVL>) for ADVL that resemble s-booundaries. Mainverb to the right. 
- 
    -fobj> (@-FOBJ>) for Acc 
- 
    -fobj> (@-FOBJ>) for Acc 
- 
    advl>mainV (@ADVL>) if; finite mainverb not found to the left, but the finite mainverb is found to the right. 
- 
    <advl (@<ADVL) if; finite mainverb found to the left. Not if a comma is found immediately to the left and a finite mainverb is located somewhere to the right of this comma. 
- <advlPoPr (@<ADVL) if mainverb to the left.
- 
    advlPoPr> (@<ADVL) if mainverb to the right. 
- 
    advlEss> (@<ADVL) for weather and time Ess, if FMAINV to the left. 
- 
    advl>inbetween (@ADVL>) for Adv; if inbetween two sentenceboundaries where no mainverb is present. 
- 
    comma<advlEOS (@<ADVL) if; comma found to the left and the finite mainverb to the left of comma. To the right is the end of the sentence. 
- 
    advlBOS> (@ADVL>) if; you are N Ill and found sentnece initially. First one to your right is a clause. 
- 
    <advlPoEOS (@<ADVL) for Po; if you are found at the very end of a sentence. A mainverb is needed to the right though. 
- 
    cleanupILL<advl (@<ADVL) for N Ill if; there are no boundarysymbols to your left, if you arent already @N< OR @APP-N<, and no mainverb is to yor left. 
- <opredAAcc (@<OPRED) for A Acc; if an other accusative to the left, and a transtive verb to the left of it. OR: if a transitive verb to the left, and an accusative to the left of it.
sma object
- <advlEss (@<ADVL) for ESS-ADVL if; FMAINV to the left
- <spredEss (@<SPRED) for N Ess if; FMAINV to the left is intransitive or bargat
SUBJ MAPPING - leftovers
OBJ MAPPING - leftovers
HNOUN MAPPING
This (part of) documentation was generated from src/cg3/functions.cg3
src-fst-morphology-affixes-nouns.lexc.md
Noun inflection The Old French language nouns inflect in cases.
- nouns examples:*
- fille: fille+N+Fem+Nom+Sg
- file: fille+N+Fem+Nom+Sg
- fille: fille+N+Fem+Obl+Sg
- file: fille+N+Fem+Obl+Sg
- filles: fille+N+Fem+Nom+Pl
- files: fille+N+Fem+Nom+Pl
- filles: fille+N+Fem+Obl+Pl
- files: fille+N+Fem+Obl+Pl
- Alemaigne: Alemaigne+N+Prop+Fem+Nom+Sg
- Alemaigne: Alemaigne+N+Prop+Fem+Obl+Sg
- Alemaigne: Alemaigne+N+Prop+Fem+Nom+Pl
- Alemaigne: Alemaigne+N+Prop+Fem+Obl+Pl
- Alamaigne: Alemaigne+N+Prop+Fem+Nom+Sg
- Alamaigne: Alemaigne+N+Prop+Fem+Obl+Sg
- Alamaigne: Alemaigne+N+Prop+Fem+Nom+Pl
- Alamaigne: Alemaigne+N+Prop+Fem+Obl+Pl
- Gieffroy: Gieffroy+N+Prop+Masc+Nom+Sg
- Gieffroy: Gieffroy+N+Prop+Masc+Obl+Sg
- Gieffroy: Gieffroy+N+Prop+Masc+Nom+Pl
- Gieffroy: Gieffroy+N+Prop+Masc+Obl+Pl
- lei: lei+N+Fem+Nom+Sg
- lei: lei+N+Fem+Obl+Sg
- ley: lei+N+Fem+Nom+Sg
- ley: lei+N+Fem+Obl+Sg
- leis: lei+N+Fem+Nom+Pl
- leis: lei+N+Fem+Obl+Pl
- loi: lei+N+Fem+Nom+Sg
- loi: lei+N+Fem+Obl+Sg
- loy: lei+N+Fem+Nom+Sg
- loy: lei+N+Fem+Obl+Sg
- lois: lei+N+Fem+Nom+Pl
- lois: lei+N+Fem+Obl+Pl
- termes: terme+N+Masc+Nom+Sg
- terme: terme+N+Masc+Obl+Sg
- terme: terme+N+Masc+Nom+Pl
- termes: terme+N+Masc+Obl+Pl
- vespre: vespre+N+Masc+Nom+Sg
- vespre: vespre+N+Masc+Obl+Sg
- vespre: vespre+N+Masc+Nom+Pl
- vespres: vespre+N+Masc+Obl+Pl
- ancestre: ancessor+N+Masc+Nom+Sg
- ancessor: ancessor+N+Masc+Obl+Sg
- ancessor: ancessor+N+Masc+Nom+Pl
- ancessors: ancessor+N+Masc+Obl+Pl
- bers: bers+N+Masc+Nom+Sg
- bers: bers+N+Masc+Obl+Sg
- bers: bers+N+Masc+Nom+Pl
- bers: bers+N+Masc+Obl+Pl
- cortiers: cort+N+Agent+Masc+Nom+Sg
- corteor: cort+N+Agent+Masc+Obl+Sg
- courteor: cort+N+Agent+Masc+Obl+Sg
- courtiers: cort+N+Agent+Masc+Nom+Sg
- corteors: cort+N+Agent+Masc+Obl+Pl
- courteors: cort+N+Agent+Masc+Obl+Pl
- cortière: cort+N+Agent+Fem+Nom+Sg
- cortière: cort+N+Agent+Fem+Obl+Sg
- cortiere: cort+N+Agent+Fem+Nom+Sg
- cortiere: cort+N+Agent+Fem+Obl+Sg
- courtière: cort+N+Agent+Fem+Nom+Sg
- courtière: cort+N+Agent+Fem+Obl+Sg
- courtiere: cort+N+Agent+Fem+Nom+Sg
- courtiere: cort+N+Agent+Fem+Obl+Sg
This (part of) documentation was generated from src/fst/morphology/affixes/nouns.lexc
src-fst-morphology-affixes-verbs.lexc.md
Verb Inflection
- -er_verbs examples:*
- durer: durer+V+Inf
- dur: durer+V+1s+Pres+Ind
- durat: durer+V+3s+Perf+Ind
- dura: durer+V+3s+Perf+Ind
- durerons: durer+V+1p+Fut+Ind
- 
    dure: durer+V+2s+Pres+Imp
- -ier_verbs examples:*
- aidier: aidier+V+Inf
- aidastes: aidier+V+2p+Perf+Ind
- aidoit: aidier+V+3s+Ipfv+Ind
- aideit: aidier+V+3s+Ipfv+Ind
- aida: aidier+V+3s+Perf+Ind
- aidat: aidier+V+3s+Perf+Ind
- aidierons: aidier+V+1p+Fut+Ind
- aidieront: aidier+V+3p+Fut+Ind
- 
    aidasse: aidier+V+1s+Ipfv+Sbjv
- -ir_verbs examples:*
- choisir: choisir+V+Inf
- choisis: choisir+V+1s+Pres+Ind
- choisis: choisir+V+2s+Pres+Ind
- choisis: choisir+V+2s+Perf+Ind
- choisirez: choisir+V+2p+Fut+Ind
- choisireiz: choisir+V+2p+Fut+Ind
- choisiroiz: choisir+V+2p+Fut+Ind
- choisireie: choisir+V+1s+Pres+Cond
- choisiroie: choisir+V+1s+Pres+Cond
- choisist: choisir+V+3s+Ipfv+Sbjv
- 
    choisist: choisir+V+3s+Pres+Ind
- -re_verbs examples:*
- perdre: perdre+V+Inf
- perds: perdre+V+2s+Pres+Ind
- perdrons: perdre+V+1p+Fut+Ind
- perdreie: perdre+V+1s+Pres+Cond
- perdroie: perdre+V+1s+Pres+Cond
- perdu: perdre+V+Part+Past+Masc+Obl+Sg
- perdu: perdre+V+Part+Past+Masc+Nom+Pl
- perdust: perdre+V+3s+Ipfv+Sbjv
- perduxt: perdre+V+3s+Ipfv+Sbjv
- perdxt: perdre+V+3s+Ipfv+Sbjv
- perdu: perdre+V+3s+Perf+Ind
- 
    perdut: perdre+V+3s+Perf+Ind
- misc_verbs examples:*
- aveir: avoir+V+Inf
- avoir: avoir+V+Inf
- escrivant: escrire+V+Part+Pres+Masc+Obl+Sg
- escrivant: escrire+V+Part+Pres+Masc+Nom+Pl
- escrivant: escrire+V+Part+Pres+Fem+Nom+Sg
- escrivant: escrire+V+Part+Pres+Fem+Obl+Sg
- escrivans: escrire+V+Part+Pres+Fem+Nom+Sg
- escrivanz: escrire+V+Part+Pres+Fem+Nom+Sg
This (part of) documentation was generated from src/fst/morphology/affixes/verbs.lexc
src-fst-morphology-phonology.twolc.md
=================================== ! The Old French morphophonological/twolc rules file ! =================================== !
{T}+s and s{T}+s become z
p, f, c, b, and v disappear before inflectional -s FIXME: change to labials + c
NASAL ALTERATIONS
Silent t and d
Palatalized vowels
Ordinals
This (part of) documentation was generated from src/fst/morphology/phonology.twolc
src-fst-morphology-root.lexc.md
General features
Noun Features
Verb Features
Article Features
Pronoun Features
Adverb Features
Adjective Features
Conjunction Features
Numeral Features
Interjection Features
Archiphonemes
This (part of) documentation was generated from src/fst/morphology/root.lexc
src-fst-morphology-stems-nouns.lexc.md
Nouns
This (part of) documentation was generated from src/fst/morphology/stems/nouns.lexc
src-fst-morphology-stems-verbs.lexc.md
Verbs
Add: doloir, rire, trover, asseoir conduire, criembre, tordre
This (part of) documentation was generated from src/fst/morphology/stems/verbs.lexc
src-fst-phonetics-txt2ipa.xfscript.md
retroflex plosive, voiceless			t  ʈ	    0288, 648 ( = ASCII 096)
retroflex plosive, voiced			d	ɖ		0256, 598
labiodental nasal					F 	ɱ		0271, 625
retroflex nasal						n 	ɳ		0273, 627
palatal nasal						J 	ɲ		0272, 626
velar nasal							N 	ŋ		014B, 331
uvular nasal							N\	ɴ		0274, 628
bilabial trill						B\ 	ʙ		0299, 665
uvular trill							R\ 	ʀ		0280, 640
alveolar tap							4	ɾ		027E, 638
retroflex flap						r 	ɽ		027D, 637
bilabial fricative, voiceless		p\ 	ɸ		0278, 632
bilabial fricative, voiced			B 	β		03B2, 946
dental fricative, voiceless			T 	θ		03B8, 952
dental fricative, voiced				D 	ð		00F0, 240
postalveolar fricative, voiceless	S	ʃ		0283, 643
postalveolar fricative, voiced		Z 	ʒ		0292, 658
retroflex fricative, voiceless		s 	ʂ		0282, 642
retroflex fricative, voiced			z` 	ʐ		0290, 656
palatal fricative, voiceless			C 	ç		00E7, 231
palatal fricative, voiced			j\ 	ʝ		029D, 669
velar fricative, voiced	        	G 	ɣ		0263, 611
uvular fricative, voiceless			X	χ		03C7, 967
uvular fricative, voiced				R 	ʁ		0281, 641
pharyngeal fricative, voiceless		X\ 	ħ		0127, 295
pharyngeal fricative, voiced			?\ 	ʕ		0295, 661
glottal fricative, voiced			h\	ɦ		0266, 614
alveolar lateral fricative, vl. K alveolar lateral fricative, vd. K\
labiodental approximant P (or v) alveolar approximant r\ retroflex approximant r` velar approximant M\
retroflex lateral approximant		l` 
palatal lateral approximant			L 
velar lateral approximant			L
Clicks
bilabial								O\	(O = capital letter) 
dental								|
(post)alveolar						!\ 
palatoalveolar						=\ 
alveolar lateral						||
Ejectives, implosives
ejective > e.g. ejective p p> implosive < e.g. implosive b b< Vowels
close back unrounded M close central unrounded 1 close central rounded } lax i I lax y Y lax u U
close-mid front rounded 2 close-mid central unrounded @\ close-mid central rounded 8 close-mid back unrounded 7
schwa ə @
open-mid front unrounded E open-mid front rounded 9 open-mid central unrounded 3 open-mid central rounded 3\ open-mid back unrounded V open-mid back rounded O
ash (ae digraph) { open schwa (turned a) 6
open front rounded & open back unrounded A open back rounded Q Other symbols
voiceless labial-velar fricative W voiced labial-palatal approx. H voiceless epiglottal fricative H\ voiced epiglottal fricative <\ epiglottal plosive >\
alveolo-palatal fricative, vl. s\ alveolo-palatal fricative, voiced z\ alveolar lateral flap l\ simultaneous S and x x\ tie bar _ Suprasegmentals
primary stress						“ 
secondary stress						% 
long									: 
half-long							:\ 
extra-short							_X 
linking mark							-
Tones and word accents
level extra high _T level high _H level mid _M level low _L level extra low _B downstep ! upstep ^ (caret, circumflex)
contour, rising contour, falling _F contour, high rising _H_T contour, low rising _B_L
contour, rising-falling				_R_F 
(NB Instead of being written as diacritics with _, all prosodic 
marks can alternatively be placed in a separate tier, set off 
by < >, as recommended for the next two symbols.)
global rise						
voiceless 0 (0 = figure), e.g. n_0 voiced _v aspirated _h more rounded _O (O = letter) less rounded _c advanced _+ retracted _- centralized _” syllabic = (or _=) e.g. n= (or n=) non-syllabic _^ rhoticity `
breathy voiced _t creaky voiced _k linguolabial _N labialized _w palatalized ‘ (or _j) e.g. t’ (or t_j) velarized _G pharyngealized _?\
dental d apical _a laminal _m nasalized ~ (or _~) e.g. A~ (or A~) nasal release _n lateral release _l no audible release _}
velarized or pharyngealized _e velarized l, alternatively 5 raised _r lowered _o advanced tongue root _A retracted tongue root _q
This (part of) documentation was generated from src/fst/phonetics/txt2ipa.xfscript
src-fst-transcriptions-transcriptor-abbrevs2text.lexc.md
We describe here how abbreviations are in Old French are read out, e.g. for text-to-speech systems.
For example:
- s.:syntynyt # ;
- os.:omaa% sukua # ;
- v.:vuosi # ;
- v.:vuonna # ;
- esim.:esimerkki # ;
- esim.:esimerkiksi # ;
This (part of) documentation was generated from src/fst/transcriptions/transcriptor-abbrevs2text.lexc
src-fst-transcriptions-transcriptor-numbers-digit2text.lexc.md
% komma% :, Root ; % tjuohkkis% :%. Root ; % kolon% :%: Root ; % sárggis% :%- Root ; % násti% :%* Root ;
This (part of) documentation was generated from src/fst/transcriptions/transcriptor-numbers-digit2text.lexc
tools-tokenisers-tokeniser-disamb-gt-desc.pmscript.md
Tokeniser for fro
Usage:
$ make
$ echo "ja, ja" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
$ echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa boasttu olmmoš, man mielde lahtuid." | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
$ echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
$ echo "márffibiillagáffe" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
Pmatch documentation: https://github.com/hfst/hfst/wiki/HfstPmatch
Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words:
- Punct contains ASCII punctuation marks
- The symbol after m-dash is soft-hyphen U+00AD
- The symbol following {•} is byte-order-mark / zero-width no-break space
U+FEFF.
Whitespace contains ASCII white space and the List contains some unicode white space characters
- En Quad U+2000 to Zero-Width Joiner U+200d’
- Narrow No-Break Space U+202F
- Medium Mathematical Space U+205F
- Word joiner U+2060
Apart from what’s in our morphology, there are
- unknown word-like forms, and
- unmatched strings
We want to give 1) a match, but let 2) be treated specially by
hfst-tokenise -aUnknowns are made of:- lower-case ASCII
- upper-case ASCII
- select extended latin symbols
- fro specific characters ASCII digits
- select symbols
- Combining diacritics as individual symbols,
- various symbols from Private area (probably Microsoft), so far:
- U+F0B7 for “x in box”
 
Unknown handling
Unknowns are tagged ?? and treated specially with hfst-tokenise
hfst-tokenise –giella-cg will treat such empty analyses as unknowns, and
remove empty analyses from other readings. Empty readings are also
legal in CG, they get a default baseform equal to the wordform, but
no tag to check, so it’s safer to let hfst-tokenise handle them.
Finally we mark as a token any sequence making up a:
- known word in context
- unknown (OOV) token in context
- sequence of word and punctuation
- URL in context
This (part of) documentation was generated from tools/tokenisers/tokeniser-disamb-gt-desc.pmscript
tools-tokenisers-tokeniser-gramcheck-gt-desc.pmscript.md
Grammar checker tokenisation for fro
Requires a recent version of HFST (3.10.0 / git revision>=3aecdbc) Then just:
$ make
$ echo "ja, ja" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
More usage examples:
$ echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa boasttu olmmoš, man mielde lahtuid." | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
$ echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
$ echo "márffibiillagáffe" | hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
Pmatch documentation: https://github.com/hfst/hfst/wiki/HfstPmatch
Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words:
- Punct contains ASCII punctuation marks
- The symbol after m-dash is soft-hyphen U+00AD
- The symbol following {•} is byte-order-mark / zero-width no-break space
U+FEFF.
Whitespace contains ASCII white space and the List contains some unicode white space characters
- En Quad U+2000 to Zero-Width Joiner U+200d’
- Narrow No-Break Space U+202F
- Medium Mathematical Space U+205F
- Word joiner U+2060
Apart from what’s in our morphology, there are 1) unknown word-like forms, and 2) unmatched strings We want to give 1) a match, but let 2) be treated specially by hfst-tokenise -a
- select extended latin symbols
- select symbols
- various symbols from Private area (probably Microsoft), so far:
- U+F0B7 for “x in box”
TODO: Could use something like this, but built-in’s don’t include šžđčŋ:
Simply give an empty reading when something is unknown: hfst-tokenise –giella-cg will treat such empty analyses as unknowns, and remove empty analyses from other readings. Empty readings are also legal in CG, they get a default baseform equal to the wordform, but no tag to check, so it’s safer to let hfst-tokenise handle them.
Finally we mark as a token any sequence making up a:
- known word in context
- unknown (OOV) token in context
- sequence of word and punctuation
- URL in context
This (part of) documentation was generated from tools/tokenisers/tokeniser-gramcheck-gt-desc.pmscript
tools-tokenisers-tokeniser-tts-cggt-desc.pmscript.md
TTS tokenisation for smj
Requires a recent version of HFST (3.10.0 / git revision>=3aecdbc) Then just:
make
echo "ja, ja" \
| hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
More usage examples:
echo "Juos gorreválggain lea (dárbbašlaš) deavdit gáibádusa \
boasttu olmmoš, man mielde lahtuid." \
| hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
echo "(gáfe) 'ja' ja 3. ja? ц jaja ukjend \"ukjend\"" \
| hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
echo "márffibiillagáffe" \
| hfst-tokenise --giella-cg tokeniser-disamb-gt-desc.pmhfst
Pmatch documentation: https://kitwiki.csc.fi/twiki/bin/view/KitWiki/HfstPmatch
Characters which have analyses in the lexicon, but can appear without spaces before/after, that is, with no context conditions, and adjacent to words:
- Punct contains ASCII punctuation marks
- The symbol after m-dash is soft-hyphen U+00AD
- The symbol following {•} is byte-order-mark / zero-width no-break space
U+FEFF.
Whitespace contains ASCII white space and the List contains some unicode white space characters
- En Quad U+2000 to Zero-Width Joiner U+200d’
- Narrow No-Break Space U+202F
- Medium Mathematical Space U+205F
- Word joiner U+2060
Apart from what’s in our morphology, there are 1) unknown word-like forms, and 2) unmatched strings We want to give 1) a match, but let 2) be treated specially by hfst-tokenise -a
- select extended latin symbols
- select symbols
- various symbols from Private area (probably Microsoft), so far:
- U+F0B7 for “x in box”
TODO: Could use something like this, but built-in’s don’t include šžđčŋ:
Simply give an empty reading when something is unknown: hfst-tokenise –giella-cg will treat such empty analyses as unknowns, and remove empty analyses from other readings. Empty readings are also legal in CG, they get a default baseform equal to the wordform, but no tag to check, so it’s safer to let hfst-tokenise handle them.
Needs hfst-tokenise to output things differently depending on the tag they get
This (part of) documentation was generated from tools/tokenisers/tokeniser-tts-cggt-desc.pmscript