# GRAMDIVVUN-møte 19.05.2021 10:30-12:00 Til stades: Linda, Sjur, Duommá ## Regeltyper real-adnui-atnui.yaml real-johttui-johtui.yaml real-Derh-Inf.yaml real-DerNomActSgGen-PrfPrc.yaml real-DerNomActSgGen-PrsSg1.yaml real-DerNomActSgGen-PrtSg1.yaml real-Ess-PrfPrc.yaml real-ImprtDu1-DerPassPrsSg3.yaml real-ImprtDu1-NSgNom.yaml real-ImprtPl2-Inf.yaml real-ImprtPl2-PrsPl3.yaml ---- Precision 80,6 % real-NomAgIll-PrtSg3.yaml real-PrtPl3-PrsSg3.yaml msyn-congruence_subj-verb.yaml syn-compound.yaml ## evaluering av regeltyper - kor mange eksempler + moteksempel (50-4300) - 50-200 er litt lite - kor bra er presisjon (her ønsker vi oss mer enn 80, over 90 er veldig bra) - kor bra gjør vi det generelt (være minst på 70, helst 80) ### real-adnui-atnui.yaml Total passes: 807, Total fails: 140, Total: 947 True positive: 749 True negative: 58 False positive 1: 2 False positive 2: 31 False negative 1: 7 False negative 2: 100 Precision: 95.8% Recall: 87.5% F₁ score: 91.5% ### real-johttui-johtui.yaml Total passes: 355, Total fails: 39, Total: 394 True positive: 353 True negative: 2 False positive 1: 2 False positive 2: 3 False negative 1: 5 False negative 2: 29 Precision: 98.6% Recall: 91.2% F₁ score: 94.8% ### real-Derh-Inf.yaml Total passes: 873, Total fails: 72, Total: 945 True positive: 423 True negative: 450 False positive 1: 5 False positive 2: 41 False negative 1: 5 False negative 2: 21 Precision: 90.2% Recall: 94.2% F₁ score: 92.2% ### real-DerNomActSgGen-PrfPrc.yaml Total passes: 1262, Total fails: 301, Total: 1563 True positive: 711 True negative: 551 False positive 1: 14 False positive 2: 126 False negative 1: 5 False negative 2: 156 Precision: 83.5% Recall: 81.5% F₁ score: 82.5% --- evtl. sjekke real-DerNomActSgGen-PrsSg1.yaml Total passes: 98, Total fails: 34, Total: 132 True positive: 78 True negative: 20 False positive 1: 0 False positive 2: 22 False negative 1: 0 False negative 2: 12 Precision: 78.0% Recall: 86.7% F₁ score: 82.1% --- sjekke real-DerNomActSgGen-PrtSg1.yaml Total passes: 1217, Total fails: 153, Total: 1370 True positive: 437 True negative: 780 False positive 1: 8 False positive 2: 115 False negative 1: 6 False negative 2: 24 Precision: 78.0% Recall: 93.6% F₁ score: 85.1% ---- sjekke real-Ess-PrfPrc.yaml Total passes: 1217, Total fails: 153, Total: 1370 True positive: 437 True negative: 780 False positive 1: 8 False positive 2: 115 False negative 1: 6 False negative 2: 24 Precision: 78.0% Recall: 93.6% F₁ score: 85.1% ---- sjekke --- ser bra ut real-ImprtDu1-DerPassPrsSg3.yaml Total passes: 273, Total fails: 77, Total: 350 True positive: 266 True negative: 7 False positive 1: 3 False positive 2: 47 False negative 1: 0 False negative 2: 27 Precision: 84.2% Recall: 90.8% F₁ score: 87.4% real-ImprtDu1-NSgNom.yaml Total passes: 526, Total fails: 111, Total: 637 True positive: 370 True negative: 156 False positive 1: 9 False positive 2: 70 False negative 1: 1 False negative 2: 31 Precision: 82.4% Recall: 92.0% F₁ score: 87.0% msyn-number_congruence-subj-verb) Test 632 - Passes: 1, Fails: 1, Total: 2 Total passes: 675, Total fails: 149, Total: 824 True positive: 448 True negative: 227 False positive 1: 11 False positive 2: 70 False negative 1: 7 False negative 2: 61 Precision: 84.7% Recall: 86.8% F₁ score: 85.7% ---- sjekke real-ImprtPl2-PrsPl3.yaml Total passes: 83, Total fails: 57, Total: 140 True positive: 76 True negative: 7 False positive 1: 13 False positive 2: 17 False negative 1: 1 False negative 2: 26 Precision: 71.7% Recall: 73.8% F₁ score: 72.7% --- sjekke -- ser greit nok ut tools/grammarcheckers/tests/real-NomAgIll-PrtSg3.yaml Total passes: 323, Total fails: 28, Total: 351 True positive: 147 True negative: 176 False positive 1: 3 False positive 2: 14 False negative 1: 1 False negative 2: 10 Precision: 89.6% Recall: 93.0% F₁ score: 91.3% real-ImprtPl2-Inf.yaml Total passes: 1147, Total fails: 285, Total: 1432 True positive: 954 True negative: 193 False positive 1: 19 False positive 2: 68 False negative 1: 3 False negative 2: 195 Precision: 91.6% Recall: 82.8% F₁ score: 87.0% tools/grammarcheckers/tests/real-PrtPl3-PrsSg3.yaml Total passes: 2111, Total fails: 300, Total: 2411 True positive: 799 True negative: 1312 False positive 1: 5 False positive 2: 269 False negative 1: 7 False negative 2: 19 Precision: 74.5% Recall: 96.8% F₁ score: 84.2% syn-compound.yaml -c Total passes: 4885, Total fails: 543, Total: 5428 True positive: 3432 True negative: 1453 False positive 1: 20 False positive 2: 211 False negative 1: 9 False negative 2: 303 Precision: 93.7% Recall: 91.7% F₁ score: 92.7% Linda Wiechetek: cat real-Ess-PrfPrc-eval-2021-05-18.txt | grep 'errorortreal' | grep 'tp' | wc -l 473 cat real-Ess-PrfPrc-eval-2021-05-18.txt | grep 'errorortreal' | grep 'fn' | wc -l 84 cat real-Ess-PrfPrc-eval-2021-05-18.txt | grep 'real-Ess-PrfPrc' | grep 'fp' | wc -l 5 Precision: TP/TP+FP = 473/(473+5)=0,9895 Recall: TP/TP+FN=473/(473+84)=0,85 cat real-ImprtPl2-Inf-eval-2021-05-18.txt | grep 'errorortreal' | grep 'fn' | wc -l 188 cat real-ImprtPl2-Inf-eval-2021-05-18.txt | grep 'errorortreal' | grep 'tp' | wc -l 467 cat real-ImprtPl2-Inf-eval-2021-05-18.txt | grep 'real-ImprtPl2-Inf' | grep 'fp' | wc -l 39 Precision: TP/TP+FP = 467/(467+39)=467/506=0.92 Recall: TP/TP+FN=467/(467+188)=467/655=0.71 cat real-PrtPl3-PrsSg3-eval-2021-05-18.txt | grep 'errorortreal' | grep 'fn' | wc -l 4 cat real-PrtPl3-PrsSg3-eval-2021-05-18.txt | grep 'errorortreal' | grep 'tp' | wc -l 32 cat real-PrtPl3-PrsSg3-eval-2021-05-18.txt | grep 'real-PrtPl3-PrsSg3' | grep 'fp' | wc -l 190 Precision: TP/TP+FP = 32/(32+190)=32/222=0.14 Recall: TP/TP+FN=32/(32+4)=32/36=0.88 cat real-NomAgIll-PrtSg3-eval-2021-05-18.txt | grep 'real-NomAgIll-PrtSg3' | grep 'fp' | wc -l 5 cat real-NomAgIll-PrtSg3-eval-2021-05-18.txt | grep 'errorortreal' | grep 'tp' | wc -l 21 cat real-NomAgIll-PrtSg3-eval-2021-05-18.txt | grep 'errorortreal' | grep 'fn' | wc -l 7 Precision: TP/TP+FP = 21/(21+5)=21/26=0.807 Recall: TP/TP+FN=21/(21+4)=21/25=0.84 # Slepp-prosedyre: 1. grammatikkontrollen/cg-filene er klar(e) - alt utkommentert som ikkje skal vera med 1. sjekk ein siste gong at alt fungerer: `make + make check` (Duommá og Linda) 1. etter det gi beskjed til Sjur og ikkje rør filan til han har kompilert og kjørt make check 1. tagg gramcheck-versjon: GramDivvun-v.1.1.0, inkl. giella-core og giella-shared (Sjur) 1. kopier zcheck-fila til server => (Gøteborg/Sjur) 1. sjekk at ting fungerer i GDocs/MS Word (Linda, Duommá) 1. SoMe-meldingar: ta med dei viktigaste nye feiltypane (Linda, Duommá) Planlagt sleppdato: 25.5. # Feil som Linda har meldt til Sjur ## Manglande analyse ```sh echo "Smihtten, ahte leagoson dat maid váikkuhan dasa ahte mo olggobeale olbmot oidnet gávtti ja gávttegeavaheami eará láhkai go mii sámit. " \ | tools/grammarcheckers/modes/trace-smegramrelese-dev.mode | less ``` `leagoson` (auxiliar + partikler) får ingen analyse: ``` "" "leagoson" ? &typo #4->4 ADD:9530:uncorrected-typos ``` icke enklitiske partiklar uppför sej enklitisk. Sjå i Bugzilla eller https://github.com/giellalt/lang-sme/issues ## Manglande retteforslag > Ádelhearrá healkkehii nu ahte goasii nisttihii luhkkárgirjii láhttái, muhto son oaččut fas árjjaid go fas jurdilii ledjona mearkka birra. også luhkkárgirjii:luhkkárgirjji blir ikkje generert sjøl om det et får en feil ``` "" "girje" N Sem/Ani Sg Ill ADD:2210 ADD:2210 @6 ADD:8293:real-girjji "luhkkár" N Sem/Hum Cmp/SgGen Cmp #6->6 real-girjji "girji" Sem/Ani ADD:2210 ADD:2210 @6 ADD:8293:real-girjji COPY:8301:real-girjjii "luhkkár" Sem/Hum Cmp/SgGen Cmp #6->6 luhkkár+Cmp/SgGen+Cmp#girji+N+Sg+Gen ? "girje" N Sem/Ani Sg Ill ADD:2210 ADD:2210 @6 ADD:8293:real-girjji "luhkkár" N Sem/Hum Cmp/SgNom Cmp #6->6 real-girjji "girji" Sem/Ani ADD:2210 ADD:2210 @6 ADD:8293:real-girjji COPY:8301:real-girjjii "luhkkár" Sem/Hum Cmp/SgNom Cmp #6->6 luhkkár+Cmp/SgNom+Cmp#girji+N+Sg+Gen ? ``` har det noe med retting av underlesinger å gjøre? Problem: retteforslag blir ikkje generert, jf inputdata - ordklassetag manglar. Kvifor? ## Ikkje stor bokstav i forslag problem - ikkje stor bokstav ved forslag, sjølv om feilen har stor bokstav (Test 376/616): ``` Finnmárkku dearvvašvuohta galgá geahččat mot sii doaibmet iešguđege suorggis ja leat bidjan johtui prošeavtta "Psykalaš dearvvašvuođasuddjema ja fágaidrasttideaddji spesialiserejuvvon gárrenmirkodivššodeami ovdánahttin ja ođđasit organiseren". ``` Psykalaš:Psyhkalaš er feilen. Den blir retta, men vi får ikkje store bokstaver: ``` "<">" """ PUNCT &punct-aistton-left #15->15 ID:15 R:RIGHT :16 ADD:9170:punct-aistton-left ADDRELATION(RIGHT):9183:punct-aistt on-left-rel ADD:9170:punct-aistton-left ADD:9170:punct-aistton-left punct-aistton-left """ PUNCT "”Psykalaš"S &punct-aistton-left &SUGGESTWF #15->15 ID:15 R:RIGHT:16 ADD:9170:punct-aistton-left ADDRELATION(RIGHT):9183:punct-aistton-left-rel COPY:9197:punct-aistton-left-sugg punct-aistton-left "”" PUNCT RIGHT Err/Orth #15->15 ID:15 R:RIGHT:16 "" "psyhkalaš" A Err/Orth Sem/Dummytag Attr @>N MAP:22170:r86 &LINK &punct-aistton-left &typo #16->16 ID:16 ADD:9178:punct-aistton-left-link ADD:9178:punct-aistton-left-link ADD:9427:Err/Orth-any ADD:9178:punct-aistton-left-link punct-aistton-left typo "psyhkalaš" A Sem/Dummytag Attr @>N MAP:22170:r86 &typo &SUGGEST #16->16 ID:16 ADD:9178:punct-aistton-left-link ADD:9178:punct-aistton-left-link ADD:9427:Err/Orth-any COPY:9436:Err/Orth-any psyhkalaš+A+Attr psyhkalaš "psyhkalaš" A Err/Orth Sem/Dummytag Sg Nom @16 ID:16 ADD:9178:punct-aistton-left-link ADD:9178:punct-aistton-left-link ADD:9427:Err/Orth-any ADD:9178:punct-aistton-left-link punct-aistton-left typo "psyhkalaš" A Sem/Dummytag Sg Nom @16 ID:16 ADD:9178:punct-aistton-left-link ADD:9178:punct-aistton-left-link ADD:9427:Err/Orth-any COPY:9436:Err/Orth-any psyhkalaš+A+Sg+Nom psyhkalaš ```