====== Korpus SYNv10 ====== Většina práce se dělá na ''lovelace.korpus.cz''. Zdrojové texty a výsledky jsou uchovány na grimmovi. Korpus sestává z těchto dílčích korpusů: * ''grimm'': * ''/store/corp/SYNv10/NEWTON2019'': * ''vert-rules0-frazrl-rules-mdita-kolok'' zkontrolováno * ''vert-rules0-frazrl-rules-mdita-kolok-segm-sublm-agr'': 14.1.2022 --- zkontrolováno * **''vert-corr''**: 17.1.2022 * ''/store/corp/SYNv10/NEWTON2020'': * ''vert-rules0-frazrl-rules-mdita-kolok'' zkontrolováno * ''vert-rules0-frazrl-rules-mdita-kolok-segm-sublm-agr'': 14.1.2022 --- zkontrolováno * **''vert-corr''**: 17.1.2022 * ''/store/corp/SYNv10/SYN2020'': * ''vert-rules0-frazrl-rules-mdita-kolok'' zkontrolováno * ''vert-rules0-frazrl-rules-mdita-kolok-segm-sublm-agr'': 14.1.2022 --- zkontrolováno * **''vert-corr''**: 17.1.2022 * ''/store/corp/SYNv10/SYNv8_new'': * ''SYNv8_a-i_'' * ''vert-rules0-frazrl-rules-mdita-kolok'' zkontrolováno * ''vert-rules0-frazrl-rules-mdita-kolok-segm-sublm-agr'': 14.1.2022 --- zkontrolováno * **''vert-corr''**: 17.1.2022 * ''SYNv8_j-ln-z'' * ''vert-rules0-frazrl-rules-mdita-kolok-segm-sublm-agr'': 17.1.2022 --- zkontrolováno * **''vert-corr''**: 17.1.2022 * ''SYNv8_m'' * ''vert-rules0-frazrl-rules-mdita-kolok'' zkontrolováno * ''vert-rules0-frazrl-rules-mdita-kolok-segm-sublm-agr'': 14.1.2022 --- zkontrolováno * **''vert-corr''**: 17.1.2022 ===== Opravy vadných XML souborů ===== Původní XML soubory je třeba opravit (odstranit kontrolní znaky a ligatury). K tomu slouží skripty v adresáři ''/usr/local/syn2020/repair_xml'': * ''repair_control_chars.pl'' * ''repair_xml.pl'' Oprava se provádí takto:cd /store/corp/SYNv9/SYNv8_new/SYNv8_a-i_ parallel-filtr.sh -C "repair_control_chars.pl | repair_xml.pl" -s orig -t ../../../SYNv10/SYNv8_new/SYNv8_a-i_/orig -v -p45V pracovním adresáři vznikne soubor ''stat_repair.txt'', ve kterém jsou zapsány všechny opravy. Pro porovnání výsledku dvou oprav nejdříve vyrobíme seznam odlišných souborů a potom vygrepneme z diffu jednotlivá slovíčka:cd .../SYNv10/SYNv8_new/SYNv8_a-i_/orig diff -q . ../orig-11 | cut -f2 -d' ' | cut -f2 -d'/' > ../diff-orig-orig-11.lst for ff in $(cat ../diff-orig-orig-11.lst); do diff -u $ff ../orig-11/$ff | diff-highlight; done \ | grep -- "^-

" | grep -oP '[^\s]*\e\[7m \e\[27m[^\s]+' | perl -pe 's/^

//' | perl -pe 's:

$::' \ | perl -pe 's/(\x9B|\x1B\[)[0-?]*[ -\/]*[@-~]//g' | sort -u > ../repair-missing.txt ===== Opravy na části souborů ===== * Změněné soubory s ligaturami * Chybějící tagy * Soubory, které neprošly programem ''no_space_...'' ===== Postup zpracování ===== Zpracovává se skripty pro ''SYN2020'': - ''process_text.sh'' provede lemmatizaci a tagování, pouští se s parametrem ''-t in'' - ''process_kolok.sh'' přidá kolokace Datum poslední úpravy: * 2021, Dec Hotové korpusy jsou v adresáři ''vert-'': * ''SYN2020'': 2021, Dec ===== Průběžné opravy ===== Některé soubory měly vadné znaky, některé měly chybějící nabídku tagů a některé měly neúplnou nabídku tagů. Tyto soubory se musí opravit s těmito kroky: * projet znovu skripty ''repair_control_chars.pl'' a ''repair_xml.pl'' * doplnit tagy do seznamu a zkompilovat znovu ''LEXy'' Seznam souborů, ''/home/skoumal/PROJEKTY/INFRASTRUKTURA/SYNv10/doc/znovu-all.lst'' (711 položek): 3dil2.txt 4dil1.txt aa060925.txt aa061204.txt aa061218.txt aa061227.txt aa070602.txt aa070623.txt aa070728.txt aa080126.txt aa081021.txt aa081122.txt aa081223.txt aa090113.txt aa090418.txt aa090514.txt aa090831.txt aa090911.txt aa100305.txt aa100623.txt aa101118.txt aa110616.txt aa120503.txt aa130815.txt aa140117.txt aa140312.txt aa161003.txt aa170307.txt aa180813.txt abneumsm.txt ad0601.txt ad0704.txt ad0803.txt adva1707.txt adva1806.txt ahne0619.txt ahne0642.txt ahne0916.txt ahne0919.txt ahne0920.txt ahne0926.txt ahne0943.txt ahne0950.txt ahne1004.txt ahne1048.txt ahne1049.txt ahne1116.txt ahne1206.txt ahne1523.txt ahne1627.txt ahne1652.txt ahne1710.txt ahne1806.txt ahne1831.txt ajla01.txt ajvaz.txt ajvaz_vrazdavhot.txt aleje.txt alvar_35mrtvych.txt atip1411.txt atwood.txt auto1509.txt backm_muzjmenemo.txt bank0201.txt bank0406.txt bank0604.txt bank1001.txt bank1002.txt bank1104.txt bank1110.txt bank1406.txt bank1704.txt bayar_cernavez.txt bellona.txt berkova.txt bezestop.txt bezpros.txt bl010205.txt bl030212.txt bl050212.txt bl050226.txt bl050416.txt bl060114.txt bl060428.txt bl060602.txt bl060610.txt bl060902.txt bl060909.txt bl090103.txt bl090321.txt bl090404.txt bl090406.txt bl090502.txt bl090530.txt bl090926.txt bl091029.txt bl110125.txt bl120504.txt bl120608.txt bl120623.txt bl121229.txt bl140110.txt bl140214.txt bl151007.txt bl151014.txt bl151105.txt bl161110.txt bl161223.txt bl170303.txt bl180917.txt bl181214.txt bl971217.txt bladrun2.txt blmg0412.txt blmg1020.txt blmg1328.txt blmg1406.txt blmg1527.txt blmg1536.txt blmg1747.txt blmg9609.txt blmg9714.txt blmg9825.txt blne0237.txt blne0516.txt blne0551.txt blne1146.txt blne1409.txt blne1432.txt blne1608.txt blne1644.txt blne1647.txt blne1717.txt blne1730.txt blzd1106.txt blzd1303.txt blzd1309.txt blzd1502.txt blzd1509.txt blzd1611.txt blzd1708.txt blzd1709.txt blzd1807.txt blze0509.txt blze0510.txt blze0512.txt blze0530.txt blze0536.txt blze0540.txt blze0618.txt blze0622.txt blze0623.txt blze0634.txt blze0639.txt blze0643.txt blze1016.txt blze1050.txt blze1139.txt blze1234.txt blze1304.txt blze1329.txt blze1641.txt blze1725.txt blze1812.txt blze1838.txt bosk0444.txt bouka_interakcea.txt bromf_stoprozmaz.txt buhsro.txt burdo_poselstvij.txt camus_cizinec.txt carof_nevedomysv.txt carsvet.txt carus_magickyctv.txt cechov.txt cechu_drsherlock.txt celon_chybejicid.txt chatin.txt child.txt chip1812.txt choch_jakovranan.txt chovani.txt chrl0510.txt chrz0103.txt chvaly.txt chytme.txt cinm0400.txt cinm1200.txt cizpri.txt comam.txt ctyricet.txt cunyl_venecky.txt cvvi0803.txt db040116.txt db040209.txt db040211.txt db040217.txt db040517.txt db040601.txt db040719.txt db040818.txt db040828.txt db050507.txt db050712.txt db050804.txt db050820.txt db051001.txt db051229.txt db060109.txt db060124.txt db060216.txt db060816.txt db060819.txt db060826.txt db061004.txt db061013.txt db061014.txt db061016.txt db061017.txt db061018.txt db061019.txt db061027.txt db061031.txt db061103.txt db061104.txt db061120.txt db061222.txt db070103.txt db070109.txt db070123.txt db070205.txt db070420.txt db070516.txt db070626.txt db070723.txt db070728.txt db070814.txt db070829.txt db070831.txt db070903.txt db070905.txt db070906.txt db070907.txt db070908.txt db070910.txt db070911.txt db070914.txt db070924.txt db071017.txt db071109.txt db071113.txt db071116.txt db071119.txt db071123.txt db071211.txt db071228.txt db071229.txt db080102.txt db080104.txt db080307.txt db080314.txt db080321.txt db080331.txt db080401.txt db080404.txt db080411.txt db080418.txt db080425.txt db080502.txt db080509.txt db080516.txt db080523.txt db080528.txt db080530.txt db080603.txt db080606.txt db080609.txt db080613.txt db080730.txt db080823.txt db080901.txt db080903.txt db080905.txt db080910.txt db080920.txt db081018.txt db081121.txt db081122.txt db081204.txt db081218.txt db090129.txt db090220.txt db090228.txt db090310.txt db090311.txt db090324.txt db090408.txt db090420.txt db090505.txt db090509.txt db090526.txt db090608.txt db090620.txt db090622.txt db090625.txt db090626.txt db090729.txt db090815.txt db090926.txt db091031.txt db091106.txt db100128.txt db100129.txt db100312.txt db100413.txt db100518.txt db100602.txt db100628.txt db100629.txt db101001.txt db101005.txt db101021.txt db101102.txt db101103.txt db101218.txt db110107.txt db110110.txt db110121.txt db110405.txt db110418.txt db110429.txt db110623.txt db110701.txt db110716.txt db110722.txt db110803.txt db110809.txt db110810.txt db110817.txt db110824.txt db110831.txt db110907.txt db110929.txt db111022.txt db111104.txt db111105.txt db111112.txt db111119.txt db111126.txt db111203.txt db111210.txt db111217.txt db111231.txt db120105.txt db120106.txt db120107.txt db120114.txt db120121.txt db120128.txt db120204.txt db120207.txt db120211.txt db120216.txt db120224.txt db120225.txt db120303.txt db120306.txt db120310.txt db120317.txt db120324.txt db120403.txt db120407.txt db120412.txt db120414.txt db120421.txt db120502.txt db120630.txt db120714.txt db120719.txt db120901.txt db120906.txt db121010.txt db121122.txt db121213.txt db121217.txt db121221.txt db130110.txt db130111.txt db130211.txt db130221.txt db130315.txt db130319.txt db130320.txt db130409.txt db130412.txt db130424.txt db130617.txt db130619.txt db130621.txt db130622.txt db130710.txt db130713.txt db130807.txt db130913.txt db131111.txt db131114.txt db131125.txt db131129.txt db131212.txt db131219.txt db140122.txt db140127.txt db140207.txt db140212.txt db140226.txt db140308.txt db140415.txt db140603.txt db140723.txt db140819.txt db141007.txt db141031.txt db141104.txt db141129.txt db141231.txt db150112.txt db150226.txt db150722.txt db151003.txt db151009.txt db151103.txt db151119.txt db151201.txt db151222.txt db151231.txt db160109.txt db160118.txt db160820.txt db160913.txt db161004.txt db161014.txt db161223.txt db170206.txt db170424.txt db170809.txt db170830.txt db170930.txt db180222.txt db181005.txt db181011.txt dc070108.txt dc080304.txt dc080804.txt dejdivad.txt delena.txt dempolmk.txt denikotc.txt deti1005.txt deti1008.txt deti1107.txt deti1205.txt deti1207.txt deti1210.txt df96.txt diet0910.txt diet1002.txt diet1702.txt diet1814.txt divis1.txt dode0538.txt dode1336.txt dode1606.txt dode1726.txt doktglas.txt drboh_migraceaim.txt drews_otazkyaodp.txt droga.txt duma1209.txt durrell.txt dvno0519.txt echt1806.txt egoo1609.txt ek0537.txt ek1026.txt ek1108.txt ek1143.txt ek1202.txt ek1207.txt ek1220.txt ek1232.txt ek1240.txt ek1407.txt ek1444.txt ek1507.txt ek1604.txt ek1607.txt ek1707.txt ek1807.txt emoke.txt ep100513.txt ep101029.txt ep101124.txt ep101125.txt ep101206.txt ep101223.txt ep101229.txt ep101230.txt ep110303.txt ep110318.txt ep110826.txt ep130604.txt ep140226.txt ep140902.txt ep150716.txt eura1326.txt extr0806.txt extr1002.txt extr1404.txt extr1408.txt fazi1001.txt fazi1003.txt fc93.txt fc94.txt fc95.txt fipo1006.txt fipo1208.txt fnkohout.txt food1502.txt fotr.txt frank_mujotec.txt frazi_bilynaceln.txt geniuskn.txt grass_pribehyzte.txt hajeni.txt hajic_vzpominkyn.txt halas_snovaexist.txt hampl_nabozenstv.txt hano150102.txt hano150123.txt hano150205.txt hano150220.txt hano150319.txt hano150516.txt hano150718.txt hano150725.txt hano151007.txt hano151104.txt hano151205.txt hano160224.txt hano160330.txt hano160406.txt hano160420.txt hano160427.txt hano160503.txt hano161222.txt hano170225.txt hano170802.txt hano170822.txt hano170829.txt hano170901.txt hano170905.txt hano170908.txt hano170915.txt hano170916.txt hano170919.txt hano170922.txt hano170926.txt hano171003.txt hano171006.txt hano171010.txt hano171013.txt hano171017.txt hano171216.txt hano171230.txt hano180814.txt hano180821.txt hano180824.txt hano180829.txt hano180831.txt hano180904.txt hano180911.txt hano180918.txt hano180919.txt hano180920.txt hano180921.txt hano180925.txt hano181002.txt hano181003.txt hano181108.txt harri_krysarovys.txt hasko_osetrovate.txt hasse_gestapo.txt hawki_jirkuvtajn.txt hearts6.txt heller.txt hemingw3.txt hepac.txt herri_zverolekas.txt hn001020.txt hn010124.txt hn020104.txt hn020412.txt hn020425.txt hn030306.txt hn040304.txt hn040618.txt hn050120.txt hn051013.txt hn060227.txt hn060413.txt hn060724.txt hn061003.txt hn070316.txt hn070517.txt hn071101.txt hn080107.txt hn080115.txt hn080429.txt hn080613.txt hn080721.txt hn080901.txt hn080902.txt hn080904.txt hn080926.txt hn081024.txt hn081223.txt hn090105.txt hn090106.txt hn090406.txt hn091002.txt hn100319.txt hn100527.txt hn101201.txt hn101220.txt hn110329.txt hn111021.txt hn111128.txt hn121220.txt hn130104.txt hn140311.txt hn140618.txt hn150320.txt hn150713.txt hn150825.txt hn151002.txt hn160218.txt hn161116.txt hn170911.txt hn171031.txt hn181119.txt hn181129.txt hn960207.txt hn961113.txt hn970127.txt hn970627.txt hn980320.txt hn980918.txt hn990106.txt hn990219.txt hn990628.txt hn990827.txt hnh6.txt hnvi0201.txt hnvi0214.txt hnvi1349.txt hodekant.txt hodiny.txt holand.txt homostup.txt honzak.txt hordubal.txt hosp1009.txt hrdli_bohemistic.txt hrozny.txt ikar0808.txt ikar1008.txt ikariec.txt ikariep.txt inf1207.txt inf1808.txt inma1145.txt insi1048.txt insi1106.txt insi1119.txt insi1202.txt insi1226.txt insi1309.txt insi1322.txt insi1335.txt insi1415.txt insi1438.txt insi1440.txt insi1528.txt insi1533.txt insi1614.txt insi1624.txt insi1643.txt insi1647.txt insi1704.txt insi1810.txt irving.txt islavika.txt atlashud.txt bank0201.txt bront_navetrnehu.txt cocte_sbohemvyhr.txt db071204.txt db080516.txt db080613.txt db090310.txt db090311.txt db110405.txt db130110.txt db150216.txt db180222.txt df96.txt dzibr_jezissyncl.txt ep101223.txt hn060413.txt hn080721.txt hn140311.txt holub_aktualnipr.txt hyeny.txt irmis_stresmezil.txt