-
Notifications
You must be signed in to change notification settings - Fork 4
LapDevelopment_Giellatekno
In wrapping the UiT Giellatekno (GT) pipeline for Sami segmentation, tokenization, morphological analysis, and dependency parsing, we need to (a) decide how many individual steps of processing to distinguish (from the LAP point of view) and (b) how to map the input and output data from the GT pipeline to LAF-compatible annotations.
Following is a sample invocation (against the LAP Tree), which was originally suggested by Trond Trosterud of UiT:
cat $LAPTREE/gt/etc/test.txt \
| $LAPTREE/perl/lap/perl $LAPTREE/gt/script/preprocess --abbr=$LAPTREE/gt/sme/abbr.txt \
| $LAPTREE/hfst/lap/hfst-lookup $LAPTREE/gt/sme/analyser-gt-norm.hfstol \
| $LAPTREE/perl/lap/perl $LAPTREE/gt/script/lookup2cg | sed 's/0.000000//g' \
| $LAPTREE/vislcg3/lap/vislcg3 -g $LAPTREE/gt/sme/disambiguation.cg3 \
| $LAPTREE/vislcg3/lap/vislcg3 -g $LAPTREE/gt/sme/functions.cg3 \
| $LAPTREE/vislcg3/lap/vislcg3 -g $LAPTREE/gt/sme/dependency.cg3
It probably makese sense to conceptually distinguish the following layers:
-
(a) (maybe normalization and) tokenization (preprocess)
(b) morphological analysis (hfst-lookup)
(c) segmentation and morpho-syntactic tagging (disambiguation.cg3)
(d) grammatical function analysis (functions.cg3)
(e) dependency parsing (dependency.cg3)
Dán skábma čuojaha Mari Boine ovttas Kai Sombyn ja Ája joavkkuin. Go beakkán artistta guovttos čuojaheaba oktasaš konsearttaid vuosttaš gearddi, de šaddaba garvit sámi báikkiid gos girkuin lea juoigangielddus. Dattetge illudeaba sakka ovttasbargui.
"<Dán>"
"dát" Pron Dem Sg Loc Attr
"dát" Pron Dem Sg Ill Attr
"dát" Pron Dem Sg Acc
"dát" Pron Dem Sg Gen
"<skábma>"
"skábma" N Sg Nom
"<čuojaha>"
"čuodjat" V* IV* Der/h V TV Ind Prs Sg3
"čuojahit" V TV Ind Prs Sg3
"<Mari>"
"Mari" N Prop Sem/Fem Sg Acc
"Mari" N Prop Sem/Fem Sg Nom
"Mari" N Prop Sem/Fem Attr
"Mari" N Prop Sem/Fem Sg Gen
"<Boine>"
"Boine" N Prop Sem/Sur Sg Gen
"Boine" N Prop Sem/Sur Sg Nom
"Boine" N Prop Sem/Sur Sg Acc
"<ovttas>"
"ovttas" N Coll Sg Nom
"ovttastit" V TV Imprt ConNeg
"ovttastit" V TV Ind Prs ConNeg
"ovttastit" V TV Imprt Sg2
"ovttas" Adv
"okta" Num Sg Loc
"ovttastit" V TV VGen
"<Kai>"
"Kai" N Prop Sem/Mal Sg Nom
"Kai" N Prop Sem/Mal Sg Gen
"Kai" N Prop Sem/Mal Attr
"Kai" N Prop Sem/Mal Sg Acc
"<Sombyn>"
"Somby" N Prop Sem/Sur Ess
"<ja>"
"ja" CC
"<Ája>"
"Ája" N Prop Sem/Org Sg Gen
"ája" N Sg Nom
"Ája" N Prop Sem/Org Sg Nom
"Ája" N Prop Sem/Org Sg Acc
"<joavkkuin>"
"joavku" N Pl Loc
"joavku" N Sg Com
"<.>"
"." CLB
"<Go>"
"go" Pcle Qst
"go" CS
"<beakkán>"
"beakkán" A Sg Gen
"beakkán" A Sg Nom
"beakkán" A Sg Acc
"beaggit" V IV Ind Prs Sg1
"beakkán" A Attr
"<artistta>"
"artista" N Sg Gen
"artista" N Sg Acc
"<guovttos>"
"guovttos" N Coll Sg Loc
"guovttos" N Coll Sg Gen PxSg3
"guovttos" N Coll Sg Nom
"guovttos" N Coll Sg Acc PxSg3
"<čuojaheaba>"
"čuojahit" V TV Ind Prs Du3
"čuodjat" V* IV* Der/h V TV Ind Prs Du3
"<oktasaš>"
"oktasaš" A Attr
"oktasaš" A Sg Nom
"<konsearttaid>"
"konsearta" N Pl Acc
"konsearta" N Pl Gen
"<vuosttaš>"
"vuosttaš" A Ord Sg Nom
"vuosttaš" A Ord Attr
"<gearddi>"
"gearddi" Adv
"geardi" N Sg Gen
"geardi" N Sg Acc
"<,>"
"," CLB
"<de>"
"de" Adv
"<šaddaba>"
"šaddat" V IV Ind Prs Du3
"<garvit>"
"garvit" V TV Inf
"garvit" V TV Ind Prs Pl1
"garvit" V* TV* Der/NomAg N Pl Nom
"garvit" V TV Imprt Pl2
"<sámi>"
"sápmi" N Sg Acc
"sápmi" N Sg Gen
"<báikkiid>"
"báiki" N Pl Gen
"báiki" N Pl Acc
"<gos>"
"gos" Adv
"<girkuin>"
"girku" N Sg Com
"girku" N Pl Loc
"<lea>"
"leat" V IV Ind Prs Sg3
"<juoigangielddus>"
"juoigan#gieldu" N Sg Acc PxSg3
"juoigan#gieldu" N Sg Loc
"juoigan#gieldu" N Sg Gen PxSg3
"juoigan#gielddus" N Sg Nom
"<.>"
"." CLB
"<Dattetge>"
"dattetge" Adv
"<illudeaba>"
"illudit" V IV Ind Prs Du3
"<sakka>"
"sakka" Adv
"<ovttasbargui>"
"ovttasbargu" N Sg Ill
"ovttasbargat" V* IV* Der/PassS V IV Ind Prt Sg3
"<.>"
"." CLB
"<Dán>"
"dát" Pron Dem Sg Acc @OBJ> #1->3
"<skábma>"
"skábma" N Sg Nom @SUBJ> #2->3
"<čuojaha>"
"čuojahit" V TV Ind Prs Sg3 @FMV #3->0
"<Mari>"
"Mari" N Prop Sem/Fem Attr @>N #4->5
"<Boine>"
"Boine" N Prop Sem/Sur Sg Nom @<SUBJ #5->3
"<ovttas>"
"ovttas" Adv @<ADVL #6->3
"<Kai>"
"Kai" N Prop Sem/Mal Attr @>N #7->8
"<Sombyn>"
"Somby" N Prop Sem/Sur Ess @<OPRED #8->1
"<ja>"
"ja" CC @CNP #9->8
"<Ája>"
"Ája" N Prop Sem/Org Sg Gen @>N #10->11
"<joavkkuin>"
"joavku" N Sg Com @<ADVL #11->3
"<.>"
"." CLB #12->3
"<Go>"
"go" CS @CVP #1->5
"<beakkán>"
"beakkán" A Attr @>N #2->3
"<artistta>"
"artista" N Sg Acc @OBJ> #3->5
"<guovttos>"
"guovttos" N Coll Sg Nom @SUBJ> #4->5
"<čuojaheaba>"
"čuojahit" V TV Ind Prs Du3 @FS-ADVL> #5->13
"<oktasaš>"
"oktasaš" A Attr @>N #6->7
"<konsearttaid>"
"konsearta" N Pl Gen @>N #7->9
"<vuosttaš>"
"vuosttaš" A Ord Attr @>N #8->9
"<gearddi>"
"geardi" N Sg Gen @<ADVL #9->5
"gearddi" Adv @<ADVL #9->5
"<,>"
"," CLB #10->1
"<de>"
"de" Adv @ADVL> #11->13
"<šaddaba>"
"šaddat" V IV Ind Prs Du3 @FAUX #12->0
"<garvit>"
"garvit" V TV Inf @IMV #13->12
"<sámi>"
"sápmi" N Sg Gen @>N #14->15
"<báikkiid>"
"báiki" N Pl Acc @<OBJ #15->13
"<gos>"
"gos" Adv @ADVL> #16->18
"<girkuin>"
"girku" N Pl Loc @ADVL> #17->18
"<lea>"
"leat" V IV Ind Prs Sg3 @FS-<ADVL #18->13
"<juoigangielddus>"
"juoigan#gielddus" N Sg Nom <ext> @<SUBJ #19->18
"<.>"
"." CLB #20->12
"<Dattetge>"
"dattetge" Adv @ADVL> #1->2
"<illudeaba>"
"illudit" V IV Ind Prs Du3 @FMV #2->0
"<sakka>"
"sakka" Adv @>N #3->4
"<ovttasbargui>"
"ovttasbargu" N Sg Ill @<ADVL #4->2
"<.>"
Granularity: Should steps (a) to (e) above be wrapped as a single (complex) component, or each individually. We opt for the latter, to enable users to only run parts of the pipeline and have access to the corresponding intermediate analyses layers.
Sequencing: Unlike in analysis pipelines we have constructed earlier (e.g. English dependency parsing), sentence segmentation in Giellatekno only happens as a bi-product of step (c), i.e. morpho-syntactic disambiguation. This should not cause problems in the LAP context, however; step (a) only creates token regions and corresponding nodes; steps (b) and (c) receive as their inputs the token sequences for a full document (i.e. cannot expect to iterate over sentence regions); and step (c) inserts multiple layers of annotation into LAP Store: in addition to annotation nodes for morpho-syntactic analysis, it also creates regions and corresponding nodes for sentence segments.
Ambiguity: Morphological analysis in Giellatekno (step (b), also often called multi-tagging) creates ambiguous outputs, i.e. multiple possible morpho-syntactic analyses per token. Because downstream processing may need to refer back to one specific analysis (i.e. the part of speech and morphological features used as the leaves of a parse tree), ambiguity must be represented as a set of nodes, i.e. each individual morpho-syntactic analysis establishes its own annotation node. This begs the question of how to represent the relations among alternatives, i.e. minimally an ordering relation (assuming an n-best list).
Interpretation:
Home | Forum | Discussions | Events