³ò 4ÒÇIc@s‹dZddkZddkTdd?d„ƒYZedjoKddkZddkZddkZeiƒZei ddd d d d d dƒei ddd dd dd dƒei ddd dddd gd dƒei ddd dddd gd dƒei ddd d d d!ƒei d"d#d d$d d%d d&ƒei d'd(d d)d d*d d+ƒei d,d-d d.d d/d d0ƒei d1d2d d3d dd d4ƒei d5d6d d7d e dd8d d9ƒei ƒ\Z Ze ipeiƒeƒneid:d;d<ee iƒƒgZxãee iƒD]ÒZeie iƒZed=eed>d!ƒeediƒie iƒƒZZ Z!e i"e i#jpLe$e i"ƒd=joee i"jp&e$e i#ƒd=jo*ee i#joei%ee e!fƒq4q4We i&o(eee'ee iƒe i&ƒƒZ(neee'ee iƒƒZ(e i)one'e(e i*ƒƒGHei+ƒndS(@sp Implementations of inter-annotator agreement coefficients surveyed by Artstein and Poesio (2007), Inter-Coder Agreement for Computational Linguistics. An agreement coefficient calculates the amount that annotators agreed on label assignments beyond what is expected by chance. In defining the AnnotationTask class, we use naming conventions similar to the paper's terminology. There are three types of objects in an annotation task: the coders (variables "c" and "C") the items to be annotated (variables "i" and "I") the potential categories to be assigned (variables "k" and "K") Additionally, it is often the case that we don't want to treat two different labels as complete disagreement, and so the AnnotationTask constructor can also take a distance metric as a final argument. Distance metrics are simply functions that take two arguments, and return a value between 0.0 and 1.0 indicating the distance between them. If not supplied, the default is binary comparison between the arguments. The simplest way to initialize an AnnotationTask is with a list of equal-length lists, each containing a coder's assignments for all objects in the task: task = AnnotationTask([],[],[]) Alpha (Krippendorff 1980) Kappa (Cohen 1960) S (Bennet, Albert and Goldstein 1954) Pi (Scott 1955) TODO: Describe handling of multiple coders and missing data Expected results from the Artstein and Poesio survey paper: >>> t = AnnotationTask(data=[x.split() for x in open("%sartstein_poesio_example.txt" % (__file__.replace("__init__.py", "")))]) >>> t.avg_Ao() 0.88 >>> t.pi() 0.7995322418977614 >>> t.S() 0.81999999999999984 iÿÿÿÿN(t*tAnnotationTaskcBsÈeZdZeed„Zd„Zd„Zd„Zeeed„Z d„Z d„Z d„Z d d „Z d d „Zd „Zd „Zd„Zd„Zd„Zd„Zd d„Zd„ZRS(s3Represents an annotation task, i.e. people assign labels to items. Notation tries to match notation in Artstein and Poesio (2007). In general, coders and items can be represented as any hashable object. Integers, for example, are fine, though strings are more readable. Labels must support the distance functions applied to them, so e.g. a string-edit-distance makes no sense if your labels are integers, whereas interval distance needs numeric values. A notable case of this is the MASI metric, which requires Python sets. cCsX||_tƒ|_tƒ|_tƒ|_g|_|djo|i|ƒndS(s.Initialize an empty annotation task. N(tdistancetsettItKtCtdatatNonet load_array(tselfRR((s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyt__init__Fs      cCsditd„|iƒƒS(Ns cSs2d|d|diddƒdi|dƒfS(s%s %s %stcodertitemt_s t,tlabels(treplacetjoin(tx((s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pytSs(RtmapR(R ((s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyt__str__RscCsyxr|D]j\}}}|ii|ƒ|ii|ƒ|ii|ƒ|iih|d<|d<|d<ƒqWdS(s­Load the results of annotation. The argument is a list of 3-tuples, each representing a coder's labeling of an item: (coder,item,label) R RR N(RtaddRRRtappend(R tarrayR R R((s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyR Ws cs¹t‡‡fd†|iƒd}t‡‡fd†|iƒd}dt|i|d|dƒƒ}tidˆˆˆ|ƒtiddi|dƒdi|dƒd|ƒ|S( s6Agreement between two coders on a given item cs|dˆjo|dˆjS(R R ((R(titcA(s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyRgsics|dˆjo|dˆjS(R R ((R(RtcB(s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyRhsgð?Rs.Observed agreement between %s and %s on %s: %fs"Distance between "%s" and "%s": %fR(tfilterRtfloatRtloggingtdebugR(R RRRtkAtkBtret((RRRs,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pytagrcs""$  +csˆtjo?ˆtjo2ˆtjo%tt‡fd†|iƒƒ}n¤ˆtjoBˆtjo5ˆtjo(tt‡‡fd†|iƒƒ}nUˆtjoBˆtjo5ˆtjo(tt‡‡fd†|iƒƒ}ndGHtidˆˆˆ|ƒt|ƒS(sHImplements the "n-notation" used in Artstein and Poesio (2007) csˆ|djS(R((R(tk(s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyRuscsˆ|djoˆ|djS(RR ((R(RR%(s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyRwscsˆ|djoˆ|djS(RR ((R(R%tc(s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyRyss&You must pass either i or c, not both!sCount on N[%s,%s,%s]: %d(RtlenRRRR R(R R%RR&R#((R&RR%s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pytNps'%'('(csZttt‡‡‡fd†ˆiƒƒƒttˆiƒƒ}tidˆˆ|ƒ|S(s=Observed agreement between two coders on all items. csˆiˆˆ|ƒS((R$(R(R RR(s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyRƒss(Observed agreement between %s and %s: %f(RtsumRRR'RR (R RRR#((RR Rs,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pytAos@cCs‹|iiƒ}d}d}xO|iD]D}|i|ƒx.|D]&}||i||ƒ7}|d7}q?Wq%W||}tid|ƒ|S(sAAverage observed agreement across all coders and items. ggð?sAverage observed agreement: %f(RtcopytremoveR*RR (R tstcounterttotalRRR#((s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pytavg_Ao‡s   c CsÙd}x|iD]v}xm|iD]b}xY|iD]N}|t|id|d|ƒ|id|d|ƒƒ|i||ƒ7}q0Wq WqWdtt|iƒt|iƒt|iƒdƒ|}tid|ƒ|S(s©The observed disagreement for the alpha coefficient. The alpha coefficient, unlike the other metrics, uses this rather than observed agreement. gRR%gð?isObserved disagreement: %f( RRRR(RR'RRR (R R/RtjtlR#((s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pytDo_alpha˜s   T;gð?c s¡d}xg|iD]\‰||it‡‡fd†|iƒddt‡‡fd†|iƒddƒ7}qW|t|iƒ|}tidˆˆ|ƒ|S(sGThe observed disagreement for the weighted kappa coefficient. gcs|dˆjo|dˆjS(R R ((R(RR(s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyR­siRcs|dˆjo|dˆjS(R R ((R(RR(s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyR®ss+Observed disagreement between %s and %s: %f(RRRRR'RR (R RRt max_distanceR/R#((RRRs,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pytDo_Kw_pairwise§s ,.cCs´h}x{|iD]p}xg|iD]\}t||gƒ|iƒjo7||j o)|i|||ƒ|t||gƒR@RCRERIRJ(((s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pyR9s&          t__main__s-ds --distancetdestRtdefaultRNthelpsdistance metric to uses-as --agreementt agreementRCs"agreement coefficient to calculates-es --excludetexcludetactionRs8coder names to exclude (may be specified multiple times)s-is --includetincludes.coder names to include, same format as excludes-fs--filetfilesPfile to read labelings from, each line with three columns: 'labeler item labels's-vs --verbosetverboset0s+how much debugging to print on stderr (0-4)s-cs --columnsept columnseps sIchar/string that separates the three columns in the file, defaults to tabs-ls --labelseptlabelsepRs[char/string that separates labels (if labelers can assign more than one), defaults to commas-ps --presencetpresences=convert each labeling into 1 or 0, based on presence of LABELs-Ts --thoroughtthorought store_trues6calculate agreement for every subset of the annotatorstleveli2i ii((,RMRRRRKtretoptparset OptionParsertparsert add_optionRtFalset parse_argstoptionst remainderRWt print_helptexitt basicConfigtintRXRtopenR2tsplitRZttokststrR6tstripR[R tobjectRRVRTR'RR\tgetattrttaskR]RStshutdown(((s,/p/zhu/06/nlp/nltk/nltk/metrics/agreement.pys4sb  á      !!   !A&& (