³ò ŽB_Kc @sidZddkZddkZddklZlZddklZddkTddk Tde fd„ƒYZ de fd „ƒYZ d e ei fd „ƒYZd e ei fd „ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZde ei fd„ƒYZde ei fd„ƒYZde efd„ƒYZdefd„ƒYZdS(s5 Classes for tagging sentences sequentially, left to right. The abstract base class L{SequentialBackoffTagger} serves as the base class for all the taggers in this module. Tagging of individual words is performed by the method L{choose_tag() }, which is defined by subclasses of L{SequentialBackoffTagger}. If a tagger is unable to determine a tag for the specified token, then its I{backoff tagger} is consulted instead. Any C{SequentialBackoffTagger} may serve as a backoff tagger for any other C{SequentialBackoffTagger}. iÿÿÿÿN(tFreqDisttConditionalFreqDist(tNaiveBayesClassifier(t*tSequentialBackoffTaggercBsPeZdZed„Zd„ZeeddƒZd„Zd„Z d„Z RS(sÆ An abstract base class for taggers that tags words sequentially, left to right. Tagging of individual words is performed by the method L{choose_tag()}, which should be defined by subclasses. If a tagger is unable to determine a tag for the specified token, then its backoff tagger is consulted. @ivar _taggers: A list of all the taggers that should be tried to tag a token (i.e., C{self} and its backoff taggers). cCs4|djo|g|_n|g|i|_dS(N(tNonet_taggers(tselftbackoff((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyt__init__,s cCs-t|iƒdjodSn |idSdS(Nii(tlenRR(R((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyt _get_backoff2stdocs, The backoff tagger for this tagger.cCsLg}x6tt|ƒƒD]"}|i|i|||ƒƒqWt||ƒS(N(trangeR tappendttag_onetzip(Rttokensttagsti((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyttag8s  cCsEt}x8|iD]-}|i|||ƒ}|tj oPqqW|S(sG Determine an appropriate tag for the specified token, and return that tag. If this tagger is unable to determine a tag for the specified token, then its backoff tagger is consulted. @rtype: C{str} @type tokens: C{list} @param tokens: The list of words that are being tagged. @type index: C{int} @param index: The index of the word whose tag should be returned. @type history: C{list} of C{str} @param history: A list of the tags for all words before C{index}. (RRt choose_tag(RRtindexthistoryRttagger((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR?s  cCstdƒ‚dS(s³ Decide which tag should be used for the specified token, and return that tag. If this tagger is unable to determine a tag for the specified token, return C{None} -- do I{not} consult the backoff tagger. This method should be overridden by subclasses of C{SequentialBackoffTagger}. @rtype: C{str} @type tokens: C{list} @param tokens: The list of words that are being tagged. @type index: C{int} @param index: The index of the word whose tag should be returned. @type history: C{list} of C{str} @param history: A list of the tags for all words before C{index}. s,SequentialBackoffTagger is an abstract classN(tAssertionError(RRRR((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRUs( t__name__t __module__t__doc__RR R tpropertyRRRR(((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR!s       t ContextTaggercBsMeZdZed„Zd„Zd„Zd„Zd„Zde d„Z RS(s2 An abstract base class for sequential backoff taggers that choose a tag for a token based on the value of its "context". Different subclasses are used to define different contexts. A C{ContextTagger} chooses the tag for a token by calculating the token's context, and looking up the corresponding tag in a table. This table can be constructed manually; or it can be automatically constructed based on a training corpus, using the L{_train()} factory method. @ivar _context_to_tag: Dictionary mapping contexts to tags. cCs1ti||ƒ|o ||_n h|_dS(sŸ @param context_to_tag: A dictionary mapping contexts to tags. @param backoff: The backoff tagger that should be used for this tagger. N(RR t_context_to_tag(Rtcontext_to_tagR((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR xs cCstdƒ‚dS(sá @return: the context that should be used to look up the tag for the specified token; or C{None} if the specified token should not be handled by this tagger. @rtype: (hashable) sAbstract base classN(R(RRRR((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pytcontextƒscCs%|i|||ƒ}|ii|ƒS(N(R!Rtget(RRRRR!((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRŒscCs t|iƒS(sw @return: The number of entries in the table used by this tagger to map from contexts to tags. (R R(R((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pytsizescCsd|ii|iƒfS(Ns <%s: size=%d>(t __class__RR#(R((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyt__repr__—sic Cs©d}}tƒ}tƒ}xÏ|D]Ç}t|Œ\} } x¬t|ƒD]ž\} \} } |d7}|i| | | | ƒ}|djoqHn||i| ƒ|idjp#| |ii| | | | ƒjo|i |ƒqHqHWq#WxT|D]L}||i ƒ}|||}||jo||i |<||7}qõqõW|oYt |i ƒ}d|d|}d|dt |i ƒƒ}dGd|||fGHndS(sð Initialize this C{ContextTagger}'s L{_context_to_tag} table based on the given training data. In particular, for each context C{I{c}} in the training data, set C{_context_to_tag[I{c}]} to the most frequent tag for that context. However, exclude any contexts that are already tagged perfectly by the backoff tagger(s). The old value of C{self._context_to_tag} (if any) is discarded. @param tagged_corpus: A tagged corpus. Each item should be a C{list} of C{(word, tag)} tuples. @param cutoff: If the most likely tag for a context occurs fewer than C{cutoff} times, then exclude it from the context-to-tag table for the new tagger. iiidgY@s[Trained Unigram tagger:s(size=%d, backoff=%.2f%%, pruning=%.2f%%]N(tsetRRt enumerateR!RtincRRtaddtmaxRR t conditions(Rt tagged_corpustcutofftverboset token_countt hit_counttuseful_contextstfdtsentenceRRRttokenRR!tbest_tagthitsR#Rtpruning((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyt_trainšs:     #  ( RRRRR R!RR#R%tFalseR8(((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRjs    t DefaultTaggercBs/eZdZdZd„Zd„Zd„ZRS(s< A tagger that assigns the same tag to every token. s!nltk.DefaultTaggercCs||_ti|dƒdS(sK Construct a new tagger that assigns C{tag} to all tokens. N(t_tagRR R(RR((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR Þs cCs|iS(N(R;(RRRR((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRåscCs d|iS(Ns(R;(R((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR%ès(RRRtyaml_tagR RR%(((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR:Øs   t NgramTaggercBs5eZdZdZeeeded„Zd„ZRS(s- A tagger that chooses a token's tag based on its word string and on the preceeding I{n} word's tags. In particular, a tuple C{(tags[i-n:i-1], words[i])} is looked up in a table, and the corresponding tag is returned. N-gram taggers are typically trained on a tagged corpus. s!nltk.NgramTaggericCsN||_|i||ƒti|||ƒ|o|i|||ƒndS(s@ Train a new C{NgramTagger} using the given training data or the supplied model. In particular, construct a new tagger whose table maps from each context C{(tag[i-n:i-1], word[i])} to the most frequent tag for that context. But exclude any contexts that are already tagged perfectly by the backoff tagger. @param train: A tagged corpus consisting of a C{list} of tagged sentences, where each sentence is a C{list} of C{(word, tag)} tuples. @param backoff: A backoff tagger, to be used by the new tagger if it encounters an unknown context. @param cutoff: If the most likely tag for a context occurs fewer than C{cutoff} times, then exclude it from the context-to-tag table for the new tagger. N(t_nt _check_paramsRR R8(RtnttraintmodelRR-R.((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR ös  cCs5t|td||idƒ|!ƒ}|||fS(Nii(ttupleR*R>(RRRRt tag_context((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR!s'(RRRR<RR9R R!(((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR=ìs t UnigramTaggercBs5eZdZdZeeeded„Zd„ZRS(s‚ A tagger that chooses a token's tag based its word string. Unigram taggers are typically trained on a tagged corpus. s!nltk.UnigramTaggericCs#ti|d|||||ƒdS(Ni(R=R (RRARBRR-R.((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR scCs||S(N((RRRR((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR!!s(RRRR<RR9R R!(((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyREs t BigramTaggercBs)eZdZdZeeded„ZRS(s3 A tagger that chooses a token's tag based its word string and on the preceeding words' tag. In particular, a tuple consisting of the previous tag and the word is looked up in a table, and the corresponding tag is returned. Bigram taggers are typically trained on a tagged corpus. s!nltk.BigramTaggericCs#ti|d|||||ƒdS(Ni(R=R (RRARBRR-R.((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR /s(RRRR<RR9R (((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRF%st TrigramTaggercBs,eZdZdZeeeded„ZRS(sC A tagger that chooses a token's tag based its word string and on the preceeding two words' tags. In particular, a tuple consisting of the previous two tags and the word is looked up in a table, and the corresponding tag is returned. Trigram taggers are typically trained them on a tagged corpus. s!nltk.TrigramTaggericCs#ti|d|||||ƒdS(Ni(R=R (RRARBRR-R.((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR ?s(RRRR<RR9R (((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRG5st AffixTaggercBs;eZdZdZeeddeded„Zd„ZRS(sË A tagger that chooses a token's tag based on a leading or trailing substring of its word string. (It is important to note that these substrings are not necessarily "true" morphological affixes). In particular, a fixed-length substring of the word is looked up in a table, and the corresponding tag is returned. Affix taggers are typically constructed by training them on a tagged corpus; see L{the constructor <__init__>}. s!nltk.AffixTaggeriýÿÿÿiicCsa|i||ƒti|||ƒ||_|t|ƒ|_|o|i|||ƒndS(s• Construct a new affix tagger. @param affix_length: The length of the affixes that should be considered during training and tagging. Use negative numbers for suffixes. @param min_stem_length: Any words whose length is less than C{min_stem_length+abs(affix_length)} will be assigned a tag of C{None} by this tagger. N(R?RR t _affix_lengthtabst_min_word_lengthR8(RRARBt affix_lengthtmin_stem_lengthRR-R.((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR Qs  cCsV||}t|ƒ|ijodSn+|idjo||i Sn ||iSdS(Ni(R RKRRI(RRRRR4((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR!gs  (RRRR<RR9R R!(((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRHEs   t RegexpTaggercBs2eZdZdZed„Zd„Zd„ZRS(sa A tagger that assigns tags to words based on regular expressions over word strings. s!nltk.RegexpTaggercCs||_ti||ƒdS(sÌ Construct a new regexp tagger. @type regexps: C{list} of C{(str, str)} @param regexps: A list of C{(regexp, tag)} pairs, each of which indicates that a word matching C{regexp} should be tagged with C{tag}. The pairs will be evalutated in order. If none of the regexps match a word, then the optional backoff tagger is invoked, else it is assigned the tag C{None}. N(t_regexpsRR (RtregexpsR((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR ws cCs=x6|iD]+\}}ti|||ƒo|Sq q WdS(N(ROtretmatchR(RRRRtregexpR((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR†s   cCsdt|iƒS(Ns(R RO(R((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR%Œs(RRRR<RR RR%(((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRNqs   tClassifierBasedTaggercBs\eZdZeeeieeeed„Zd„Zd„Z d„Z d„Z d„Z RS(sì A sequential tagger that uses a classifier to choose the tag for each token in a sentence. The featureset input for the classifier is generated by a feature detector function:: feature_detector(tokens, index, history) -> featureset Where C{tokens} is the list of unlabeled tokens in the sentence; C{index} is the index of the token for which feature detection should be performed; and C{history} is list of the tags for all tokens before C{index}. cCsœ|i||ƒti||ƒ|o|p| o| otdƒ‚n|dj o ||_n||_||_|o|i|||ƒndS(sb Construct a new classifier-based sequential tagger. @param feature_detector: A function used to generate the featureset input for the classifier:: feature_detector(tokens, index, history) -> featureset @param train: A tagged corpus consisting of a C{list} of tagged sentences, where each sentence is a C{list} of C{(word, tag)} tuples. @param backoff: A backoff tagger, to be used by the new tagger if it encounters an unknown context. @param classifier_builder: A function used to train a new classifier based on the data in C{train}. It should take one argument, a list of labeled featuresets (i.e., C{(featureset, label)} tuples). @param classifier: The classifier that should be used by the tagger. This is only useful if you want to manually construct the classifier; normally, you would use C{train} instead. @param backoff: A backoff tagger, used if this tagger is unable to determine a tag for a given token. @param cutoff_prob: If specified, then this tagger will fall back on its backoff tagger if the probability of the most likely tag is less than C{cutoff_prob}. s8Must specify either training data or trained classifier.N( R?RR t ValueErrorRt_feature_detectort _cutoff_probt _classifierR8(Rtfeature_detectorRAtclassifier_buildert classifierRt cutoff_probR.((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR œs"    cCs€|i|||ƒ}|idjo|ii|ƒSnD|ii|ƒ}|iƒ}|i|ƒ|ijo|SndSdS(N(RYRWRRXtclassifyt prob_classifyR*tprob(RRRRt featuresettpdistR((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRÕs c CsÆg}|o dGHnx€|D]x}g}t|Œ\}}xWtt|ƒƒD]C} |i|| |ƒ} |i| || fƒ|i|| ƒqNWqW|odt|ƒGHn||ƒ|_dS(sf Build a new classifier, based on the given training data (C{tagged_corpus}). s,Constructing training corpus for classifier.s"Training classifier (%d instances)N(RR R RYRRX( RR,RZR.tclassifier_corpusR3Rtuntagged_sentenceRRR`((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR8æs    cCs d|iS(Ns(RX(R((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR%ýscCs|i|||ƒS(s Return the feature detector that this tagger uses to generate featuresets for its classifier. The feature detector is a function with the signature:: feature_detector(tokens, index, history) -> featureset @see: L{classifier()} (RV(RRRR((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRYs cCs|iS(sñ Return the classifier that this tagger uses to choose a tag for each word in a sentence. The input for this classifier is generated using this tagger's feature detector. @see: L{feature_detector()} (RX(R((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyR[ s( RRRRRRAR9R RR8R%RYR[(((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRTs 6    tClassifierBasedPOSTaggercBseZdZd„ZRS(s3 A classifier based part of speech tagger. c Cs||}|djod}}d}}n„|djo2||diƒ}d}||d}d}nE||diƒ}||diƒ}||d}||d}tid|ƒo d} n{tid|ƒo d} n^tid|ƒo d } nAtid |ƒo d } n$tid |ƒo d } nd} h|d<|d<|d<|iƒd<|iƒdd<|iƒdd<|iƒdd<|d<|d<d||iƒfd<d||iƒfd<d||iƒfd<| d<} | S( Niiis![0-9]+(\.[0-9]*)?|[0-9]*\.[0-9]+$tnumbers\W+$tpuncts [A-Z][a-z]+$tupcases[a-z]+$tdowncases\w+$t mixedcasetothertprevtagt prevprevtagtwords word.loweriýÿÿÿtsuffix3iþÿÿÿtsuffix2iÿÿÿÿtsuffix1t prevprevwordtprevwords%s+%ss prevtag+wordsprevprevtag+words prevword+wordtshape(RtlowerRQRR( RRRRRmRrRqRkRlRstfeatures((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRYsN                (RRRRY(((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyRds(RRQtyamltnltk.probabilityRRtnltk.classify.naivebayesRtapitutiltTaggerIRRt YAMLObjectR:R=RERFRGRHRNtFeaturesetTaggerIRTRd(((s)/p/zhu/06/nlp/nltk/nltk/tag/sequential.pyss   In),‡