³ò ŽB_Kc@sudZddkZddkTdefd„ƒYZed„Zd„Zd„Zd „Z e d jo eƒndS( s~ Implementation of 'TnT - A Statisical Part of Speech Tagger' by Thorsten Brants http://acl.ldc.upenn.edu/A/A00/A00-1031.pdf iÿÿÿÿN(t*tTnTcBsbeZdZeeded„Zd„Zd„Zd„Zd„Z d„Z d„Z d „Z RS( sŸ TnT - Statistical POS tagger IMPORTANT NOTES: * DOES NOT AUTOMATICALLY DEAL WITH UNSEEN WORDS It is possible to provide an untrained POS tagger to create tags for unknown words, see __init__ function * SHOULD BE USED WITH SENTENCE-DELIMITED INPUT - Due to the nature of this tagger, it works best when trained over sentence delimited input. - However it still produces good results if the training data and testing data are separated on all punctuation eg: [,.?!] - Input for training is expected to be a list of sentences where each sentence is a list of (word, tag) tuples - Input for tag function is a single sentence Input for tagdata function is a list of sentences Output is of a similar form * Function provided to process text that is unsegmented - Please see basic_sent_chop() TnT uses a second order Markov model to produce tags for a sequence of input, specifically: argmax [Proj(P(t_i|t_i-1,t_i-2)P(w_i|t_i))] P(t_T+1 | t_T) IE: the maximum projection of a set of probabilities The set of possible tags for a given word is derived from the training data. It is the set of all tags that exact word has been assigned. The probability of a tag for a given word is the linear interpolation of 3 markov models; a zero-order, first-order, and a second order model. P(t_i| t_i-1, t_i-2) = l1*P(t_i) + l2*P(t_i| t_i-1) + l3*P(t_i| t_i-1, t_i-2) A beam search is used to limit the memory usage of the algorithm. The degree of the beam can be changed using N in the initialization. N represents the maximum number of possible solutions to maintain while tagging. It is possible to differentiate the tags which are assigned to capitalized words. However this does not result in a significant gain in the accuracy of the results. iècCs¯tiiƒ|_tiiƒ|_tiiƒ|_tiiƒ|_tiiƒ|_d|_ d|_ d|_ ||_ ||_ ||_||_d|_d|_dS(sø Construct a TnT statistical tagger. Tagger must be trained before being used to tag input. @param unk: instance of a POS tagger, conforms to TaggerI @type unk:(TaggerI) @param Trained: Indication that the POS tagger is trained or not @type Trained: boolean @param N: Beam search degree (see above) @type N:(int) @param C: Capitalization flag @type C: boolean Initializer, creates frequency distributions to be used for tagging _lx values represent the portion of the tri/bi/uni taggers to be used to calculate the probability N value is the number of possible solutions to maintain while tagging. A good value for this is 1000 C is a boolean value which specifies to use or not use the Capitalization of the word as additional information for tagging. NOTE: using capitalization may not increase the accuracy of the tagger giN(tnltkt probabilitytFreqDistt_unitConditionalFreqDistt_bit_trit_wdt_eost_l1t_l2t_l3t_Nt_Ct_Tt_unktunknowntknown(tselftunktTrainedtNtC((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pyt__init__Is        cCsCt}|idjo$|itjo|ii|ƒnxø|D]ð}ddg}xÇ|D]¿\}}|io|diƒo t}n|i|i |ƒ|i i ||fƒ|i |di ||fƒ|i t |ƒi ||fƒ|i||fƒ|idƒt}qZW|i|i dƒqAW|iƒdS(sü Uses a set of tagged data to train the tagger. If an unknown word tagger is specified, it is trained on the same data. @param data: List of lists of (word, tag) tuples @type data: L{tuple} of L{str} tBOSiitEOSN(tFalseRtNoneRttrainRtisuppertTrueR tincRRRttupletappendtpopR t_compute_lambda(RtdataRtsentthistorytwtt((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pyRys$    %   c Cs“d}d}d}x;|iiƒD]*}|\}}x|i|iƒD]}|i|djoqHn|i|i||d|i|iƒdƒ}|i|i||d|i|iƒdƒ} |i|i|d|iiƒdƒ} | |jo&| | jo||i||7}qH| |jo&| | jo||i||7}qH|| jo&|| jo||i||7}qH|| joO|| joB|t|i||ƒd7}|t|i||ƒd7}qH| | joO| |joB|t|i||ƒd7}|t|i||ƒd7}qHqHWq"W|||||_|||||_ |||||_ dS(s  creates lambda values based upon training data NOTE: no need to explicitly reference C, it is contained within the tag variable :: tag == (tag,C) for each tag trigram (t1, t2, t3) depending on the maximum value of - f(t1,t2,t3)-1 / f(t1,t2)-1 - f(t2,t3)-1 / f(t2)-1 - f(t3)-1 / N-1 increment l3,l2, or l1 by f(t1,t2,t3) ISSUES -- Resolutions: if 2 values are equal, increment both lambda values by (f(t1,t2,t3) / 2) gig@N( Rt conditionstsamplesRt _safe_divRRtfloatR R R ( Rttl1ttl2ttl3R(th1th2ttagtc3tc2tc1((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pyR%¨s: 22*##cCs-|djodSnt|ƒt|ƒSdS(s€ Safe floating point division function, does not allow division by 0 returns -1 if the denominator is 0 iiÿÿÿÿN(R.(Rtv1tv2((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pyR-ûs cCs7g}x*|D]"}|i|ƒ}|i|ƒq W|S(sk Tags each sentence in a list of sentences @param data:list of list of words @type data: [[string,],] @return: list of list of (word, tag) tuples Invokes tag(sent) function for each sentence compiles the results into a list of tagged sentences each tagged sentence is a list of (word, tag) tuples (R4R#(RR&tresR'tres1((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pyttagdatas c Cs…ddgdfg}t|ƒ}|i||ƒ}g}xEtt|ƒƒD]1}||d\}}|i|||fƒqLW|S(so Tags a single sentence @param data: list of words @type data: [string,] @return: [(word, tag),] Calls recursive function '_tagword' to produce a list of tags Associates the sequence of returned tags with the correct words in the input sequence returns a list of (word, tag) tuples Rgð?i(tlistt_tagwordtrangetlenR#( RR&t current_stateR'ttagsR:tiR*R((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pyR4s c Cs«|gjo|d\}}|Sn|d}|d}g}t}|io|diƒo t}n||iiƒjoU|id7_xÒ|D]7\}} g} xí|i|iƒD]Ø} |ii | |fƒ} |i |di | |fƒ} |i t |dƒi | |fƒ}t |i|| ƒt |i| |fƒ}|i| |i| |i|}||}| i| |f|fƒq¼Wx2| D]*\}}|i||g| |fƒqŸWq–Wn|id7_d}|itjod|f}n1t|ii|gƒƒ\\}} | |f}x!|D]\}}|i|ƒqAW|}|i|iƒt|ƒ|ijo||i }n|i||ƒS(s. @param sent : List of words remaining in the sentence @type sent : [word,] @param current_states : List of possible tag combinations for the sentence so far, and the probability associated with each tag combination @type current_states : [([tag, ],prob), ] Tags the first word in the sentence and recursively tags the reminder of sentence Uses formula specified above to calculate the probability of a particular tag iiiÿÿÿÿiþÿÿÿtUnk(RRRR R R+RR,RtfreqRRR"R.R R R R#RRRR=R4tsortt_cmp_tupR@RR>(RR'tcurrent_statesthtptwordt new_statesRR(tcurr_sent_probtprobsR*tp_unitp_bitp_tritp_wdtp2R4tprobt_w((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pyR>9sN   %  &.#  *$   cCs9|\}}|\}}||djodSndSdS(s´ comparison function @params : (_, prob) @types : (_, int) tuple used to sort a list of these tuples into descending order iiiÿÿÿÿN((Rt.1t.2t_hqtp1t_h2RS((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pyRG¢s ( t__name__t __module__t__doc__RRRRR%R-R<R4R>RG(((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pyRs30 / S  ! icCsÝg}g}ddddg}|oSx±|D]D}||jo$|i|ƒ|i|ƒg}q,|i|ƒq,Wnbx^|D]V\}}||jo*|i||fƒ|i|ƒg}q|i||fƒqW|S(s’ Basic method for tokenizing input into sentences for this tagger: @param data: list of tokens tokens can be either words or (word, tag) tuples @type data: [string,] or [(string, string),] @param raw: boolean flag marking the input data as a list of words or a list of tagged words @type raw: Boolean @ret : list of sentences sentences are a list of tokens tokens are the same as the input Function takes a list of tokens and separates the tokens into lists where each list represents a sentence fragment This function can separate both tagged and raw sequences into basic sentences. Sentence markers are the set of [,.!?] This is a simple method which enhances the performance of the TnT tagger. Better sentence tokenization will further enhance the results. t,t.t?t!(R#(R&trawtnew_datat curr_sentt sent_markRKR4((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pytbasic_sent_chop¶s&        c Csæddkl}ddkl}t|iƒƒ}t|iƒƒ}|iƒ}|i|dd!ƒ|i |dd!ƒ}xet t |ƒƒD]Q}||}||d}x/t t |ƒƒD]} || GdG|| GHq¾WHqWdS( Niÿÿÿÿ(ttnt(tbrowniÈièidixs--( tnltk.tagRgt nltk.corpusRhR=t tagged_sentstsentsRRR<R?R@( RgRhRlttestttaggert tagged_datatjtsR*RC((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pytdemoîs  c Csddkl}ddkl}ddkl}t|iƒƒ}|idddt ƒ}|idddt ƒ}|i |dƒ|i |dƒxht d ƒD]Z}|i |||d |d d !ƒ}t|iƒt|i|iƒ}t|iƒt|i|iƒ} d |_d |_d GHdG|GHdG| GHdG|GHdG|| GH|i |||d |d d !ƒ} t|iƒt|i|iƒ} t|iƒt|i|iƒ} d |_d |_dGHdG| GHdG| GHdG| GHdG| | GHq¡WdS(Niÿÿÿÿ(R4(Rg(ttreebankRièRi idi iisCapitalization off:s Accuracy:sPercentage known:sPercentage unknown:sAccuracy over known words:sCapitalization on:iLiL(RR4RiRgRjRsR=RkRRR RR?taccuracyR.RR( R4RgRstdR*RqRCttaccttp_unttp_kntsacctsp_untsp_kn((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pytdemo2s< %##      %##     cCs÷ddkl}ddkl}l}ddkl}t|iƒƒ}t|iƒƒ}|d }|d }t t |ƒdƒ}t t |ƒdƒ}d}d} d} d} d} d} xät dƒD]Ö}|i d dd t ƒ}|i d dd t ƒ}||||d |!}||||d |!}||| ||d |}||| ||d |}|i|ƒ|i|ƒ|i||ƒ}t|iƒt|i|iƒ}t|iƒt|i|iƒ}| |7} d|_d|_|i||ƒ}t|iƒt|i|iƒ}t|iƒt|i|iƒ}| |7} d|_d|_|||7}| ||7} | |7} | |7} qËWd Gd|GHd Gd| GHdGd| GHdGd| GHdGd| GHdGd| GHdS(Niÿÿÿÿ(R4(RsRh(Rgiègš™™™™™¹?ii RRisbrown: acc over words known:s : overall accuracy:s : words known:streebank: acc over words known:s : overall accuracy:s : words known:(RR4RjRsRhRiRgR=RktintR@R?RRRRtR.RR(R4RsRhRgRutetd10te10ttknacctsknaccttallacctsallaccttknowntsknownRCR*RqtdtesttetesttdtraintetrainRvRwRxRyRzR{((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pytdemo3)s^     ##   ##         t__main__( R]RtapitTaggerIRR RfRrR|R‹R[(((s"/p/zhu/06/nlp/nltk/nltk/tag/tnt.pyss  ÿ£ 8  % A