³ò 4ÒÇIc @sddkZddklZddklZddklZlZlZddk l Z ddk Tde fd„ƒYZ d „Zed jo eƒndS( iÿÿÿÿN(tchain(tlog(tConditionalProbDisttConditionalFreqDistt MLEProbDist(tingrams(t*t NgramModelcBsweZdZed„Zd„Zd„Zd„Zd„Zd d„Z d„Z d„Z d „Z d „Z d „ZRS( sN A processing interface for assigning a probability to the next word. cCsö||_|djo d„}ntƒ}tƒ|_d|d|_x^tt|i|ƒ|ƒD]A}|ii|ƒt |d ƒ}|d}||i |ƒqeWt ||t t |ƒƒ|_|djot|d||ƒ|_ndS(sx Creates an ngram language model to capture patterns in n consecutive words of training text. An estimator smooths the probabilities derived from the text and may allow generation of ngrams not seen during training. @param n: the order of the language model (ngram size) @type n: C{int} @param train: the training text @type train: C{list} of C{list} of C{string} @param estimator: a function for generating a probability distribution @type estimator: a function that takes a C{ConditionalFreqDist} and returns a C{ConditionalProbDist} cSs t|ƒS((R(tfdisttbins((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyt)stiiÿÿÿÿN(R (t_ntNoneRtsett_ngramst_prefixRRtaddttupletincRtFalsetlent_modelRt_backoff(tselftnttraint estimatortcfdtngramtcontextttoken((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyt__init__s       cCs“t|ƒ}||f|ijo||i|ƒSnX|idjo(|i|ƒ|ii||d ƒSn td|di|ƒfƒ‚dS(s6Evaluate the probability of this word in this context.iiÿÿÿÿs5No probability mass assigned to word %s in context %st N(RRtprobR t_alphaRt RuntimeErrortjoin(RtwordR((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyR"<s  (cCs!|i|ƒ|ii|d ƒS(Niÿÿÿÿ(t_betaR(Rttokens((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyR#GscCs'||jo||iƒSndSdS(Ni(tdiscount(RR(((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyR'Js cCst|i||ƒdƒ S(sEEvaluate the (negative) log probability of this word in this context.i(RR"(RR&R((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pytlogprobPscCs=t|ƒ}x*t|ƒD]}|i|i|ƒƒqW|S(s1Generate random text based on the language model.(tlisttrangetappendt _generate_one(Rt num_wordsRttextti((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pytgenerateWs   cCsn|it|ƒ|i d}||jo||iƒSn-|idjo|ii|dƒSndSdS(Nit.(RRR R2RR.(RR((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyR.^s  cCssd}xft|idt|ƒƒD]H}t|||id|d!ƒ}||}||i||ƒ7}q#W|S(s‹Evaluate the total entropy of a text with respect to the model. This is the sum of the log probability of each word in the message.gi(R,R RRR*(RR0teR1RR((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pytentropyhs" cCst|ƒ|ijS(N(RR(Rtitem((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyt __contains__sscCs|it|ƒS(N(RR(RR6((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyt __getitem__vscCsdt|iƒ|ifS(Ns(RRR (R((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyt__repr__ys((t__name__t __module__t__doc__R R R"R#R'R*R2R.R5R7R8R9(((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyRs %      cs—ddkl}ddkl‰l}‡fd†}td|iddƒ|ƒ}|GH|idƒ}ddk}d i |i d i |ƒƒƒGHdS( Niÿÿÿÿ(tbrown(tLidstoneProbDisttWittenBellProbDistcs ˆ|dƒS(gš™™™™™É?((RR (R>(s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyR sit categoriestnewsids R!( t nltk.corpusR=tnltk.probabilityR>R?RtwordsR2ttextwrapR%twrap(R=R?RtlmR0RE((R>s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pytdemo|s t__main__(trandomt itertoolsRtmathRRCRRRt nltk.utilRtapitModelIRRHR:(((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyss  k