³ò ŽB_Kc @sddkZddklZddklZddklZlZlZddk l Z ddk Tde fd„ƒYZ d „Zed jo eƒndS( iÿÿÿÿN(tchain(tlog(tConditionalProbDisttConditionalFreqDistt MLEProbDist(tingrams(t*t NgramModelcBsweZdZed„Zd„Zd„Zd„Zd„Zd d„Z d„Z d„Z d „Z d „Z d „ZRS( sN A processing interface for assigning a probability to the next word. cCsó||_|djo d„}ntƒ}tƒ|_d|d|_x^tt|i|ƒ|ƒD]A}|ii|ƒt |d ƒ}|d}||i |ƒqeWt ||t |ƒƒ|_ |djot|d||ƒ|_ndS(su Creates an ngram language model to capture patterns in n consecutive words of training text. An estimator smooths the probabilities derived from the text and may allow generation of ngrams not seen during training. @param n: the order of the language model (ngram size) @type n: C{int} @param train: the training text @type train: C{list} of C{string} @param estimator: a function for generating a probability distribution @type estimator: a function that takes a C{ConditionalFreqDist} and returns a C{ConditionalProbDist} cSs t|ƒS((R(tfdisttbins((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyt+stiiÿÿÿÿN(R (t_ntNoneRtsett_ngramst_prefixRRtaddttupletincRtlent_modelRt_backoff(tselftnttraint estimatortcfdtngramtcontextttoken((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyt__init__s       cCs—t|ƒ}||f|ijo||i|ƒSn\|idjo(|i|ƒ|ii||dƒSn$tdd|di|ƒfƒ‚dS(sH Evaluate the probability of this word in this context. is+No probability mass assigned to word %s in s context %st N(RRtprobR t_alphaRt RuntimeErrortjoin(RtwordR((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyR!>s (cCs!|i|ƒ|ii|dƒS(Ni(t_betaR(Rttokens((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyR"LscCs'||jo||iƒSndSdS(Ni(tdiscount(RR'((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyR&Os cCst|i||ƒdƒ S(sW Evaluate the (negative) log probability of this word in this context. i(RR!(RR%R((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pytlogprobUscCs=t|ƒ}x*t|ƒD]}|i|i|ƒƒqW|S(s1Generate random text based on the language model.(tlisttrangetappendt _generate_one(Rt num_wordsRttextti((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pytgenerate^s   cCsn|it|ƒ|i d}||jo||iƒSn-|idjo|ii|dƒSndSdS(Nit.(RRR R1RR-(RR((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyR-es  cCssd}xft|idt|ƒƒD]H}t|||id|d!ƒ}||}||i||ƒ7}q#W|S(s Evaluate the total entropy of a text with respect to the model. This is the sum of the log probability of each word in the message. gi(R+R RRR)(RR/teR0RR((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pytentropyos" cCst|ƒ|ijS(N(RR(Rtitem((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyt __contains__|scCs|it|ƒS(N(RR(RR5((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyt __getitem__scCsdt|iƒ|ifS(Ns(RRR (R((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyt__repr__‚s((t__name__t __module__t__doc__R RR!R"R&R)R1R-R4R6R7R8(((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyRs &      cs—ddkl}ddkl‰l}‡fd†}td|iddƒ|ƒ}|GH|idƒ}ddk}d i |i d i |ƒƒƒGHdS( Niÿÿÿÿ(tbrown(tLidstoneProbDisttWittenBellProbDistcs ˆ|dƒS(gš™™™™™É?((RR (R=(s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyR ˆsit categoriestnewsids R ( t nltk.corpusR<tnltk.probabilityR=R>RtwordsR1ttextwrapR$twrap(R<R>RtlmR/RD((R=s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pytdemo…s t__main__(trandomt itertoolsRtmathRRBRRRt nltk.utilRtapitModelIRRGR9(((s&/p/zhu/06/nlp/nltk/nltk/model/ngram.pyss  s