³ò 3ÒÇIc@sÌedƒZddkZddkZddkZddklZddklZde fd„ƒYZ de fd„ƒYZ d e fd „ƒYZ d e fd „ƒYZd e fd„ƒYZde fd„ƒYZdefd„ƒYZdefd„ƒYZde fd„ƒYZde fd„ƒYZde fd„ƒYZde fd„ƒYZde fd„ƒYZd„Zd „Zd!e fd"„ƒYZd#e fd$„ƒYZd%efd&„ƒYZd'efd(„ƒYZeid)d*ƒZd+„Z d,„Z!d-e fd.„ƒYZ"d/e"fd0„ƒYZ#d1„Z$d2„Z%d3„Z&d4d5d6„Z'e(d7joe'd4d8ƒe'd9d:ƒnd!d%d#dd'd ddddd/ddd ddd-d dd;d<d=d>gZ)dS(?s-1e300iÿÿÿÿN(t itemgetter(tislicetFreqDistcBsCeZdZed„Zdd„Zd„Zd„Zd„Zd„Z ed„Z d „Z d „Z ed „Z d „Zd „Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Z d„Z!d „Z"d!„Z#d"„Z$RS(#s A frequency distribution for the outcomes of an experiment. A frequency distribution records the number of times each outcome of an experiment has occurred. For example, a frequency distribution could be used to record the frequency of each word type in a document. Formally, a frequency distribution can be defined as a function mapping from each sample to the number of times that sample occurred as an outcome. Frequency distributions are generally constructed by running a number of experiments, and incrementing the count for a sample every time it is an outcome of an experiment. For example, the following code will produce a frequency distribution that encodes how often each word occurs in a text: >>> fdist = FreqDist() >>> for word in tokenize.whitespace(sent): ... fdist.inc(word.lower()) An equivalent way to do this is with the initializer: >>> fdist = FreqDist(word.lower() for word in tokenize.whitespace(sent)) cCs^ti|ƒd|_d|_d|_d|_|o"x|D]}|i|ƒq?WndS(sd Construct a new frequency distribution. If C{samples} is given, then the frequency distribution will be initialized with the count of each object in C{samples}; otherwise, it will be initialized to be empty. In particular, C{FreqDist()} returns an empty frequency distribution; and C{FreqDist(samples)} first creates an empty frequency distribution, and then calls C{inc} for each element in the list C{samples}. @param samples: The samples to initialize the frequency distribution with. @type samples: Sequence iN(tdictt__init__t_NtNonet _Nr_cachet _max_cachet _item_cachetinc(tselftsamplestsample((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRLs     icCs]|djodSn|i|7_|i|dƒ|||RR ti((s&/p/zhu/06/nlp/nltk/nltk/probability.pyttabulate@s %cCs td‚dS(NsjUse FreqDist.keys(), or iterate over the FreqDist to get its samples in sorted order (most frequent first)(R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pytsorted_samples_scCs td‚dS(NsjUse FreqDist.keys(), or iterate over the FreqDist to get its samples in sorted order (most frequent first)(R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pytsortedbscCs<|ip.tti|ƒdtdƒdtƒ|_ndS(Ntkeyitreverse(R RDRtitemsRR4(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt_sort_keys_by_valuees cCs |iƒttdƒ|iƒS(s¡ Return the samples sorted in decreasing order of frequency. @return: A list of samples, in sorted order @rtype: C{list} of any i(RHtmapRR (R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRjs cCs |iƒttdƒ|iƒS(s¡ Return the samples sorted in decreasing order of frequency. @return: A list of samples, in sorted order @rtype: C{list} of any i(RHRIRR (R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pytvaluests cCs|iƒ|iS(s¢ Return the items sorted in decreasing order of frequency. @return: A list of items, in sorted order @rtype: C{list} of C{tuple} (RHR (R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRG~s cCst|iƒƒS(s¥ Return the samples sorted in decreasing order of frequency. @return: An iterator over the samples, in sorted order @rtype: C{iter} (titerR(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt__iter__ˆscCst|iƒƒS(s¥ Return the samples sorted in decreasing order of frequency. @return: An iterator over the samples, in sorted order @rtype: C{iter} (RKR(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pytiterkeys‘scCst|iƒƒS(s– Return the values sorted in decreasing order. @return: An iterator over the values, in sorted order @rtype: C{iter} (RKRJ(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt itervaluesšscCs|iƒt|iƒS(s¨ Return the items sorted in decreasing order of frequency. @return: An iterator over the items, in sorted order @rtype: C{iter} of any (RHRKR (R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt iteritems£s cCs.t|tƒptSn|iƒ|iƒjS(N(t isinstanceRR2RG(R tother((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt__eq__±scCs ||j S(N((R RQ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt__ne__´scsKtˆtƒptSntˆƒiˆƒot‡‡fd†ˆDƒƒS(Nc3s'x |]}ˆ|ˆ|jVqWdS(N((t.0RE(R RQ(s&/p/zhu/06/nlp/nltk/nltk/probability.pys ¸s(RPRR2tsettissubsettall(R RQ((R RQs&/p/zhu/06/nlp/nltk/nltk/probability.pyt__le__¶scCs/t|tƒptSn||jo ||jS(N(RPRR2(R RQ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt__lt__¹scCs"t|tƒptSn||jS(N(RPRR2(R RQ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt__ge__¼scCs"t|tƒptSn||jS(N(RPRR2(R RQ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt__gt__¿scCsd|iƒS(s^ @return: A string representation of this C{FreqDist}. @rtype: string s(R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt__repr__ÃscCs@g}|D]}|d|||fq ~}ddi|ƒS(s^ @return: A string representation of this C{FreqDist}. @rtype: string s%r: %rss, (tjoin(R RR@RG((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt__str__Ês/cCs|i|dƒS(Ni(R(R R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt __getitem__Òs(%t__name__t __module__t__doc__RRR RRR RRRRRR!R$R5RBRCRDRHRRJRGRLRMRNRORRRSRXRYRZR[R\R^R_(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR3sD         .             t ProbDistIcBsSeZdZeZd„Zd„Zd„Zd„Zd„Z d„Z d„Z RS(sŽ A probability distribution for the outcomes of an experiment. A probability distribution specifies how likely it is that an experiment will have any given outcome. For example, a probability distribution could be used to predict the probability that a token in a document will have a given type. Formally, a probability distribution can be defined as a function mapping from samples to nonnegative real numbers, such that the sum of every number in the function's range is 1.0. C{ProbDist}s are often used to model the probability distribution of the experiment used to generate a frequency distribution. cCs!|itjo td‚ndS(Ns Interfaces can't be instantiated(t __class__RctAssertionError(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRêscCs tƒ‚dS(s  @return: the probability for a given sample. Probabilities are always real numbers in the range [0, 1]. @rtype: float @param sample: The sample whose probability should be returned. @type sample: any N(Re(R R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pytprobîs cCs8|i|ƒ}|djotSnti|dƒSdS(s. @return: the base 2 logarithm of the probability for a given sample. Log probabilities range from negitive infinity to zero. @rtype: float @param sample: The sample whose probability should be returned. @type sample: any iiN(Rft_NINFtmathtlog(R R tp((s&/p/zhu/06/nlp/nltk/nltk/probability.pytlogprobùs  cCs tƒ‚dS(sÞ @return: the sample with the greatest probability. If two or more samples have the same probability, return one of them; which sample is returned is undefined. @rtype: any N(Re(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR$ scCs tƒ‚dS(s¶ @return: A list of all samples that have nonzero probabilities. Use C{prob} to find the probability of each sample. @rtype: C{list} N(Re(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR scCsdS(sm @return: The ratio by which counts are discounted on average: c*/c @rtype: C{float} g((R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pytdiscount scCs¢tiƒ}x<|iƒD].}||i|ƒ8}|djo|SqqW|djo|Sn|iotid|d|fƒntit|iƒƒƒS(sÄ @return: A randomly selected sample from this probabilitiy distribution. The probability of returning each sample C{samp} is equal to C{self.prob(samp)}. ig-Cëâ6?sTProbability distribution %r sums to %r; generate() is returning an arbitrary sample.i(trandomR Rft SUM_TO_ONEtwarningstwarntchoiceR0(R RjR ((s&/p/zhu/06/nlp/nltk/nltk/probability.pytgenerate)s     ( R`RaRbR4RnRRfRkR$R RlRr(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRcÙs    tUniformProbDistcBs;eZdZd„Zd„Zd„Zd„Zd„ZRS(s— A probability distribution that assigns equal probability to each sample in a given set; and a zero probability to all other samples. cCsbt|ƒdjotddƒ‚nt|ƒ|_dt|iƒ|_t|iƒ|_dS(s5 Construct a new uniform probability distribution, that assigns equal probability to each sample in C{samples}. @param samples: The samples that should be given uniform probability. @type samples: C{list} @raise ValueError: If C{samples} is empty. is(A Uniform probability distribution must shave at least one sample.gð?N(RR/RUt _samplesett_probR0t_samples(R R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRBs cCs#||ijo |iSndSdS(Ni(RtRu(R R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRfSscCs |idS(Ni(Rv(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR$VscCs|iS(N(Rv(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR WscCsdt|iƒS(Ns!(RRt(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\Xs(R`RaRbRRfR$R R\(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRs<s     tDictionaryProbDistcBsMeZdZeeed„Zd„Zd„Zd„Zd„Z d„Z RS(s£ A probability distribution whose probabilities are directly specified by a given dictionary. The given dictionary maps samples to probabilities. c Cs\|iƒ|_||_|o9|ot|iiƒƒ}|tjoDtidt|ƒdƒ}xX|i ƒD]}||i|“si(thasattrR$RyRGRƒ(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR$‘s*cCs |iiƒS(N(RyR(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR •scCsdt|iƒS(Ns(RRy(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\—s( R`RaRbRR2RRfRkR$R R\(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRw[s!    t MLEProbDistcBsDeZdZd„Zd„Zd„Zd„Zd„Zd„ZRS(s& The maximum likelihood estimate for the probability distribution of the experiment used to generate a frequency distribution. The X{maximum likelihood estimate} approximates the probability of each sample as the frequency of that sample in the frequency distribution. cCs ||_dS(s+ Use the maximum likelihood estimate to create a probability distribution for the experiment used to generate C{freqdist}. @type freqdist: C{FreqDist} @param freqdist: The frequency distribution that the probability estimates should be based on. N(t _freqdist(R tfreqdist((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR¢s cCs|iS(s @return: The frequency distribution that this probability distribution is based on. @rtype: C{FreqDist} (R‡(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRˆ­scCs|ii|ƒS(N(R‡R!(R R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRfµscCs |iiƒS(N(R‡R$(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR$¸scCs |iiƒS(N(R‡R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR »scCsd|iiƒS(sa @rtype: C{string} @return: A string representation of this C{ProbDist}. s!(R‡R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\¾s( R`RaRbRRˆRfR$R R\(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR†šs    tLidstoneProbDistcBsVeZdZeZed„Zd„Zd„Zd„Z d„Z d„Z d„Z RS(sA The Lidstone estimate for the probability distribution of the experiment used to generate a frequency distribution. The C{Lidstone estimate} is paramaterized by a real number M{gamma}, which typically ranges from 0 to 1. The X{Lidstone estimate} approximates the probability of a sample with count M{c} from an experiment with M{N} outcomes and M{B} bins as M{(c+gamma)/(N+B*gamma)}. This is equivalant to adding M{gamma} to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. cCsA|djp |d jo;|iƒdjo(|iid }td|dƒ‚n|d j oQ||iƒjo>|iid }td|d|dd|iƒƒ‚n||_t|ƒ|_|iiƒ|_ |d jo|iƒ}n||_ |i |||_ |i d jod|_d |_ nd S( sÎ Use the Lidstone estimate to create a probability distribution for the experiment used to generate C{freqdist}. @type freqdist: C{FreqDist} @param freqdist: The frequency distribution that the probability estimates should be based on. @type gamma: C{float} @param gamma: A real number used to paramaterize the estimate. The Lidstone estimate is equivalant to adding M{gamma} to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. @type bins: C{int} @param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If C{bins} is not specified, it defaults to C{freqdist.B()}. iiøÿÿÿsA %s probability distribution smust have at least one bin.s) The number of bins in a %s distribution s&(%d) must be greater than or equal to s(the number of bins in the FreqDist used sto create it (%d).giN( RRRdR`R/RR‡R t_gammaRt_binst_divisor(R RˆtgammaRtname((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRÒs$-     cCs|iS(s @return: The frequency distribution that this probability distribution is based on. @rtype: C{FreqDist} (R‡(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRˆscCs|i|}||i|iS(N(R‡RŠRŒ(R R R((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRfs cCs |iiƒS(N(R‡R$(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR$ scCs |iiƒS(N(R‡R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR scCs|i|i}||i|S(N(RŠR‹R(R tgb((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRlscCsd|iiƒS(sa @rtype: C{string} @return: A string representation of this C{ProbDist}. s&(R‡R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\s( R`RaRbR2RnRRRˆRfR$R RlR\(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR‰Ås  .     tLaplaceProbDistcBs#eZdZed„Zd„ZRS(s¿ The Laplace estimate for the probability distribution of the experiment used to generate a frequency distribution. The X{Lidstone estimate} approximates the probability of a sample with count M{c} from an experiment with M{N} outcomes and M{B} bins as M{(c+1)/(N+B)}. This is equivalant to adding one to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. cCsti||d|ƒdS(s‘ Use the Laplace estimate to create a probability distribution for the experiment used to generate C{freqdist}. @type freqdist: C{FreqDist} @param freqdist: The frequency distribution that the probability estimates should be based on. @type bins: C{int} @param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If C{bins} is not specified, it defaults to C{freqdist.B()}. iN(R‰R(R RˆR((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR+scCsd|iiƒS(sa @rtype: C{string} @return: A string representation of this C{ProbDist}. s%(R‡R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\<s(R`RaRbRRR\(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR!s  t ELEProbDistcBs#eZdZed„Zd„ZRS(sÚ The expected likelihood estimate for the probability distribution of the experiment used to generate a frequency distribution. The X{expected likelihood estimate} approximates the probability of a sample with count M{c} from an experiment with M{N} outcomes and M{B} bins as M{(c+0.5)/(N+B/2)}. This is equivalant to adding 0.5 to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. cCsti||d|ƒdS(s Use the expected likelihood estimate to create a probability distribution for the experiment used to generate C{freqdist}. @type freqdist: C{FreqDist} @param freqdist: The frequency distribution that the probability estimates should be based on. @type bins: C{int} @param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If C{bins} is not specified, it defaults to C{freqdist.B()}. gà?N(R‰R(R RˆR((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRMscCsd|iiƒS(sa @rtype: C{string} @return: A string representation of this C{ProbDist}. s!(R‡R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\^s(R`RaRbRRR\(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR‘Cs  tHeldoutProbDistcBsqeZdZeZed„Zd„Zd„Zd„Z d„Z d„Z d„Z d„Z d „Zd „ZRS( s£ The heldout estimate for the probability distribution of the experiment used to generate two frequency distributions. These two frequency distributions are called the "heldout frequency distribution" and the "base frequency distribution." The X{heldout estimate} uses uses the X{heldout frequency distribution} to predict the probability of each sample, given its frequency in the X{base frequency distribution}. In particular, the heldout estimate approximates the probability for a sample that occurs M{r} times in the base distribution as the average frequency in the heldout distribution of all samples that occur M{r} times in the base distribution. This average frequency is M{Tr[r]/(Nr[r]*N)}, where: - M{Tr[r]} is the total count in the heldout distribution for all samples that occur M{r} times in the base distribution. - M{Nr[r]} is the number of samples that occur M{r} times in the base distribution. - M{N} is the number of outcomes recorded by the heldout frequency distribution. In order to increase the efficiency of the C{prob} member function, M{Tr[r]/(Nr[r]*N)} is precomputed for each value of M{r} when the C{HeldoutProbDist} is created. @type _estimate: C{list} of C{float} @ivar _estimate: A list mapping from M{r}, the number of times that a sample occurs in the base distribution, to the probability estimate for that sample. C{_estimate[M{r}]} is calculated by finding the average frequency in the heldout distribution of all samples that occur M{r} times in the base distribution. In particular, C{_estimate[M{r}]} = M{Tr[r]/(Nr[r]*N)}. @type _max_r: C{int} @ivar _max_r: The maximum number of times that any sample occurs in the base distribution. C{_max_r} is used to decide how large C{_estimate} must be. c Cs“||_||_||iƒ|_|iƒ}g}t|idƒD]}||i||ƒqI~}|iƒ}|i|||ƒ|_ dS(sæ Use the heldout estimate to create a probability distribution for the experiment used to generate C{base_fdist} and C{heldout_fdist}. @type base_fdist: C{FreqDist} @param base_fdist: The base frequency distribution. @type heldout_fdist: C{FreqDist} @param heldout_fdist: The heldout frequency distribution. @type bins: C{int} @param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If C{bins} is not specified, it defaults to C{freqdist.B()}. iN( t _base_fdistt_heldout_fdistR$t_max_rt _calculate_TrR7RRt_calculate_estimatet _estimate( R t base_fdistt heldout_fdistRtTrRRRR((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRs   : cCsPdg|id}x5|iD]*}|i|}||c|i|7(R“RR”(R R@((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\øs(R`RaRbR2RnRRR–R—R™RšR RfR$RlR\(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR’es( !       tCrossValidationProbDistcBsJeZdZeZd„Zd„Zd„Zd„Zd„Z d„Z RS(sB The cross-validation estimate for the probability distribution of the experiment used to generate a set of frequency distribution. The X{cross-validation estimate} for the probability of a sample is found by averaging the held-out estimates for the sample in each pair of frequency distributions. cCsk||_g|_xR|D]J}xA|D]9}||j o&t|||ƒ}|ii|ƒq&q&WqWdS(s¥ Use the cross-validation estimate to create a probability distribution for the experiment used to generate C{freqdists}. @type freqdists: C{list} of C{FreqDist} @param freqdists: A list of the frequency distributions generated by the experiment. @type bins: C{int} @param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If C{bins} is not specified, it defaults to C{freqdist.B()}. N(t _freqdistst_heldout_probdistsR’Rœ(R t freqdistsRtfdist1tfdist2tprobdist((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR s   cCs|iS(s” @rtype: C{list} of C{FreqDist} @return: The list of frequency distributions that this C{ProbDist} is based on. (R (R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR¢$scCs7ttg}|iD]}||iƒq~gƒƒS(N(RUR|R R(R Rtfd((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR ,scCs>d}x$|iD]}||i|ƒ7}qW|t|iƒS(Ng(R¡RfR(R R Rftheldout_probdist((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRf0s  cCs tƒ‚dS(N(Rž(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRl8scCsdt|iƒS(sa @rtype: C{string} @return: A string representation of this C{ProbDist}. s!(RR (R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\;s( R`RaRbR2RnRR¢R RfRlR\(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRŸs     tWittenBellProbDistcBsPeZdZed„Zd„Zd„Zd„Zd„Zd„Z d„Z RS(s¬ The Witten-Bell estimate of a probability distribution. This distribution allocates uniform probability mass to as yet unseen events by using the number of events that have only been seen once. The probability mass reserved for unseen events is equal to: - M{T / (N + T)} where M{T} is the number of observed event types and M{N} is the total number of observed events. This equates to the maximum likelihood estimate of a new type event occuring. The remaining probability mass is discounted such that all probability estimates sum to one, yielding: - M{p = T / Z (N + T)}, if count = 0 - M{p = c / (N + T)}, otherwise cCsÙ|djp||iƒjp td‚|djo|iƒ}n||_|iiƒ|_||iiƒ|_|iiƒ|_|idjod|i|_n(|it |i|i|iƒ|_dS(s¦ Creates a distribution of Witten-Bell probability estimates. This distribution allocates uniform probability mass to as yet unseen events by using the number of events that have only been seen once. The probability mass reserved for unseen events is equal to: - M{T / (N + T)} where M{T} is the number of observed event types and M{N} is the total number of observed events. This equates to the maximum likelihood estimate of a new type event occuring. The remaining probability mass is discounted such that all probability estimates sum to one, yielding: - M{p = T / Z (N + T)}, if count = 0 - M{p = c / (N + T)}, otherwise The parameters M{T} and M{N} are taken from the C{freqdist} parameter (the C{B()} and C{N()} values). The normalising factor M{Z} is calculated using these values along with the C{bins} parameter. @param freqdist: The frequency counts upon which to base the estimation. @type freqdist: C{FreqDist} @param bins: The number of possible event types. This must be at least as large as the number of bins in the C{freqdist}. If C{None}, then it's assumed to be equal to that of the C{freqdist} @type bins: C{Int} s1Bins parameter must not be less than freqdist.B()igð?N( RRReR‡t_Tt_ZRRt_P0R (R RˆR((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRTs#  cCsA|i|}|djo |iSn|t|i|iƒSdS(Ni(R‡R«R RR©(R R R((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRf‚s   cCs |iiƒS(N(R‡R$(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR$ŠscCs |iiƒS(N(R‡R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR scCs|iS(N(R‡(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRˆscCs tƒ‚dS(N(Rž(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRl“scCsd|iiƒS(sa @rtype: C{string} @return: A string representation of this C{ProbDist}. s((R‡R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\–s( R`RaRbRRRfR$R RˆRlR\(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR¨Bs .     tGoodTuringProbDistcBsPeZdZed„Zd„Zd„Zd„Zd„Zd„Z d„Z RS(sí The Good-Turing estimate of a probability distribution. This method calculates the probability mass to assign to events with zero or low counts based on the number of events with higher counts. It does so by using the smoothed count M{c*}: - M{c* = (c + 1) N(c + 1) / N(c)} where M{c} is the original count, M{N(i)} is the number of event types observed with count M{i}. These smoothed counts are then normalised to yield a probability distribution. cCs]|djp||iƒjp td‚|djo|iƒ}n||_||_dS(s' Creates a Good-Turing probability distribution estimate. This method calculates the probability mass to assign to events with zero or low counts based on the number of events with higher counts. It does so by using the smoothed count M{c*}: - M{c* = (c + 1) N(c + 1) / N(c)} where M{c} is the original count, M{N(i)} is the number of event types observed with count M{i}. These smoothed counts are then normalised to yield a probability distribution. The C{bins} parameter allows C{N(0)} to be estimated. @param freqdist: The frequency counts upon which to base the estimation. @type freqdist: C{FreqDist} @param bins: The number of possible event types. This must be at least as large as the number of bins in the C{freqdist}. If C{None}, then it's taken to be equal to C{freqdist.B()}. @type bins: C{Int} s1Bins parameter must not be less than freqdist.B()N(RRReR‡R‹(R RˆR((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR­s #  cCs|i|}|ii||iƒ}|ii|d|iƒ}|djp|iiƒdjodSnt|dƒ|||iiƒS(Niig(R‡RR‹RR (R R Rtnctncn((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRfÌs  #cCs |iiƒS(N(R‡R$(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR$ØscCs |iiƒS(N(R‡R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR ÛscCs tƒ‚dS(N(Rž(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRlÞscCs|iS(N(R‡(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRˆáscCsd|iiƒS(sa @rtype: C{string} @return: A string representation of this C{ProbDist}. s((R‡R(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\äs( R`RaRbRRRfR$R RlRˆR\(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR¬s      tMutableProbDistcBsAeZdZed„Zd„Zd„Zd„Zed„ZRS(sÕ An mutable probdist where the probabilities may be easily modified. This simply copies an existing probdist, storing the probability values in a mutable dictionary and providing an update method. c síyddk}Wntj odGHtƒnXˆ|_t‡fd†ttˆƒƒDƒƒ|_|itˆƒ|i ƒ|_ xYttˆƒƒD]E}|o|i ˆ|ƒ|i |s(tnumpyR.texitRvRR7Rt _sample_dicttzerostfloat64t_dataRkRft_logs(R t prob_distR t store_logsR°RA((R s&/p/zhu/06/nlp/nltk/nltk/probability.pyRòs  +cCs|iS(N(Rv(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR scCsS|ii|ƒ}|djo,|iod|i|SqO|i|SndSdS(Nig(R²RRR¶Rµ(R R RA((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRfs   cCsa|ii|ƒ}|djo4|io|i|Sq]ti|i|dƒSn tdƒSdS(Nis-inf(R²RRR¶RµRhRiR (R R RA((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRks   cCs’|ii|ƒ}|djpt‚|io5|o||i|Js (RPRcR/R|R(R»Rº((R»Rºs&/p/zhu/06/nlp/nltk/nltk/probability.pytlog_likelihoodEs c Csfg}|iƒD]}||i|ƒq~}tg}|D]}||ti|dƒq>~ƒ S(Ni(R RfR|RhRi(tpdistRR@tprobsR?Rj((s&/p/zhu/06/nlp/nltk/nltk/probability.pytentropyMs0tConditionalFreqDistcBs˜eZdZed„Zd„Zd„Zd„Zd„Zd„Z d„Z d„Z d „Z d „Z d „Zd „Zd „Zd„Zd„ZRS(sø A collection of frequency distributions for a single experiment run under different conditions. Conditional frequency distributions are used to record the number of times each sample occurred, given the condition under which the experiment was run. For example, a conditional frequency distribution could be used to record the frequency of each word (type) in a document, given its length. Formally, a conditional frequency distribution can be defined as a function that maps from each condition to the C{FreqDist} for the experiment under that condition. The frequency distribution for each condition is accessed using the indexing operator: >>> cfdist[3] >>> cfdist[3].freq('the') 0.4 >>> cfdist[3]['dog'] 2 When the indexing operator is used to access the frequency distribution for a condition that has not been accessed before, C{ConditionalFreqDist} creates a new empty C{FreqDist} for that condition. Conditional frequency distributions are typically constructed by repeatedly running an experiment under a variety of conditions, and incrementing the sample outcome counts for the appropriate conditions. For example, the following code will produce a conditional frequency distribution that encodes how often each word type occurs, given the length of that word type: >>> cfdist = ConditionalFreqDist() >>> for word in tokenize.whitespace(sent): ... condition = len(word) ... cfdist[condition].inc(word) An equivalent way to do this is with the initializer: >>> cfdist = ConditionalFreqDist((len(word), word) for word in tokenize.whitespace(sent)) cCs@h|_|o,x)|D]\}}||i|ƒqWndS(sN Construct a new empty conditional frequency distribution. In particular, the count for every sample, under every condition, is zero. @param cond_samples: The samples to initialize the conditional frequency distribution with @type cond_samples: Sequence of (condition, sample) tuples N(t_fdistsR (R t cond_samplestcondR ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRs  cCs/||ijotƒ|i|¾s(R|RÁRJ(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR¸scsðyddk}Wntj otdƒ‚nXt|dtƒ}t|dˆiƒƒ}t|ddƒ}t|dtt‡fd †|Dƒƒƒƒ}d |jod |d ×sR)isCumulative Countss lower rightR&s upper righttlabeltlocR'R(R*iZR,(R-R.R/R1R2RÅRDRUR0RR5tlegendR3R4R6R7RR8R+R9R:R;(R R<R=R-R%RÅR+R RÄR>R:t legend_locRR R?R@((R s&/p/zhu/06/nlp/nltk/nltk/probability.pyR5Às<  %  ) C  c s&t|dtƒ}t|dˆiƒƒ}t|dtt‡fd†|Dƒƒƒƒ}td„|Dƒƒ}d|Gx|D]}dt|ƒGq}WHx‰|D]}d|t|ƒfG|otˆ|i|ƒƒ} n*g} |D]} | ˆ|| qæ~ } x| D]} d | Gq WHqWd S( s\ Tabulate the given samples from the conditional frequency distribution. @param samples: The samples to plot @type samples: C{list} @param title: The title for the graph @type title: C{str} @param conditions: The conditions to plot (default is all) @type conditions: C{list} R%RÅR c3s.x'|] }xˆ|D] }|VqWqWdS(N((RTRR„(R (s&/p/zhu/06/nlp/nltk/nltk/probability.pys scss%x|]}tt|ƒƒVqWdS(N(RR8(RTR((s&/p/zhu/06/nlp/nltk/nltk/probability.pys st s%4ss%*ss%4dN( R1R2RÅRDRUR$R8R0R( R R<R=R%RÅR tcondition_sizeR@RR>RR tf((R s&/p/zhu/06/nlp/nltk/nltk/probability.pyRBñs(  %) csTtˆtƒptSnˆiƒˆiƒjo#t‡‡fd†ˆiƒDƒƒS(Nc3s'x |]}ˆ|ˆ|jVqWdS(N((RTR(RQR (s&/p/zhu/06/nlp/nltk/nltk/probability.pys s(RPRÀR2RÅRW(R RQ((R RQs&/p/zhu/06/nlp/nltk/nltk/probability.pyRRscCs ||j S(N((R RQ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRSscs]tˆtƒptSntˆiƒƒiˆiƒƒo#t‡‡fd†ˆiƒDƒƒS(Nc3s'x |]}ˆ|ˆ|jVqWdS(N((RTR(RQR (s&/p/zhu/06/nlp/nltk/nltk/probability.pys s(RPRÀR2RURÅRVRW(R RQ((R RQs&/p/zhu/06/nlp/nltk/nltk/probability.pyRXs"cCs/t|tƒptSn||jo ||jS(N(RPRÀR2(R RQ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRYscCs"t|tƒptSn||jS(N(RPRÀR2(R RQ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRZscCs"t|tƒptSn||jS(N(RPRÀR2(R RQ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR["scCsd|iƒS(sx @return: A string representation of this C{ConditionalProbDist}. @rtype: C{string} s((RÆ(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\&scCst|iƒ}d|S(sx @return: A string representation of this C{ConditionalFreqDist}. @rtype: C{string} s((RRÁ(R tn((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR\/s(R`RaRbRRR_RÅRÆRR5RBRRRSRXRYRZR[R\(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRÀUs +     1 !       tConditionalProbDistIcBs2eZdZd„Zd„Zd„Zd„ZRS(sn A collection of probability distributions for a single experiment run under different conditions. Conditional probability distributions are used to estimate the likelihood of each sample, given the condition under which the experiment was run. For example, a conditional probability distribution could be used to estimate the probability of each word type in a document, given the length of the word type. Formally, a conditional probability distribution can be defined as a function that maps from each condition to the C{ProbDist} for the experiment under that condition. cCs td‚dS(Ns$ConditionalProbDistI is an interface(Re(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyREscCs t‚dS(s @return: The probability distribution for the experiment run under the given condition. @rtype: C{ProbDistI} @param condition: The condition whose probability distribution should be returned. @type condition: any N(Re(R RÄ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR_Hs cCs t‚dS(s‹ @return: The number of conditions that are represented by this C{ConditionalProbDist}. @rtype: C{int} N(Re(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRÆSscCs t‚dS(sñ @return: A list of the conditions that are represented by this C{ConditionalProbDist}. Use the indexing operator to access the probability distribution for a given condition. @rtype: C{list} N(Re(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRÅ[s(R`RaRbRR_RÆRÅ(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRÑ8s    tConditionalProbDistcBs>eZdZed„Zd„Zd„Zd„Zd„ZRS(sË A conditional probability distribution modelling the experiments that were used to generate a conditional frequency distribution. A C{ConditoinalProbDist} is constructed from a C{ConditionalFreqDist} and a X{C{ProbDist} factory}: - The B{C{ConditionalFreqDist}} specifies the frequency distribution for each condition. - The B{C{ProbDist} factory} is a function that takes a condition's frequency distribution, and returns its probability distribution. A C{ProbDist} class's name (such as C{MLEProbDist} or C{HeldoutProbDist}) can be used to specify that class's constructor. The first argument to the C{ProbDist} factory is the frequency distribution that it should model; and the remaining arguments are specified by the C{factory_args} parameter to the C{ConditionalProbDist} constructor. For example, the following code constructs a C{ConditionalProbDist}, where the probability distribution for each condition is an C{ELEProbDist} with 10 bins: >>> cpdist = ConditionalProbDist(cfdist, ELEProbDist, 10) >>> print cpdist['run'].max() 'NN' >>> print cpdist['run'].prob('NN') 0.0813 cGs‰||_||_||_||_h|_xU|iƒD]G}|o|||||Œ}n||||Œ}||i| probdist N(t_dict(R t probdist_dict((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRÅscCs |i|S(N(RÞ(R RÄ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR_ÍscCs |iiƒS(N(RÞR(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRÅÒs(R`RaRbRR_RÅ(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRÝ¿s  g ÂëþKH´9icCsi||tjo|Sn||tjo|Snt||ƒ}|tid||d||dƒS(s Given two numbers C{logx}=M{log(x)} and C{logy}=M{log(y)}, return M{log(x+y)}. Conceptually, this is the same as returning M{log(2**(C{logx})+2**(C{logy}))}, but the actual implementation avoids overflow errors that could result from direct computation. i(t_ADD_LOGS_MAX_DIFFtminRhRi(tlogxtlogytbase((s&/p/zhu/06/nlp/nltk/nltk/probability.pytadd_logsÝs cCs7t|ƒdjotSntt|d|dƒSdS(Nii(RRgtreduceRå(tlogs((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR{ëstProbabilisticMixIncBs;eZdZd„Zd„Zd„Zd„Zd„ZRS(s A mix-in class to associate probabilities with other classes (trees, rules, etc.). To use the C{ProbabilisticMixIn} class, define a new class that derives from an existing class and from ProbabilisticMixIn. You will need to define a new constructor for the new class, which explicitly calls the constructors of both its parent classes. For example: >>> class A: ... def __init__(self, x, y): self.data = (x,y) ... >>> class ProbabilisticA(A, ProbabilisticMixIn): ... def __init__(self, x, y, **prob_kwarg): ... A.__init__(self, x, y) ... ProbabilisticMixIn.__init__(self, **prob_kwarg) See the documentation for the ProbabilisticMixIn L{constructor<__init__>} for information about the arguments it expects. You should generally also redefine the string representation methods, the comparison methods, and the hashing method. cKs{d|jo5d|jotdƒ‚qwti||dƒn6d|joti||dƒnd|_|_dS(sª Initialize this object's probability. This initializer should be called by subclass constructors. C{prob} should generally be the first argument for those constructors. @kwparam prob: The probability associated with the object. @type prob: C{float} @kwparam logprob: The log of the probability associated with the object. @type logprob: C{float} RfRks.Must specify either prob or logprob (not both)N(t TypeErrorRètset_probt set_logprobRt_ProbabilisticMixIn__probt_ProbabilisticMixIn__logprob(R R=((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRs   cCs||_d|_dS(s“ Set the probability associated with this object to C{prob}. @param prob: The new probability @type prob: C{float} N(RìRRí(R Rf((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRê&s cCst|_d|_dS(sÿ Set the log probability associated with this object to C{logprob}. I.e., set the probability associated with this object to C{2**(logprob)}. @param logprob: The new log probability @type logprob: C{float} N(RfRíRRì(R Rk((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRë/s cCsC|itjo,|itjotSnd|i|_n|iS(s` @return: The probability associated with this object. @rtype: C{float} i(RìRRí(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRf:scCsK|itjo4|itjotSnti|idƒ|_n|iS(s‚ @return: C{log(p)}, where C{p} is the probability associated with this object. @rtype: C{float} i(RíRRìRhRi(R ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRkDs(R`RaRbRRêRëRfRk(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRè÷s   tImmutableProbabilisticMixIncBseZd„Zd„ZRS(cCstd|ii‚dS(Ns%s is immutable(R/RdR`(R Rf((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRêQscCstd|ii‚dS(Ns%s is immutable(R/RdR`(R Rf((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRëSs(R`RaRêRë(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyRîPs cCs,||jo||}||=n|}|S(N((R=REtdefaulttarg((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR1Xs    cCs{ddk}ddkl}tƒ}xOt|ƒD]A}|idd|dƒ|id|dƒ}|i|ƒq2W|S(sÑ Create a new frequency distribution, with random samples. The samples are numbers from 1 to C{numsamples}, and are generated by summing two numbers, each of which has a uniform distribution. iÿÿÿÿN(tsqrtiii(RmRhRñRR7trandintR (t numsamplest numoutcomesRmRñRÇRty((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt_create_rand_fdistds   cCsltƒ}xVtdd|ddƒD]9}x0td|ddƒD]}|i||ƒqCWq%Wt|ƒS(so Return the true probability distribution for the experiment C{_create_rand_fdist(numsamples, x)}. iii(RR7R R†(RóRÇRRõ((s&/p/zhu/06/nlp/nltk/nltk/probability.pyt_create_sum_pdistss iiôcsÕt||ƒ}t||ƒ}t||ƒ}t|ƒt|d|ƒt|||ƒt|||ƒt|||g|ƒt|ƒg}g}xetd|dƒD]P}|it||i |ƒgg}|D]‰|ˆi |ƒqÍ~ƒƒq¡Wd|||fGHdt |ƒdGHddt |ƒdd } | td „|d DƒƒGHdt |ƒdGHd dt |ƒdd} x|D]} | | GHq‰Wt |Œ} d„} g} | dD]} | | | ƒqÀ~ }dt |ƒdGHddt |ƒd} | t|ƒGHd t |ƒdGHt t |ƒ ƒdjo1dGt |ƒGHdGt |ƒGHdGt |ƒGHnHdGHxS|D]K‰t‡fd†tdƒDƒƒ}dˆiid t |ƒd fGHqWHdS(!sQ A demonstration of frequency distributions and probability distributions. This demonstration creates three frequency distributions with, and uses them to sample a random process with C{numsamples} samples. Each frequency distribution is sampled C{numoutcomes} times. These three frequency distributions are then used to build six probability distributions. Finally, the probability estimates of these distributions are compared to the actual probability of each sample. @type numsamples: C{int} @param numsamples: The number of samples to use in each demo frequency distributions. @type numoutcomes: C{int} @param numoutcomes: The total number of outcomes for each demo frequency distribution. These outcomes are divided into C{numsamples} bins. @rtype: C{None} gà?is=%d samples (1-%d); %d outcomes were sampled for each FreqDistt=i is FreqDist s%8s s | Actualcss!x|]}| dd!VqWdS(ii N((RTR½((s&/p/zhu/06/nlp/nltk/nltk/probability.pys ­siÿÿÿÿt-s %3d %8.6f s%8.6f s| %8.6fcSstd„|dƒS(NcSs||S(((RRõ((s&/p/zhu/06/nlp/nltk/nltk/probability.pytµsi(Ræ(tlst((s&/p/zhu/06/nlp/nltk/nltk/probability.pyR|µssTotal iFs fdist1:s fdist2:s fdist3:s Generating:c3sx|]}ˆiƒVqWdS(N(Rr(RTRA(R½(s&/p/zhu/06/nlp/nltk/nltk/probability.pys Åsiˆs%20s %sii7Ns =========s ---------s ---------s =========(RöR†R‰R’RŸR÷R7RœttupleR!RfRtzipR8RRdR`(RóRôR£R¤tfdist3tpdiststvalsRÐRt FORMATSTRtvaltzvalsR|R?tsumsRÇ((R½s&/p/zhu/06/nlp/nltk/nltk/probability.pytdemo~sT 3   +"'t__main__i iiˆRåR¼R{R¿(*R RgRhRmRotoperatorRt itertoolsRRRtobjectRcRsRwR†R‰RR‘R’RŸR¨R¬R¯R¼R¿RÀRÑRÒRÝRiRàRåR{RèRîR1RöR÷RR`t__all__(((s&/p/zhu/06/nlp/nltk/nltk/probability.pyssZ    ÿ§c?+\""›B[NZ  ã1V  Y  K