³ò 3ÒÇIc@sLddklZddkZddklZlZddklZddkl Z ddk l Z l Z l Z ddklZddklZdd klZlZd efd „ƒYZd efd „ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZd„Zedjo eƒnd d dddgZdS(iÿÿÿÿ(tlogN(tFreqDisttLidstoneProbDist(tConditionalFreqDist(t defaultdict(tngramst tokenwraptLazyConcatenation(t NgramModel(t f_measure(tBigramCollocationFindertbigram_measurest ContextIndexcBs\eZdZed„ƒZeed„d„Zd„Zd„Zdd„Z e d„Z RS( s A bidrectional index between words and their 'contexts' in a text. The context of a word is usually defined to be the words that occur in a fixed window around the word; but other definitions may also be used by providing a custom context function. cCsj|djo d}n||diƒ}|t|ƒdjo d}n||diƒ}||fS(s;One left token and one right token, normalized to lowercaseis*START*is*END*(tlowertlen(ttokenstitlefttright((s/p/zhu/06/nlp/nltk/nltk/text.pyt_default_contexts !cCs|S(((tx((s/p/zhu/06/nlp/nltk/nltk/text.pyt$scs±|ˆ_|pˆiˆ_n|o6g}ˆD]}||ƒo ||q2q2~‰nt‡‡fd†tˆƒDƒƒˆ_t‡‡fd†tˆƒDƒƒˆ_dS(Nc3s:x3|],\}}ˆi|ƒˆiˆ|ƒfVqWdS(N(t_keyt _context_func(t.0Rtw(tselfR(s/p/zhu/06/nlp/nltk/nltk/text.pys *s c3s:x3|],\}}ˆiˆ|ƒˆi|ƒfVqWdS(N(RR(RRR(RR(s/p/zhu/06/nlp/nltk/nltk/text.pys ,s (RRRtCFDt enumeratet_word_to_contextst_context_to_words(RRt context_functfiltertkeyt_[1]tt((RRs/p/zhu/06/nlp/nltk/nltk/text.pyt__init__$s 6cCs|iS(s€ @rtype: C{list} of token @return: The document that this context index was created from. (t_tokens(R((s/p/zhu/06/nlp/nltk/nltk/text.pyR/scCse|i|ƒ}t|i|ƒ}h}x6|iiƒD]%\}}t|t|ƒƒ||esN( RR&RtrangeRtreducet intersectiont ValueErrortjoinR( Rtwordstfail_on_unknownR"Rt_[2]tcontextst_[3]Rtemptytfd((RR4s/p/zhu/06/nlp/nltk/nltk/text.pytcommon_contextsOs *.@ ( t__name__t __module__t__doc__t staticmethodRtNoneR$RR,R2tFalseRA(((s/p/zhu/06/nlp/nltk/nltk/text.pyR s    tConcordanceIndexcBsGeZdZd„d„Zd„Zd„Zd„Zddd„ZRS( ss An index that can be used to look up the offset locations at which a given word occurs in a document. cCs|S(((R((s/p/zhu/06/nlp/nltk/nltk/text.pyRoscCse||_||_ttƒ|_x=t|ƒD]/\}}|i|ƒ}|i|i|ƒq.WdS(sÞ Construct a new concordance index. @param tokens: The document (list of tokens) that this concordance index was created from. This list can be used to access the context of a given word occurance. @param key: A function that maps each token to a normalized version that will be used as a key in the index. E.g., if you use C{key=str.lower}, then the index will be case-insensitive. N(R%RRtlistt_offsetsRtappend(RRR!tindexR(((s/p/zhu/06/nlp/nltk/nltk/text.pyR$os   cCs|iS(s„ @rtype: C{list} of token @return: The document that this concordance index was created from. (R%(R((s/p/zhu/06/nlp/nltk/nltk/text.pyR‹scCs|i|ƒ}|i|S(sì @rtype: C{list} of C{int} @return: A list of the offset positions at which the given word occurs. If a key function was specified for the index, then given word's key will be looked up. (RRJ(RR(((s/p/zhu/06/nlp/nltk/nltk/text.pytoffsets“scCs dt|iƒt|iƒfS(Ns+(RR%RJ(R((s/p/zhu/06/nlp/nltk/nltk/text.pyt__repr__siKic Cs|t|ƒdd}|d}|i|ƒ}|oÃt|t|ƒƒ}d|t|ƒfGHx›|D]Š}d|di|i|||!ƒ}di|i|d||!ƒ} || }| | } |G|G| GH|d8}|djoPqiqiWndGHdS( s` Print a concordance for C{word} with the specified context window. @param word: The target word @type word: C{str} @param width: The width of each line, in characters (default=80) @type width: C{int} @param lines: The number of lines to display (default=25) @type lines: C{int} iisDisplaying %s of %s matches:R3iis No matchesN(RRMtminR9R%( RR(twidthtlinest half_widthtcontextRMRRR((s/p/zhu/06/nlp/nltk/nltk/text.pytprint_concordance¡s$  !      (RBRCRDR$RRMRNRT(((s/p/zhu/06/nlp/nltk/nltk/text.pyRHjs   t TokenSearchercBs eZdZd„Zd„ZRS(sÝ A class that makes it easier to use regular expressions to search over tokenized strings. The tokenized string is converted to a string where tokens are marked with angle brackets -- e.g., C{''}. The regular expression passed to the L{findall()} method is modified to treat angle brackets as nongrouping parentheses, in addition to matching the token boundaries; and to have C{'.'} not match the angle brackets. cCs did„|Dƒƒ|_dS(Ntcss!x|]}d|dVqWdS(tN((RR((s/p/zhu/06/nlp/nltk/nltk/text.pys Ës(R9t_raw(RR((s/p/zhu/06/nlp/nltk/nltk/text.pyR$ÊscCsàtidd|ƒ}tidd|ƒ}tidd|ƒ}tidd|ƒ}ti||iƒ}x?|D]7}|idƒ o |idƒotd ƒ‚qpqpWg}|D]}||d d !id ƒq¶~}|S( s¯ Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> ts.findall("<.*><.*>") ['you rule bro', ['telling you bro; u twizted bro >>> ts.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the @param regexp: A regular expression @type regexp: C{str} s\sRVRWs(?:<(?:RXs)>)s (?]s$Bad regexp for TokenSearcher.findalliiÿÿÿÿs><(tretsubtfindallRYt startswithtendswithR8tsplit(RtregexpthitsthR"((s/p/zhu/06/nlp/nltk/nltk/text.pyR\Ís!1(RBRCRDR$R\(((s/p/zhu/06/nlp/nltk/nltk/text.pyRUÀs  tTextcBsÚeZdZeZed„Zd„Zd„Zddd„Z dd„Z d „Z d „Z d „Z d d „Zd„Zdd„Zdd„Zd„Zd„Zd„Zd„ZeidƒZd„Zd„ZRS(s A wrapper around a sequence of simple (string) tokens, which is intened to support initial exploration of texts (via the interactive console). Its methods perform a variety of analyses on the text's contexts (e.g., counting, concordancing, collocation discovery), and display the results. If you wish to write a program which makes use of these analyses, then you should bypass the C{Text} class, and use the appropriate analysis function or class directly instead. C{Text}s are typically initialized from a given document or corpus. E.g.: >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt')) cCs¨|iot|ƒ}n||_|o ||_nnd|d jo9|d idƒ}ditt|d|!ƒƒ|_n$ditt|d ƒƒd|_dS(s„ Create a Text object. @param tokens: The source text. @type tokens: C{sequence} of C{str} t]iR3iis...N(t _COPY_TOKENSRIRtnameRLR9tmaptstr(RRRftend((s/p/zhu/06/nlp/nltk/nltk/text.pyR$ s   &cCs7t|tƒo|i|i|i!Sn |i|SdS(N(t isinstancetsliceRtstarttstop(RR((s/p/zhu/06/nlp/nltk/nltk/text.pyt __getitem__!scCs t|iƒS(N(RR(R((s/p/zhu/06/nlp/nltk/nltk/text.pyt__len__'siOicCsNd|ijo$dGHt|idtiƒ|_n|ii|||ƒdS(sz Print a concordance for C{word} with the specified context window. @seealso: L{ConcordanceIndex} t_concordance_indexsBuilding index...R!N(t__dict__RHRRhR RpRT(RR(RPRQ((s/p/zhu/06/nlp/nltk/nltk/text.pyt concordance.s  icsÌd|ijoudGHddkl}|idƒ‰ti|iƒ}|idƒ|i‡fd†ƒ|i t i |ƒ|_ ng}|i D]\}}||d|q“~}t |d d ƒGHd S( sb Print collocations derived from the text. @seealso: L{find_collocaitons} t _collocationssBuilding collocations listiÿÿÿÿ(t stopwordstenglishics#t|ƒdjp|iƒˆjS(i(RR (R(t ignored_words(s/p/zhu/06/nlp/nltk/nltk/text.pyREsR3t separators; N(Rqt nltk.corpusRtR:R t from_wordsRtapply_freq_filtertapply_word_filtertnbestR tlikelihood_ratioRsR(RtnumRttfinderR"tw1tw2tcolloc_strings((Rvs/p/zhu/06/nlp/nltk/nltk/text.pyt collocations:s 2cCs|ii|ƒS(sJ Count the number of times this word appears in the text. (Rtcount(RR(((s/p/zhu/06/nlp/nltk/nltk/text.pyR„JscCs|ii|ƒS(sQ Find the index of the first occurrence of the word in the text. (RRL(RR(((s/p/zhu/06/nlp/nltk/nltk/text.pyRLPscCs t‚dS(N(tNotImplementedError(Rtmethod((s/p/zhu/06/nlp/nltk/nltk/text.pyt readabilityVsidcCsXd|ijo'dGHd„}td||ƒ|_n|ii|ƒ}t|ƒGHdS(sØ Print random text, generated using a trigram language model. @param length: The length of text to generate (default=100) @type length: C{int} @seealso: L{NgramModel} t_trigram_modelsBuilding ngram index...cSs t|dƒS(gš™™™™™É?(R(tfdisttbins((s/p/zhu/06/nlp/nltk/nltk/text.pyRdsiN(RqRRˆtgenerateR(Rtlengtht estimatorttext((s/p/zhu/06/nlp/nltk/nltk/text.pyR‹Zs  cCs?d|ijodGHt|iƒ|_n|ii|ƒdS(s Search for instances of the regular expression pattern in the text. @seealso: L{TokenSearcher} t_token_searchersLoading data...N(RqRURRR\(Rtpattern((s/p/zhu/06/nlp/nltk/nltk/text.pytsearchiscsÈd|ijo-dGHt|idd„dtiƒ|_nˆiƒ‰|ii‰ˆˆiƒjoWtˆˆƒ‰t ‡‡‡fd†ˆiƒDƒƒ}|i ƒ| }t |ƒGHndGHdS( s Distributional similarity: find other words which appear in the same contexts as the specified word; list most similar words first. @param word: The word used to seed the similarity search @type word: C{str} @param num: The number of words to generate (default=20) @type num: C{int} @seealso: L{ContextIndex.similar_words()} t_word_context_indexsBuilding word-context index...R cSs |iƒS((tisalpha(R((s/p/zhu/06/nlp/nltk/nltk/text.pyR‚sR!c3sMxF|]?}x6ˆ|D]*}|ˆjo|ˆj o |VqqWqWdS(N((RRR1(R(R=twci(s/p/zhu/06/nlp/nltk/nltk/text.pys Šss No matchesN( RqR RRhR R’Rt conditionsR&RtkeysR(RR(R~R@R:((R(R=R”s/p/zhu/06/nlp/nltk/nltk/text.pytsimilarus (  (cCs£d|ijo$dGHt|idtiƒ|_nyN|ii|tƒ}|p dGHn&|iƒ| }t d„|DƒƒGHWnt j o}|GHnXdS(sh Find contexts where the specified words appear; list most frequent common contexts first. @param word: The word used to seed the similarity search @type word: C{str} @param num: The number of words to generate (default=20) @type num: C{int} @seealso: L{ContextIndex.common_contexts()} R’sBuilding word-context index...R!sNo common contexts were foundcss'x |]\}}|d|VqWdS(t_N((RR€R((s/p/zhu/06/nlp/nltk/nltk/text.pys §sN( RqR RRhR R’RAtTrueR–RR8(RR:R~R@tranked_contextste((s/p/zhu/06/nlp/nltk/nltk/text.pyRA’s  cCs!ddkl}|||ƒdS(s Produce a plot showing the distribution of the words through the text. Requires pylab to be installed. @param words: The words to be plotted @type word: C{str} @seealso: L{nltk.draw.dispersion_plot()} iÿÿÿÿ(tdispersion_plotN(t nltk.drawRœ(RR:Rœ((s/p/zhu/06/nlp/nltk/nltk/text.pyRœ¬s cGs|iƒi|ŒdS(sf See documentation for FreqDist.plot() @seealso: L{nltk.prob.FreqDist.plot()} N(tvocabtplot(Rtargs((s/p/zhu/06/nlp/nltk/nltk/text.pyRŸ¸scCs/d|ijodGHt|ƒ|_n|iS(s1 @seealso: L{nltk.prob.FreqDist} t_vocabsBuilding vocabulary index...(RqRR¡(R((s/p/zhu/06/nlp/nltk/nltk/text.pyRž¿scCsqd|ijot|ƒ|_n|ii|ƒ}g}|D]}|di|ƒq@~}t|dƒGHdS(s° Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the @param regexp: A regular expression @type regexp: C{str} RR3s; N(RqRURR\R9R(RR`RaR"Rb((s/p/zhu/06/nlp/nltk/nltk/text.pyR\Ès *s \w+|[\.\!\?]cCsÚ|d}x4|djo&|ii||ƒ o|d}q W|djo d}n ||}|d}x:|t|ƒjo&|ii||ƒ o|d}qoW|t|ƒjo d}n ||}||fS(sÕ One left & one right token, both case-normalied. Skip over non-sentence-final punctuation. Used by the L{ContextIndex} that is created for L{similar()} and L{common_contexts()}. iis*START*s*END*(t _CONTEXT_REtmatchR(RRRtjRR((s/p/zhu/06/nlp/nltk/nltk/text.pyt_contextês (  . cCs d|iS(s^ @return: A string representation of this C{FreqDist}. @rtype: string s (Rf(R((s/p/zhu/06/nlp/nltk/nltk/text.pyRNs(RBRCRDR™ReRFR$RnRoRrRƒR„RLR‡R‹R‘R—RARœRŸRžR\RZtcompileR¢R¥RN(((s/p/zhu/06/nlp/nltk/nltk/text.pyRcôs*            ! tTextCollectioncBs;eZdZed„Zed„Zed„Zd„ZRS(sÉA collection of texts, which can be loaded with list of texts, or with a corpus consisting of one or more texts, and which supports counting, concordancing, collocation discovery, etc. Initialize a TextCollection as follows: >>> gutenberg = TextCollection(nltk.corpus.gutenberg) >>> mytexts = TextCollection([text1, text2, text3]) Iterating over a TextCollection produces all the tokens of all the texts in order. cCsgt|dƒo4g}|iƒD]}||i|ƒq!~}n||_ti|t|ƒƒdS(NR:(thasattrtfilesR:t_textsRcR$R(RtsourceRfR"tf((s/p/zhu/06/nlp/nltk/nltk/text.pyR$s4 cCst|i|ƒƒt|ƒS(N(tfloatR„R(RttermRŽR†((s/p/zhu/06/nlp/nltk/nltk/text.pyttfscs0t‡fd†|iDƒƒtt|iƒƒS(Nc3s*x#|]}ˆ|jo tVqqWdS(N(R™(RRŽ(R®(s/p/zhu/06/nlp/nltk/nltk/text.pys "s(RRªR­(RR®R†((R®s/p/zhu/06/nlp/nltk/nltk/text.pytdf!scCs#|i||ƒt|i|ƒƒS(N(R¯RR°(RR®RŽ((s/p/zhu/06/nlp/nltk/nltk/text.pyttf_idf%s(RBRCRDRFR$R¯R°R±(((s/p/zhu/06/nlp/nltk/nltk/text.pyR§ s     cCsßddkl}t|iddƒƒ}|GHHdGH|idƒHdGH|idƒHdGH|iƒHdGH|iƒHd GH|idd d d gƒHd GH|i dƒHdGHdG|dGHdG|dd!GHdG|i ƒdGHdS(Niÿÿÿÿ(tbrownt categoriestnewss Concordance:sDistributionally similar words:s Collocations:sAutomatically generated text:sDispersion plot:treporttsaidt announcedsVocabulary plot:i2s Indexing:stext[3]:is text[3:5]:istext.vocab()['news']:( RxR²RcR:RrR—RƒR‹RœRŸRž(R²RŽ((s/p/zhu/06/nlp/nltk/nltk/text.pytdemo(s4      t__main__(tmathRRZtnltk.probabilityRRRRt nltk.compatRt nltk.utilRRRt nltk.modelRt nltk.metricsR tnltk.collocationsR R tobjectR RHRURcR§R¸RBt__all__(((s/p/zhu/06/nlp/nltk/nltk/text.pyss, VV4ÿ