³ò RÔjKc@sådZddkTddkTddkTddkTddkTddkZdddddd d d d d ddddddddddgZyddkZWne j onXddk Tedg7Zd„Z e ƒi Zd„ZdS(sK Functions for X{tokenizing}, i.e., dividing text strings into substrings. iÿÿÿÿ(t*NtWhitespaceTokenizertSpaceTokenizert TabTokenizert LineTokenizertRegexpTokenizertBlanklineTokenizertWordPunctTokenizert WordTokenizertblankline_tokenizetwordpunct_tokenizetregexp_tokenizet word_tokenizetSExprTokenizertsexpr_tokenizet line_tokenizetPunktWordTokenizertPunktSentenceTokenizertTreebankWordTokenizert sent_tokenizetTextTilingTokenizercCstiidƒ}|i|ƒS(sŸ Use NLTK's currently recommended sentence tokenizer to tokenize sentences in the given text. Currently, this uses L{PunktSentenceTokenizer}. stokenizers/punkt/english.pickle(tnltktdatatloadttokenize(ttextt tokenizer((s,/p/zhu/06/nlp/nltk/nltk/tokenize/__init__.pyR%scCs t|ƒS(sÙ Use NLTK's currently recommended word tokenizer to tokenize words in the given sentence. Currently, this uses L{TreebankWordTokenizer}. This tokenizer should be fed a single sentence at a time. (t_word_tokenize(R((s,/p/zhu/06/nlp/nltk/nltk/tokenize/__init__.pyR 0s(t__doc__tsimpletregexptpunkttsexprttreebankRt__all__tnumpyt ImportErrort texttilingRRRRR (((s,/p/zhu/06/nlp/nltk/nltk/tokenize/__init__.pys s(