³ò 4ÒÇIc@sxdZddkTdefd„ƒYZdefd„ƒYZdefd„ƒYZd efd „ƒYZd d „Zd S(sÎ Tokenizers that divide strings into substrings using the string C{split()} method. These tokenizers follow the standard L{TokenizerI} interface, and so can be used with any code that expects a tokenizer. For example, these tokenizers can be used to specify the tokenization conventions when building a L{CorpusReader}. But if you are tokenizing a string yourself, consider using string C{split()} method directly instead. iÿÿÿÿ(t*tWhitespaceTokenizercBseZdZd„ZRS(s¯ A tokenizer that divides a string into substrings by treating any sequence of whitespace characters as a separator. Whitespace characters are space (C{' '}), tab (C{'\t'}), and newline (C{'\n'}). If you are performing the tokenization yourself (rather than building a tokenizer to pass to some other piece of code), consider using the string C{split()} method instead: >>> words = s.split() cCs |iƒS(N(tsplit(tselfts((s*/p/zhu/06/nlp/nltk/nltk/tokenize/simple.pyttokenize#s(t__name__t __module__t__doc__R(((s*/p/zhu/06/nlp/nltk/nltk/tokenize/simple.pyRs tSpaceTokenizercBseZdZd„ZRS(sR A tokenizer that divides a string into substrings by treating any single space character as a separator. If you are performing the tokenization yourself (rather than building a tokenizer to pass to some other piece of code), consider using the string C{split()} method instead: >>> words = s.split(' ') cCs |idƒS(Nt (R(RR((s*/p/zhu/06/nlp/nltk/nltk/tokenize/simple.pyR0s(RRRR(((s*/p/zhu/06/nlp/nltk/nltk/tokenize/simple.pyR &s t TabTokenizercBseZdZd„ZRS(sQ A tokenizer that divides a string into substrings by treating any single tab character as a separator. If you are performing the tokenization yourself (rather than building a tokenizer to pass to some other piece of code), consider using the string C{split()} method instead: >>> words = s.split('\t') cCs |idƒS(Ns (R(RR((s*/p/zhu/06/nlp/nltk/nltk/tokenize/simple.pyR=s(RRRR(((s*/p/zhu/06/nlp/nltk/nltk/tokenize/simple.pyR 3s t LineTokenizercBs#eZdZdd„Zd„ZRS(sÆ A tokenizer that divides a string into substrings by treating any single newline character as a separator. Handling of blank lines may be controlled using a constructor parameter. tdiscardcCs=d}||jotddi|ƒƒ‚n||_dS(s  @param blanklines: Indicates how blank lines should be handled. Valid values are: - C{'discard'}: strip blank lines out of the token list before returning it. A line is considered blank if it contains only whitespace characters. - C{'keep'}: leave all blank lines in the token list. - C{'discard-eof'}: if the string ends with a newline, then do not generate a corresponding token C{''} after that newline. R tkeeps discard-eofsBlank lines must be one of: %sR N(sdiscardRs discard-eof(t ValueErrortjoint _blanklines(Rt blanklinestvalid_blanklines((s*/p/zhu/06/nlp/nltk/nltk/tokenize/simple.pyt__init__Fs  cCs”|idƒ}|idjo6g}|D]}|iƒo ||q*q*~}n<|idjo+|o |diƒ o|iƒqn|S(Ns R s discard-eofiÿÿÿÿ(RRtrstriptstriptpop(RRtlinest_[1]tl((s*/p/zhu/06/nlp/nltk/nltk/tokenize/simple.pyRZs 6+(RRRRR(((s*/p/zhu/06/nlp/nltk/nltk/tokenize/simple.pyR @s R cCst|ƒi|ƒS(N(R R(ttextR((s*/p/zhu/06/nlp/nltk/nltk/tokenize/simple.pyt line_tokenizegsN(Rtapit TokenizerIRR R R R(((s*/p/zhu/06/nlp/nltk/nltk/tokenize/simple.pyss    '