³ò 4ÒÇIc@sØdZddkZddkZddklZlZddkTdefd„ƒYZdefd„ƒYZ d efd „ƒYZ d eefd „ƒYZ e e eieiBeiBd „Ze ƒiZe ƒiZdS(s… Tokenizers that divide strings into substrings using regular expressions that can match either tokens or separators between tokens. iÿÿÿÿN(tconvert_regexp_to_nongroupingt Deprecated(t*tRegexpTokenizercBsCeZdZeeeieiBeiBd„Z d„Z d„Z RS(sV A tokenizer that splits a string into substrings using a regular expression. The regular expression can be specified to match either tokens or separators between tokens. Unlike C{re.findall()} and C{re.split()}, C{RegexpTokenizer} does not treat regular expressions that contain grouping parenthases specially. cCs™t|d|ƒ}||_||_||_||_d|_t|ƒ}yti ||ƒ|_Wn.ti j o}t d||fƒ‚nXdS(s‰ Construct a new tokenizer that splits strings using the given regular expression C{pattern}. By default, C{pattern} will be used to find tokens; but if C{gaps} is set to C{False}, then C{patterns} will be used to find separators between tokens instead. @type pattern: C{str} @param pattern: The pattern used to build this tokenizer. This pattern may safely contain grouping parenthases. @type gaps: C{bool} @param gaps: True if this tokenizer's pattern should be used to find separators between tokens; False if this tokenizer's pattern should be used to find the tokens themselves. @type discard_empty: C{bool} @param discard_empty: True if any empty tokens (C{''}) generated by the tokenizer should be discarded. Empty tokens can only be generated if L{_gaps} is true. @type flags: C{int} @param flags: The regexp flags used to compile this tokenizer's pattern. By default, the following flags are used: C{re.UNICODE | re.MULTILINE | re.DOTALL}. tpatterns"Error in regular expression %r: %sN( tgetattrt_patternt_gapst_discard_emptyt_flagstNonet_regexpRtretcompileterrort ValueError(tselfRtgapst discard_emptytflagstnongrouping_patternte((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pyt__init__ s      cCsv|ioX|io:g}|ii|ƒD]}|o ||q+q+~Sqr|ii|ƒSn|ii|ƒSdS(N(RRR tsplittfindall(Rttextt_[1]ttok((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pyttokenize[s   :cCs)d|ii|i|i|i|ifS(Ns3%s(pattern=%r, gaps=%r, discard_empty=%r, flags=%r)(t __class__t__name__RRRR (R((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pyt__repr__gs( Rt __module__t__doc__tFalsetTrueR tUNICODEt MULTILINEtDOTALLRRR(((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pyRs  : tBlanklineTokenizercBseZdZd„ZRS(sù A tokenizer that divides a string into substrings by treating any sequence of blank lines as a separator. Blank lines are defined as lines containing no characters, or containing only space (C{' '}) or tab (C{' '}) characters. cCsti|ddtƒdS(Ns \s*\n\s*\n\s*R(RRR#(R((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pyRss(RR R!R(((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pyR'lstWordPunctTokenizercBseZdZd„ZRS(sÜ A tokenizer that divides a text into sequences of alphabetic and non-alphabetic characters. E.g.: >>> WordPunctTokenizer().tokenize("She said 'hello'.") ['She', 'said', "'", 'hello', "'."] cCsti|dƒdS(Ns \w+|[^\w\s]+(RR(R((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pyR~s(RR R!R(((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pyR(vst WordTokenizercBseZdZd„ZRS(s` B{If you want to tokenize words, you should probably use TreebankWordTokenizer or word_tokenize() instead.} A tokenizer that divides a text into sequences of alphabetic characters. Any non-alphabetic characters are discarded. E.g.: >>> WordTokenizer().tokenize("She said 'hello'.") ['She', 'said', 'hello'] cCsti|dƒdS(Ns\w+(RR(R((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pyRŒs(RR R!R(((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pyR)s cCs"t||||ƒ}|i|ƒS(s¹ Split the given text string, based on the given regular expression pattern. See the documentation for L{RegexpTokenizer.tokenize()} for descriptions of the arguments. (RR(RRRRRt tokenizer((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pytregexp_tokenize“s(R!R t sre_constantstnltk.internalsRRtapit TokenizerIRR'R(R)R"R#R$R%R&R+Rtblankline_tokenizetwordpunct_tokenize(((s*/p/zhu/06/nlp/nltk/nltk/tokenize/regexp.pys s   V