³ò 4ÒÇIc@s0dZddklZdefd„ƒYZdS(s Tokenizer Interface iÿÿÿÿ(t overriddent TokenizerIcBs eZdZd„Zd„ZRS(sÈ A procesing interface for I{tokenizing} a string, or dividing it into a list of substrings. Subclasses must define: - either L{tokenize()} or L{batch_tokenize()} (or both) cCs5t|iƒo|i|gƒdSn tƒ‚dS(sp Divide the given string into a list of substrings. @return: C{list} of C{str} iN(Rtbatch_tokenizetNotImplementedError(tselfts((s'/p/zhu/06/nlp/nltk/nltk/tokenize/api.pyttokenizescCs(g}|D]}||i|ƒq ~S(sº Apply L{self.tokenize()} to each element of C{strings}. I.e.: >>> return [self.tokenize(s) for s in strings] @rtype: C{list} of C{list} of C{str} (R(Rtstringst_[1]R((s'/p/zhu/06/nlp/nltk/nltk/tokenize/api.pyR s(t__name__t __module__t__doc__RR(((s'/p/zhu/06/nlp/nltk/nltk/tokenize/api.pyR s N(R tnltk.internalsRtobjectR(((s'/p/zhu/06/nlp/nltk/nltk/tokenize/api.pys s