³ò B_Kc @s˜dZddkZddkZddkZddklZddklZddk Tddk Tddk Tde fd„ƒYZ defd „ƒYZdS( sN A reader for corpora that contain chunked (and optionally tagged) documents. iÿÿÿÿN(tBracketParseCorpusReader(tTree(t*tChunkedCorpusReadercBs¹eZdZdeiieddeƒee d„Z e d„Z e d„Z e d„Z e d„Ze d „Ze d „Ze d „Ze d „Ze d „Ze d„Zd„ZRS(s% Reader for chunked (and optionally tagged) corpora. Paragraphs are split using a block reader. They are then tokenized into sentences using a sentence tokenizer. Finally, these sentences are parsed into chunk trees using a string-to-chunktree conversion function. Each of these steps can be performed using a default function or a custom function. By default, paragraphs are split on blank lines; sentences are listed one per line; and sentences are parsed into chunk trees using L{nltk.chunk.tagstr2tree}. ts tgapscCs,ti||||ƒ|||f|_dS(s’ @param root: The root directory for this corpus. @param fileids: A list or regexp specifying the fileids in this corpus. N(t CorpusReadert__init__t_cv_args(tselftroottfileidst extensiont str2chunktreetsent_tokenizertpara_block_readertencoding((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyR"s cCsk|tjo |i}nt|tƒo |g}ntg}|D]}||i|ƒiƒqE~ƒS(sW @return: the given file(s) as a single string. @rtype: C{str} (tNonet_fileidst isinstancet basestringtconcattopentread(R R t_[1]tf((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pytraw1sc CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s† @return: the given file(s) as a list of words and punctuation symbols. @rtype: C{list} of C{str} i(RtabspathstTruetChunkedCorpusViewR(R R RRtenc((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pytwords:s c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(sÁ @return: the given file(s) as a list of sentences or utterances, each encoded as a list of word strings. @rtype: C{list} of (C{list} of C{str}) ii(RRRRR(R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pytsentsCs c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(sò @return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as lists of word strings. @rtype: C{list} of (C{list} of (C{list} of C{str})) ii(RRRRR(R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pytparasMs c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(sÀ @return: the given file(s) as a list of tagged words and punctuation symbols, encoded as tuples C{(word,tag)}. @rtype: C{list} of C{(str,str)} ii(RRRRR(R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt tagged_wordsWs c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s @return: the given file(s) as a list of sentences, each encoded as a list of C{(word,tag)} tuples. @rtype: C{list} of (C{list} of C{(str,str)}) ii(RRRRR(R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt tagged_sentsas c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s @return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as lists of C{(word,tag)} tuples. @rtype: C{list} of (C{list} of (C{list} of C{(str,str)})) ii(RRRRR(R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt tagged_parasks c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s| @return: the given file(s) as a list of tagged words and chunks. Words are encoded as C{(word, tag)} tuples (if the corpus has tags) or word strings (if the corpus has no tags). Chunks are encoded as depth-one trees over C{(word,tag)} tuples or word strings. @rtype: C{list} of (C{(str,str)} and L{Tree}) ii(RRRRR(R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt chunked_wordsus c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s@ @return: the given file(s) as a list of sentences, each encoded as a shallow C{Tree}. The leaves of these trees are encoded as C{(word, tag)} tuples (if the corpus has tags) or word strings (if the corpus has no tags). @rtype: C{list} of L{Tree} ii(RRRRR(R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt chunked_sentss c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s€ @return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a shallow C{Tree}. The leaves of these trees are encoded as C{(word, tag)} tuples (if the corpus has tags) or word strings (if the corpus has no tags). @rtype: C{list} of (C{list} of L{Tree}) i(RRRRR(R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt chunked_parass cCs1g}t|ƒD]}|tii|ƒq~S(N(tread_blankline_blocktnltktchunkt tagstr2tree(R tstreamRtt((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt _read_block™s(t__name__t __module__t__doc__R)R*R+tRegexpTokenizerRR(RRRRR R!R"R#R$R%R&R'R.(((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyRs"   RcBs#eZd„Zd„Zd„ZRS(c CsYti||d|ƒ||_||_||_||_||_||_| |_dS(NR( tStreamBackedCorpusViewRt_taggedt_group_by_sentt_group_by_parat_chunkedt_str2chunktreet_sent_tokenizert_para_block_reader( R tfileidRttaggedt group_by_sentt group_by_paratchunkedR RR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyRžs      cCsÝg}xÐ|i|ƒD]¿}g}xˆ|ii|ƒD]t}|i|ƒ}|ip|i|ƒ}n|ip|iƒ}n|io|i |ƒq5|i |ƒq5W|i o|i |ƒq|i |ƒqW|S(N( R:R9ttokenizeR8R4t_untagR7tleavesR5tappendtextendR6(R R,tblocktpara_strtparatsent_strtsent((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt read_blockªs$    cCspxit|ƒD][\}}t|tƒo|i|ƒq t|tƒo|d|| s    †