³ò 4ÒÇIc @sœdZddkZddkZddklZddklZddkTddk l Z ddk Tddk Tde fd„ƒYZd efd „ƒYZdS( sN A reader for corpora that contain chunked (and optionally tagged) documents. iÿÿÿÿN(tBracketParseCorpusReader(tTree(t*(tchunktChunkedCorpusReadercBs¶eZdZdeieddeƒeed„Z ed„Z ed„Z ed„Z ed„Z ed „Zed „Zed „Zed „Zed „Zed„Zd„ZRS(s  Reader for chunked (and optionally tagged) corpora. Paragraphs are split using a block reader. They are then tokenized into sentences using a sentence tokenizer. Finally, these sentences are parsed into chunk trees using a string-to-chunktree conversion function. Each of these steps can be performed using a default function or a custom function. By default, paragraphs are split on blank lines; sentences are listed one per line; and sentences are parsed into chunk trees using L{chunk.tagstr2tree}. ts tgapscCs,ti||||ƒ|||f|_dS(s’ @param root: The root directory for this corpus. @param fileids: A list or regexp specifying the fileids in this corpus. N(t CorpusReadert__init__t_cv_args(tselftroottfileidst extensiont str2chunktreetsent_tokenizertpara_block_readertencoding((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyR#s cCsk|tjo |i}nt|tƒo |g}ntg}|D]}||i|ƒiƒqE~ƒS(sW @return: the given file(s) as a single string. @rtype: C{str} (tNonet_fileidst isinstancet basestringtconcattopentread(R R t_[1]tf((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pytraw2sc CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s† @return: the given file(s) as a list of words and punctuation symbols. @rtype: C{list} of C{str} i(RtabspathstTruetChunkedCorpusViewR (R R RRtenc((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pytwords;s c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(sÁ @return: the given file(s) as a list of sentences or utterances, each encoded as a list of word strings. @rtype: C{list} of (C{list} of C{str}) ii(RRRRR (R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pytsentsDs c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(sò @return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as lists of word strings. @rtype: C{list} of (C{list} of (C{list} of C{str})) ii(RRRRR (R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pytparasNs c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(sÀ @return: the given file(s) as a list of tagged words and punctuation symbols, encoded as tuples C{(word,tag)}. @rtype: C{list} of C{(str,str)} ii(RRRRR (R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt tagged_wordsXs c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s @return: the given file(s) as a list of sentences, each encoded as a list of C{(word,tag)} tuples. @rtype: C{list} of (C{list} of C{(str,str)}) ii(RRRRR (R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt tagged_sentsbs c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s @return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as lists of C{(word,tag)} tuples. @rtype: C{list} of (C{list} of (C{list} of C{(str,str)})) ii(RRRRR (R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt tagged_parasls c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s| @return: the given file(s) as a list of tagged words and chunks. Words are encoded as C{(word, tag)} tuples (if the corpus has tags) or word strings (if the corpus has no tags). Chunks are encoded as depth-one trees over C{(word,tag)} tuples or word strings. @rtype: C{list} of (C{(str,str)} and L{Tree}) ii(RRRRR (R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt chunked_wordsvs c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s@ @return: the given file(s) as a list of sentences, each encoded as a shallow C{Tree}. The leaves of these trees are encoded as C{(word, tag)} tuples (if the corpus has tags) or word strings (if the corpus has no tags). @rtype: C{list} of L{Tree} ii(RRRRR (R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt chunked_sents‚s c CsRtg}|i|tƒD].\}}|t||dddd|iŒq~ƒS(s€ @return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a shallow C{Tree}. The leaves of these trees are encoded as C{(word, tag)} tuples (if the corpus has tags) or word strings (if the corpus has no tags). @rtype: C{list} of (C{list} of L{Tree}) i(RRRRR (R R RRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt chunked_parasŽs cCs.g}t|ƒD]}|ti|ƒq~S(N(tread_blankline_blockRt tagstr2tree(R tstreamRtt((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt _read_blockšs(t__name__t __module__t__doc__RR*tRegexpTokenizerRR)RRRR R!R"R#R$R%R&R'R(R-(((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyRs"  RcBs#eZd„Zd„Zd„ZRS(c CsYti||d|ƒ||_||_||_||_||_||_| |_dS(NR( tStreamBackedCorpusViewRt_taggedt_group_by_sentt_group_by_parat_chunkedt_str2chunktreet_sent_tokenizert_para_block_reader( R tfileidRttaggedt group_by_sentt group_by_paratchunkedRRR((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyRŸs      cCsÝg}xÐ|i|ƒD]¿}g}xˆ|ii|ƒD]t}|i|ƒ}|ip|i|ƒ}n|ip|iƒ}n|io|i |ƒq5|i |ƒq5W|i o|i |ƒq|i |ƒqW|S(N( R9R8ttokenizeR7R3t_untagR6tleavesR4tappendtextendR5(R R+tblocktpara_strtparatsent_strtsent((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/chunked.pyt read_block«s$    cCspxit|ƒD][\}}t|tƒo|i|ƒq t|tƒo|d|| s   †