³ò B_Kc@sšdZdZddkZddkiiZddkTddkTddk Tde fd„ƒYZ dd„Z defd „ƒYZd efd „ƒYZdS( sC Corpus reader for the XML version of the British National Corpus. s epytext eniÿÿÿÿN(t*tBNCCorpusReadercBsqeZdZed„Zeeed„Zeeeed„Zeeed„Z eeeed„Z d„Z RS(s Corpus reader for the XML version of the British National Corpus. For access to the complete XML data structure, use the L{xml()} method. For access to simple word lists and tagged word lists, use L{words()}, L{sents()}, L{tagged_words()}, and L{tagged_sents()}. cCs ti|||ƒ||_dS(N(tXMLCorpusReadert__init__t_lazy(tselftroottfileidstlazy((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyRsc Cs•|ioDtg}|i|ƒD]}|t|td||ƒq!~ƒSnDtg}|i|ƒD]"}||i|td||ƒqe~ƒSdS(sd @return: the given file(s) as a list of words and punctuation symbols. @rtype: C{list} of C{str} @param strip_space: If true, then strip trailing spaces from word tokens. Otherwise, leave the spaces on the tokens. @param stem: If true, then use word stems instead of word strings. N(Rtconcattabspathst BNCWordViewtFalsetNonet_words(RRt strip_spacetstemt_[1]tfileidt_[2]((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pytwords s  : c Cs¬|o d}nd}|ioDtg}|i|ƒD]}|t|t|||ƒq8~ƒSnDtg}|i|ƒD]"}||i|t|||ƒq|~ƒSdS(s* @return: the given file(s) as a list of tagged words and punctuation symbols, encoded as tuples C{(word,tag)}. @rtype: C{list} of C{(str,str)} @param c5: If true, then the tags used will be the more detailed c5 tags. Otherwise, the simplified tags will be used. @param strip_space: If true, then strip trailing spaces from word tokens. Otherwise, leave the spaces on the tokens. @param stem: If true, then use word stems instead of word strings. tc5tposN(RR R R R R( RRRRRttagRRR((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyt tagged_words3s   : c Cs•|ioDtg}|i|ƒD]}|t|td||ƒq!~ƒSnDtg}|i|ƒD]"}||i|td||ƒqe~ƒSdS(sŸ @return: the given file(s) as a list of sentences or utterances, each encoded as a list of word strings. @rtype: C{list} of (C{list} of C{str}) @param strip_space: If true, then strip trailing spaces from word tokens. Otherwise, leave the spaces on the tokens. @param stem: If true, then use word stems instead of word strings. N(RR R R tTrueR R(RRRRRRR((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pytsentsIs  : c Cs¬|o d}nd}|ioDtg}|i|ƒD]}|t|t|||ƒq8~ƒSnDtg}|i|ƒD]"}||i|t|||ƒq|~ƒSdS(s# @return: the given file(s) as a list of sentences, each encoded as a list of C{(word,tag)} tuples. @rtype: C{list} of (C{list} of C{(str,str)}) @param c5: If true, then the tags used will be the more detailed c5 tags. Otherwise, the simplified tags will be used. @param strip_space: If true, then strip trailing spaces from word tokens. Otherwise, leave the spaces on the tokens. @param stem: If true, then use word stems instead of word strings. RRN(RR R R RR( RRRRRRRRR((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyt tagged_sents[s   : c CsYg}ti|ƒiƒ}x#|idƒD]}g} xÎt|ƒD]À} | i} | p d} n|p|o| iƒ} n|o| id| ƒ} n|djo| | idƒf} n3|djo%| | id| idƒƒf} n| i| ƒqDW|o!|it |i d| ƒƒq+|i | ƒq+Wt |jpt ‚|S(sØ Helper used to implement the view methods -- returns a list of words or a list of sentences, optionally tagged. @param fileid: The name of the underlying file. @param bracket_sent: If true, include sentence bracketing. @param tag: The name of the tagset to use, or None for no tags. @param strip_space: If true, strip spaces from word tokens. @param stem: If true, then substitute stems for words. s.//stthwRRtn(t ElementTreetparsetgetroottfindallt_all_xmlwords_inttexttstriptgettappendt BNCSentencetattribtextendR tAssertionError( RRt bracket_sentRRRtresulttxmldoctxmlsenttsenttxmlwordtword((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyRqs,      %!( t__name__t __module__t__doc__RRR R RRRRR(((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyRs   cCsZ|djo g}nx<|D]4}|idjo|i|ƒqt||ƒqW|S(Ntctw(R6R7(R RR'R#(teltR-tchild((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyR#”s !R(cBseZdZd„ZRS(s‰ A list of words, augmented by an attribute C{num} used to record the sentence identifier (the C{n} attribute from the XML). cCs||_ti||ƒdS(N(tnumtlistR(RR:titems((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyR s (R3R4R5R(((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyR(›sR cBsSeZdZd„ZeZeZeZeZd„Z d„Z d„Z d„Z RS(sN A stream backed corpus view specialized for use with the BNC corpus. cCs‘|o d}nd}||_||_||_||_ti|||ƒ|iƒ|i|id|i ƒ|i ƒhdd<|_ dS(sG @param fileid: The name of the underlying file. @param sent: If true, include sentence bracketing. @param tag: The name of the tagset to use, or None for no tags. @param strip_space: If true, strip spaces from word tokens. @param stem: If true, then substitute stems for words. s.*/ss.*/s/(.*/)?(c|w)s .*/teiHeader$iN(( t_sentt_tagt _strip_spacet_stemt XMLCorpusViewRt_opent read_blockt_streamt handle_headertcloset _tag_context(RRR0RRRttagspec((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyR¨s      cCsh|idƒ}|o:dig}|D]}||iiƒq'~ƒ|_n|idƒ}|o:dig}|D]}||iiƒqw~ƒ|_n|idƒ} | o:dig} | D]} | | iiƒqÇ~ ƒ|_n|idƒ} | o^dig} | D]:}| dig}|D]}||iiƒq1~ƒq~ ƒ|_ndS(NstitleStmt/titles stitleStmt/authorstitleStmt/editorstitleStmt/respStmts (R"tjoinR$R%ttitletauthorteditortresps(RR8tcontextttitlesRRJtauthorsRRKteditorst_[3]RLRMt_[4]trespt_[5]tresp_elt((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyREÇs 4 4 4 cCs,|io|i|ƒSn|i|ƒSdS(N(R=t handle_sentt handle_word(RR8RN((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyt handle_eltÚscCsÀ|i}|p d}n|ip |io|iƒ}n|io|id|ƒ}n|idjo||idƒf}n6|idjo%||id|idƒƒf}n|S(NRRRR(R$R?R@R%R&R>(RR8R2((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyRXÞs   %cCsªg}x|D]…}|idjo2|g}|D]}||i|ƒq1~7}q |idjo|i|i|ƒƒq td|iƒ‚q Wt|id|ƒS(NtmwR7R6sUnexpected element %sR(R7R6(RRXR't ValueErrorR(R)(RR8R0R9RR7((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyRWìs2( R3R4R5RR RJRKRLRMRERYRXRW(((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pyR ¤s    (R5t __docformat__tretnltk.etree.ElementTreetetreeRtETtapitutiltxmldocsRRR R#R;R(RAR (((s,/p/zhu/06/nlp/nltk/nltk/corpus/reader/bnc.pys s