³ò B_Kc @sÌdZddkZyddklZWn#ej oddklZnXddklZddk l Z ddk l Z l Z ddklZdd kTd efd „ƒYZd efd „ƒYZdS(s‚ Corpus reader for corpora whose documents are xml files. (note -- not named 'xml' to avoid conflicting w/ standard xml package) iÿÿÿÿN(t cElementTree(t ElementTree(tSeekableUnicodeStreamReader(tWordPunctTokenizer(t deprecatedtElementWrapper(t CorpusReader(t*tXMLCorpusReadercBsYeZdZed„Zed„Zed„Zed„Ze dƒedd„ƒZ RS(s  Corpus reader for corpora whose documents are xml files. Note that the C{XMLCorpusReader} constructor does not take an C{encoding} argument, because the unicode encoding is specified by the XML files themselves. See the XML specs for more info. cCs ||_ti|||ƒdS(N(t _wrap_etreeRt__init__(tselftroottfileidst wrap_etree((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/xmldocs.pyR #s cCs–|djo't|iƒdjo|id}nt|tƒptdƒ‚nti|i|ƒi ƒƒi ƒ}|i ot |ƒ}n|S(Niis(Expected a single file identifier string( tNonetlent_fileidst isinstancet basestringt TypeErrorRtparsetabspathtopentgetrootR R(R tfileidtelt((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/xmldocs.pytxml's#$ c Csu|i|ƒ}tƒ}|iƒ}g}xD|D]<}|i}|tj o |i|ƒ}|i|ƒq1q1W|S(sM Returns all of the words and punctuation symbols in the specified file that were in text nodes -- ie, tags are ignored. Like the xml() method, fileid can only specify one file. @return: the given file's text nodes as a list of words and punctuation symbols @rtype: C{list} of C{str} (RRt getiteratorttextRttokenizetextend( R RRtword_tokenizertiteratortouttnodeRttoks((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/xmldocs.pytwords5s     cCsk|djo |i}nt|tƒo |g}ntg}|D]}||i|ƒiƒqE~ƒS(N(RRRRtconcatRtread(R R t_[1]tf((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/xmldocs.pytrawKssUse .raw() or .xml() instead.RcCsP|djo|i|ƒSn|djo|i|ƒSntd|ƒ‚dS(NR*Rs bad format %r(R*Rt ValueError(R titemstformat((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/xmldocs.pyR'Qs( t__name__t __module__t__doc__tFalseR RRR%R*RR'(((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/xmldocs.pyRs     t XMLCorpusViewcBs—eZdZeZdZed„Zd„Zd„Z e i de i e i BƒZe i dƒZe i de i e i BƒZd„Zeed „ZRS( sg A corpus view that selects out specified elements from an XML file, and provides a flat list-like interface for accessing them. (Note: C{XMLCorpusView} is not used by L{XMLCorpusReader} itself, but may be used by subclasses of L{XMLCorpusReader}.) Every XML corpus view has a X{tag specification}, indicating what XML elements should be included in the view; and each (non-nested) element that matches this specification corresponds to one item in the view. Tag specifications are regular expressions over tag paths, where a tag path is a list of element tag names, separated by '/', indicating the ancestry of the element. Some examples: - C{'foo'}: A top-level element whose tag is C{foo}. - C{'foo/bar'}: An element whose tag is C{bar} and whose parent is a top-level element whose tag is C{foo}. - C{'.*/foo'}: An element whose tag is C{foo}, appearing anywhere in the xml tree. - C{'.*/(foo|bar)'}: An wlement whose tag is C{foo} or C{bar}, appearing anywhere in the xml tree. The view items are generated from the selected XML elements via the method L{handle_elt()}. By default, this method returns the element as-is (i.e., as an ElementTree object); but it can be overridden, either via subclassing or via the C{elt_handler} constructor parameter. icCse|o ||_nti|dƒ|_hdd<|_|i|ƒ}ti||d|ƒdS(s[ Create a new corpus view based on a specified XML file. Note that the C{XMLCorpusView} constructor does not take an C{encoding} argument, because the unicode encoding is specified by the XML files themselves. @type tagspec: C{str} @param tagspec: A tag specification, indicating what XML elements should be included in the view. Each non-nested element that matches this specification corresponds to one item in the view. @param elt_handler: A function used to transform each element to a value for the view. If no handler is specified, then L{self.handle_elt()} is called, which returns the element as an ElementTree object. The signature of elt_handler is:: elt_handler(elt, tagspec) -> value s\ZitencodingN((t handle_elttretcompilet_tagspect _tag_contextt_detect_encodingtStreamBackedCorpusViewR (R Rttagspect elt_handlerR3((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/xmldocs.pyR |s cCst|tƒo|iƒiƒ}nt|dƒiƒ}|itiƒodSn|itiƒodSn|itiƒodSn|iti ƒodSn|iti ƒodSnt i d|ƒ}|o|i dƒSnt i d |ƒ}|o|i dƒSndS( Ntrbs utf-16-bes utf-16-les utf-32-bes utf-32-lesutf-8s \s*) | # comment () | # doctype decl (<[^>]*>)) # tag or PI [^<]*)* \Zs<\s*/?\s*([^\s>]+)s6 # Include these so we can skip them: (?P )| (?P )| (?P <\?.*?\?> )| (?P )| # These are the ones we actually care about: (?P <\s*[^>/\?!\s][^>]*/\s*> )| (?P <\s*[^>/\?!\s][^>]*> )| (?P <\s*/[^>/\?!\s][^>]*> )cCsud}xhto`t|tƒo|iƒ}n|i|iƒ}||7}|ii|ƒo|Snti d|ƒi dƒdjo@|iƒt |ƒti d|ƒi ƒ}t d|ƒ‚n|pt dƒ‚n|idƒ}|djop|ii|| ƒoUt|tƒo|i|ƒ|i|ƒn|it |ƒ| dƒ|| Sqlq q Wd S( sz Read a string from the given stream that does not contain any un-closed tags. In particular, this function first reads a block from the stream of size L{self._BLOCK_SIZE}. It then checks if that block contains an un-closed tag. If it does, then this function either backtracks to the last '<', or reads another block. ts[<>]it>sUnexpected ">" near char %ss&Unexpected end of file: tag not closedtiÿÿÿÿsUnmatched tag <%s>...t EMPTY_ELT_TAGit i$s (backtrack)tasciitxmlcharrefreplace(RR7R4tlistR8tgetRPtAssertionErrorRRR^R+t _XML_PIECEtfinditert_DEBUGtjoinRHt _XML_TAG_NAMERGtappendR5tstartRRTtpopRVRWttupleRt fromstringtencode(R RXR;R<RKteltst elt_startt elt_depthtelt_textRZt xml_fragmenttpiecetnameR\R(R((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/xmldocs.pyt read_blocks€ &!   !   ! $+       %(R.R/R0R1RkRQRR R9R4R5R6tDOTALLtVERBOSERRRmRiR^R{(((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/xmldocs.pyR2Xs #    0(R0RAt xml.etreeRRt ImportErrort nltk.etreet nltk.dataRt nltk.tokenizeRtnltk.internalsRRtnltk.corpus.reader.apiRtnltk.corpus.reader.utilRR:R2(((s0/p/zhu/06/nlp/nltk/nltk/corpus/reader/xmldocs.pys s # =