Ic@sdZddklZdZdZdZdZddklZl Z ddk Z ddk Z ddk Z ddk Z ddkZydd klZWnej o hZnXyeWn#ej odd klZnXe id e_e id ie _d ZdfdYZdeefdYZdefdYZdefdYZdefdYZdefdYZ defdYZ!dfdYZ"de#fdYZ$d Z%d!Z&d"Z'd#e!efd$YZ(d%e(fd&YZ)d'e*fd(YZ+d)e)fd*YZ,d+e)fd,YZ-d-e(fd.YZ.d/e(fd0YZ/d1e)fd2YZ0d3e,fd4YZ1d5e-fd6YZ2d7e.fd8YZ3yddk4Z4Wnej o e5Z4nXyddk6Z7Wnej onXyddk8Z8Wnej onXd9fd:YZ9e:d;jo*ddk;Z;e)e;i<Z=e=i>GHndS(<s Beautiful Soup Elixir and Tonic "The Screen-Scraper's Friend" http://www.crummy.com/software/BeautifulSoup/ Beautiful Soup parses a (possibly invalid) XML or HTML document into a tree representation. It provides methods and Pythonic idioms that make it easy to navigate, search, and modify the tree. A well-formed XML/HTML document yields a well-formed data structure. An ill-formed XML/HTML document yields a correspondingly ill-formed data structure. If your document is only locally well-formed, you can use this library to find and process the well-formed part of it. Beautiful Soup works with Python 2.2 and up. It has no external dependencies, but you'll have more success at converting data to UTF-8 if you also install these three packages: * chardet, for auto-detecting character encodings http://chardet.feedparser.org/ * cjkcodecs and iconv_codec, which add more encodings to the ones supported by stock Python. http://cjkpython.i18n.org/ Beautiful Soup defines classes for two main parsing strategies: * BeautifulStoneSoup, for parsing XML, SGML, or your domain-specific language that kind of looks like XML. * BeautifulSoup, for parsing run-of-the-mill HTML code, be it valid or invalid. This class has web browser-like heuristics for obtaining a sensible parse tree in the face of common HTML errors. Beautiful Soup also defines a class (UnicodeDammit) for autodetecting the encoding of an HTML or XML document, and converting it to Unicode. Much of this code is taken from Mark Pilgrim's Universal Feed Parser. For more than you ever wanted to know about Beautiful Soup, see the documentation: http://www.crummy.com/software/BeautifulSoup/documentation.html Here, have some legalese: Copyright (c) 2004-2008, Leonard Richardson All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the the Beautiful Soup Consortium and All Night Kosher Bakery nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE, DAMMIT. i(t generatorss*Leonard Richardson (leonardr@segfault.org)s3.0.7as*Copyright (c) 2004-2008 Leonard Richardsons New-style BSD(t SGMLParsertSGMLParseErrorN(tname2codepoint(tSets[a-zA-Z][-_.:a-zA-Z0-9]*s-zA-Z][-_.:a-zA-Z0-9]*\s*sutf-8t PageElementcBsveZdZeedZdZdZdZdZdZ ehedZ eheedZ ehed Z eheed Z e Zehed Zeheed ZeZehed ZeheedZeZehdZehedZeZdZdZdZdZdZdZdZedZedZ RS(seContains the navigational information for some part of the page (either a tag or a piece of text)cCsk||_||_d|_d|_d|_|io0|iio#|iid|_||i_ndS(sNSets up the initial relations between this element and other elements.iN(tparenttprevioustNonetnexttpreviousSiblingt nextSiblingtcontents(tselfRR((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pytsetupqs     cCs|i}|iii|}t|doN|i|ijo;|iii|}|o||jo|d}q|n|i|i||dS(NRi(RR tindexthasattrtextracttinsert(R t replaceWitht oldParenttmyIndexR((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR}s # cCs|io1y|iii|Wq;tj oq;Xn|i}|i}|io||i_n|o|i|_nt|_t|_t|_|io|i |i_ n|i o|i|i _nt|_|_ |S(s0Destructively rips this element out of the tree.( RR tremovet ValueErrort_lastRecursiveChildR RRR R (R t lastChildt nextElement((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs(          cCs9|}x,t|do|io|id}q W|S(s8Finds the last element beneath this object to be parsed.R i(RR (R R((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRsc Cs<t|tpt|to!t|t ot|}nt|t|i}t|doc|idjoS|i|jo5|i |}|o||jo|d}qn|i n||_d}|djod|_ ||_ n6|i|d}||_ ||i _|i|_ |i o||i _n|i}|t|ijocd|_|}d}x*|p"|i}|i}|pPqqW|o ||_q d|_n:|i|}||_|io||i_ n||_|io||i_ n|ii||dS(NRii(t isinstancet basestringtunicodetNavigableStringtmintlenR RRRtfindRR RR RR R( R tpositiontnewChildRt previousChildtnewChildsLastElementRtparentsNextSiblingt nextChild((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRsV                     cCs|it|i|dS(s2Appends the given tag to the contents of this tag.N(RR R (R ttag((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pytappendscKs|i|i||||S(sjReturns the first item that matches the given criteria and appears after this Tag in the document.(t_findOnet findAllNext(R tnametattrsttexttkwargs((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pytfindNextscKs|i|||||i|S(sbReturns all items that match the given criteria and appear after this Tag in the document.(t_findAllt nextGenerator(R R,R-R.tlimitR/((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR+scKs|i|i||||S(s{Returns the closest sibling to this Tag that matches the given criteria and appears after this Tag in the document.(R*tfindNextSiblings(R R,R-R.R/((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pytfindNextSiblingscKs|i|||||i|S(sqReturns the siblings of this Tag that match the given criteria and appear after this Tag in the document.(R1tnextSiblingGenerator(R R,R-R.R3R/((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR4scKs|i|i||||S(skReturns the first item that matches the given criteria and appears before this Tag in the document.(R*tfindAllPrevious(R R,R-R.R/((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt findPrevious scKs|i|||||i|S(scReturns all items that match the given criteria and appear before this Tag in the document.(R1tpreviousGenerator(R R,R-R.R3R/((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR7scKs|i|i||||S(s|Returns the closest sibling to this Tag that matches the given criteria and appears before this Tag in the document.(R*tfindPreviousSiblings(R R,R-R.R/((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pytfindPreviousSiblingscKs|i|||||i|S(srReturns the siblings of this Tag that match the given criteria and appear before this Tag in the document.(R1tpreviousSiblingGenerator(R R,R-R.R3R/((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR:scKs4t}|i||d}|o|d}n|S(sOReturns the closest parent of this Tag that matches the given criteria.ii(Rt findParents(R R,R-R/trtl((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt findParent$s cKs|i||t||i|S(sFReturns the parents of this Tag that match the given criteria.(R1RtparentGenerator(R R,R-R3R/((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR=/scKs7d}||||d|}|o|d}n|S(Nii(R(R tmethodR,R-R.R/R>R?((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR*9s c Kst|to |}nt||||}t|}|} xto|y| i} Wntj oPnX| oJ|i| } | o0|i| |ot||joPqqqGqGW|S(s8Iterates over a generator looking for things that match.( Rt SoupStrainert ResultSettTrueR t StopIterationtsearchR)R ( R R,R-R.R3t generatorR/tstrainertresultstgtitfound((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR1@s"     ccs'|}x|o|i}|Vq WdS(N(R (R RL((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR2Ys  ccs'|}x|o|i}|Vq WdS(N(R (R RL((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR6_s  ccs'|}x|o|i}|Vq WdS(N(R(R RL((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR9es  ccs'|}x|o|i}|Vq WdS(N(R (R RL((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR<ks  ccs'|}x|o|i}|Vq WdS(N(R(R RL((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRAqs  cCs|pd}|id|S(Nsutf-8s%SOUP-ENCODING%(treplace(R tstrtencoding((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pytsubstituteEncodingxs cCst|to|o|i|}qnjt|to*|o|i|}qt|}n0|o|it||}n t|}|S(sHEncodes an object to a string in some encoding, or to Unicode. .(RRtencodeROt toEncoding(R tsRP((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRS|s (!t__name__t __module__t__doc__RRRRRRR)R0R+R5R4tfetchNextSiblingsR8R7t fetchPreviousR;R:tfetchPreviousSiblingsR@R=t fetchParentsR*R1R2R6R9R<RARQRS(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRms>    <            RcBs8eZdZdZdZdZedZRS(cCs7t|toti||Snti||tS(s-Create a new NavigableString. When unpickling a NavigableString, this method is called with the string in DEFAULT_OUTPUT_ENCODING. That encoding needs to be passed in to the superclass's __new__ or the superclass won't know how to handle non-ASCII characters. (RRt__new__tDEFAULT_OUTPUT_ENCODING(tclstvalue((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR\scCsti|fS(N(Rt__str__(R ((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt__getnewargs__scCs2|djo|Sntd|ii|fdS(stext.string gives you text. This is for backwards compatibility for Navigable*String, but for CData* it lets you get the string without the CData wrapper.tstrings!'%s' object has no attribute '%s'N(tAttributeErrort __class__RU(R tattr((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt __getattr__s cCst|itS(N(ROtdecodeR](R ((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt __unicode__scCs |o|i|Sn|SdS(N(RR(R RP((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR`s(RURVR\RaRfRhR]R`(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs   tCDatacBseZedZRS(cCsdti||S(Ns(RR`(R RP((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR`s(RURVR]R`(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRistProcessingInstructioncBseZedZRS(cCs=|}d|jo|i||}nd|i||S(Ns%SOUP-ENCODING%s(RQRS(R RPtoutput((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR`s (RURVR]R`(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRjstCommentcBseZedZRS(cCsdti||S(Ns (RR`(R RP((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR`s(RURVR]R`(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRlst DeclarationcBseZedZRS(cCsdti||S(Ns(RR`(R RP((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR`s(RURVR]R`(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRmstTagcBseZdZdZhdd<dd<dd<dd <d d /afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt_inverts  t'tapost"tquott&tampttgtcCs |id}|io|tjott|Sn||ijo%|io|i|Sqd|Snt|djoh|ddjoWt|djo,|ddjott|ddSqtt|dSn|io d|Sn d|Sd S( sUsed in a call to re.sub to replace HTML, XML, and numeric entities with the appropriate Unicode characters. If HTML entities are being converted, any unrecognized entities are escaped.iu&%s;it#txiiu&%s;N( tgrouptconvertHTMLEntitiesRtunichrtXML_ENTITIES_TO_SPECIAL_CHARStconvertXMLEntitiesR tinttescapeUnrecognizedEntities(R tmatchR((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt_convertEntitiess  $$  cs|i_|i|_|_|djo g}n|_g_i||t _ t _ |i _ |i _ |i_fd}t|i_dS(sBasic constructor.cs(|\}}|tidi|fS(s&(#\d+|#x[0-9a-fA-F]+|\w+);(tretsubR(t.0Rqtval(R (s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyts N(Rdt parserClasstisSelfClosingTagt isSelfClosingR,RR-R RtFalsethiddentcontainsSubstitutionsRRRtmap(R tparserR,R-RRtconvert((R s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt__init__s           cCs|ii||S(sReturns the value of the 'key' attribute for the tag, or the value given for 'default' if it doesn't have that attribute.(t _getAttrMaptget(R tkeytdefault((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR scCs|ii|S(N(Rthas_key(R R((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRscCs|i|S(sqtag[key] returns the value of the 'key' attribute for the tag, and throws an exception if it's not there.(R(R R((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt __getitem__scCs t|iS(s0Iterating over a tag iterates over its contents.(titerR (R ((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt__iter__scCs t|iS(s:The length of a tag is the length of its list of contents.(R R (R ((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt__len__scCs ||ijS(N(R (R R((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt __contains__"scCstS(s-A tag is non-None even if it has no contents.(RE(R ((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt __nonzero__%scCs|i||i|/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt __setitem__)s  cCshxa|iD]V}|d|jo|ii|n|i|ii|o|i|=q q WdS(s;Deleting tag[key] deletes all 'key' attributes for the tag.iN(R-RRRR(R Rtitem((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt __delitem__7s  cOst|i||S(sCalling a tag like a function is the same as calling its findAll() method. Eg. tag('a') returns a list of all the A tags found within this tag.(tapplytfindAll(R targsR/((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt__call__BscCst|djo5|idt|djo|i|d Sn(|iddjo|i|Sntd|i|fdS(NiRnit__is!'%s' object has no attribute '%s'(R trfindR!RcRd(R R(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRfHs 3cCst|d pat|d pPt|d p?|i|ijp,|i|ijpt|t|jotSnxCtdt|iD])}|i||i|jotSqqWtS(sReturns true iff this tag has the same name, the same attributes, and the same contents (recursively) as the given tag. NOTE: right now this will return false if two tags have the same attributes in a different order. Should this be fixed?R,R-R i(RR,R-R RRR RE(R totherRL((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt__eq__Psr cCs ||j S(sZReturns true iff this tag is not identical to the other tag, as defined in __eq__.((R R((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt__ne__]scCs |i|S(sRenders this tag as a string.(R`(R RP((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt__repr__bscCs |idS(N(R`R(R ((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRhfss([<>]|s&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)t)cCs d|i|idddS(smUsed with a regular expression to substitute the appropriate XML entity for an XML special character.Rxit;(tXML_SPECIAL_CHARS_TO_ENTITIESR(R R((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt _sub_entitymsicCs|i|i|}g}|iox|iD]\}}d}t|o|io#d|jo|i||}nd|jo-d}d|jo|idd}qn|ii|i |}n|i ||i|||i||fq/Wnd} d} |i o d} n d |} d\} } |o"|} d | d } | d } n|i ||| }|i o |}ng}d}|od d i|}n|o|i | n|i d ||| f|o|i dn|i ||o)|o"|ddjo|i dn|o| o|i | n|i | |o"| o|io|i dndi|}|S(sReturns a string or Unicode representation of this tag and its contents. To get Unicode, pass None for encoding. NOTE: since Python's HTML parser consumes whitespace, this method is not certain to reproduce the whitespace present in the original string.s%s="%s"s%SOUP-ENCODING%Rvs%s='%s'Rts&squot;ts /sit is<%s%s%s>s i(ii(RSR,R-tisStringRRQRNtBARE_AMPERSAND_OR_BRACKETRRR)RtrenderContentsRtjoinR (R RPt prettyPrintt indentLevelt encodedNameR-RRtfmttclosetcloseTagt indentTagtindentContentstspaceR RTtattributeString((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR`rs`              cCskg}|iD] }||q~}x6|D].}t|to|iq+|iq+W|idS(s/Recursively destroys the contents of this tree.N(R RRnt decomposeR(R t_[1]RLR ((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs$cCs|i|tS(N(R`RE(R RP((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pytprettifyscCsg}x|D]}t}t|to|i|}n1t|to |i|i|||n|o|o|i}n|oI|o|id|dn|i||o|idqq q Wdi|S(s{Renders the contents of this tag as a string in the given encoding. If encoding is None, returns a Unicode string..Ris R(RRRR`RnR)tstripR(R RPRRRTtcR.((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs"  cKs=t}|i||||d|}|o|d}n|S(sLReturn only the first child of this Tag matching the given criteria.ii(RR(R R,R-t recursiveR.R/R>R?((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR!s cKs9|i}|p |i}n|i||||||S(sExtracts a list of Tag objects that match the given criteria. You can specify the name of the Tag and any attributes you want the Tag to have. The value of a key-value pair in the 'attrs' map can be a string, a list of strings, a regular expression object, or a callable that takes a string and returns whether or not the string matches for some custom definition of 'matches'. The same is true of the tag name.(trecursiveChildGeneratortchildGeneratorR1(R R,R-RR.R3R/RH((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs  cCs|id|d|d|S(NR.RR3(R(R R.RR3((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt fetchText scCs|id|d|S(NR.R(R!(R R.R((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt firstTextscCsKt|dp4h|_x(|iD]\}}||i|/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs   ccs9x,tdt|iD]}|i|VqWtdS(Ni(RR R RF(R RL((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRsccs|dfg}x|o|i\}}t|toxt|t|iD]}|i|}|Vt|toW|ioM|t|idjo|i||dfn|i|dfPqTqTWqqWtdS(Nii(tpopRRnRR R R)RF(R tstackR(tstartRLta((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR"s  (.RURVRWRsRRRRRRRRRRRRRRRRfRRR]RRhRtcompileRRRR`RRRRER!t findChildRt findChildrentfirsttfetchRRRRR(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRns^                       T     RCcBsJeZdZehedZdZehdZdZdZRS(sMEncapsulates a number of ways of matching a markup element (tag or text).cKsu||_t|o||d/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR6s       cCs-|io |iSnd|i|ifSdS(Ns%s|%s(R.R,R-(R ((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR`Ds  c Csd}d}t|to|}|}nt|iot|t }|i pB|p;|o|i||ip| o|i||io|o|i||}nt}d}x|iiD]\}} |pEt |do |}q*h}x"|D]\} } | || /afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt searchTagJsB       cCsd}t|oQt|t o@x|D]1}t|to|i|o |}Pq+q+Wnt|to!|ip|i|}qnRt|tp t|o$|i ||io |}qnt d|i |S(Ns&I don't know how to match against a %s( RtisListRRnRRGR.RRRt ExceptionRd(R RRMtelement((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRGos$    cCsat}|tjo&t|tijo|dj}n%t|o||}nt|to |i }n|ot | ot |}nt |do|o |i |}nt|o||j}nht |do|i|}nE|o=t |o0t|t ot |}qFt|}n|p||j}n|S(NRRo(RREttypettypest BooleanTypeRRRRnR,RRRRGRRRO(R RRtresult((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs,#   ( RURVRWRRR`RRGR(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRC2s  % RDcBseZdZdZRS(sTA ResultSet is just a list that keeps track of the SoupStrainer that created it.cCstig||_dS(N(tlistRtsource(R R((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs (RURVRWR(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRDscCs,t|dpt|titifjS(stConvenience method that works with all 2.x versions of Python to determine whether or not something is listlike.R(RRRtListTypet TupleType(R?((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRscCsHy!t|tp t|tSWn tj ot|tSnXdS(svConvenience method that works with all 2.x versions of Python to determine whether or not something is stringlike.N(RRRt NameErrorRO(RT((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs!cGsh}x|D]w}t|do+x^|iD]\}}|||/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt buildTagMaps  tBeautifulStoneSoupc BseZdZhZhZhZhZgZei ddfei ddfgZ dZ dZ dZ dZeZhed <ed <ed <ed <ed " actually means "". [Another possible explanation is "", but since this class defines no SELF_CLOSING_TAGS, it will never use that explanation.] This class is useful for parsing XML or made-up markup languages, or when BeautifulSoup makes an assumption counter to what you were expecting.s (<[^<>]*)/>cCs|iddS(is />(R(R((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRss]*)>cCsd|iddS(s/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRsu [document]thtmltxmltxhtmli i i i i Rc Csk||_||_||_||_|iod|_||ijot|_t|_ t|_ q||i jot|_t|_ t|_ q||i jot|_t|_ t|_ qnt|_t|_ t|_ t d||_ti|t|do|i}n||_||_y|id|Wntj onXd|_dS(sVThe Soup object is initialized as the 'root tag', and the provided markup (which can be a string or a file-like object) is fed into the underlying parser. sgmllib will process most bad HTML, and the BeautifulSoup class has some tricks for dealing with some HTML that kills sgmllib, but Beautiful Soup can nonetheless choke or lose data if your data uses self-closing tags or declarations incorrectly. By default, Beautiful Soup uses regexes to sanitize input, avoiding the vast majority of these problems. If the problems don't apply to you, pass in False for markupMassage, and you'll get better performance. The default parser massage techniques fix the two most common instances of invalid HTML that choke sgmllib:
(No space between name of closing tag and tag close) (Extraneous whitespace in declaration) You can pass in a custom list of (RE object, replace method) tuples to get Beautiful Soup to scrub your input the way you want.treadtisHTMLN(tparseOnlyTheset fromEncodingt smartQuotesTotconvertEntitiesRt HTML_ENTITIESRRRERRtXHTML_ENTITIESt XML_ENTITIESRtinstanceSelfClosingTagsRRRRRt markupMassaget_feedt StopParsing( R RRRRRRtselfClosingTagsR((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs@                    cCs]yt|}Wntj o dSnXd|jo djnpdSn|i|S(s/This method fixes a bug in Python's SGMLParser.Nii(RRtconvert_codepoint(R R,tn((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pytconvert_charrefFs cCs=|i}t|to!t|dp d|_qnIt||i|gd|id|}|i}|i|_|i |_ |od|i oVt |i p|i |_ nx)|i D]\}}|i ||}qW|` qn|iti|||ix%|ii|ijo|iqWdS(NtoriginalEncodingRR(RRRRRR t UnicodeDammitRRtdeclaredHTMLEncodingRRtMARKUP_MASSAGERtresetRtfeedtendDatat currentTagR,t ROOT_TAG_NAMEtpopTag(R tinDocumentEncodingRRtdammittfixtm((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRPs.        cCs|iddjp,|iddjp|iddjoti||Sn1|iddjoti||SntdS(sThis method routes method call requests to either the SGMLParser superclass or the Tag superclass, depending on the method name.tstart_itend_tdo_RN(R!RRfRnRc(R t methodName((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRfqs ,cCs#|ii|p|ii|S(seReturns true iff the given string is the name of a self-closing tag according to this parser.(tSELF_CLOSING_TAGSRR(R R,((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR~scCsati|||id|_ti|g|_d|_g|_ g|_ |i |dS(Ni( RnRRRRRt currentDataRRttagStackt quoteStacktpushTag(R ((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs      cCs|ii}t|iidjo4t|iidto|iid|i_n|io|id|_n|iS(Niii(RRR RR RRRb(R R(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs cCsE|io|iii|n|ii||id|_dS(Ni(RR R)R(R R(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR!s cCsD|io6di|i}|i|idjo\tg}|iD]}||iqF~i|i o!d|jo d}qd}ng|_|i o@t |idjo*|i i p|i i | odSn||}|i |i|i|io||i_n||_|iii|ndS(NuRs Ri(RRt translatetSTRIP_ASCII_SPACEStsetRR,t intersectiontPRESERVE_WHITESPACE_TAGSRR R.RGRRRR R R)(R tcontainerClassRRR(to((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs& -        cCs||ijodSnd}d}xVtt|idddD]5}||i|ijot|i|}PqDqDW|p|d}nx#td|D]}|i}qW|S(sPops the tag stack up to and including the most recent instance of the given tag. If inclusivePop is false, pops the tag stack up to but *not* including the most recent instqance of the given tag.Niii(RRRR RR,R(R R,t inclusivePoptnumPopst mostRecentTagRL((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt _popToTags  c Cs!|ii|}|dj}|ii|}d}t}xtt|idddD]}|i|}| p|i |jo| o |}Pn|djo|i |jp*|djo1|o*|ii|i o|i }t }Pn|i }q\W|o|i ||ndS(sWe need to pop up to the previous tag of this type, unless one of this tag's nesting reset triggers comes between this tag and the previous tag of this type, OR unless this tag is a generic nesting trigger and another generic nesting trigger comes between this tag and the previous tag of this type. Examples:

FooBar *

* should pop to 'p', not 'b'.

FooBar *

* should pop to 'table', not 'p'.

Foo

Bar *

* should pop to 'tr', not 'p'.

    • *
    • * should pop to 'ul', not the first 'li'.
    • ** should pop to 'table', not the first 'tr' tag should implicitly close the previous tag within the same
    ** should pop to 'tr', not the first 'td' iiiN( t NESTABLE_TAGSRRtRESET_NESTING_TAGSRRERR RR,RRR,( R R,tnestingResetTriggerst isNestabletisResetNestingtpopTot inclusiveRLtp((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt _smartPops*       icCsh|io:ditd|}|id||fdSn|i|i| o| o|i|n|ioBt|i djo,|ii p|ii || odSnt ||||i |i}|io||i_n||_|i||p|i|o|in||ijo|ii|d|_n|S(NRcSs|\}}d||fS(s %s="%s"((RRty((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRss<%s%s>i(R RRt handle_dataRRR5RR RR.RRnRRR R!Rt QUOTE_TAGSR)tliteral(R R,R-t selfClosingR(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pytunknown_starttags*   $    cCs|io-|id|jo|id|dSn|i|i||io=|id|jo)|iit|idj|_ndS(Nisi(R R7RR,RR R9(R R,((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pytunknown_endtags   cCs|ii|dS(N(RR)(R tdata((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR7)scCs(|i|i||i|dS(sOAdds a certain piece of text to the tree as a NavigableString subclass.N(RR7(R R.tsubclass((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt_toStringSubclass,s  cCs/|d djo d}n|i|tdS(sHandle a processing instruction as a ProcessingInstruction object, possibly one with a %SOUP-ENCODING% slot into which an encoding will be plugged later.iRu,xml version='1.0' encoding='%SOUP-ENCODING%'N(R?Rj(R R.((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt handle_pi3s cCs|i|tdS(s#Handle comments as Comment objects.N(R?Rl(R R.((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pythandle_comment;scCs;|iott|}n d|}|i|dS(s$Handle character references as data.s&#%s;N(RRRR7(R trefR=((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pythandle_charref?s  cCsd}|io.ytt|}Wq>tj oq>Xn| o |io|ii|}n| o,|io"|ii| od|}n|pd|}n|i|dS(sHandle entity references as data, possibly converting known HTML and/or XML entity references to the corresponding Unicode characters.s&%ss&%s;N( RRRRtKeyErrorRRRR7(R RBR=((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pythandle_entityrefGs  cCs|i|tdS(s4Handle DOCTYPEs and the like as Declaration objects.N(R?Rm(R R=((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt handle_declrscCst}|i||d!djog|iid|}|djot|i}n|i|d|!}|d}|i|tnWyti||}Wn=tj o1|i|}|i ||t|}nX|S(s`Treat a bogus SGML declaration as raw data. Treat a CDATA declaration as a CData object.i s ii( RtrawdataR!R R?RiRtparse_declarationRR7(R RLtjRqR=ttoHandle((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRHvs    (*RURVRWRR-R.R8R&RRRRRRRt ALL_ENTITIESRR#RERRR RRfRRRR!RRR,R5R;R<R7R?R@RARCRERFRH(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRsN   3 C !      .       + t BeautifulSoupc BseZdZdZeedddddddd d g Zed d gZhed <ed tag should implicitly close the previous

    tag.

    Para1

    Para2 should be transformed into:

    Para1

    Para2 Some tags can be nested arbitrarily. For instance, the occurance of a

    tag should _not_ implicitly close the previous
    tag. Alice said:
    Bob said:
    Blah should NOT be transformed into: Alice said:
    Bob said:
    Blah Some tags can be nested, but the nesting is reset by the interposition of other tags. For instance, a
    , but not close a tag in another table.
    BlahBlah should be transformed into:
    BlahBlah but, Blah
    Blah should NOT be transformed into Blah
    Blah Differing assumptions about tag nesting rules are a major source of problems with the BeautifulSoup class. If BeautifulSoup is not treating as nestable a tag your page author treats as nestable, try ICantBelieveItsBeautifulSoup, MinimalSoup, or BeautifulStoneSoup before writing your own subclass.cOsB|idp|i|d/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRs tbrthrtinputtimgtmetatspacertlinktframetbasetprettextareatscripttspantfonttqtobjecttbdoRtsuptcentert blockquotetdivtfieldsettinstdeltoltultlitdltddtdtttablettbodyttfootttheadttrttdtthtaddresstformR4tnoscripts((^|;)\s*charset=)([^;]*)cCsd}d}d}t}xmtdt|D]V}||\}}|i}|djo |}q.|djo|}|}q.q.W|o|o|ii|} | o|idj p|i|i jo@d} |ii | |} ||d| f||/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pytrewritesiRQN(RRRR tlowert CHARSET_RERGR R RRRERRRR;R(R R-t httpEquivt contentTypetcontentTypeIndexttagNeedsEncodingSubstitutionRLRR_RRvtnewAttrt newCharsetR(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyt start_metas@          (RURVRWRRRRR$R&R8tNESTABLE_INLINE_TAGStNESTABLE_BLOCK_TAGStNESTABLE_LIST_TAGStNESTABLE_TABLE_TAGStNON_NESTABLE_BLOCK_TAGSR.R-RRtMRxR(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyRLs@.               RcBseZRS((RURV(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR#stICantBelieveItsBeautifulSoupcBsheZdZddddddddd d d d d ddddgZdgZegeieeZRS(syThe BeautifulSoup class is oriented towards skipping over common HTML errors like unclosed tags. However, sometimes it makes errors of its own. For instance, consider this fragment: FooBar This is perfectly valid (if bizarre) HTML. However, the BeautifulSoup class will implicitly close the first b tag when it encounters the second 'b'. It will think the author wrote "FooBar", and didn't close the first 'b' tag, because there's no real-world reason to bold something that's already bold. When it encounters '' it will close two more 'b' tags, for a grand total of three tags closed instead of two. This can throw off the rest of your document structure. The same is true of a number of other tags, listed below. It's much more common for someone to forget to close a 'b' tag than to actually use nested 'b' tags, and the BeautifulSoup class handles the common case. This class handles the not-co-common case: where you can't believe someone wrote what they did, but it's valid HTML and BeautifulSoup screwed up by assuming it wouldn't be.temtbigRLtsmallttttabbrtacronymtstrongtcitetcodetdfntkbdtsamptvartbRt(RURVRWt*I_CANT_BELIEVE_THEYRE_NESTABLE_INLINE_TAGSt)I_CANT_BELIEVE_THEYRE_NESTABLE_BLOCK_TAGSRRLR-(((s>/afs/cs.wisc.edu/p/zhu/public/html/space2/TTP/BeautifulSoup.pyR&s   t MinimalSoupcBs eZdZedZhZRS(sThe MinimalSoup class is for parsing HTML that contains pathologically bad markup. It makes no assumptions about tag nesting, but it does know which tags are self-closing, that