B_Kc%@sdZddkZddkZddkZddkZddkZddkZddkZddkZddk l Z yddk l Z Wnddk l Z nXy>e ZddkTddkTddklZddklZWneZeZnXddklZddkZeiid Zd efd YZd efd YZ defdYZ!de!fdYZ"de!fdYZ#de!fdYZ$de!fdYZ%de!fdYZ&de!fdYZ'de!fdYZ(de!fdYZ)d e!fd!YZ*d"e!fd#YZ+d$e!fd%YZ,d&e!fd'YZ-d(e!fd)YZ.d*efd+YZ/d,efd-YZ0d.efd/YZ1d0Z2e d1Z3e d2Z4ddk5Z5ddkZd3Z6d4d5Z7d6Z8d7Z9d8Z:d9Z;e/Z<e<i=Z=d:Z>d;Z?d<Z@eAd=jo_dd>kBlCZCeCZDeDiEd?d@dAdBdCdDdEdFeDiEdGdHdAdIdJdKdLedCdMeDiEdNdOdAdPdJdKdLedCdQeDiEdRdSdAdTdJdKdLedCdUeDiF\ZGZHeHodx]eHD]UZIe=dVeIdWeGiJdIeGiKdPeGiLdTeGiMZNeNejoeGiMoPnqGWn,e=dWeGiJdIeGiKdPeGiLdTeGiMndS(XsN The NLTK corpus and module downloader. This module defines several interfaces which can be used to download corpora, models, and other data packages that can be used with NLTK. Downloading Packages ==================== If called with no arguments, L{download() } function will display an interactive interface which can be used to download and install new packages. If Tkinter is available, then a graphical interface will be shown; otherwise, a simple text interface will be provided. Individual packages can be downloaded by calling the C{download()} function with a single argument, giving the package identifier for the package that should be downloaded: >>> download('treebank') # doctest: +SKIP [nltk_data] Downloading package 'treebank'... [nltk_data] Unzipping corpora/treebank.zip. NLTK also provides a number of "package collections", consisting of a group of related packages. To download all packages in a colleciton, simply call C{download()} with the collection's identifier: >>> download('all-corpora') # doctest: +SKIP [nltk_data] Downloading package 'abc'... [nltk_data] Unzipping corpora/abc.zip. [nltk_data] Downloading package 'alpino'... [nltk_data] Unzipping corpora/alpino.zip. ... [nltk_data] Downloading package 'words'... [nltk_data] Unzipping corpora/words.zip. Download Directory ================== By default, packages are installed in either a system-wide directory (if Python has sufficient access to write to it); or in the current user's home directory. However, the C{download_dir} argument may be used to specify a different installation target, if desired. See L{Downloader.default_download_dir()} for more a detailed description of how the default download directory is chosen. NLTK Download Server ==================== Before downloading any packages, the corpus and module downloader contacts the NLTK download server, to retrieve an index file describing the available packages. By default, this index file is loaded from C{}. If necessary, it is possible to create a new L{Downloader} object, specifying a different URL for the package index file. Usage:: python nltk/downloader.py [-d DATADIR] [-q] [-f] [-k] PACKAGE_IDS or with py2.5+: python -m nltk.downloader [-d DATADIR] [-q] [-f] [-k] PACKAGE_IDS iN(tStringIO(tmd5(t*(tTable(tShowText(t ElementTreeturllib2tPackagec BsPeZdZedeeeedddded ZedZdZRS(s3 A directory entry for a downloadable package. These entries are extracted from the XML index file that is downloaded by L{Downloader}. Each package consists of a single file; but if that file is a zip file, then it can be automatically decompressed when the package is installed. ttUnknowncKs||_|p||_||_||_t||_t||_||_||_| |_ | |_ | |_ | |_ t ii|iddd}t ii||||_tt| |_|ii|dS(Nt/ii(tidtnametsubdirturltinttsizet unzipped_sizetchecksumt svn_revisiont copyrighttcontacttlicensetauthortostpathtsplitexttsplittjointfilenametbooltunzipt__dict__tupdate(tselfR RR R RRRRRRRRRtkwtext((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt__init__s          #cCs0t|toti|}nt|iS(N(t isinstancet basestringRtparseRtattrib(txml((s%/p/zhu/06/nlp/nltk/nltk/downloader.pytfromxmlscCs d|iS(Ns (R (R"((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt__repr__s( t__name__t __module__t__doc__tNonetTrueR%t staticmethodR+R,(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRs 6t CollectioncBs2eZdZedZedZdZRS(s A directory entry for a collection of downloadable packages. These entries are extracted from the XML index file that is downloaded by L{Downloader}. cKs?||_|p||_||_d|_|ii|dS(N(R R tchildrenR0tpackagesR R!(R"R R4R R#((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%s    cCsit|toti|}ng}|idD]}||idq7~}td||iS(NtitemtrefR4(R&R'RR(tfindalltgetR3R)(R*t_[1]tchildR4((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR+ s3cCs d|iS(Ns(R (R"((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR,'s(R-R.R/R0R%R2R+R,(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR3s tDownloaderMessagecBseZdZRS(sUA status message object, used by L{incr_download} to communicate its progress.(R-R.R/(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR<.stStartCollectionMessagecBseZdZdZRS(s<Data server has started working on a collection of packages.cCs ||_dS(N(t collection(R"R>((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%3s(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR=1stFinishCollectionMessagecBseZdZdZRS(s=Data server has finished working on a collection of packages.cCs ||_dS(N(R>(R"R>((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%6s(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR?4stStartPackageMessagecBseZdZdZRS(s-Data server has started working on a package.cCs ||_dS(N(tpackage(R"RA((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%9s(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR@7stFinishPackageMessagecBseZdZdZRS(s.Data server has finished working on a package.cCs ||_dS(N(RA(R"RA((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%<s(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRB:stStartDownloadMessagecBseZdZdZRS(s.Data server has started downloading a package.cCs ||_dS(N(RA(R"RA((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%?s(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRC=stFinishDownloadMessagecBseZdZdZRS(s/Data server has finished downloading a package.cCs ||_dS(N(RA(R"RA((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%Bs(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRD@stStartUnzipMessagecBseZdZdZRS(s,Data server has started unzipping a package.cCs ||_dS(N(RA(R"RA((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%Es(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRECstFinishUnzipMessagecBseZdZdZRS(s-Data server has finished unzipping a package.cCs ||_dS(N(RA(R"RA((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%Hs(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRFFstUpToDateMessagecBseZdZdZRS(s/The package download file is already up-to-datecCs ||_dS(N(RA(R"RA((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%Ks(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRGIst StaleMessagecBseZdZdZRS(s3The package download file is out-of-date or corruptcCs ||_dS(N(RA(R"RA((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%Ns(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRHLst ErrorMessagecBseZdZdZRS(s Data server encountered an errorcCs9||_t|tot||_n ||_dS(N(RAR&t Exceptiontstrtmessage(R"RARL((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%Qs (R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRIOstProgressMessagecBseZdZdZRS(s4Indicates how much progress the data server has madecCs ||_dS(N(tprogress(R"RN((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%Zs(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRMXstSelectDownloadDirMessagecBseZdZdZRS(s:Indicates what download directory the data server is usingcCs ||_dS(N(t download_dir(R"RP((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%]s(R-R.R/R%(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRO[st DownloadercBseZdZd'ZdZdZdZdZdZe e dZ e e e e e e dZ d Zd Zd Zd Zd Ze e dZdZdZdZe e e e de e dZe dZe dZe dZe dZdZe ddZe dZdZdZ dZ!dZ"e#de"d d!Z$d"Z%d#Z&e#d$e&d d%Z'd&Z(RS((sy A class used to access the NLTK data server, which can be used to download corpora and other data packages. i<s8http://nltk.googlecode.com/svn/trunk/nltk_data/index.xmlt installeds not installeds out of datetpartialcCsy|p|i|_h|_h|_||_d|_d|_h|_d|_ |idjo|i |_ndS(N( t DEFAULT_URLt_urlt _collectionst _packagest _download_dirR0t_indext_index_timestampt _status_cachet_errorstdefault_download_dir(R"tserver_index_urlRP((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%s       cCsd}|djo|i}d|GHn|oFddt|iGHd|iGHddt|iGH|d7}nt}} g} |o| idn|o| idnx| D]} d | iGH|d 7}xrtt|| D]X} |i | |} | |i jo|oqn| |i jo t }n| |i jo t } nhd |i <d |i <d |i <d|i<| }tid d| ip| iddddd}d|| iidd|fGH|t|id7}|o>|djo1td}|idjodSnd}qqWHqWd}|o|d7}n| o|d7}nti|dddddGHdS( Nis!Using default data directory (%s)t=is Data server index for <%s>iR5t collectionss%s:iRt-tPt iiKtsubsequent_indents [%s] %s %sit.s sHit Enter to continue: txtqs([*] marks installed packagess+; [-] marks out-of-date or corrupt packagess+; [P] marks partially installed collectionst)twidthiL(RfRg(R0RXtlenRUtFalsetappendt capitalizetsortedtgetattrtstatust INSTALLEDtSTALER1tPARTIALt NOT_INSTALLEDttextwraptfillR R tljustRt raw_inputtlower(R"RPt show_packagestshow_collectionstheadert more_prompttskip_installedtlineststaleRSt categoriestcategorytinfoRptprefixR t user_inputtmsg((s%/p/zhu/06/nlp/nltk/nltk/downloader.pytlistsL      ! cCs|i|iiS(N(t _update_indexRWtvalues(R"((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR5s cCsL|ig}|iiD]'\}}|idjo ||qq~S(Ntcorpora(RRWtitemsR (R"R:R tpkg((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRs  cCsL|ig}|iiD]'\}}|idjo ||qq~S(NR(RRWRR (R"R:R R((s%/p/zhu/06/nlp/nltk/nltk/downloader.pytmodelss  cCs|i|iiS(N(RRVR(R"((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR`s cCs)t|to|i|Sn|SdS(N(R&R'R(R"t info_or_id((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt _info_or_idsc cs0|djo|i}t|Vnt|ttfo-x"|i|||D] }|VqQWdSny|i|}Wn7tt fj o%}t dd||fVdSnXt|t oBt |Vx%|i |i||D] }|VqWt|Vn&x"|i|||D] }|VqWdS(NsError loading %s: %s(R0RXROR&Rttuplet_download_listRtIOErrort ValueErrorRIR3R=t incr_downloadR4R?t_download_package(R"RRPtforceRRte((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRs,       cCs)t|todSnt|iSdS(Ni(R&RRjR5(R"R6((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt _num_packagessc #sCxitt|D]U}yi||||+sig?id(trangeRjRRRRItsumt enumerateR&RtfloatR5RRMRN( R"RRPRtiRt num_packagesRNR6tdeltaR((R"s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR"s$   c cs-t|VtdV|i||}| o9||ijo)t|VtdVt|VdSn|ii|idt i i ||i }t i i|o0||ijot|Vnt i|nt i i|pt i|nt i it i i ||ip#t it i i ||int|VtdVyti|i}t|d}tdt|id}xqtiD]c} |id} |i| | pPn| ddjo$tt d dd | |VqqW|i!|i!Wn:t"j o.} t#|d |i|i| fVdSnXt$|Vtd V|i i%d ot i i ||i} |i&p%t i it i i | |ioHt'|Vx+t(|| d t)D]} || _*| VqWt+|Vqnt|VdS(NiiditwbiiiiiPiKs$Error downloading %r from <%s>: %ss.ziptverbosei@i@(,R@RMRpRqRGRBR[tpopR R0RRRRtexistsRrRHtremovetmkdirR RCRturlopenRtopentmaxRRt itertoolstcounttreadtwritetmintcloseRRIRDtendswithRREt _unzip_iterRkRARF(R"RRPRRptfilepathtinfiletoutfilet num_blockstblocktsRtzipdirR((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR8sb     %#     (     /   s [nltk_data] c ss|djo,|dj o ||_n|itSn7dfd}x!|i|||D] } t| to|| i|ot| in|ot Snt|_ |pndGHt i } | djo4|i | ii|||||pt Sq'q+| djot Sq+q/n|p5t| to'|d| iid 7GHqkt| toIGHd |i o|d | iiqg|d | iiqkt| to|d | ii|fqkt| to|d| iidqkt| to|d| iidqkt| to | i}qkqaqaWtS(NRcs*ti|d|d|dGHdS(Ntinitial_indentRdRcis (RuRv(Rtprefix2(R(s%/p/zhu/06/nlp/nltk/nltk/downloader.pytshowss(Error installing package. Retry? [n/y/e]tytYRtEsDownloading collection %rs | is$Downloaded collection %r with errorssDone downloading collection %rsDownloading package %r to %s...s!Package %s is already up-to-date!s s Unzipping %s.(RR(RR(R0RXt_interactive_downloadR1RR&RIRLRRkR\RxtstriptdownloadRAR R=R>R?R@RGRERRORP( R"RRPtquietRRt halt_on_errortraise_on_errorRRtchoice((Rs%/p/zhu/06/nlp/nltk/nltk/downloader.pyR{s`             cCs|i|||ijS(N(RpRr(R"RRP((s%/p/zhu/06/nlp/nltk/nltk/downloader.pytis_stalescCs|i|||ijS(N(RpRq(R"RRP((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt is_installedscCs5|djo|iin|ii|ddS(N(R0R[tclearR(R"R ((s%/p/zhu/06/nlp/nltk/nltk/downloader.pytclear_status_caches c Css|djo |i}n|i|}t|tog}|iD]}||i|iqG~}|i|jo |iSqo|i |jo |i Sqo|i |jo|i |jo |i Sqo|i |jo |i Sqo|i Snt i i||i}||ijo|i||}nB|i|ijo |i|||i|i s (RRRRtRtOSErrorRRRRrt md5_hexdigestRRRqtisdirRtwalkR(R"RRtfilestattunzipdirR((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRs&       cCs[|ixJ|iD]<}|i||ijo|i|d|d|qqWdS(sA Re-download any packages whose status is STALE. RRN(RR5RpRrR(R"RRR((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR!s   c CsG|idjp'|dj pti|i|ijpdSn|p|i|_tiit i t i |ii |_ti|_g}|iidD]}|ti|q~}td|D|_g}|iidD]}|ti|q~}td|D|_x|iiD]v}xmt|iD]\\} } | |ijo|i| |i| 4sscollections/collectioncss"x|]}|i|fVqWdS(N(R (Rtc((s%/p/zhu/06/nlp/nltk/nltk/downloader.pys 9s(RYR0ttimeRZt INDEX_TIMEOUTRUtnltkt internalstElementWrapperRR(RRtgetrootR8RR+tdictRWR3RVRRR4R&textendR R5R[R( R"RR:RR5t_[2]RR`R>Rtchild_idtqueueR;((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR s@ '//   cCs|i|iS(s Return the XML index describing the packages available from the data server. If necessary, this index will be downloaded from the data server. (RRY(R"((s%/p/zhu/06/nlp/nltk/nltk/downloader.pytindexQs cCs\|i||ijo|i|Sn||ijo|i|Sntd|dS(sLReturn the L{Package} or L{Collection} record for the given item.sPackage %r not found in indexN(RRWRVR(R"R ((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRZs cCs|ix8|iidD]$}|id|jo|SqqWx8|iidD]$}|id|jo|SqXqXWtd|dS(s-Return the XML info record for the given itemspackages/packageR scollections/collectionsPackage %r not found in indexN(RRYR8R9R(R"R RAR>((s%/p/zhu/06/nlp/nltk/nltk/downloader.pytxmlinfobs   cCs4|i}y|i|Wn||_nXdS(N(RUR(R"Rt original_url((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt_set_urlqs   cCs|iS((RU(R"((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt{stdocs2 The URL for the data server's index file.cCsxBtiiD]4}tii|otii|o|Sq q Wtidjo!dti joti d}n0tii d}|djot dntii |dS(s  Return the directory to which packages will be downloaded by default. This value can be overridden using the constructor, or on a case-by-case basis using the C{download_dir} argument when calling L{download()}. On Windows, the default download directory is C{I{PYTHONHOME}/lib/nltk}, where C{I{PYTHONHOME}} is the directory containing Python (e.g. C{C:\Python25}). On all other platforms, the default directory is determined as follows: - If C{/usr/share} exists and is writable, then return C{/usr/share/nltk} - If C{/usr/local/share} exists and is writable, then return C{/usr/local/share/nltk} - If C{/usr/lib} exists and is writable, then return C{/usr/lib/nltk} - If C{/usr/local/lib} exists and is writable, then return C{/usr/local/lib/nltk} - Otherwise, return C{~/nltk_data}, where C{~} is the current user's home directory. twin32tAPPDATAs~/s+Could not find a default download directoryt nltk_data( RtdataRRRRt is_writabletsystplatformtenviront expanduserRR(R"tnltkdirthomedir((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR]~s    cCs||_|iidS(N(RXR[R(R"RP((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt_set_download_dirs cCs|iS((RX(R"((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRss  The default directory to which packages will be downloaded. This defaults to the value returned by L{default_download_dir()}. To override this default on a case-by-case basis, use the C{download_dir} argument when calling L{download()}.cCsYto>yt|iWqUtj ot|iqUXnt|idS(N(tTKINTERt DownloaderGUItmainlooptTclErrortDownloaderShelltrun(R"((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRs i()R-R.R/RRTRqRtRrRsR0R%R1RkRR5RRR`RRRRRRRRRRpRR!RRRRRtpropertyRR]RRPR(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRQcsT %  '        C   G    #  1     -   RcBsGeZdZdZdZdZdZdZdZRS(cCs ||_dS(N(t_ds(R"t dataserver((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%scGsTddGHdtd|Dt|dd}d|i|GHddGHdS(NRaiKiDcssx|]}t|VqWdS(N(Rj(Rto((s%/p/zhu/06/nlp/nltk/nltk/downloader.pys siRcs (RRjR(R"toptionstspc((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt_simple_interactive_menus ,cCsdGHxto |idddddtdi}|pHqn|iid}|id }|d jo'H|ii|iid t d tns|d jo|i nX|djo|i n=|djodSn(|djo|i |n d|GHHqWdS(NsNLTK Downloaders d) Downloadsl) Lists c) Configsh) Helpsq) Quits Downloader> iitlR|R}thRRgRfRsCommand %r unrecogmized(RgRf( R1RRxRRyRRRRPRkt_simple_interactive_helpt_simple_interactive_configt_simple_interactive_download(R"Rtcommandtargs((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRs,        cCs;|oRx-|D]C}y|ii|ddWqttfj o}|GHqXqWnxtoHdGHtd}|idjo/|ii|iidt dtdtq\q\|id jodSq\|oYxQ|i D]C}y|ii|ddWqttfj o}|GHqXqWPq\q\WdS( NRs s*Download which package (l=list; x=cancel)?s Identifier> RR|R}R~RfRgR(RfRgR( RRRRR1RxRyRRPRkR(R"R targRRR ((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRs*(    $cCsHdGHdGHdGHdGHdS(Ns Commands:s: d) Download a package or collection h) Helps: l) List packages & collections q) Quits c) View & Modify Configuration((R"((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRs cCs^HdGHd|iiGHdt|iiGHdt|iiGHHdGHd|iiGHdS(Ns Data Server:s - URL: <%s>s$ - %d Package Collections Availables$ - %d Individual Packages AvailablesLocal Machine:s - Data directory: %s(RRRjR`R5RP(R"((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt _show_config scCsx|ixgto_H|iddddtdii}|djo|iq |djo^tdii}|djo d GHqotii|o||i _ qod |GHq |djotdii}|djo d GHqo|i dpd|}ny||i _ Wqot j o}d||fGHqoXq |djoPq q WdS(Nss) Show Configsu) Set Server URLsd) Set Data Dirs m) Main MenusConfig> RRs New Directory> RRfRgs Cancelled!s)Directory %r not found! Create it first.tus New URL> shttp://sError reading <%r>: %stm(RRfRg(RRfRg(R R1RRxRRyRRRRRPt startswithRRJ(R"Rt new_dl_dirtnew_urlR((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRs8            ( R-R.R%RRRRR R(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRs     Rc BseZdZddddddddd d d d d g Zhdd<dd<dd<ddZ3d?Z4d@Z5dAZ6dBZ7e8i9dCZ:dDZ;dEZ<dZ=dFZ>dGZ?dHZ@dIZAdJZBdKeCiDfdLYZEdMZFdNZGRS(XsU Graphical interface for downloading packages from the NLTK data server. Rt IdentifiertNametSizetStatuss Unzipped Sizet CopyrighttContacttLicensetAuthors SVN RevisiontSubdirtChecksumiiiii-i i is#000s#cccs#afas#080s#ffas#880s#faas#800s#fffs#888s#45cs#aaas#67as#f00shelvetica -16 boldcCs9||_||_ti|_g|_g|_t|_h|_ g|_ d|_ |i dt }|_|id|id|id|id|id|i|id|it|_h|_|i|i|i|i|i|iid|iid |idS( NisNLTK Downloader Started!s+50+50sNLTK Downloadert backgroundis s s (Rt _use_threadst threadingtLockt_download_lockt_download_msg_queuet_download_abort_queueRkt _downloadingt_afteridt _log_messagest _log_indentt_logtTkttoptgeometryttitlet configuret_BACKDROP_COLORtbindtdestroyt _destroyedt _column_varst _init_widgetst _init_menut _fill_tablet _show_infot_select_columnst_tabletselectt_destroy(R"Rt use_threadsR)((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR%~s2                  cCs.|iidtid|i|fdS(Ns%s %s%ss | (R%RlRtctimeR&(R"R((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR'scstidddddddd}|id d d td d |iddd|idddt|ddiddddt|}|iddddddt|}|iddddddt|}|iddddddt|ddiddddt|}|iddddddt|ddiddddtidddddid}|iddd ddidd?id,d_'i'idd$t&|d"d@d?i(d,d_)i)iddAt |d"dBd+iddid_*t+|d,dCdd)di,dddDdd_-i.i-iddAi*idd$dS(ENtrelieftraisedtborderitpadxitpadyitsidR)texpandRvtbothtweightitheighttcolumntrowtstickytnewsiiiiRtsidetbottomRfthighlightthicknesst CollectionstCorporatModelss All Packagesttexttfonttlefti s tcolumn_weightstlistbox_heightitreprfunct foregroundRissssss Rs Server Index:RPsDownload Directory:Rtcouriertgroovetdisabledforegroundtblackcs i|S((t _info_edit(Rtkey(R"(s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRstewtDownloadR tRefreshtrightRitsunken(/tFrameR)tpackR1tgrid_rowconfiguretgrid_columnconfiguretgridR-t _tab_namest_tabsRtLabelt _TAB_FONTR.t _select_tabRytCOLUMNStCOLUMN_WEIGHTSR9Rt_table_reprfuncR7t columnconfigt _MARK_COLORt COLUMN_WIDTHStDEFAULT_COLUMN_WIDTHtfocustbind_to_listboxest _downloadt _table_markt _prev_tabt _next_tabt _mark_allRRt_infotEntryt _info_savetButtont_download_buttont_refresht_refresh_buttont_progresslabeltCanvast_PROGRESS_COLORt _progressbart_init_progressbar(R"tf1ttabframet tableframet buttonframet infoframet progressframeRttabtlabelR:RFRSRiRR\tcallbacktentry((R"s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR2s'"   " "    ",       (  !!   cs)ti}t|dd}|idddddidd|i|idd dd dfd |idd dddfd |i|idddddi|i|idddddidd|iddddd|t|dd}xii dD]}t i}|i jpt |i |<|i jo|idn|id|ddd|diq1W|iddddd|t|dd}x>ii dD],}|idd|d|fdqW|ix>ii dD],}|idd|d|fdq>W|iddddd|t|dd}|idddddi|idddddidd|idd ddd|iid!iiid|dS("NttearoffiRR^t underlineR t acceleratortReturnsChange Server Indexics idS(R(R[((R"(s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRssChange Download Directorycs idS(RP(R[((R"(s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRssShow LogitExitisCtrl-xtFiletmenuitvariabletViews Sort by %scsii|dS(t ascending(R7tsort_by(R(R"(s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR3ssReverse sort by %scsii|dS(t descending(R7R(R(R"(s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR9stSorttAboutt InstructionstF1tHelps(tMenuR)t add_commandRut add_separatort _show_logR/t add_cascadeR7t column_namestIntVarR1tAssertionErrortINITIAL_COLUMNStsettadd_checkbuttonR6taboutthelpR.tconfig(R"tmenubartfilemenutviewmenuRFtvartsortmenuthelpmenu((R"s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR3 sZ     !  cCsUxN|iiD]=\}}|io|ii|q|ii|qWdS(N(R1RR9R7t show_columnt hide_column(R"RFR((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR6Gs   cCs+|ii|i|iiddS(Ni(RRR4R7R8(R"((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRNs  cCs?|i|i|\}}d|dR&R@RGRDRERR?t _clear_markRBt _DL_DELAYR(R"RtidsRtafteridR((R"s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRsF      cCsVxOtt|iD]8}|i|df|jo|ii|dSqqWdS(NR(RRjR7R8(R"R RG((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR s c Csxtt|iD]}|i|i|df\}}d \}}|ii|d|d|d|d||ii|dd|idd|id qWdS( NRRZtwhiteRVtselectforegroundRtselectbackgroundii(sblackR(RRjR7t _ROW_COLORt rowconfigt itemconfigureRp(R"RGtbgtsbgtfgtsfg((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRs    cCsUxNtt|iD]7}|i|df|jod|i|dfR&R@RRRGRDRERRFR?RRBRt_MONITOR_QUEUE_DELAYR(R"RRR((R"s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRsn               (s#000s#ccc(s#afas#080(s#ffas#880(s#faas#800(s#fffs#888(s#000s#ccc(s#fffs#45c(s#aaas#67a(s#f00s#aaa(HR-R.R/RlRmRqRrRRRR-RQRqRsRrRtRRpRRRRjR1R%R'R2R3R6RR[R0R|RnRRR5RwRxRkRt_rowsR4RRuRRRRRRyRvRRR/R9RRutdedentRRRRRRRRRRRRRR(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR8s *    +  \ :       '   +             & cCsmt|tot|d}nt}x4to,|id}|pPn|i|q/W|iS(sz Calculate and return the MD5 checksum for a given file. C{file} may either be a filename or an open stream. trbiii@(R&R'RRR1RR!t hexdigest(tfilet md5_digestR((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRXs   cCs>x7t|||D]#}t|to t|qqWdS(sZ Extract the contents of the zip file C{filename} into the directory C{root}. N(RR&RIRJ(RtrootRRL((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRjsccs|o5tiidtii|dtiinyti|}WnPti j o}t |dVdSn(t j o}t ||VdSnX|i }g}|D]!}|i do ||qq~}g} |D]!}|i dp | |qq~ } tii|pti|nxt|D]w} | d id} x[tt| D]G} tii|| | d }tii|pti|ququWqIWxt| D]\} }tii||id}t|d}y|i|}Wn't j o}t ||VdSnX|i||i|oN| dt| | ddt| jo!tiidtiiqqW|oHndS( Ns Unzipping %sisError with downloaded zip fileR iRi Re(RtstdoutRRRRtflushtzipfiletZipFileterrorRIRJtnamelistRRRRnRRjRRRRR(RR"RtzfRR(R:RftdirlistRtfilelisttdirnametpiecesRtdirpathRtouttcontents((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyRssL$  55     4c Csg}xttii|dD]\}}}ti|i}d||tii|idf}td|iD}|i dd||i dd|i |i ddt |i|i d ||i d ||i |q"Wt ttii|d } t } x[|| D]O} | id | jotd | id n| i| id q3Wtid} | i tidx|D]} | di | qW| i tid x| D]}| di |qWt| | S(s  Create a new data.xml index file, by combining the xml description files for various packages and collections. C{root} should be the path to a directory containing the package xml and zip files; and the collection xml files. The C{root} directory is expected to have the following subdirectories:: root/ packages/ .................. subdirectory for packages corpora/ ................. zip & xml files for corpora grammars/ ................ zip & xml files for grammars taggers/ ................. zip & xml files for taggers tokenizers/ .............. zip & xml files for tokenizers etc. collections/ ............... xml files for collections For each package, there should be two files: C{I{package}.zip} contains the package itself, as a compressed zip file; and C{I{package}.xml} is an xml description of the package. The zipfile C{I{package}.zip} should expand to a single subdirectory named C{I{package/}}. The base filename C{I{package}} must match the identifier given in the package's xml file. For each collection, there should be a single file C{I{collection}.zip}, describing the collection. All identifiers (for both packages and collections) must be unique. R5s%s/%s/%sicssx|]}|iVqWdS(N(t file_size(Rtzf_info((s%/p/zhu/06/nlp/nltk/nltk/downloader.pys sRs%sRRR RR`R sDuplicate UID: %sRi(t_find_packagesRRRRRRRtinfolistRRRRlRt_find_collectionsR9RtaddRtElementt _indent_xml(R"tbase_urlR5tpkg_xmlR)R tzipstatRRR`tuidsR6ttop_eltRAR>((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt build_indexs8&!   RcCst|djo|ipdid|d|_x|D]}t||dq?Wx7|d D]+}|ipdid|d|_qeW|dipdid||d_ndS(s Helper for L{build_index()}: Given an XML ElementTree, modify it (and its descendents) C{text} and C{tail} attributes to generate an indented tree, where each nested element is indented by 2 spaces with respect to its parent. iRs s iN(RjRPRR8ttail(R*RR;((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR8s% )cstiitii|dd|idjo#td|idfntfd|iDotdfndS(sq Helper for L{build_index()}: Perform some checks to make sure that the given package is consistent. iiR s&package identifier mismatch (%s vs %s)c3s4x-|]&}|jo|id VqWdS(R N(R(RR (tuid(s%/p/zhu/06/nlp/nltk/nltk/downloader.pys s s;Zipfile %s.zip does not expand to a single subdirectory %s/N(RRRRR9RRR((R:t zipfilenameR)((R@s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt_check_packages&cCstiddd|gdtidti}|i\}}|idjp|p| o3tdtii|dt i |fn|id S( s Helper for L{build_index()}: Calculate the subversion revision number for a given file (by using C{subprocess} to run C{svn}). tsvnRps-vR#tstderris)Error determining svn_revision for %s: %sii( t subprocesstPopentPIPEt communicatet returncodeRRRRRuRv(RRR#RD((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt _svn_revision s  -ccs{g}xnti|D]]\}}}xK|D]C}|ido-tii||}ti|iVq,q,WqWdS(s Helper for L{build_index()}: Yield a list of ElementTree.Element objects, each holding the xml for a single package collection. s.xmlN(RRRRRRR(R(R"R5R,tsubdirsRRtxmlfile((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR5sc #sddkl}g}xti|D]\}}}di|||}xk|D]c}|idoMtii||}|d d} yti| } Wn+t j o} t d| | fnXyt i |i } Wn+t j o} t d|| fnXtii|d d| id jo#t d | id fntfd | iDot d fn| | |fVqTqTWy|id Wq&t j oq&Xq&WdS(st Helper for L{build_index()}: Yield a list of tuples C{(pkg_xml, zf, subdir)}, where: - C{pkg_xml} is an ElementTree.Element holding the xml for a package - C{zf} is a zipfile.ZipFile for the package's contents. - C{subdir} is the subdirectory (relative to C{root}) where the package was found (e.g. 'corpora' or 'grammars'). i(t _path_fromR s.xmlis.zipsError reading file %r! %siR s&package identifier mismatch (%s vs %s)c3s4x-|]&}|jo|id VqWdS(R N(R(RR (R@(s%/p/zhu/06/nlp/nltk/nltk/downloader.pys Is s;Zipfile %s.zip does not expand to a single subdirectory %s/s.svnN(tnltk.corpus.reader.utilRMRRRRRR%R&RJRRR(RRR9RR(R( R"RMR5R,RKRtrelpathRt xmlfilenameRAR)RR:((R@s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR3%s: cCsttidS(N(Rt _downloaderR(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pytdownload_shell\scCsttidS(N(RRQR(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyt download_gui]scCstidS(N(RQR!(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pyR!^st__main__(t OptionParsers-ds--dirtdesttdirRs!download package to directory DIRtmetavartDIRs-qs--quietRtactiont store_truetdefaults work quietlys-fs--forceRs"download even if already installeds-es--exit-on-errorRsexit if an error occursRRP(OR/RtreRR%RRuRRt cStringIORthashlibRR1RtTkinterRtnltk.draw.tableRt nltk.drawRRkRRt nltk.etreeRRRtimport_from_stdlibRtobjectRR3R<R=R?R@RBRCRDRERFRGRHRIRMRORQRRRRRRER>R8RBRJR5R3RQRRRRSR!R-toptparseRUtparsert add_optiont parse_argsRR tpkg_idRWRRRtrv(((s%/p/zhu/06/nlp/nltk/nltk/downloader.pysEs]`    L& er#  2 C    5