B_Kc@s'ddkZddkZddkZddkZddkZddkZddkZddkZddkZddk l Z yddk l Z Wn#ej oddkl Z nXdZdagadddZddddedZeiieiieiie dd Zd efd YZeid Zd Z eidZ!dZ"eidZ#dZ$dZ%dZ&dZ'dZ(de)fdYZ*dd!dYZ+dd"d#ddedZ,dZ-dZ.de)fdYZ/dZ0d Z1dS($iN(t__file__(t cElementTree(t ElementTreecCskxLtid|D]8}|ddjp |djotd|qqWd}tid||S(s Convert all grouping parenthases in the given regexp pattern to non-grouping parenthases, and return the result. E.g.: >>> convert_regexp_to_nongrouping('ab(c(x+)(z*))?d') 'ab(?:c(?:x+)(?:z*))?d' @type pattern: C{str} @rtype: C{str} s \\.|\(\?P=it 0123456789s(?P=s>Regular expressions with back-references are not supported: %rcSstidd|iS(Ns^\((\?P<[^>]*>)?$s(?:(tretsubtgroup(tm((s$/p/zhu/06/nlp/nltk/nltk/internals.pytsubfunc0ss(?x) \\. | # Backslashed character \(\?P<[^>]*> | # Named group \(\? | # Extension group \( # Grouping parenthasis(Rtfindallt ValueErrorR(tpatterntsR((s$/p/zhu/06/nlp/nltk/nltk/internals.pytconvert_regexp_to_nongroupings  cCs\td|dddga|dj o0t|to|i}nt|andS(s  Configure nltk's java interface, by letting nltk know where it can find the C{java} binary, and what extra options (if any) should be passed to java when it is run. @param bin: The full path to the C{java} binary. If not specified, then nltk will search the system for a C{java} binary; and if one is not found, it will raise a C{LookupError} exception. @type bin: C{string} @param options: A list of options that should be passed to the C{java} binary when it is called. A common value is C{['-Xmx512m']}, which tells the C{java} binary to increase the maximum heap size to 512 megabytes. If no options are specified, then do not modify the options list. @type options: C{list} of C{string} tjavatenv_varstJAVAHOMEt JAVA_HOMEN(t find_binaryt _java_bintNonet isinstancet basestringtsplittlistt _java_options(tbintoptions((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt config_javaFs  cCsI|djo ti}n|djo ti}n|djo ti}nt|totdnttjo tn|tjo t}n|dt7}t |}d|g|}tgt |}ti |d|d|d|}|p|Sn|i \}}|i djo|GHtd n||fS( s Execute the given java command, by opening a subprocess that calls C{java}. If java has not yet been configured, it will be configured by calling L{config_java()} with no arguments. @param cmd: The java command that should be called, formatted as a list of strings. Typically, the first string will be the name of the java class; and the remaining strings will be arguments for that java class. @type cmd: C{list} of C{string} @param classpath: A C{':'} separated list of directories, JAR archives, and ZIP archives to search for class files. @type classpath: C{string} @param stdin, stdout, stderr: Specify the executed programs' standard input, standard output and standard error file handles, respectively. Valid values are C{subprocess.PIPE}, an existing file descriptor (a positive integer), an existing file object, and C{None}. C{subprocess.PIPE} indicates that a new pipe to the child should be created. With C{None}, no redirection will occur; the child's file handles will be inherited from the parent. Additionally, stderr can be C{subprocess.STDOUT}, which indicates that the stderr data from the applications should be captured into the same file handle as for stdout. @param blocking: If C{false}, then return immediately after spawning the subprocess. In this case, the return value is the C{Popen} object, and not a C{(stdout, stderr)} tuple. @return: If C{blocking=True}, then return a tuple C{(stdout, stderr)}, containing the stdout and stderr outputs generated by the java command if the C{stdout} and C{stderr} parameters were set to C{subprocess.PIPE}; or C{None} otherwise. If C{blocking=False}, then return a C{subprocess.Popen} object. @raise OSError: If the java command returns a nonzero return code. tpipescmd should be a list of stringst:s-cptstdintstdouttstderrisJava command failed!(t subprocesstPIPERRt TypeErrorRRRtNLTK_JARRRtPopent communicatet returncodetOSError(tcmdt classpathRR R!tblockingtp((s$/p/zhu/06/nlp/nltk/nltk/internals.pyR_s()     !isnltk.jart ParseErrorcBs eZdZdZdZRS(s Exception raised by parse_* functions when they fail. @param position: The index in the input string where an error occured. @param expected: What was expected when an error occured. cCs)ti|||||_||_dS(N(R t__init__texpectedtposition(tselfR0R1((s$/p/zhu/06/nlp/nltk/nltk/internals.pyR/s cCsd|i|ifS(NsExpected %s at %s(R0R1(R2((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt__str__s(t__name__t __module__t__doc__R/R3(((s$/p/zhu/06/nlp/nltk/nltk/internals.pyR.s s[uU]?[rR]?(\"\"\"|\'\'\'|\"|\')cCsti||}|ptd|n|id}tid|}|i}xcto[|i||}|ptd|n|iddjo|id}q]Pq]Wy't |||i!|ifSWn(t j o}td|t nXdS( s If a Python string literal begins at the specified position in the given string, then return a tuple C{(val, end_position)} containing the value of the string literal and the position where it ends. Otherwise, raise a L{ParseError}. s open quoteis\\|%ss close quoteis\svalid string (%s)N( t_STRING_START_REtmatchR.RRtcompiletendtTruetsearchtevalR tstart(R tstart_positionRt quotemarkt_STRING_END_RER1R8te((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt parse_strs  *'s-?\d+cCsHti||}|ptd|nt|i|ifS(s If an integer begins at the specified position in the given string, then return a tuple C{(val, end_position)} containing the value of the integer and the position where it ends. Otherwise, raise a L{ParseError}. tinteger(t _PARSE_INT_RER8R.tintRR:(R R?R((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt parse_intss-?(\d*)([.]?\d*)?cCsti||}| p!|idp |id otd|n|ido t|i|ifSnt|i|ifSdS(s If an integer or float begins at the specified position in the given string, then return a tuple C{(val, end_position)} containing the value of the number and the position where it ends. Otherwise, raise a L{ParseError}. iitnumberN(t_PARSE_NUMBER_VALUER8RR.tfloatR:RF(R R?R((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt parse_numbers )0cCst|tior|idj ob|i}g}t|iD](}||ijo||i|q@q@~}t|djSn t ddS(s @return: True if C{method} overrides some method with the same name in a base class. This is typically used when defining abstract base classes or interfaces, to allow subclasses to define either of two related methods: >>> class EaterI: ... '''Subclass must define eat() or batch_eat().''' ... def eat(self, food): ... if overridden(self.batch_eat): ... return self.batch_eat([food])[0] ... else: ... raise NotImplementedError() ... def batch_eat(self, foods): ... return [self.eat(food) for food in foods] @type method: instance method isExpected an instance method.N( Rttypest MethodTypetim_classRR4t_mrot__dict__tlenR$(tmethodtnamet_[1]tclstfuncs((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt overridden s# +cCsSt|to |iSn5|g}x$|iD]}|it|q.W|SdS(sX Return the I{method resolution order} for C{cls} -- i.e., a list containing C{cls} and all its base classes, in the order in which they would be checked by C{getattr}. For new-style classes, this is just cls.__mro__. For classic classes, this can be obtained by a depth-first left-to-right traversal of C{__bases__}. N(Rttypet__mro__t __bases__textendRO(RUtmrotbase((s$/p/zhu/06/nlp/nltk/nltk/internals.pyRO's    c Csd}|ioL|iid|_tid|ii}|ot|}qen d|_|itid||fd|d|d7_dS( s4Add an epytext @field to a given object's docstring.ts s(?<=\n)[ ]+(?!\s)s@%s: %stinitial_indenttsubsequent_indents N(R6trstripRR t expandtabstminttextwraptfill(tobjtfieldtmessagetindenttindents((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt_add_epytext_field;s  csfd}|S(s A decorator used to mark functions as deprecated. This will cause a warning to be printed the when the function is used. Usage: >>> @deprecated('Use foo() instead') >>> def bar(x): ... print x/10 csdifdtiddddfd}|iiii|_i|_t|_t|d|S(Ns&Function %s() has been deprecated. %ss R_s R`cs&tidtdd||S(Ntcategoryt stackleveli(twarningstwarntDeprecationWarning(targstkwargs(tfunctmsg(s$/p/zhu/06/nlp/nltk/nltk/internals.pytnewFuncZst deprecated( R4RdReRPtupdateR6R;t__deprecated__Rk(RsRu(Rh(RsRts$/p/zhu/06/nlp/nltk/nltk/internals.pyt decoratorUs    ((RhRy((Rhs$/p/zhu/06/nlp/nltk/nltk/internals.pyRvLs t DeprecatedcBseZdZdZRS(sK A base class used to mark deprecated classes. A typical usage is to alert users that the name of a class has changed: >>> class OldClassName(Deprecated, NewClassName): ... "Use NewClassName instead." The docstring of the deprecated class will be used in the deprecation warning message. c Osd}x/t|D]!}t|ijo |}PqqW|p td|ip di}tidd|}tidd|}d|i }||jo|d|i 7}nd||f}dt i |d d d d }t i |d td dti|||S(Ns-Unable to determine which base is deprecated.R^s\A\s*@deprecated:s(?m)^\s*sClass %ss (base class for %s)s%s has been deprecated. %ss R_s R`RlRmi(RRORzRZtAssertionErrorR6tstripRRR4RdReRnRoRptobjectt__new__(RURqRrtdep_clsR]tdocRSRt((s$/p/zhu/06/nlp/nltk/nltk/internals.pyR~ss"    (R4R5R6R~(((s$/p/zhu/06/nlp/nltk/nltk/internals.pyRzhs tCountercBs#eZdZddZdZRS(sE A counter that auto-increments each time its value is read. icCs ||_dS(N(t_value(R2t initial_value((s$/p/zhu/06/nlp/nltk/nltk/internals.pyR/scCs|id7_|iS(Ni(R(R2((s$/p/zhu/06/nlp/nltk/nltk/internals.pytgets(R4R5R6R/R(((s$/p/zhu/06/nlp/nltk/nltk/internals.pyRs cCse|djo |g}nt|tptt|t ptt|t ptt|to|i}n|dj otii|o|Snx|D]~}tiitii||otii||Sntiitii|d|otii|d|SqqWt d||fnx!|D]}|ti joti |}tii|o)|od||fGHnti |Sqlx|D]} tiiti || }tii|o"|od||fGHn|Sntiiti |d| }tii|o"|od||fGHn|SqqWqSqSWxO|D]G} x>|D]6}tii| |}tii|o|SqqWqwWti djox|D]}yt i d|gdt i} | i\} } | i}|i|o5tii|o"|od||fGHn|SnWqtj o }qqXqWnd||f}|o|d|d 7}nti|d d d d d }|d||f7}|o+|d7}|did|D7}n|o|d||f7}ndd}td|||fdS(sp Search for the binary for a program that is used by nltk. @param name: The name of the program @param path_to_bin: The user-supplied binary location, or None. @param env_vars: A list of environment variable names to check @param binary_names: A list of alternative binary names to check. @param searchpath: List of directories to search. RsCould not find %s binary at %ss[Found %s: %s]tposixtwhichR s;NLTK was unable to find the %s executable! Use config_%s()s# or set the %s environment variableit.R_s R`s" >>> config_%s('/path/to/%s')s Searched in:R^cssx|]}d|VqWdS(s - %sN((t.0td((s$/p/zhu/06/nlp/nltk/nltk/internals.pys ss. For more information, on %s, see: <%s>t=iKs %s %s %sN(RRRR{RtostpathtisfiletjoinR tenvironRSR"R&R#R'R|tendswithtexiststKeyboardInterruptRdRet LookupError(RSt path_to_binRt searchpatht binary_namesturltverboseRtenv_vartbin_namet directoryR-R R!Rt SystemExitRttdiv((s$/p/zhu/06/nlp/nltk/nltk/internals.pyRs  "%   #     !  cCsZti}g}tiD]}|djo ||qq~t_t|}|t_|S(s When python is run from within the nltk/ directory tree, the current directory is included at the beginning of the search path. Unfortunately, that means that modules within nltk can sometimes shadow standard library modules. As an example, the stdlib 'inspect' module will attempt to import the stdlib 'tokenzie' module, but will instead end up importing NLTK's 'tokenize' module instead (causing the import to fail). R^R(R^R(tsysRt __import__(tmoduletold_pathRTRR((s$/p/zhu/06/nlp/nltk/nltk/internals.pytimport_from_stdlibs 8  cBsed}|i|\}}}}d|i}|i|||d}d||fdU|ie_|ie_|ie_ee_e eddeS(s; A decorator used to mark methods as abstract. I.e., methods that are marked by this decorator must be overridden by subclasses. If an abstract method is called (either in the base class or in a subclass that does not override the base class method), it will raise C{NotImplementedError}. tinspects%s is an abstract method.s,def newfunc%s: raise NotImplementedError(%r)NtnotesThis method is abstract.(( Rt getargspecR4t formatargspect func_defaultstnewfuncR6R;t __abstract__Rk(RsRRqtvarargstvarkwtdefaultsRtt signature((s$/p/zhu/06/nlp/nltk/nltk/internals.pytabstracts      tElementWrappercBseZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZedZdZdZdZRS(s A wrapper around ElementTree Element objects whose main purpose is to provide nicer __repr__ and __str__ methods. In addition, any of the wrapped Element's methods that return other Element objects are overridden to wrap those values before returning them. This makes Elements more convenient to work with in interactive sessions and doctests, at the expense of some efficiency. cCs,t|to|Sntit|SdS(s Create and return a wrapper around a given Element object. If C{etree} is an C{ElementWrapper}, then C{etree} is returned as-is. N(RRR}R~(RUtetree((s$/p/zhu/06/nlp/nltk/nltk/internals.pyR~CscCs4t|toti|}n||id(RttostringRRQtrfind(R2R RB((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt__repr__bs !cCsti|iiS(sr @return: the result of applying C{ElementTree.tostring()} to the wrapped Element object. (RRRRa(R2((s$/p/zhu/06/nlp/nltk/nltk/internals.pyR3jscCst|i|S(N(tgetattrR(R2tattrib((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt __getattr__uscCst|i||S(N(tsetattrR(R2tattrtvalue((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt __setattr__xscCst|i|S(N(tdelattrR(R2R((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt __delattr__{scCs||i|tstoptelements((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt __setslice__scCs|i||5dS(N(R(R2R>R((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt __delslice__scCs t|iS(N(RQR(R2((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt__len__scCst|i|S(N(RR(R2R((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt __getitem__scCs/g}|i||!D]}|t|q~S(N(RR(R2R>RRTtelt((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt __getslice__scCs(g}|iD]}|t|q~S(N(RR(R2RTR((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt getchildrenscCsd|ii|DS(Ncssx|]}t|VqWdS(N(R(RR((s$/p/zhu/06/nlp/nltk/nltk/internals.pys s (Rt getiterator(R2ttag((s$/p/zhu/06/nlp/nltk/nltk/internals.pyRscCst|ii||S(N(RRt makeelement(R2RR((s$/p/zhu/06/nlp/nltk/nltk/internals.pyRscCs5|ii|}|djo|Sn t|SdS(N(RtfindRR(R2RR((s$/p/zhu/06/nlp/nltk/nltk/internals.pyRscCs1g}|ii|D]}|t|q~S(N(RR R(R2RRTR((s$/p/zhu/06/nlp/nltk/nltk/internals.pyR s(R4R5R6R~R/RRR3RRRRRRRRRRRRRRRR (((s$/p/zhu/06/nlp/nltk/nltk/internals.pyR6s*                cCs!|itj otd|iin|i|i}}|tjo d}n|tjot|}n|djotdt||}n|djotdt||}n|djo6y||dWqt j ot|}qXnt ||}||fS(sl Given a slice, return the corresponding (start, stop) bounds, taking into account None indices and negative indices. The following guarantees are made for the returned start and stop values: - 0 <= start <= len(sequence) - 0 <= stop <= len(sequence) - start <= stop @raise ValueError: If C{slice_obj.step} is not C{None}. s)slices with steps are not supported by %sii( tstepRR t __class__R4R>RRQtmaxt IndexErrorRc(tsequencet slice_objR>R((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt slice_boundss ** #cCstii|ptSnttdoti|}ti|i}|d@otSq|i ti jo|d@otSq|i ti jo|d@otSqtSntS(Ntgetuidiii( RRRtFalsethasattrtstattS_IMODEtst_modeR;tst_uidRtst_gidtgetgid(Rtstatdatatperm((s$/p/zhu/06/nlp/nltk/nltk/internals.pyt is_writables !!((((2R"Rtos.pathRRnRdRLRRtnltkRt xml.etreeRRt ImportErrort nltk.etreeR RRRRR;RRtabspathRRR%R R.R9R7RCRERGRIRKRWRORkRvR}RzRRRRRRR(((s$/p/zhu/06/nlp/nltk/nltk/internals.pys sP         # &  L%         *  [  %w *