³ņ 3ŅĒIc@sŗdZddkZddklZddklZddklZlZl Z l Z l Z de fd„ƒYZ d„Zd e fd „ƒYZd efd „ƒYZd eefd„ƒYZde fd„ƒYZdefd„ƒYZde fd„ƒYZde fd„ƒYZdefd„ƒYZd„Zd„Zd„Zd„Zd„Zed„Zed„Zd „Zed!ƒd"„ƒZ d#ee fd$„ƒYZ!d%„Z"dddd&„Z$ei%d'ei&ƒZ'ei%d(ƒZ(d)„Z)d*„Z*d+„Z+ed,ƒZ,ed-ƒZ-d.„Z.d/„Z/d0„Z0d1„Z1d2„Z2e3d3jo e2ƒndd4d d d ddddd5d6d7d8d9d:d;d<d=d>d?d@dAdBdCdDdEdFgZ4dS(Gs Basic data classes for representing context free grammars. A X{grammar} specifies which trees can represent the structure of a given text. Each of these trees is called a X{parse tree} for the text (or simply a X{parse}). In a X{context free} grammar, the set of parse trees for any piece of a text can depend only on that piece, and not on the rest of the text (i.e., the piece's context). Context free grammars are often used to find possible syntactic structures for sentences. In this context, the leaves of a parse tree are word tokens; and the node values are phrasal categories, such as C{NP} and C{VP}. The L{ContextFreeGrammar} class is used to encode context free grammars. Each C{ContextFreeGrammar} consists of a start symbol and a set of productions. The X{start symbol} specifies the root node value for parse trees. For example, the start symbol for syntactic parsing is usually C{S}. Start symbols are encoded using the C{Nonterminal} class, which is discussed below. A Grammar's X{productions} specify what parent-child relationships a parse tree can contain. Each production specifies that a particular node can be the parent of a particular set of children. For example, the production C{ -> } specifies that an C{S} node can be the parent of an C{NP} node and a C{VP} node. Grammar productions are implemented by the C{Production} class. Each C{Production} consists of a left hand side and a right hand side. The X{left hand side} is a C{Nonterminal} that specifies the node type for a potential parent; and the X{right hand side} is a list that specifies allowable children for that parent. This lists consists of C{Nonterminals} and text types: each C{Nonterminal} indicates that the corresponding child may be a C{TreeToken} with the specified node type; and each text type indicates that the corresponding child may be a C{Token} with the with that type. The C{Nonterminal} class is used to distinguish node values from leaf values. This prevents the grammar from accidentally using a leaf value (such as the English word "A") as the node of a subtree. Within a C{ContextFreeGrammar}, all node values are wrapped in the C{Nonterminal} class. Note, however, that the trees that are specified by the grammar do B{not} include these C{Nonterminal} wrappers. Grammars can also be given a more procedural interpretation. According to this interpretation, a Grammar specifies any tree structure M{tree} that can be produced by the following procedure: - Set M{tree} to the start symbol - Repeat until M{tree} contains no more nonterminal leaves: - Choose a production M{prod} with whose left hand side M{lhs} is a nonterminal leaf of M{tree}. - Replace the nonterminal leaf with a subtree, whose node value is the value wrapped by the nonterminal M{lhs}, and whose children are the right hand side of M{prod}. The operation of replacing the left hand side (M{lhs}) of a production with the right hand side (M{rhs}) in a tree (M{tree}) is known as X{expanding} M{lhs} to M{rhs} in M{tree}. i’’’’N(t deprecated(tImmutableProbabilisticMixIn(t FeatStructtFeatDicttFeatStructParsertSLASHtTYPEt NonterminalcBs_eZdZd„Zd„Zd„Zd„Zd„Zd„Zd„Z d„Z d „Z RS( s: A non-terminal symbol for a context free grammar. C{Nonterminal} is a wrapper class for node values; it is used by C{Production}s to distinguish node values from leaf values. The node value that is wrapped by a C{Nonterminal} is known as its X{symbol}. Symbols are typically strings representing phrasal categories (such as C{"NP"} or C{"VP"}). However, more complex symbol types are sometimes used (e.g., for lexicalized grammars). Since symbols are node values, they must be immutable and hashable. Two C{Nonterminal}s are considered equal if their symbols are equal. @see: L{ContextFreeGrammar} @see: L{Production} @type _symbol: (any) @ivar _symbol: The node value corresponding to this C{Nonterminal}. This value must be immutable and hashable. cCs||_t|ƒ|_dS(só Construct a new non-terminal from the given symbol. @type symbol: (any) @param symbol: The node value corresponding to this C{Nonterminal}. This value must be immutable and hashable. N(t_symbolthasht_hash(tselftsymbol((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt__init__fs cCs|iS(sf @return: The node value corresponding to this C{Nonterminal}. @rtype: (any) (R(R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR rscCsEy'|i|ijot||iƒSWntj o tSnXdS(s @return: True if this non-terminal is equal to C{other}. In particular, return true iff C{other} is a C{Nonterminal} and this non-terminal's symbol is equal to C{other}'s symbol. @rtype: C{boolean} N(Rt isinstancet __class__tAttributeErrortFalse(R tother((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt__eq__ys cCs ||j S(s @return: True if this non-terminal is not equal to C{other}. In particular, return true iff C{other} is not a C{Nonterminal} or this non-terminal's symbol is not equal to C{other}'s symbol. @rtype: C{boolean} ((R R((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt__ne__‡scCs)yt|i|iƒSWn dSnXdS(Ni’’’’(tcmpR(R R((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt__cmp__‘scCs|iS(N(R (R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt__hash__—scCs7t|itƒod|ifSnd|ifSdS(se @return: A string representation for this C{Nonterminal}. @rtype: C{string} s%ss%rN(RRt basestring(R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt__repr__šscCs7t|itƒod|ifSnd|ifSdS(se @return: A string representation for this C{Nonterminal}. @rtype: C{string} s%ss%rN(RRR(R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt__str__¤scCstd|i|ifƒS(sa @return: A new nonterminal whose symbol is C{M{A}/M{B}}, where C{M{A}} is the symbol for this nonterminal, and C{M{B}} is the symbol for rhs. @rtype: L{Nonterminal} @param rhs: The nonterminal used to form the right hand side of the new nonterminal. @type rhs: L{Nonterminal} s%s/%s(RR(R trhs((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt__div__®s ( t__name__t __module__t__doc__R R RRRRRRR(((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRSs     cCsWd|jo|idƒ}n |iƒ}g}|D]}|t|iƒƒq7~S(sź Given a string containing a list of symbol names, return a list of C{Nonterminals} constructed from those symbols. @param symbols: The symbol name string. This string can be delimited by either spaces or commas. @type symbols: C{string} @return: A list of C{Nonterminals} constructed from the symbol names given in C{symbols}. The C{Nonterminals} are sorted in the same order as the symbols names. @rtype: C{list} of L{Nonterminal} t,(tsplitRtstrip(tsymbolst symbol_listt_[1]ts((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt nonterminalsŗs  t ProductioncBs_eZdZd„Zd„Zd„Zd„Zd„Zd„Zd„Z d„Z d „Z RS( s A grammar production. Each production maps a single symbol on the X{left-hand side} to a sequence of symbols on the X{right-hand side}. (In the case of context-free productions, the left-hand side must be a C{Nonterminal}, and the right-hand side is a sequence of terminals and C{Nonterminals}.) X{terminals} can be any immutable hashable object that is not a C{Nonterminal}. Typically, terminals are strings representing words, such as C{"dog"} or C{"under"}. @see: L{ContextFreeGrammar} @see: L{DependencyGrammar} @see: L{Nonterminal} @type _lhs: L{Nonterminal} @ivar _lhs: The left-hand side of the production. @type _rhs: C{tuple} of (C{Nonterminal} and (terminal)) @ivar _rhs: The right-hand side of the production. cCs]t|ttfƒotdƒ‚n||_t|ƒ|_t|i|ifƒ|_dS(s Construct a new C{Production}. @param lhs: The left-hand side of the new C{Production}. @type lhs: L{Nonterminal} @param rhs: The right-hand side of the new C{Production}. @type rhs: sequence of (C{Nonterminal} and (terminal)) s9production right hand side should be a list, not a stringN( Rtstrtunicodet TypeErrort_lhsttuplet_rhsR R (R tlhsR((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR ćs  cCs|iS(sc @return: the left-hand side of this C{Production}. @rtype: L{Nonterminal} (R,(R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR/óscCs|iS(s @return: the right-hand side of this C{Production}. @rtype: sequence of (C{Nonterminal} and (terminal)) (R.(R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRśscCs^d|if}xG|iD]<}t|tƒo|d|f7}q|d|f7}qW|S(sv @return: A verbose string representation of the C{Production}. @rtype: C{string} s%s ->s %ss %r(R,R.RR(R R)telt((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRs cCsd|S(sw @return: A concise string representation of the C{Production}. @rtype: C{string} s%s((R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRscCs6t||iƒo#|i|ijo|i|ijS(sf @return: true if this C{Production} is equal to C{other}. @rtype: C{boolean} (RRR,R.(R R((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRscCs ||j S(N((R R((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR scCs@t||iƒpdSnt|i|if|i|ifƒS(Ni’’’’(RRRR,R.(R R((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR#scCs|iS(sU @return: A hash value for the C{Production}. @rtype: C{int} (R (R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR's( RRRR R/RRRRRRR(((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR(Ļs       tDependencyProductioncBseZdZd„ZRS(s‹ A dependency grammar production. Each production maps a single head word to an unordered list of one or more modifier words. cCs9d|if}x"|iD]}|d|f7}qW|S(s @return: A verbose string representation of the C{DependencyProduction}. @rtype: C{string} s'%s' ->s '%s'(R,R.(R R)R0((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR4s  (RRRR(((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR1/stWeightedProductioncBs;eZdZd„Zd„Zd„Zd„Zd„ZRS(sļ A probabilistic context free grammar production. PCFG C{WeightedProduction}s are essentially just C{Production}s that have probabilities associated with them. These probabilities are used to record how likely it is that a given production will be used. In particular, the probability of a C{WeightedProduction} records the likelihood that its right-hand side is the correct instantiation for any given occurance of its left-hand side. @see: L{Production} cKs'ti||ti|||ƒdS(s{ Construct a new C{WeightedProduction}. @param lhs: The left-hand side of the new C{WeightedProduction}. @type lhs: L{Nonterminal} @param rhs: The right-hand side of the new C{WeightedProduction}. @type rhs: sequence of (C{Nonterminal} and (terminal)) @param prob: Probability parameters of the new C{WeightedProduction}. N(RR R((R R/Rtprob((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR Ms cCsti|ƒd|iƒS(Ns [%s](R(RR3(R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRZscCsOt||iƒo<|i|ijo)|i|ijo|iƒ|iƒjS(N(RRR,R.R3(R R((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR]scCs ||j S(N((R R((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRcscCst|i|i|iƒfƒS(N(R R,R.R3(R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRfs(RRRR RRRR(((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR2As     tContextFreeGrammarcBsSeZdZd„Zd„Zeed„Zd„Zd„Zd„Z d„Z RS(s# A context-free grammar. A Grammar consists of a start state and a set of productions. The set of terminals and nonterminals is implicitly specified by the productions. If you need efficient key-based access to productions, you can use a subclass to implement it. cCsÜ||_||_h|_h|_x±|iD]¦}|i|ijog|i|i­ss3Grammar does not cover some of the input words: %r.N(tlenR;tjoint ValueError(R ttokensR%ttoktmissing((s"/p/zhu/06/nlp/nltk/nltk/grammar.pytcheck_coverage„s 3cCs<x5|D]-}t|id|ƒƒdjotSqqWtS(sÕ Check whether the grammar rules cover the given list of tokens. @param tokens: the given list of tokens. @type tokens: a C{list} of C{string} objects. @return: True/False Ri(R@R;RtTrue(R RCttoken((s"/p/zhu/06/nlp/nltk/nltk/grammar.pytcovers³s  cCsdt|iƒS(Ns(R@R6(R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRĄscCsJdt|iƒ}|d|i7}x|iD]}|d|7}q.W|S(NsGrammar with %d productionss (start state = %s)s %s(R@R6R5(R R)t production((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRĆs  ( RRRR R:tNoneR;RFRIRR(((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR4ns    tGrammarcBseZedƒd„ƒZRS(s$Use nltk.ContextFreeGrammar instead.cOsti|||ŽdS(N(R4R (R targstkwargs((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR Ės(RRRR (((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRLŹstDependencyGrammarcBs;eZdZd„Zd„Zd„Zd„Zd„ZRS(sÆ A dependency grammar. A DependencyGrammar consists of a set of productions. Each production specifies a head/modifier relationship between a pair of words. cCs ||_dS(sÜ Create a new dependency grammar, from the set of C{Production}s. @param productions: The list of productions that defines the grammar @type productions: C{list} of L{Production} N(R6(R R;((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR ÖscCsQxJ|iD]?}x6|iD]+}|i|jo||jotSqqWq WtS(sB @param head: A head word. @type head: C{string}. @param mod: A mod word, to test as a modifier of 'head'. @type mod: C{string}. @return: true if this C{DependencyGrammar} contains a C{DependencyProduction} mapping 'head' to 'mod'. @rtype: C{boolean}. (R6R.R,RGR(R theadtmodRJt possibleMod((s"/p/zhu/06/nlp/nltk/nltk/grammar.pytcontainsßs  cCsQxJ|iD]?}x6|iD]+}|i|jo||jotSqqWq WtS(sB @param head: A head word. @type head: C{string}. @param mod: A mod word, to test as a modifier of 'head'. @type mod: C{string}. @return: true if this C{DependencyGrammar} contains a C{DependencyProduction} mapping 'head' to 'mod'. @rtype: C{boolean}. (R6R.R,RGR(R RPRQRJRR((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt __contains__šs  cCs9dt|iƒ}x|iD]}|d|7}qW|S(s| @return: A verbose string representation of the C{DependencyGrammar} @rtype: C{string} s&Dependency grammar with %d productionss %s(R@R6(R R)RJ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR s  cCsdt|iƒS(sb @return: A concise string representation of the C{DependencyGrammar} s&Dependency grammar with %d productions(R@R6(R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRs(RRRR RSRTRR(((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyROŠs    tStatisticalDependencyGrammarcBs2eZdZd„Zd„Zd„Zd„ZRS(s cCs||_||_||_dS(N(R6t_eventst_tags(R R;teventsttags((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR %s  cCsQxJ|iD]?}x6|iD]+}|i|jo||jotSqqWq WtS(sB @param head: A head word. @type head: C{string}. @param mod: A mod word, to test as a modifier of 'head'. @type mod: C{string}. @return: true if this C{DependencyGrammar} contains a C{DependencyProduction} mapping 'head' to 'mod'. @rtype: C{boolean}. (R6R.R,RGR(R RPRQRJRR((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRS*s  c Cs«dt|iƒ}x|iD]}|d|7}qW|d7}x,|iD]!}|d|i||f7}qIW|d7}x,|iD]!}|d||i|f7}q‚W|S(s‡ @return: A verbose string representation of the C{StatisticalDependencyGrammar} @rtype: C{string} s2Statistical dependency grammar with %d productionss %ss Events:s %d:%ss Tags:s %s: (%s)(R@R6RVRW(R R)RJteventttag_word((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR;s     cCsdt|iƒS(sm @return: A concise string representation of the C{StatisticalDependencyGrammar} s2Statistical Dependency grammar with %d productions(R@R6(R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRLs(RRRR RSRR(((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRU s    tWeightedGrammarcBseZdZdZd„ZRS(sŃ A probabilistic context-free grammar. A Weighted Grammar consists of a start state and a set of weighted productions. The set of terminals and nonterminals is implicitly specified by the productions. PCFG productions should be C{WeightedProduction}s. C{WeightedGrammar}s impose the constraint that the set of productions with any given left-hand-side must have probabilities that sum to 1. If you need efficient key-based access to productions, you can use a subclass to implement it. @type EPSILON: C{float} @cvar EPSILON: The acceptable margin of error for checking that productions with a given left-hand side have probabilities that sum to 1. g{®Gįz„?cCs·ti|||ƒh}x:|D]2}|i|iƒdƒ|iƒ||iƒ B C in a PCFG is: | count(A -> B C) | P(B, C | A) = --------------- where * is any right hand side | count(A -> *) @param start: The start symbol @type start: L{Nonterminal} @param productions: The list of productions that defines the grammar @type productions: C{list} of L{Production} iiR3(R=R/R2RtfloatR\(R:R;tpcounttlcountR<R%R`tprods((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt induce_pcfg…s&McCs t|tƒS(s' Returns a list of productions (tparse_productiontstandard_nonterm_parser(R&((s"/p/zhu/06/nlp/nltk/nltk/grammar.pytparse_cfg_production«scCs"t|tƒ\}}t||ƒS(N(t parse_grammarRgR4(R&R:R;((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt parse_cfg±scCst|tdtƒS(s, Returns a list of PCFG productions t probabilistic(RfRgRG(R&((s"/p/zhu/06/nlp/nltk/nltk/grammar.pytparse_pcfg_production·scCs(t|tdtƒ\}}t||ƒS(NRk(RiRgRGR\(R&R:R;((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt parse_pcfg½s cCs¾d}|||ƒ\}}tidƒi||ƒ}|ptdƒ‚n|iƒ}dg}t}}gg} xŅ|t|ƒjo¾tidƒi||ƒ}|oc|o\|iƒ}t|idƒdd!ƒ|d<|ddjotd |dfƒ‚q'n||d jo€tidƒi||ƒ}|ptd ƒ‚n|otdƒ‚nt }| di |idƒdd!ƒ|iƒ}n‡||djoI|i dƒt}}| i gƒtidƒi||ƒiƒ}n-|||ƒ\} }| di | ƒt }|o|otdƒ‚qxqxW|oAg} t | |ƒD]"\} } | t || d| ƒqe~ Sn)g}| D]} |t || ƒq~SdS(Nis\s*->\s*sExpected an arrowgs(\[[\d\.]+\])\s*ii’’’’gš?s9Production probability %f, should not be greater than 1.0s'"s ("[^"]+"|s '[^']+')\s*sUnterminated strings7Bad right-hand-side: do not use a sequence of terminalst|s\|\s*s;Bad right-hand-side: do not mix terminals and non-terminalsR3s("[^"]+"|'[^']+')\s*(tretcompiletmatchRBtendRR@RatgroupRGR9tzipR2R((tlinetnonterm_parserRktposR/tmt probabilitiestfound_terminaltfound_non_terminaltrhsidestnontermR%Rt probabilityt_[2]((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRfÄsL      !   %:c CsĖt|tƒo|idƒ}n|}t}g}d}xTt|ƒD]F\}}||iƒ}|idƒp |djoqHn|idƒo|d iƒd}qHnd}y |ddjou|d itd ƒ\} } | d jo<|| dƒ\}} | t | ƒjot d ƒ‚q>qXt d ƒ‚n|t |||ƒ7}WqHt j o&} t d |d || fƒ‚qHXqHW|p t d‚n|p|di ƒ}n||fS(sS Return a starting category and a list of C{Production}s. @param input: a grammar, either in the form of a string or else as a list of strings. @param nonterm_parser: a function for parsing nonterminals. It should take a C{(string,position)} as argument and return a C{(nonterminal,position)} as result. s tt#s\i’’’’t it%iR:sBad argument to start directives Bad directivesUnable to parse line %s: %s %ssNo productions found!( RR)R!RKt enumerateR"t startswithtendswithtrstripR@RBRfR/( tinputRvRktlinesR:R;t continue_linetlinenumRut directiveRMRwte((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRis>  $   cCsYtidƒi||ƒ}|ptd||ƒ‚nt|idƒƒ|iƒfS(Ns ([\w/]+)\s*sExpected a nonterminal, found: i(RoRpRqRBRRsRr(tstringRwRx((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRg0s sUse nltk.parse_fcfg() instead.cCs t|ƒS(N(t parse_fcfg(Rˆ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt parse_featcfg6stFeatStructNonterminalcBs eZdZd„Zd„ZRS(s|A feature structure that's also a nonterminal. It acts as its own symbol, and automatically freezes itself when hashed.cCs|iƒti|ƒS(N(tfreezeRR(R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR@s cCs|S(N((R ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR Cs(RRRRR (((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR‘=s cCs t||ƒS(N(Rf(Rˆtfstruct_parser((s"/p/zhu/06/nlp/nltk/nltk/grammar.pytparse_fcfg_productionFscCs…|tjottf}n|tjot|td|ƒ}n|tj otdƒ‚nt||iƒ\}}t||ƒS(s» Return a feature structure grammar. @param input: a grammar, either in the form of a string or else as a list of strings. @param features: a tuple of features (default: SLASH, TYPE) @param logic_parser: a parser for lambda-expressions (default: LogicParser()) @param fstruct_parser: a feature structure parser (only if features and logic_parser is None) t logic_parsers8'logic_parser' and 'fstruct_parser' must not both be set( RKRRRR‘t ExceptionRit partial_parseR4(RˆtfeaturesR•R“R:R;((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyRIs     sH^\s* # leading whitespace ('[^']+')\s* # single-quoted lhs (?:[-=]+>)\s* # arrow (?:( # rhs: "[^"]+" # doubled-quoted terminal | '[^']+' # single-quoted terminal | \| # disjunction ) \s*) # trailing space *$s"('[^']'|[-=]+>|"[^"]+"|'[^']+'|\|)cCsĆg}xt|idƒƒD]y\}}|iƒ}|idƒp |djoqny|t|ƒ7}Wqtj otd||f‚qXqWt|ƒdjo td‚nt|ƒS(Ns RR€sUnable to parse line %s: %sisNo productions found!(R„R!R"R…tparse_dependency_productionRBR@RO(R&R;R‹Ru((s"/p/zhu/06/nlp/nltk/nltk/grammar.pytparse_dependency_grammarts  $ c Cs’ti|ƒp td‚nti|ƒ}g}t|ƒD](\}}|ddjo ||q=q=~}|didƒ}gg}xJ|dD]>}|djo|igƒq•|di|idƒƒq•Wg}|D]} |t|| ƒqā~S(NsBad production stringiiis'"Rni’’’’( t _PARSE_DG_RERqRBt _SPLIT_DG_RER!R„R"R9R1( R&tpiecesR%tiR`tlhsideR|tpieceRtrhside((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyR™€s B   c Csddkl}l}l}|dƒ\}}}}|dƒ\}}} } ||} dG||||||| | ||g GHdG|iƒ GHH|||gƒGH|dƒ} dG| GHd G| iƒ GHd G| iƒ id d d dƒGHHdGH| iddgƒGH| iddgƒGHdS(sU A demonstration showing how C{ContextFreeGrammar}s can be created and used. i’’’’(R'R(Rjs S, NP, VP, PPs N, V, P, DetsSome nonterminals:s S.symbol() =>sæ S -> NP VP PP -> P NP NP -> Det N | NP PP VP -> V NP | VP PP Det -> 'a' | 'the' N -> 'dog' | 'cat' V -> 'chased' | 'sat' P -> 'on' | 'in' s A Grammar:s grammar.start() =>s grammar.productions() =>R s, R‚is%Coverage of input words by a grammar:tatdogttoyN( tnltkR'R(RjR R:R;treplaceRI( R'R(RjtStNPtVPtPPtNtVtPtDett VP_slash_NPtgrammar((s"/p/zhu/06/nlp/nltk/nltk/grammar.pytcfg_demo’s$ (   s7 S -> NP VP [1.0] NP -> Det N [0.5] | NP PP [0.25] | 'John' [0.1] | 'I' [0.15] Det -> 'the' [0.8] | 'my' [0.2] N -> 'man' [0.5] | 'telescope' [0.5] VP -> VP PP [0.1] | V NP [0.7] | V [0.2] V -> 'ate' [0.35] | 'saw' [0.65] PP -> P NP [1.0] P -> 'with' [0.61] | 'under' [0.39] sä S -> NP VP [1.0] VP -> V NP [.59] VP -> V [.40] VP -> VP PP [.01] NP -> Det N [.41] NP -> Name [.28] NP -> NP PP [.31] PP -> P NP [1.0] V -> 'saw' [.21] V -> 'ate' [.51] V -> 'ran' [.28] N -> 'boy' [.11] N -> 'cookie' [.12] N -> 'table' [.13] N -> 'telescope' [.14] N -> 'hill' [.5] Name -> 'Jack' [.52] Name -> 'Bob' [.48] P -> 'with' [.61] P -> 'under' [.39] Det -> 'the' [.41] Det -> 'a' [.31] Det -> 'my' [.28] c Csóddkl}ddkl}ddkl}ddkl}tiƒ}|d}dG| GHdG|i ƒ GHd G|i ƒ GHd G|i ƒ GHHt }d G| GHd G|i ƒ GHd G|iƒ iddddƒGHHdGH|iddgƒGH|iddgƒGHdGHg}x_|id D]P}xG|i|ƒD]6} | idtƒ| iddƒ|| iƒ7}q2WqWtdƒ} || |ƒ}|GHHdGH|i|ƒ} | idƒ|idƒdiƒ} | GHx| i| ƒD] } | GHqąWdS(sR A demonstration showing how C{WeightedGrammar}s can be created and used. i’’’’(ttreebank(ttreetransforms(Re(tpchartisA PCFG production:s pcfg_prod.lhs() =>s pcfg_prod.rhs() =>s pcfg_prod.prob() =>sA PCFG grammar:s grammar.start() =>s grammar.productions() =>R s, R‚is%Coverage of input words by a grammar:R¢tboytgirls'Induce PCFG grammar from treebank data:t collapsePOSt horzMarkovR§s%Parse sentence using induced grammar:is wsj_0001.mrgiN(t nltk.corpusR²R„R³Ret nltk.parseR“t toy_pcfg1R;R/RR3t toy_pcfg2R:R¦RIR]t parsed_sentstcollapse_unaryRtchomsky_normal_formRtInsideChartParserttracetleavest nbest_parse(R²R³ReR“t pcfg_prodst pcfg_prodR°R;titemttreeR§tparsertsenttparse((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt pcfg_demoąsR       cCs(ddk}|iidƒ}|GHHdS(Ni’’’’s!grammars/book_grammars/feat0.fcfg(t nltk.datatdatatload(R„tg((s"/p/zhu/06/nlp/nltk/nltk/grammar.pyt fcfg_demos cCstdƒ}|GHdS(s] A demonstration showing the creation and inspection of a C{DependencyGrammar}. sP 'scratch' -> 'cats' | 'walls' 'walls' -> 'the' 'cats' -> 'the' N(Rš(R°((s"/p/zhu/06/nlp/nltk/nltk/grammar.pytdg_demo#s cCs'tdƒ}|iƒ}|iƒGHdS(sh A demonstration of how to read a string representation of a CoNLL format dependency tree. sg 1 Ze ze Pron Pron per|3|evofmv|nom 2 su _ _ 2 had heb V V trans|ovt|1of2of3|ev 0 ROOT _ _ 3 met met Prep Prep voor 8 mod _ _ 4 haar haar Pron Pron bez|3|ev|neut|attr 5 det _ _ 5 moeder moeder N N soort|ev|neut 3 obj1 _ _ 6 kunnen kan V V hulp|ott|1of2of3|mv 2 vc _ _ 7 gaan ga V V hulp|inf 6 vc _ _ 8 winkelen winkel V V intrans|inf 11 cnj _ _ 9 , , Punc Punc komma 8 punct _ _ 10 zwemmen zwem V V intrans|inf 11 cnj _ _ 11 of of Conj Conj neven 7 vc _ _ 12 terrassen terras N N soort|mv|neut 11 cnj _ _ 13 . . Punc Punc punt 12 punct _ _ N(tDependencyGraphRĒtpprint(tdgRĒ((s"/p/zhu/06/nlp/nltk/nltk/grammar.pytsdg_demo/s  cCs'tƒtƒtƒtƒtƒdS(N(R±RĖRŠRŃRÕ(((s"/p/zhu/06/nlp/nltk/nltk/grammar.pytdemoFs t__main__R'ReRjRhRmRlRR”RiRfRšR™RÖR±RĖRŃRÕR»R¼(5RRotnltk.internalsRR~Rt featstructRRRRRtobjectRR'R(R1R2R4RLRORUR\ReRhRjRlRmRRfRiRgRR‘R”RKRRptVERBOSER›RœRšR™R±R»R¼RĖRŠRŃRÕRÖRt__all__(((s"/p/zhu/06/nlp/nltk/nltk/grammar.pysFsn (g `-\P41 &     = /      )   =