B_Kc@sjdZddklZddklZlZlZddklZddk Tddk Z ddk Z de fdYZ d e fd YZd e fd YZd e fdYZde fdYZdefdYZdefdYZdefdYZdefdYZdefdYZdeefdYZdefdYZdeefd YZd!efd"YZd#eefd$YZd%efd&YZd'efd(YZd)efd*YZd+efd,YZ d-efd.YZ!d/d0Z"eeeegZ#eeeegZ$eeeegZ%ee!e gZ&d1e'fd2YZ(d3e(fd4YZ)d5e(fd6YZ*d7e(fd8YZ+d9e(fd:YZ,d;e(fd<YZ-d=Z.e/e0e1e0d>d?d@dAZ2e3dBjo e2ndS(Cs Data classes and parser implementations for "chart parsers", which use dynamic programming to efficiently parse a text. A X{chart parser} derives parse trees for a text by iteratively adding "edges" to a "chart." Each X{edge} represents a hypothesis about the tree structure for a subsequence of the text. The X{chart} is a "blackboard" for composing and combining these hypotheses. When a chart parser begins parsing a text, it creates a new (empty) chart, spanning the text. It then incrementally adds new edges to the chart. A set of X{chart rules} specifies the conditions under which new edges should be added to the chart. Once the chart reaches a stage where none of the chart rules adds any new edges, parsing is complete. Charts are encoded with the L{Chart} class, and edges are encoded with the L{TreeEdge} and L{LeafEdge} classes. The chart parser module defines three chart parsers: - C{ChartParser} is a simple and flexible chart parser. Given a set of chart rules, it will apply those rules to the chart until no more edges are added. - C{SteppingChartParser} is a subclass of C{ChartParser} that can be used to step through the parsing process. i(tTree(tWeightedGrammartis_nonterminalt is_terminal(t defaultdict(t*NtEdgeIcBseZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZRS(sM A hypothesis about the structure of part of a sentence. Each edge records the fact that a structure is (partially) consistent with the sentence. An edge contains: - A X{span}, indicating what part of the sentence is consistent with the hypothesized structure. - A X{left-hand side}, specifying what kind of structure is hypothesized. - A X{right-hand side}, specifying the contents of the hypothesized structure. - A X{dot position}, indicating how much of the hypothesized structure is consistent with the sentence. Every edge is either X{complete} or X{incomplete}: - An edge is X{complete} if its structure is fully consistent with the sentence. - An edge is X{incomplete} if its structure is partially consistent with the sentence. For every incomplete edge, the span specifies a possible prefix for the edge's structure. There are two kinds of edge: - C{TreeEdges} record which trees have been found to be (partially) consistent with the text. - C{LeafEdges} record the tokens occur in the text. The C{EdgeI} interface provides a common interface to both types of edge, allowing chart parsers to treat them in a uniform manner. cCs$|itjotdndS(NsEdge is an abstract interface(t __class__Rt TypeError(tself((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt__init__[scCstddS(s @return: A tuple C{(s,e)}, where C{subtokens[s:e]} is the portion of the sentence that is consistent with this edge's structure. @rtype: C{(int, int)} sEdgeI is an abstract interfaceN(tAssertionError(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytspancscCstddS(sV @return: The start index of this edge's span. @rtype: C{int} sEdgeI is an abstract interfaceN(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytstartlscCstddS(sT @return: The end index of this edge's span. @rtype: C{int} sEdgeI is an abstract interfaceN(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytendsscCstddS(sQ @return: The length of this edge's span. @rtype: C{int} sEdgeI is an abstract interfaceN(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytlengthzscCstddS(s @return: This edge's left-hand side, which specifies what kind of structure is hypothesized by this edge. @see: L{TreeEdge} and L{LeafEdge} for a description of the left-hand side values for each edge type. sEdgeI is an abstract interfaceN(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytlhsscCstddS(s @return: This edge's right-hand side, which specifies the content of the structure hypothesized by this edge. @see: L{TreeEdge} and L{LeafEdge} for a description of the right-hand side values for each edge type. sEdgeI is an abstract interfaceN(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytrhsscCstddS(s! @return: This edge's dot position, which indicates how much of the hypothesized structure is consistent with the sentence. In particular, C{self.rhs[:dot]} is consistent with C{subtoks[self.start():self.end()]}. @rtype: C{int} sEdgeI is an abstract interfaceN(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytdotscCstddS(s @return: The element of this edge's right-hand side that immediately follows its dot. @rtype: C{Nonterminal} or X{terminal} or C{None} sEdgeI is an abstract interfaceN(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytnextscCstddS(s @return: True if this edge's structure is fully consistent with the text. @rtype: C{boolean} sEdgeI is an abstract interfaceN(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt is_completescCstddS(s @return: True if this edge's structure is partially consistent with the text. @rtype: C{boolean} sEdgeI is an abstract interfaceN(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt is_incompletescCstddS(NsEdgeI is an abstract interface(R (R tother((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt__cmp__scCstddS(NsEdgeI is an abstract interface(R (R R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt__hash__s(t__name__t __module__t__doc__R R R RRRRRRRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR6s$      tTreeEdgecBseZdZddZdZeeZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZRS(se An edge that records the fact that a tree is (partially) consistent with the sentence. A tree edge consists of: - A X{span}, indicating what part of the sentence is consistent with the hypothesized tree. - A X{left-hand side}, specifying the hypothesized tree's node value. - A X{right-hand side}, specifying the hypothesized tree's children. Each element of the right-hand side is either a terminal, specifying a token with that terminal as its leaf value; or a nonterminal, specifying a subtree with that nonterminal's symbol as its node value. - A X{dot position}, indicating which children are consistent with part of the sentence. In particular, if C{dot} is the dot position, C{rhs} is the right-hand size, C{(start,end)} is the span, and C{sentence} is the list of subtokens in the sentence, then C{subtokens[start:end]} can be spanned by the children specified by C{rhs[:dot]}. For more information about edges, see the L{EdgeI} interface. icCs.||_t||_||_||_dS(s Construct a new C{TreeEdge}. @type span: C{(int, int)} @param span: A tuple C{(s,e)}, where C{subtokens[s:e]} is the portion of the sentence that is consistent with the new edge's structure. @type lhs: L{Nonterminal} @param lhs: The new edge's left-hand side, specifying the hypothesized tree's node value. @type rhs: C{list} of (L{Nonterminal} and C{string}) @param rhs: The new edge's right-hand side, specifying the hypothesized tree's children. @type dot: C{int} @param dot: The position of the new edge's dot. This position specifies what prefix of the production's right hand side is consistent with the text. In particular, if C{sentence} is the list of subtokens in the sentence, then C{subtokens[span[0]:span[1]]} can be spanned by the children specified by C{rhs[:dot]}. N(t_lhsttuplet_rhst_spant_dot(R R RRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR s  c Cs1td||fd|id|iddS(s) @return: A new C{TreeEdge} formed from the given production. The new edge's left-hand side and right-hand side will be taken from C{production}; its span will be C{(index,index)}; and its dot position will be C{0}. @rtype: L{TreeEdge} R RRRi(RRR(t productiontindex((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytfrom_productionsc Cs9td|id|fd|id|id|idS(s% @return: A new C{TreeEdge} formed from this edge. The new edge's dot position is increased by C{1}, and its end index will be replaced by C{new_end}. @rtype: L{TreeEdge} @param new_end: The new end index. @type new_end: C{int} R iRRRi(RR RRR!(R tnew_end((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytmove_dot_forward s cCs|iS(N(R(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|iS(N(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR scCs |idS(Ni(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR scCs |idS(Ni(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|id|idS(Nii(R (R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|iS(N(R(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|iS(N(R!(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|it|ijS(N(R!tlenR(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|it|ijS(N(R!R'R(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs3|it|ijodSn|i|iSdS(N(R!R'RtNone(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR s!cCsd|i|ijodSnt|i|i|i|if|i|i|i|ifS(Ni(RtcmpR RRR!(R R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR%s$cCs(t|i|i|i|ifS(N(thashRRR R!(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR)scCsd|id|idf}|d|if7}xStt|iD]<}||ijo|d7}n|d|i|f7}qHWt|i|ijo|d7}n|S(Ns[%s:%s] iis%-2r ->s *s %r(R RtrangeR'RR!(R tstrti((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt__str__-s'cCsd|S(Ns [Edge: %s]((R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt__repr__7s(RRRR R$t staticmethodR&RR R RRRRRRRRRR.R/(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs&                tLeafEdgecBseZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZRS(s An edge that records the fact that a leaf value is consistent with a word in the sentence. A leaf edge consists of: - An X{index}, indicating the position of the word. - A X{leaf}, specifying the word's content. A leaf edge's left-hand side is its leaf value, and its right hand side is C{()}. Its span is C{[index, index+1]}, and its dot position is C{0}. cCs||_||_dS(s Construct a new C{LeafEdge}. @param leaf: The new edge's leaf value, specifying the word that is recorded by this edge. @param index: The new edge's index, specifying the position of the word that is recorded by this edge. N(t_leaft_index(R tleafR#((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR Fs cCs|iS(N(R2(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRSscCs|i|idfS(Ni(R3(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR TscCs|iS(N(R3(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR UscCs |idS(Ni(R3(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRVscCsdS(Ni((R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRWscCsdS(N(((R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRXscCsdS(Ni((R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRYscCstS(N(tTrue(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRZscCstS(N(tFalse(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR[scCsdS(N(R((R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR\scCs=t|tpdSnt|i|if|i|ifS(Ni(t isinstanceR1R)R3R2(R R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR_scCst|i|ifS(N(R*R3R2(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRbscCsd|i|id|ifS(Ns [%s:%s] %ri(R3R2(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR.fscCsd|S(Ns [Edge: %s]((R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR/hs(RRRR RR R RRRRRRRRRR.R/(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR1:s               tChartcBseZdZdZdZdZdZdZdZdZ e Z dZ d Z d Z d Zd Zd ZdZedZeedZdZdZdZedZedZedZdZRS(s A blackboard for hypotheses about the syntactic constituents of a sentence. A chart contains a set of edges, and each edge encodes a single hypothesis about the structure of some portion of the sentence. The L{select} method can be used to select a specific collection of edges. For example C{chart.select(is_complete=True, start=0)} yields all complete edges whose start indices are 0. To ensure the efficiency of these selection operations, C{Chart} dynamically creates and maintains an index for each set of attributes that have been selected on. In order to reconstruct the trees that are represented by an edge, the chart associates each edge with a set of child pointer lists. A X{child pointer list} is a list of the edges that license an edge's right-hand side. @ivar _tokens: The sentence that the chart covers. @ivar _num_leaves: The number of tokens. @ivar _edges: A list of the edges in the chart @ivar _edge_to_cpls: A dictionary mapping each edge to a set of child pointer lists that are associated with that edge. @ivar _indexes: A dictionary mapping tuples of edge attributes to indices, where each index maps the corresponding edge attribute values to lists of edges. cCs/t||_t|i|_|idS(s Construct a new chart. The chart is initialized with the leaf edges corresponding to the terminal leaves. @type tokens: L{list} @param tokens: The sentence that this chart will be used to parse. N(Rt_tokensR't _num_leavest initialize(R ttokens((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR s cCsg|_h|_h|_dS(s" Clear the chart. N(t_edgest _edge_to_cplst_indexes(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR;s  cCs|iS(s_ @return: The number of words in this chart's sentence. @rtype: C{int} (R:(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt num_leavesscCs |i|S(sc @return: The leaf value of the word at the given index. @rtype: C{string} (R9(R R#((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR4scCs|iS(s @return: A list of the leaf values of each word in the chart's sentence. @rtype: C{list} of C{string} (R9(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytleavesscCs|iS(s @return: A list of all edges in this chart. New edges that are added to the chart after the call to edges() will I{not} be contained in this list. @rtype: C{list} of L{EdgeI} @see: L{iteredges}, L{select} (R=(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytedgesscCs t|iS(s0 @return: An iterator over the edges in this chart. It is I{not} guaranteed that new edges which are added to the chart before the iterator is exhausted will also be generated. @rtype: C{iter} of L{EdgeI} @see: L{edges}, L{select} (titerR=(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt iteredgess cCs t|iS(s^ @return: The number of edges contained in this chart. @rtype: C{int} (R'R>(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt num_edgessc shjot|iSni}|it|}||ijo|i|ntfd|D}t|i|i|gS(sP @return: An iterator over the edges in this chart. Any new edges that are added to the chart before the iterator is exahusted will also be generated. C{restrictions} can be used to restrict the set of edges that will be generated. @rtype: C{iter} of L{EdgeI} @kwarg span: Only generate edges C{e} where C{e.span()==span} @kwarg start: Only generate edges C{e} where C{e.start()==start} @kwarg end: Only generate edges C{e} where C{e.end()==end} @kwarg length: Only generate edges C{e} where C{e.length()==length} @kwarg lhs: Only generate edges C{e} where C{e.lhs()==lhs} @kwarg rhs: Only generate edges C{e} where C{e.rhs()==rhs} @kwarg next: Only generate edges C{e} where C{e.next()==next} @kwarg dot: Only generate edges C{e} where C{e.dot()==dot} @kwarg is_complete: Only generate edges C{e} where C{e.is_complete()==is_complete} @kwarg is_incomplete: Only generate edges C{e} where C{e.is_incomplete()==is_incomplete} c3sx|]}|VqWdS(N((t.0tkey(t restrictions(s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pys s(RCR=tkeystsortRR?t _add_indextget(R RHt restr_keystvals((RHs&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytselects   csx/|D]'}tt|ptd|qqWh}|i|sN(thasattrRt ValueErrorR?R=Rt setdefaulttappend(R RMRGR#RN((RQs&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRK s csYxR|iiD]A\}}tfd|D}|i|giqWdS(sr A helper function for L{insert}, which registers the new edge with all existing indexes. c3s%x|]}t|VqWdS(N(RP(RFRG(RQ(s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pys "sN(R?titemsRRTRU(R RQRMR#RN((RQs&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt_register_with_indexess cCsG|i|}g}|D]}|||fq~}|i||S(sT Add a new edge to the chart, using a pointer to the previous edge. (tchild_pointer_liststinsert(R tnew_edget previous_edget child_edgetcplst_[1]tcpltnew_cpls((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytinsert_with_backpointer)s(cGs||ijo|i||i|n|ii|h}t}x;|D]3}t|}||jot||t _append_edgeRWRTR6RR5(R RQRXR]tchart_was_modifiedtchild_pointer_list((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRY1s    cCs|ii|dS(N(R=RU(R RQ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRbSscCsUg}xH|iddd|id|D]%}||i|d|dt7}q(W|S(s @return: A list of the complete tree structures that span the entire chart, and whose root node is C{root}. R iRRt tree_classtcomplete(ROR:ttreesR5(R trootReRgRQ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytparsesZs "#cCs|i||dhd|S(s> @return: A list of the tree structures that are associated with C{edge}. If C{edge} is incomplete, then the unexpanded children will be encoded as childless subtrees, whose node value is the corresponding terminal or nonterminal. @rtype: C{list} of L{Tree} @note: If two trees share a common subtree, then the same C{Tree} may be used to encode that subtree in both trees. If you need to eliminate this subtree sharing, then create a deep copy of each tree. tmemoRe(t_trees(R RQReRf((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRgdsc Cs||jo ||Sng}|o|io|Sng||RLRI(R RQ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRXscCs|tjod|id}n|i|i}}ddd|d|}||jo)|io|d7}qV|d7}n|ioN|id|ifjo2|d d |||dd |dd 7}nn|io2|d d |||dd |dd 7}n/|d d |||dd |dd7}|d|dd|i|7}|d |S(s @return: A pretty-printed string representation of a given edge in this chart. @rtype: C{string} @param width: The number of characters allotted to each index in the sentence. i2it|t.t t#t>it[t=t]t-s| %s(R(R@R RRR R:(R RQtwidthR RR,((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytpp_edges% )2 2.!cCs|tjod|id}n|itj oX|djoKd}x4|iD])}|||d i|dd7}qRW|d7}nd}|S(s @return: A pretty-printed string representation of this chart's leaves. This string can be used as a header for calls to L{pp_edge}. i2is|.R}R|t(R(R@R9tcenter(R Rtheaderttok((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt pp_leavess% 'c stjodidng}D]"}||i|i|fq0~}|ig}|D]\}}}||qp~}iddifd|DS(s @return: A pretty-printed string representation of this chart. @rtype: C{string} @param width: The number of characters allotted to each index in the sentence. i2is c3s%x|]}i|VqWdS(N(R(RFRQ(RR (s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pys s(R(R@RR RJRtjoin(R RR^teRBRrt_((R Rs&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytpps %6 *cCsd}|d7}|d7}|d7}xt|iddD]}|djo|d7}nxt|idD]g}|djp<||i|dijp||i|dijo|d ||f7}ququWq=W|d 7}xt|idD]}|d 7}xt|idD]g}|djp<||i|dijp||i|dijo|d ||f7}q,q,W|d 7}qW|d7}|d7}|d7}x;t|iD]'}|d|i||df7}qW|d7}|d7}xt|D]\}}x@t|iD],}|d||d|d|df7}qAW|d|i|d|i|d|f7}xIt|i|iD],}|d||d|d|df7}qWq"W|d 7}|S(Nsdigraph nltk_chart { s rankdir=LR; s node [height=0.1,width=0.1]; s* node [style=filled, color="lightgray"]; iis& node [style=filled, color="black"]; is %04d.%04d [label=""]; s/ x [style=invis]; x->0000.0000 [style=invis]; s {rank=same;s %04d.%04ds} s" edge [style=invis, weight=100]; s node [shape=plaintext] s 0000.0000s->%s->%04d.0000s; s edge [style=solid, weight=1]; s* %04d.%04d -> %04d.%04d [style="invis"]; s' %04d.%04d -> %04d.%04d [label="%s"]; (R+RER@R=R RR4t enumerate(R tstytxRQ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt dot_digraphsZ    +   +   %   $+( (RRRR R;R@R4RARBRDRvRERORKRWRaRYRbRRiR6RgRkRlRXR(RRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR8os2       %   "   9    t ChartRuleIcBs2eZdZdZdZdZdZRS(s A rule that specifies what new edges are licensed by any given set of existing edges. Each chart rule expects a fixed number of edges, as indicated by the class variable L{NUM_EDGES}. In particular: - A chart rule with C{NUM_EDGES=0} specifies what new edges are licensed, regardless of existing edges. - A chart rule with C{NUM_EDGES=1} specifies what new edges are licensed by a single existing edge. - A chart rule with C{NUM_EDGES=2} specifies what new edges are licensed by a pair of existing edges. @type NUM_EDGES: C{int} @cvar NUM_EDGES: The number of existing edges that this rule uses to license new edges. Typically, this number ranges from zero to two. cGs tddS(s Add the edges licensed by this rule and the given edges to the chart. @type edges: C{list} of L{EdgeI} @param edges: A set of existing edges. The number of edges that should be passed to C{apply} is specified by the L{NUM_EDGES} class variable. @rtype: C{list} of L{EdgeI} @return: A list of the edges that were added. s#ChartRuleI is an abstract interfaceN(R (R tcharttgrammarRB((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytapplyes cGs tddS(s  @return: A generator that will add edges licensed by this rule and the given edges to the chart, one at a time. Each time the generator is resumed, it will either add a new edge and yield that edge; or return. @rtype: C{iter} of L{EdgeI} @type edges: C{list} of L{EdgeI} @param edges: A set of existing edges. The number of edges that should be passed to C{apply} is specified by the L{NUM_EDGES} class variable. s#ChartRuleI is an abstract interfaceN(R (R RRRB((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt apply_iterss cCs tddS(s Add all the edges licensed by this rule and the edges in the chart to the chart. @rtype: C{list} of L{EdgeI} @return: A list of the edges that were added. s#ChartRuleI is an abstract interfaceN(R (R RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytapply_everywherescCs tddS(sL @return: A generator that will add all edges licensed by this rule, given the edges that are currently in the chart, one at a time. Each time the generator is resumed, it will either add a new edge and yield that edge; or return. @rtype: C{iter} of L{EdgeI} s#ChartRuleI is an abstract interfaceN(R (R RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytapply_everywhere_iters (RRRRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRPs    tAbstractChartRulecBs;eZdZdZdZdZdZdZRS(s An abstract base class for chart rules. C{AbstractChartRule} provides: - A default implementation for C{apply}, based on C{apply_iter}. - A default implementation for C{apply_everywhere_iter}, based on C{apply_iter}. - A default implementation for C{apply_everywhere}, based on C{apply_everywhere_iter}. Currently, this implementation assumes that C{NUM_EDGES}<=3. - A default implementation for C{__str__}, which returns a name basd on the rule's class name. cGs tddS(Ns&AbstractChartRule is an abstract class(R (R RRRB((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRsc cs]|idjo&xF|i||D] }|Vq#Wn$|idjo:x|D]+}x"|i|||D] }|VqiWqMWn|idjoNx|D]?}x6|D].}x%|i||||D] }|VqWqWqWn||idjobxh|D]S}xJ|D]B}x9|D]1}x(|i|||||D] }|Vq1WqWqWqWn tddS(Niiiis&NUM_EDGES>3 is not currently supported(t NUM_EDGESRR (R RRRZte1te2te3((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs: cGst|i|||S(N(tlistR(R RRRB((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCst|i||S(N(RR(R RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCstidd|iiS(Ns([a-z])([A-Z])s\1 \2(tretsubRR(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR.s(RRRRRRRR.(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs      tFundamentalRulecBseZdZdZdZRS(s* A rule that joins two adjacent edges to form a single combined edge. In particular, this rule specifies that any pair of edges: - [A S{->} S{alpha} * B S{beta}][i:j] - [B S{->} S{gamma} *][j:k] licenses the edge: - [A S{->} S{alpha} B * S{beta}][i:j] iccs|io<|io/|i|ijo|i|ijpdSn|i|i}|i|||o |VndS(N(RRRR RRR&Ra(R RRt left_edget right_edgeRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs  (RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs tSingleEdgeFundamentalRulecBs/eZdZdZdZdZdZRS(s A rule that joins a given edge with adjacent edges in the chart, to form combined edges. In particular, this rule specifies that either of the edges: - [A S{->} S{alpha} * B S{beta}][i:j] - [B S{->} S{gamma} *][j:k] licenses the edge: - [A S{->} S{alpha} B * S{beta}][i:j] if the other edge is already in the chart. @note: This is basically L{FundamentalRule}, with one edge left unspecified. iccsY|io&xE|i||D] }|Vq Wn#x|i||D] }|VqFWdS(N(Rt_apply_incompletet_apply_complete(R RRRQRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs  ccsmxf|id|idtd|iD]:}|i|i}|i|||o |Vq+q+WdS(NRRR(ROR R6RR&RRa(R RRRRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs ccsmxf|id|idtd|iD]:}|i|i}|i|||o |Vq+q+WdS(NR RR(RORR5RR&Ra(R RRRRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs (RRRRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs    t LeafInitRulecBseZdZdZRS(iccsUxNt|iD]:}t|i||}|i|do |VqqWdS(N((R+R@R1R4RY(R RRR#RZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs (RRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRstTopDownInitRulecBseZdZdZdZRS(s* A rule licensing edges corresponding to the grammar productions for the grammar's start symbol. In particular, this rule specifies that: - [S S{->} * S{alpha}][0:i] is licensed for each grammar production C{S S{->} S{alpha}}, where C{S} is the grammar's start symbol. iccsUxN|id|iD]4}ti|d}|i|do |VqqWdS(NRi((t productionsR RR$RY(R RRtprodRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR1s (RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR(stCachedTopDownInitRulecBseZdZRS(sUse L{TopDownInitRule} instead.(RRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR7stTopDownPredictRulecBseZdZdZdZRS(s A rule licensing edges corresponding to the grammar productions for the nonterminal following an incomplete edge's dot. In particular, this rule specifies that: - [A S{->} S{alpha} * B S{beta}][i:j] licenses the edge: - [B S{->} * S{gamma}][j:j] for each grammar production C{B S{->} S{gamma}}. @note: This rule corresponds to the Predictor Rule in Earley parsing. iccsp|iodSnxT|id|iD]:}ti||i}|i|do |Vq.q.WdS(NR((RRRRR$RRY(R RRRQRRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRGs (RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR:s tTopDownExpandRulecBseZdZRS(sUse TopDownPredictRule instead(RRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRNstCachedTopDownPredictRulecBs eZdZdZdZRS(s. A cached version of L{TopDownPredictRule}. After the first time this rule is applied to an edge with a given C{end} and C{next}, it will not generate any more edges for edges with that C{end} and C{next}. If C{chart} or C{grammar} are changed, then the cache is flushed. cCsti|h|_dS(N(RR t_done(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR Zs c csT|iodSn|i|i}}t|ipdSn|ii||fd}|d|jo|d|jodSnx|id|D]}|io|id}t |o4||i jp||i |joqqnt i ||} |i| do | VqqW||f|i||f} S{alpha} *] licenses the edge: - [B S{->} * A S{beta}] for each grammar production C{B S{->} A S{beta}}. iccsp|iodSnxT|id|iD]:}ti||i}|i|do |Vq.q.WdS(NR((RRRRR$R RY(R RRRQRRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs (RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR~s tBottomUpPredictCombineRulecBseZdZdZdZRS(s A rule licensing any edge corresponding to a production whose right-hand side begins with a complete edge's left-hand side. In particular, this rule specifies that: - [A S{->} S{alpha} *] licenses the edge: - [B S{->} A * S{beta}] for each grammar production C{B S{->} A S{beta}}. @note: This is like L{BottomUpPredictRule}, but it also applies the L{FundamentalRule} to the resulting edge. iccs|iodSnxf|id|iD]L}t|i|i|id}|i||fo |Vq.q.WdS(NRi(RRRRR RRY(R RRRQRRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs '(RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs tEmptyPredictRulecBseZdZdZdZRS(si A rule that inserts all empty productions as passive edges, in every position in the chart. iccspxi|idtD]U}xLt|idD]4}ti||}|i|do |Vq0q0WqWdS(Ntemptyi((RR5txrangeR@RR$RY(R RRRR#RZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs(RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRst!FilteredSingleEdgeFundamentalRulecBs#eZdZdZdZRS(ccse|io,xQ|i||i|D] }|Vq&Wn)x%|i||i|D] }|VqRWdS(N(RRt leftcornersR(R RRRQRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs  ccs|i}||ijo|i|}nd}x|id|idtd|iD]`}t|||i |i o8|i |i}|i |||o |VqqcqcWdS(NRRR( RR@R4R(ROR R6Rt_bottomup_filterRRR&Ra(R RRRRt nexttokenRRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs  "ccsx|id|idtd|iD]}|i}||ijo|i|}nd}t|||i|i o8|i |i}|i |||o |Vqq+q+WdS(NR RR( RORR5RR@R4R(RRRR&Ra(R RRRRRRRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs  "(RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs  t"FilteredBottomUpPredictCombineRulecBseZdZRS(c cs|iodSn|i}|i}||ijo|i|}nd}x|id|iD]i}t|||i oJt |i |i|i d}|i ||fo |VqqoqoWdS(NRi( RRRR@R4R(RRRRRR RY( R RRRQRRRRRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs  '(RRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRsicCs\t||djotSn||d}t|o||jSn|||jSdS(Ni(R'R5R(RRRRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs  t ChartParsercBsSeZdZeddeedZdZdZe dZ e e dZ RS(s A generic chart parser. A X{strategy}, or list of L{ChartRules}, is used to decide what edges to add to the chart. In particular, C{ChartParser} uses the following algorithm to parse texts: - Until no new edges are added: - For each I{rule} in I{strategy}: - Apply I{rule} to any applicable edges in the chart. - Return any complete parses in the chart ii2cCs||_||_||_||_||_||_g|_g|_x_|D]W}|idjo|ii |qO|idjo|ii |qOt |_qOWdS(s> Create a new chart parser, that uses C{grammar} to parse texts. @type grammar: L{ContextFreeGrammar} @param grammar: The grammar used to parse texts. @type strategy: C{list} of L{ChartRuleI} @param strategy: A list of rules that should be used to decide what edges to add to the chart (top-down strategy by default). @type trace: C{int} @param trace: The level of tracing that should be used when parsing a text. C{0} will generate no tracing output; and higher numbers will produce more verbose tracing output. @type trace_chart_width: C{int} @param trace_chart_width: The default total width reserved for the chart in trace output. The remainder of each line will be used to display edges. @type use_agenda: C{bool} @param use_agenda: Use an optimized agenda-based algorithm, if possible. @param chart_class: The class that should be used to create the parse charts. iiN( t_grammart _strategyt_tracet_trace_chart_widtht _use_agendat _chart_classt_axiomst_inference_rulesRRUR6(R Rtstrategyttracettrace_chart_widtht use_agendat chart_classtrule((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR s        cCs|iS(N(R(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRGscCs[|pdSn|dj}x9|D]1}|od|GHt}n|i||GHq"WdS(Nis%s:(R6R(R RRt new_edgesRt edge_widthtshould_print_rule_headerRQ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt_trace_new_edgesJs   c Cs|tjo |i}n|i}t|}|ii||i|}|i}|i|id}|o|i |GHn|i ox9|i D].}|i ||}||||||qW|i } |i} | ix| om| i} xZ| D]R} | i||| }|o&t|}||| |||n| |7} q WqWndt} xZ| oRt} xE|iD]:} | i||}t|} ||| |||qWqqW|S(s @return: The final parse L{Chart}, from which all possible parse trees can be extracted. @param tokens: The sentence to be parsed @type tokens: L{list} of L{string} @rtype: L{Chart} i(R(RRRRtcheck_coverageRRR@RRRRRRBtreversetpopRR5R6RRR'(R R<Rttrace_new_edgesRRttrace_edge_widthtaxiomRtinference_rulestagendaRQRt edges_added((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt chart_parseSsD               cCs/|i|}|i|iid|| S(NRe(RRiRR (R R<tnReR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt nbest_parses( RRRtBU_LC_STRATEGYR5R8R RRR(RRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs ,  6tTopDownChartParsercBseZdZdZRS(sj A L{ChartParser} using a top-down parsing strategy. See L{ChartParser} for more information. cKsti||t|dS(N(RR t TD_STRATEGY(R Rt parser_args((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR s(RRRR (((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRstBottomUpChartParsercBseZdZdZRS(sk A L{ChartParser} using a bottom-up parsing strategy. See L{ChartParser} for more information. cKsAt|totiddtnti||t|dS(NscBottomUpChartParser only works for ContextFreeGrammar, use BottomUpProbabilisticChartParser insteadtcategory(R7RtwarningstwarntDeprecationWarningRR t BU_STRATEGY(R RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR s  (RRRR (((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRstBottomUpLeftCornerChartParsercBseZdZdZRS(s A L{ChartParser} using a bottom-up left-corner parsing strategy. This strategy is often more efficient than standard bottom-up. See L{ChartParser} for more information. cKsti||t|dS(N(RR R(R RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR s(RRRR (((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRstLeftCornerChartParsercBseZdZRS(cKs7|iptdnti||t|dS(NsCLeftCornerParser only works for grammars without empty productions.(t is_nonemptyRSRR t LC_STRATEGY(R RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR s (RRR (((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRstSteppingChartParsercBseZdZgddZdZdZdZdZdZdZ d Z e d Z d Z d Zd Zee dZRS(s A C{ChartParser} that allows you to step through the parsing process, adding a single edge at a time. It also allows you to change the parser's strategy or grammar midway through parsing a text. The C{initialize} method is used to start parsing a text. C{step} adds a single edge to the chart. C{set_strategy} changes the strategy used by the chart parser. C{parses} returns the set of parses that has been found by the chart parser. @ivar _restart: Records whether the parser's strategy, grammar, or chart has been changed. If so, then L{step} must restart the parsing algorithm. icCs5d|_d|_t|_ti||||dS(N(R(t_chartt_current_chartruleR6t_restartRR (R RRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR s   cCs"tt||_t|_dS(sBegin parsing the given tokens.N(R8RRR5R(R R<((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR;sccs|idjo tdnxt|_d|iid}xq|iD]^}|idjo |iGHn|idjo|ii ||GHn|V|ioPqMqMWdVq dS(s @return: A generator that adds edges to the chart, one at a time. Each time the generator is resumed, it adds a single edge and yields that edge. If no more edges can be added, then it yields C{None}. If the parser's strategy, grammar, or chart is changed, then the generator will continue adding edges using the new strategy, grammar, or chart. Note that this generator never terminates, since the grammar or strategy might be changed to values that would add new edges. Instead, it yields C{None} when no more edges can be added with the current strategy and grammar. s Parser must be initialized firsti2iiN( RR(RSR6RR@t_parseRRR(R twR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytsteps   (ccs|i}|i}d}xa|djoSd}xF|iD];}||_x)|i||D]}|d7}|VqZWq8WqWdS(s A generator that implements the actual parsing algorithm. L{step} iterates through this generator, and restarts it whenever the parser's strategy, grammar, or chart is modified. iiN(RRRRR(R RRRRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs     cCs|iS(s*@return: The strategy used by this parser.(R(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR scCs|iS(s)@return: The grammar used by this parser.(R(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|iS(s/@return: The chart that is used by this parser.(R(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|iS(s>@return: The chart rule used to generate the most recent edge.(R(R ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytcurrent_chartrulescCs|ii|ii|S(s:@return: The parse trees currently contained in the chart.(RRiRR (R Re((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRiscCs/||ijodSn||_t|_dS(s Change the startegy that the parser uses to decide which edges to add to the chart. @type strategy: C{list} of L{ChartRuleI} @param strategy: A list of rules that should be used to decide what edges to add to the chart. N(RR5R(R R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt set_strategy$s cCs.||ijodSn||_t|_dS(s&Change the grammar used by the parser.N(RR5R(R R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt set_grammar0s cCs.||ijodSn||_t|_dS(s)Load a given chart into the chart parser.N(RR5R(R R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt set_chart6s cCsft|}|ii||i|x&|iD]}|djoPq6q6W|id|| S(NRe(RRRR;RR(Ri(R R<RReR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR@s   (RRRR R;RRRRRRRRiRRRR(R(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs         cCsddkl}|dS(Ni(t parse_cfgs S -> NP VP PP -> "with" NP NP -> NP PP VP -> VP PP VP -> Verb NP VP -> Verb NP -> Det Noun NP -> "John" NP -> "I" Det -> "the" Det -> "my" Det -> "a" Noun -> "dog" Noun -> "cookie" Verb -> "ate" Verb -> "saw" Prep -> "with" Prep -> "under" (t nltk.grammarR(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt demo_grammarRsis$I saw John with a dog with my cookieic Cs1ddk}ddk}ddkl} l} l} t} |odGH| GHndGH|GH|i} | GHH|djo7dGHdGHdGHd GHd GHd G|i i i }Hnt |}|d&jo dGHdSnh}hdt fd <dtfd <dtfd<}g}|i|o |g}n|djod d dg}nx|D]}d||dGHHt| ||dd|}|i}|i| }|i||||d<|o!t||jp tdn|ox&|D] }|GHqWndGt|GHHq[W|d'josdGHH|i}t| d|}|i| xtdD]}dGH|it x?t|iD]+\}}|djp |djoPqqWdGH|itx?t|iD]+\}}|djp |djoPqqWqoW|i||d <|o't|i|jp tdn|o x2|iD] }|GHqwWndGt|iGHHn|o|pdSnd!GHHtd"|iD}d#| d$}|i}|id%x#|D]\}}|||fGHqWdS((s/ A demonstration of the chart parsers. iN(t nonterminalst ProductiontContextFreeGrammars * Grammars * Sentence:s 1: Top-down chart parsers 2: Bottom-up chart parsers' 3: Bottom-up left-corner chart parsers= 4: Stepping chart parser (alternating top-down & bottom-up)s 5: All parserss Which parser (1-5)? t1t2t3t4t5sBad parser numbersTop-downs Bottom-upsBottom-up left-corners * Strategy: iiRsNot all parses founds Nr trees:s,* Strategy: Stepping (top-down vs bottom-up)is*** SWITCH TO TOP DOWNis*** SWITCH TO BOTTOM UPtSteppings* Parsing timescssx|]}t|VqWdS(N(R'(RFRG((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pys st%ss parser: %6.3fseccSst|d|dS(i(R)(tatb((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyts(RRRRR(RR( tsysttimetnltkRRRRtsplitR(tstdintreadlinetstripR,RRRthas_keyRRR'R RR;R+RRRRitmaxRIRVRJ(tchoicetshould_print_timestshould_print_grammartshould_print_treesRtsentt numparsesRR RRRRR<ttimest strategiestchoicesRRottRiRuR-tjRtmaxlentformatt times_itemstparser((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytdemois        !      #  ''   t__main__(4Rt nltk.treeRRRRRt nltk.compatRtapiRRtobjectRRR1R8RRRRRRt DeprecatedRRRRRRRRRRRRRRRtParserIRRRRRRRR(R5R6R R(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pys'sn   s5G?, &(         c