4Ic@sddklZlZlZddkTddkZdefdYZdefdYZdefd YZ d efd YZ d efd YZ defdYZ de fdYZ de fdYZde fdYZde fdYZde fdYZdefdYZdefdYZde fdYZd e fd!YZd"e fd#YZd$efd%YZd&e fd'YZd(efd)YZeeeegZeeegZeeegZeeeegZd*efd+YZd,efd-YZ e!e"d.d/Z#e$d0jo e#ndS(1i(tTreet Nonterminalt defaultdict(t*NtEdgeIcBseZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZRS(sM A hypothesis about the structure of part of a sentence. Each edge records the fact that a structure is (partially) consistent with the sentence. An edge contains: - A X{span}, indicating what part of the sentence is consistent with the hypothesized structure. - A X{left-hand side}, specifying what kind of structure is hypothesized. - A X{right-hand side}, specifying the contents of the hypothesized structure. - A X{dot position}, indicating how much of the hypothesized structure is consistent with the sentence. Every edge is either X{complete} or X{incomplete}: - An edge is X{complete} if its structure is fully consistent with the sentence. - An edge is X{incomplete} if its structure is partially consistent with the sentence. For every incomplete edge, the span specifies a possible prefix for the edge's structure. There are two kinds of edge: - C{TreeEdges} record which trees have been found to be (partially) consistent with the text. - C{LeafEdges} record the tokens occur in the text. The C{EdgeI} interface provides a common interface to both types of edge, allowing chart parsers to treat them in a uniform manner. cCs$|itjotdndS(NsEdge is an abstract interface(t __class__Rt TypeError(tself((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt__init__XscCstddS(s @return: A tuple C{(s,e)}, where C{subtokens[s:e]} is the portion of the sentence that is consistent with this edge's structure. @rtype: C{(int, int)} sEdgeI is an abstract interfaceN(tAssertionError(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytspan`scCstddS(sV @return: The start index of this edge's span. @rtype: C{int} sEdgeI is an abstract interfaceN(R (R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytstartiscCstddS(sT @return: The end index of this edge's span. @rtype: C{int} sEdgeI is an abstract interfaceN(R (R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytendpscCstddS(sQ @return: The length of this edge's span. @rtype: C{int} sEdgeI is an abstract interfaceN(R (R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytlengthwscCstddS(s @return: This edge's left-hand side, which specifies what kind of structure is hypothesized by this edge. @see: L{TreeEdge} and L{LeafEdge} for a description of the left-hand side values for each edge type. sEdgeI is an abstract interfaceN(R (R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytlhsscCstddS(s @return: This edge's right-hand side, which specifies the content of the structure hypothesized by this edge. @see: L{TreeEdge} and L{LeafEdge} for a description of the right-hand side values for each edge type. sEdgeI is an abstract interfaceN(R (R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytrhsscCstddS(s! @return: This edge's dot position, which indicates how much of the hypothesized structure is consistent with the sentence. In particular, C{self.rhs[:dot]} is consistent with C{subtoks[self.start():self.end()]}. @rtype: C{int} sEdgeI is an abstract interfaceN(R (R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytdotscCstddS(s @return: The element of this edge's right-hand side that immediately follows its dot. @rtype: C{Nonterminal} or X{terminal} or C{None} sEdgeI is an abstract interfaceN(R (R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytnextscCstddS(s @return: True if this edge's structure is fully consistent with the text. @rtype: C{boolean} sEdgeI is an abstract interfaceN(R (R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt is_completescCstddS(s @return: True if this edge's structure is partially consistent with the text. @rtype: C{boolean} sEdgeI is an abstract interfaceN(R (R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt is_incompletescCstddS(NsEdgeI is an abstract interface(R (Rtother((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt__cmp__scCstddS(NsEdgeI is an abstract interface(R (RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt__hash__s(t__name__t __module__t__doc__RR R R R RRRRRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR3s$      tTreeEdgecBseZdZddZdZeeZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZRS(se An edge that records the fact that a tree is (partially) consistent with the sentence. A tree edge consists of: - A X{span}, indicating what part of the sentence is consistent with the hypothesized tree. - A X{left-hand side}, specifying the hypothesized tree's node value. - A X{right-hand side}, specifying the hypothesized tree's children. Each element of the right-hand side is either a terminal, specifying a token with that terminal as its leaf value; or a nonterminal, specifying a subtree with that nonterminal's symbol as its node value. - A X{dot position}, indicating which children are consistent with part of the sentence. In particular, if C{dot} is the dot position, C{rhs} is the right-hand size, C{(start,end)} is the span, and C{sentence} is the list of subtokens in the sentence, then C{subtokens[start:end]} can be spanned by the children specified by C{rhs[:dot]}. For more information about edges, see the L{EdgeI} interface. icCs.||_t||_||_||_dS(s Construct a new C{TreeEdge}. @type span: C{(int, int)} @param span: A tuple C{(s,e)}, where C{subtokens[s:e]} is the portion of the sentence that is consistent with the new edge's structure. @type lhs: L{Nonterminal} @param lhs: The new edge's left-hand side, specifying the hypothesized tree's node value. @type rhs: C{list} of (L{Nonterminal} and C{string}) @param rhs: The new edge's right-hand side, specifying the hypothesized tree's children. @type dot: C{int} @param dot: The position of the new edge's dot. This position specifies what prefix of the production's right hand side is consistent with the text. In particular, if C{sentence} is the list of subtokens in the sentence, then C{subtokens[span[0]:span[1]]} can be spanned by the children specified by C{rhs[:dot]}. N(t_lhsttuplet_rhst_spant_dot(RR RRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs  c Cs1td||fd|id|iddS(s) @return: A new C{TreeEdge} formed from the given production. The new edge's left-hand side and right-hand side will be taken from C{production}; its span will be C{(index, index)}; and its dot position will be C{0}. @rtype: L{TreeEdge} R RRRi(RRR(t productiontindex((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytfrom_productionscCs|iS(N(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|iS(N(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR scCs |idS(Ni(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR scCs |idS(Ni(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR scCs|id|idS(Nii(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR scCs|iS(N(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR scCs|iS(N(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR scCs|it|ijS(N(RtlenR(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|it|ijS(N(RR#R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs3|it|ijodSn|i|iSdS(N(RR#RtNone(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs!cCsd|i|ijodSnt|i|i|i|if|i|i|i|ifS(Ni(RtcmpRRRR(RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs$cCs(t|i|i|i|ifS(N(thashRRRR(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs4d|id|idf}t|iito|d|if7}n|d|if7}xtt|iD]}||ijo|d7}nt|i|t o9t|i|ito|d|i|f7}qy|d|i|f7}qyWt|i|ijo|d7}n|S( Ns[%s:%s] iis%-2s ->s%-2r ->s *s %ss %r( Rt isinstanceRtsymbolt basestringtrangeR#RRR(Rtstrti((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt__str__s'cCsd|S(Ns [Edge: %s]((R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt__repr__.s(RRRRR"t staticmethodRR R R R RRRRRRRR-R.(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs$               tLeafEdgecBseZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZRS(s An edge that records the fact that a leaf value is consistent with a word in the sentence. A leaf edge consists of: - An X{index}, indicating the position of the word. - A X{leaf}, specifying the word's content. A leaf edge's left-hand side is its leaf value, and its right hand side is C{()}. Its span is C{[index, index+1]}, and its dot position is C{0}. cCs||_||_dS(s Construct a new C{LeafEdge}. @param leaf: The new edge's leaf value, specifying the word that is recorded by this edge. @param index: The new edge's index, specifying the position of the word that is recorded by this edge. N(t_leaft_index(RtleafR!((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR=s cCs|iS(N(R1(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRJscCs|i|idfS(Ni(R2(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR KscCs|iS(N(R2(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR LscCs |idS(Ni(R2(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR MscCsdS(Ni((R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR NscCsdS(N(((R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyROscCsdS(Ni((R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRPscCstS(N(tTrue(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRQscCstS(N(tFalse(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRRscCsdS(N(R$(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRSscCs=t|tpdSnt|i|if|i|ifS(Ni(R'R0R%R2R1(RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRVscCst|i|ifS(N(R&R2R1(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRYscCsd|i|id|ifS(Ns [%s:%s] %ri(R2R1(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR-]scCsd|S(Ns [Edge: %s]((R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR.^s(RRRRRR R R R RRRRRRRR-R.(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR01s               tChartcBseZdZdZdZdZdZdZdZeZ dZ dZ d Z d Z ed Zeed Zd ZdZdZedZedZedZdZRS(s A blackboard for hypotheses about the syntactic constituents of a sentence. A chart contains a set of edges, and each edge encodes a single hypothesis about the structure of some portion of the sentence. The L{select} method can be used to select a specific collection of edges. For example C{chart.select(is_complete=True, start=0)} yields all complete edges whose start indices are 0. To ensure the efficiency of these selection operations, C{Chart} dynamically creates and maintains an index for each set of attributes that have been selected on. In order to reconstruct the trees that are represented by an edge, the chart associates each edge with a set of child pointer lists. A X{child pointer list} is a list of the edges that license an edge's right-hand side. @ivar _tokens: The sentence that the chart covers. @ivar _num_leaves: The number of tokens. @ivar _edges: A list of the edges in the chart @ivar _edge_to_cpls: A dictionary mapping each edge to a set of child pointer lists that are associated with that edge. @ivar _indexes: A dictionary mapping tuples of edge attributes to indices, where each index maps the corresponding edge attribute values to lists of edges. cCs@t||_t|i|_g|_h|_h|_dS(s Construct a new empty chart. @type tokens: L{list} @param tokens: The sentence that this chart will be used to parse. N(tlistt_tokensR#t _num_leavest_edgest _edge_to_cplst_indexes(Rttokens((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs   cCs|iS(s_ @return: The number of words in this chart's sentence. @rtype: C{int} (R9(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt num_leavesscCs |i|S(sc @return: The leaf value of the word at the given index. @rtype: C{string} (R8(RR!((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR3scCs|iS(s @return: A list of the leaf values of each word in the chart's sentence. @rtype: C{list} of C{string} (R8(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytleavesscCs|iS(s @return: A list of all edges in this chart. New edges that are added to the chart after the call to edges() will I{not} be contained in this list. @rtype: C{list} of L{EdgeI} @see: L{iteredges}, L{select} (R:(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytedgesscCs t|iS(s @return: An iterator over the edges in this chart. Any new edges that are added to the chart before the iterator is exahusted will also be generated. @rtype: C{iter} of L{EdgeI} @see: L{edges}, L{select} (titerR:(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt iteredgesscCs t|iS(s^ @return: The number of edges contained in this chart. @rtype: C{int} (R#R;(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt num_edgesscKs|hjot|iSn|i}|it|}||ijo|i|ng}|D]}|||ql~}t|i|it|gS(sP @return: An iterator over the edges in this chart. Any new edges that are added to the chart before the iterator is exahusted will also be generated. C{restrictions} can be used to restrict the set of edges that will be generated. @rtype: C{iter} of L{EdgeI} @kwarg span: Only generate edges C{e} where C{e.span()==span} @kwarg start: Only generate edges C{e} where C{e.start()==start} @kwarg end: Only generate edges C{e} where C{e.end()==end} @kwarg length: Only generate edges C{e} where C{e.length()==length} @kwarg lhs: Only generate edges C{e} where C{e.lhs()==lhs} @kwarg rhs: Only generate edges C{e} where C{e.rhs()==rhs} @kwarg next: Only generate edges C{e} where C{e.next()==next} @kwarg dot: Only generate edges C{e} where C{e.dot()==dot} @kwarg is_complete: Only generate edges C{e} where C{e.is_complete()==is_complete} @kwarg is_incomplete: Only generate edges C{e} where C{e.is_incomplete()==is_incomplete} (RAR:tkeystsortRR<t _add_indextget(Rt restrictionst restr_keyst_[1]tktvals((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytselects   %c Csx/|D]'}tt|ptd|qqWh|i| @return: A list of the tree structures that are associated with C{edge}. If C{edge} is incomplete, then the unexpanded children will be encoded as childless subtrees, whose node value is the corresponding terminal or nonterminal. @rtype: C{list} of L{Tree} @note: If two trees share a common subtree, then the same C{Tree} may be used to encode that subtree in both trees. If you need to eliminate this subtree sharing, then create a deep copy of each tree. tmemoRZ(t_trees(RRSRZR[((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR\Asc Cs||jo ||Sng}|o|io|Sng||it[t=t]t-s| %s(R$R>R R RR R9(RRStwidthR R R+((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytpp_edges% )2 2.!cCs|tjod|id}n|itj oX|djoKd}x4|iD])}|||d i|dd7}qRW|d7}nd}|S(s @return: A pretty-printed string representation of this chart's leaves. This string can be used as a header for calls to L{pp_edge}. i2is|.RrRqRa(R$R>R8tcenter(RRztheaderttok((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt pp_leavess% 'c stjodidng}D]"}||i|i|fq0~}|ig}|D]\}}}||qp~}iddifd|DS(s @return: A pretty-printed string representation of this chart. @rtype: C{string} @param width: The number of characters allotted to each index in the sentence. i2is c3s%x|]}i|VqWdS(N(R{(t.0RS(RzR(s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pys s(R$R>R R RERtjoin(RRzRJteR@Rit_((RRzs&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytpps %6 *cCsd}|d7}|d7}|d7}xt|iddD]}|djo|d7}nxt|idD]g}|djp<||i|dijp||i|dijo|d ||f7}ququWq=W|d 7}xt|idD]}|d 7}xt|idD]g}|djp<||i|dijp||i|dijo|d ||f7}q,q,W|d 7}qW|d7}|d7}|d7}x;t|iD]'}|d|i||df7}qW|d7}|d7}xt|D]\}}x@t|iD],}|d||d|d|df7}qAW|d|i|d|i|d|f7}xIt|i|iD],}|d||d|d|df7}qWq"W|d 7}|S(Nsdigraph nltk_chart { s rankdir=LR; s node [height=0.1,width=0.1]; s* node [style=filled, color="lightgray"]; iis& node [style=filled, color="black"]; is %04d.%04d [label=""]; s/ x [style=invis]; x->0000.0000 [style=invis]; s {rank=same;s %04d.%04ds} s" edge [style=invis, weight=100]; s node [shape=plaintext] s 0000.0000s->%s->%04d.0000s; s edge [style=solid, weight=1]; s* %04d.%04d -> %04d.%04d [style="invis"]; s' %04d.%04d -> %04d.%04d [label="%s"]; (R*RCR>R:R R R3t enumerate(RtstytxRS((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt dot_digraphsZ    +   +   %   $+( (RRRRR>R3R?R@RBt__iter__RCRMRFRYRR^R5R\R`RcRUR$R{RRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR6es*      %  )  =    t ChartRuleIcBs2eZdZdZdZdZdZRS(s A rule that specifies what new edges are licensed by any given set of existing edges. Each chart rule expects a fixed number of edges, as indicated by the class variable L{NUM_EDGES}. In particular: - A chart rule with C{NUM_EDGES=0} specifies what new edges are licensed, regardless of existing edges. - A chart rule with C{NUM_EDGES=1} specifies what new edges are licensed by a single existing edge. - A chart rule with C{NUM_EDGES=2} specifies what new edges are licensed by a pair of existing edges. @type NUM_EDGES: C{int} @cvar NUM_EDGES: The number of existing edges that this rule uses to license new edges. Typically, this number ranges from zero to two. cGs tddS(s Add the edges licensed by this rule and the given edges to the chart. @type edges: C{list} of L{EdgeI} @param edges: A set of existing edges. The number of edges that should be passed to C{apply} is specified by the L{NUM_EDGES} class variable. @rtype: C{list} of L{EdgeI} @return: A list of the edges that were added. s#ChartRuleI is an abstract interfaceN(R (RtcharttgrammarR@((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytapply=s cGs tddS(s  @return: A generator that will add edges licensed by this rule and the given edges to the chart, one at a time. Each time the generator is resumed, it will either add a new edge and yield that edge; or return. @rtype: C{iter} of L{EdgeI} @type edges: C{list} of L{EdgeI} @param edges: A set of existing edges. The number of edges that should be passed to C{apply} is specified by the L{NUM_EDGES} class variable. s#ChartRuleI is an abstract interfaceN(R (RRRR@((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt apply_iterKs cCs tddS(s Add all the edges licensed by this rule and the edges in the chart to the chart. @rtype: C{list} of L{EdgeI} @return: A list of the edges that were added. s#ChartRuleI is an abstract interfaceN(R (RRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytapply_everywhereZscCs tddS(sL @return: A generator that will add all edges licensed by this rule, given the edges that are currently in the chart, one at a time. Each time the generator is resumed, it will either add a new edge and yield that edge; or return. @rtype: C{iter} of L{EdgeI} s#ChartRuleI is an abstract interfaceN(R (RRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytapply_everywhere_iterds (RRRRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR(s    tAbstractChartRulecBs;eZdZdZdZdZdZdZRS(s An abstract base class for chart rules. C{AbstractChartRule} provides: - A default implementation for C{apply}, based on C{apply_iter}. - A default implementation for C{apply_everywhere_iter}, based on C{apply_iter}. - A default implementation for C{apply_everywhere}, based on C{apply_everywhere_iter}. Currently, this implementation assumes that C{NUM_EDGES}<=3. - A default implementation for C{__str__}, which returns a name basd on the rule's class name. cGs tddS(Ns&AbstractChartRule is an abstract class(R (RRRR@((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR~sc cs]|idjo&xF|i||D] }|Vq#Wn$|idjo:x|D]+}x"|i|||D] }|VqiWqMWn|idjoNx|D]?}x6|D].}x%|i||||D] }|VqWqWqWn||idjobxh|D]S}xJ|D]B}x9|D]1}x(|i|||||D] }|Vq1WqWqWqWn tddS(Niiiis&NUM_EDGES>3 is not currently supported(t NUM_EDGESRR (RRRtnew_edgete1te2te3((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs: cGst|i|||S(N(R7R(RRRR@((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCst|i||S(N(R7R(RRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCstidd|iiS(Ns([a-z])([A-Z])s\1 \2(tretsubRR(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR-s(RRRRRRRR-(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRos      tFundamentalRulecBseZdZdZdZRS(s A rule that joins two adjacent edges to form a single combined edge. In particular, this rule specifies that any pair of edges: - [AS{->}S{alpha}*BS{beta}][i:j] - [BS{->}S{gamma}*][j:k] licenses the edge: - [AS{->}S{alpha}B*S{beta}][i:j] ic cs|i|ijo0|i|ijo|io |ipdSntd|i|ifd|id|id|id}|i |}g}|D]}|||fq~} |i || o |VndS(NR RRRi( R R RRRRRRRRURY( RRRt left_edget right_edgeRRVRJRetnew_cpls((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs((RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs tSingleEdgeFundamentalRulecBs/eZdZdZeZdZdZRS(s A rule that joins a given edge with adjacent edges in the chart, to form combined edges. In particular, this rule specifies that either of the edges: - [AS{->}S{alpha}*BS{beta}][i:j] - [BS{->}S{gamma}*][j:k] licenses the edge: - [AS{->}S{alpha}B*S{beta}][i:j] if the other edge is already in the chart. @note: This is basically L{FundamentalRule}, with one edge is left unspecified. iccs|i}|ioax|id|idtd|iD].}x%|i||||D] }|Vq`WqAWn^xZ|id|idtd|i D].}x%|i||||D] }|VqWqWdS(NR RRR R( t_fundamental_ruleRRMR R4RRR R5R(RRRtedge1tfrtedge2R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs    cCsdS(NsFundamental Rule((R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR-s(RRRRRRRR-(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs    tTopDownInitRulecBseZdZdZdZRS(s% A rule licensing edges corresponding to the grammar productions for the grammar's start symbol. In particular, this rule specifies that: - [SS{->}*S{alpha}][0:i] is licensed for each grammar production C{SS{->}S{alpha}}, where C{S} is the grammar's start symbol. iccsUxN|id|iD]4}ti|d}|i|do |VqqWdS(NRi((t productionsR RR"RY(RRRtprodR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs (RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRstTopDownExpandRulecBseZdZdZdZRS(sG A rule licensing edges corresponding to the grammar productions for the nonterminal following an incomplete edge's dot. In particular, this rule specifies that: - [AS{->}S{alpha}*BS{beta}][i:j] licenses the edge: - [BS{->}*S{gamma}][j:j] for each grammar production C{BS{->}S{gamma}}. iccsp|iodSnxT|id|iD]:}ti||i}|i|do |Vq.q.WdS(NR((RRRRR"R RY(RRRRSRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR s (RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs tTopDownMatchRulecBseZdZdZdZRS(s A rule licensing an edge corresponding to a terminal following an incomplete edge's dot. In particular, this rule specifies that: - [AS{->}S{alpha}*w{beta}][i:j] licenses the leaf edge: - [wS{->}*][j:j+1] if the C{j}th word in the text is C{w}. iccs|ip|i|ijodSn|i}|i|}|i|jo/t||}|i|do |VqndS(N((RR R>R3RR0RY(RRRRSR!R3R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs. (RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRstCachedTopDownInitRulecBs)eZdZdZdZdZRS(s A cached version of L{TopDownInitRule}. After the first time this rule is applied, it will not generate any more edges. If C{chart} or C{grammar} are changed, then the cache is flushed. cCsti|d|_dS(N(NN(RRR$t_done(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR/s ccsh|id|jo|id|jodSnx"ti|||D] }|VqFW||f|_dS(Nii(RRR(RRRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR3s 0 cCsdS(NsTop Down Init Rule((R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR-?s(RRRRRR-(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR(s  tCachedTopDownExpandRulecBs)eZdZdZdZdZRS(s- A cached version of L{TopDownExpandRule}. After the first time this rule is applied to an edge with a given C{end} and C{next}, it will not generate any more edges for edges with that C{end} and C{next}. If C{chart} or C{grammar} are changed, then the cache is flushed. cCsti|h|_dS(N(RRR(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRJs ccs|ii|i|ifd}|d|jo|d|jodSnx%ti||||D] }|VqjW||f|i|i|if}*][i:i+1] for C{w} is a word in the text, where C{i} is C{w}'s index. iccsUxNt|iD]:}t|i||}|i|do |VqqWdS(N((R*R>R0R3RY(RRRR!R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRjs (RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRbstBottomUpPredictRulecBseZdZdZdZRS(s8 A rule licensing any edge corresponding to a production whose right-hand side begins with a complete edge's left-hand side. In particular, this rule specifies that: - [AS{->}S{alpha}*] licenses the edge: - [BS{->}*AS{beta}] for each grammar production C{BS{->}AS{beta}} iccsp|iodSnxT|id|iD]:}ti||i}|i|do |Vq.q.WdS(NR((RRRRR"R RY(RRRRSRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR{s (RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRps tBottomUpPredictCombineRulecBseZdZdZRS(iccs|iodSnxf|id|iD]L}t|i|i|id}|i||fo |Vq.q.WdS(NRi(RRRRR RRY(RRRRSRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs '(RRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRst CompleterRulecBseZdZRS(cCsdS(NsCompleter Rule((R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR-s(RRR-(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRst ScannerRulecBseZdZdZdZRS(s| A rule licensing a leaf edge corresponding to a part-of-speech terminal following an incomplete edge's dot. In particular, this rule specifies that: - [AS{->}S{alpha}*PS{beta}][i:j] licenses the edges: - [PS{->}w*][j:j+1] - [wS{->}*][j:j+1] if the C{j}th word in the text is C{w}; and C{P} is a valid part of speech for C{w}. ic cs|ip|i|ijodSn|i}|i|}|ig}|id|D]}||iqi~jovt||}|i|do |Vnt ||df|i|gd} |i| |fo | VqndS(NRi(( RR R>R3RRRR0RYR( RRRRSR!R3RJRt new_leaf_edget new_pos_edge((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs. @ (RRRRR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs t PredictorRulecBseZdZdZRS(ccs |ii|i|ifd}|d|jo|d|jodSn|iodSnx|id|iD]t}t|idjo!t |idt oqnt i ||i}|i |do |VqqW||f|i|i|if}, is used to decide what edges to add to the chart. In particular, C{ChartParser} uses the following algorithm to parse texts: - Until no new edges are added: - For each I{rule} in I{strategy}: - Apply I{rule} to any applicable edges in the chart. - Return any complete parses in the chart icCs||_||_||_||_g|_g|_x_|D]W}|idjo|ii|q=|idjo|ii|q=t|_q=WdS(s Create a new chart parser, that uses C{grammar} to parse texts. @type grammar: L{ContextFreeGrammar} @param grammar: The grammar used to parse texts. @type strategy: C{list} of L{ChartRuleI} @param strategy: A list of rules that should be used to decide what edges to add to the chart (top-down strategy by default). @type trace: C{int} @param trace: The level of tracing that should be used when parsing a text. C{0} will generate no tracing output; and higher numbers will produce more verbose tracing output. @type use_agenda: C{bool} @param use_agenda: Use an optimized agenda-based algorithm, if possible. iiN( t_grammart _strategyt_tracet _use_agendat_axiomst_inference_rulesRRRR5(RRtstrategyttracet use_agendatrule((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs      cCs|iS(N(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRsc st|}ii|tt|i}g}dididjoiGHnfd}iox:iD]/}|i |}||7}|||qWx|gjoP|i }x=i D]2}|i ||}||7}|||qWqWned} x[| djoMd} x@i D]5}|i |}|||| t|7} qZWq=WS(s @return: The final parse L{Chart}, from which all possible parse trees can be extracted. @param tokens: The sentence to be parsed @type tokens: L{list} of L{string} @rtype: L{Chart} i2iicsdidjoP|gjoCidjo d|GHnx#|D]}i|GHqAWndS(Niis%s:(RR{(Rt new_edgesR(RRtw(s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt_trace_new_edgess(R7Rtcheck_coverageR6R>RRRRRtpopRRRR#( RR=RtagendaRtaxiomRRSRt edges_added((RRRs&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt chart_parses<  "        cCs5|i|}|i}|i|id|| S(NRZ(RRR^R (RR=tnRZRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt nbest_parse4s ( RRRt BU_STRATEGYR4RRRR$RR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs  #  1tSteppingChartParsercBseZdZgddZdZdZdZdZdZdZ d Z e d Z d Z d Zd Zee dZRS(s A C{ChartParser} that allows you to step through the parsing process, adding a single edge at a time. It also allows you to change the parser's strategy or grammar midway through parsing a text. The C{initialize} method is used to start parsing a text. C{step} adds a single edge to the chart. C{set_strategy} changes the strategy used by the chart parser. C{parses} returns the set of parses that has been found by the chart parser. @ivar _restart: Records whether the parser's strategy, grammar, or chart has been changed. If so, then L{step} must restart the parsing algorithm. icCs5d|_d|_t|_ti||||dS(N(R$t_chartt_current_chartruleR5t_restartRR(RRRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRNs   cCs"tt||_t|_dS(sBegin parsing the given tokens.N(R6R7RR4R(RR=((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt initializeXsccs|idjo tdnxt|_d|iid}xq|iD]^}|idjo |iGHn|idjo|ii ||GHn|V|ioPqMqMWdVq dS(s @return: A generator that adds edges to the chart, one at a time. Each time the generator is resumed, it adds a single edge and yields that edge. If no more edges can be added, then it yields C{None}. If the parser's strategy, grammar, or chart is changed, then the generator will continue adding edges using the new strategy, grammar, or chart. Note that this generator never terminates, since the grammar or strategy might be changed to values that would add new edges. Instead, it yields C{None} when no more edges can be added with the current strategy and grammar. s Parser must be initialized firsti2iiN( RR$ROR5RR>t_parseRRR{(RRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytstepas   (ccs|i}|i}d}xa|djoSd}xF|iD];}||_x)|i||D]}|d7}|VqZWq8WqWdS(s A generator that implements the actual parsing algorithm. L{step} iterates through this generator, and restarts it whenever the parser's strategy, grammar, or chart is modified. iiN(RRRRR(RRRRRR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs     cCs|iS(s*@return: The strategy used by this parser.(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|iS(s)@return: The grammar used by this parser.(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|iS(s/@return: The chart that is used by this parser.(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRscCs|iS(s>@return: The chart rule used to generate the most recent edge.(R(R((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytcurrent_chartrulescCs|ii|ii|S(s:@return: The parse trees currently contained in the chart.(RR^RR (RRZ((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR^scCs/||ijodSn||_t|_dS(s Change the startegy that the parser uses to decide which edges to add to the chart. @type strategy: C{list} of L{ChartRuleI} @param strategy: A list of rules that should be used to decide what edges to add to the chart. N(RR4R(RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt set_strategys cCs.||ijodSn||_t|_dS(s&Change the grammar used by the parser.N(RR4R(RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt set_grammars cCs.||ijodSn||_t|_dS(s)Load a given chart into the chart parser.N(RR4R(RR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyt set_charts cCsft|}|ii||i|x&|iD]}|djoPq6q6W|id|| S(NRZ(R7RRRRR$R^(RR=RRZR((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyRs   (RRRRRRRRRRRRR^RRRR$R(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyR>s         ic%CsHddk}ddk}ddkl}l}l}|d\}} } } |d\} } }}}||| | g|| || g|| || g|| | | g|| | | g|| | | g|| | g|| dg|| dg||dg||d g||d g|| d g|| d g|| d g|| dg||dg||dgg}|||}d}dGH|GH|i}|GHH|djo<dGHdGHdGHdGHdGHdGHdG|ii i }Hnt |}|d5jo d GHdSnh}hd!t fd<d"t fd<d#tfd<d$tfd<}g}|i|o |g}n|djoddddg}nx|D]}d%||d&GHHt|||d'd(|}|i}|i|}|i||||d&Ast%ss parser: %6.3fseccSst|d|dS(i(R%(Rtb((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytDs(RRRRRR(RR( tsysttimetnltkRRRtsplitR$tstdintreadlinetstripR+t TD_STRATEGYRt BUC_STRATEGYtEARLEY_STRATEGYthas_keyRRR#R RRR*RRRR^tmaxRDRTRE(%tchoicetshould_print_timesRRRRRRtStVPtNPtPPtVtNtPtNametDetRRtsentR=ttimest strategiestchoicesRRfttR^RlR,tjRtparsetmaxlentformatt times_itemstparser((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pytdemos$$$$    #       #  '#   t__main__(%RRRRtapiRtobjectRRR0R6RRRRRRRRRRRRRRRRRRRtParserIRRR$R4RR(((s&/p/zhu/06/nlp/nltk/nltk/parse/chart.pyssB  m4G>(!      mn