³ò 4ÒÇIc@sÑddkZddklZddkTdefd„ƒYZdefd„ƒYZdfd „ƒYZd efd „ƒYZd efd „ƒYZ d„Z d„Z d„Z d„Z edjo e ƒndS(iÿÿÿÿN(tparse_dependency_grammar(t*tDependencyScorerIcBs;eZdZd„Zd„Zd„Zd„Zd„ZRS(sÊ A scorer for calculated the weights on the edges of a weighted dependency graph. This is used by a C{ProbabilisticNonprojectiveParser} to initialize the edge weights of a C{DependencyGraph}. While typically this would be done by training a binary classifier, any class that can return a multidimensional list representation of the edge weights can implement this interface. As such, it has no necessary fields. cCs$|itjotdƒ‚ndS(Ns*DependencyScorerI is an abstract interface(t __class__Rt TypeError(tself((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyt__init__ scCstdƒ‚dS(s" @type graphs: A list of C{DependencyGraph} @param graphs: A list of dependency graphs to train the scorer. Typically the edges present in the graphs can be used as positive training examples, and the edges not present as negative examples. s*DependencyScorerI is an abstract interfaceN(tAssertionError(Rtgraphs((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyttrain$scCstdƒ‚dS(sa @type graph: A C{DependencyGraph} @param graph: A dependency graph whose set of edges need to be scored. @rtype: A three-dimensional list of numbers. @return: The score is returned in a multidimensional(3) list, such that the outer-dimension refers to the head, and the inner-dimension refers to the dependencies. For instance, scores[0][1] would reference the list of scores corresponding to arcs from node 0 to node 1. The node's 'address' field can be used to determine its number identification. For further illustration, a score list corresponding to Fig.2 of Keith Hall's 'K-best Spanning Tree Parsing' paper: scores = [[[], [5], [1], [1]], [[], [], [11], [4]], [[], [10], [], [5]], [[], [8], [8], []]] When used in conjunction with a MaxEntClassifier, each score would correspond to the confidence of a particular edge being classified with the positive training examples. s*DependencyScorerI is an abstract interfaceN(R(Rtgraph((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pytscore.scCstdƒ‚dS(Ns*DependencyScorerI is an abstract interface(R(Rtother((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyt__cmp__JscCstdƒ‚dS(Ns*DependencyScorerI is an abstract interface(R(RR ((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyt__hash__Ms(t__name__t __module__t__doc__RR R R R(((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyRs     tNaiveBayesDependencyScorercBs)eZdZd„Zd„Zd„ZRS(sç A dependency scorer built around a MaxEnt classifier. In this particular class that classifier is a C{NaiveBayesClassifier}. It uses head-word, head-tag, child-word, and child-tag features for classification. cCsHdS(N((R((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyR^sc Csg}xÒ|D]Ê}xÁ|iD]¶}x­tt|iƒƒD]–}|i|ƒ}||djo d}nd}|d|d|d|dg}|itd|dd|dd|dd |dƒ|fƒq9WqWq Wd d k} | i| iƒ| i i i |ƒ|_ d S( s‡ Trains a C{NaiveBayesClassifier} using the edges present in graphs list as positive examples, the edges not present as negative examples. Uses a feature vector of head-word, head-tag, child-word, and child-tag. @type graphs: A list of C{DependencyGraph} @param graphs: A list of dependency graphs to train the scorer. tdepstTtFtwordttagtatbtctdiÿÿÿÿN( tnodelisttrangetlentget_by_addresstappendtdicttnltktusaget ClassifierItclassifytNaiveBayesClassifierR t classifier( RRtlabeled_examplesR t head_nodet child_indext child_nodetlabeltfeaturesR"((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyR as   "J c CsYg}xtt|iƒƒD]†}x}tt|iƒƒD]f}|i|ƒ}|i|ƒ}|GH|GH|itd|dd|dd|dd|dƒƒq8WqWg}g}d} xš|ii|ƒD]†} d| id ƒ| id ƒfGH|it i | id ƒƒgƒ| d 7} | t|iƒjo|i|ƒg}d} qËqËW|S( sÑ Converts the graph into a feature-based representation of each edge, and then assigns a score to each based on the confidence of the classifier in assigning it to the positive label. Scores are returned in a multidimensional list. @type graph: C{DependencyGraph} @param graph: A dependency graph to score. @rtype: 3 dimensional list @return: Edge scores for the graph parameter. RRRRRRis %.4f %.4fRRi( RRRRR R!R'tbatch_prob_classifytprobtmathtlog( RR tedgestitjR)R+t edge_scorestrowtcounttpdist((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyR }s. @!"  (RRRRR R (((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyRVs  t DemoScorercBseZd„Zd„ZRS(cCs dGHdS(Ns Training...((RR((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyR §scCs[gdgdgdggggdgdgggdggdgggdgdggggS(Niii ii i((RR ((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyR ªs(RRR R (((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyR9¥s t ProbabilisticNonprojectiveParsercBsheZdZd„Zd„Zd„Zd„Zd„Zd„Zd„Z d„Z d „Z d „Z RS( sÞ A probabilistic non-projective dependency parser. Nonprojective dependencies allows for "crossing branches" in the parse tree which is necessary for representing particular linguistic phenomena, or even typical parses in some languages. This parser follows the MST parsing algorithm, outlined in McDonald(2005), which likens the search for the best non-projective parse to finding the maximum spanning tree in a weighted directed graph. cCs dGHdS(s6 Creates a new non-projective parser. s#initializing prob. nonprojective...N((R((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyR¿scCs||_|ii|ƒdS(s Trains a C{DependencyScorerI} from a set of C{DependencyGraph} objects, and establishes this as the parser's scorer. This is used to initialize the scores on a C{DependencyGraph} during the parsing procedure. @type graphs: A list of C{DependencyGraph} @param graphs: A list of dependency graphs to train the scorer. @type dependency_scorer: C{DependencyScorerI} @param dependency_scorer: A scorer which implements the C{DependencyScorerI} interface. N(t_scorerR (RRtdependency_scorer((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyR Ås cCs|ii|ƒ|_dS(s3 Assigns a score to every edge in the C{DependencyGraph} graph. These scores are generated via the parser's scorer which was assigned during the training process. @type graph: C{DependencyGraph} @param graph: A dependency graph to assign scores to. N(R;R tscores(RR ((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pytinitialize_edge_scoresÕs cCsKdGHx|D]}|i|ƒq W|ii|ƒ|i||dƒdS(sX Takes a list of nodes that have been identified to belong to a cycle, and collapses them into on larger node. The arcs of all nodes in the graph must be updated to account for this. @type new_node: Node. @param new_node: A Node (Dictionary) to collapse the cycle nodes into. @type cycle_path: A list of integers. @param cycle_path: A list of node addresses, each of which is in the cycle. @type g_graph, b_graph, c_graph: C{DependencyGraph} @param g_graph, b_graph, c_graph: Graphs which need to be updated. sCollapsing nodes...taddressN(tremove_by_addressRR t redirect_arcs(Rtnew_nodet cycle_pathtg_graphtb_graphtc_graphtcycle_node_index((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pytcollapse_nodesàs c Cs®dG|GH|i|ƒ}dG|GHdG|iGHxût|iƒD]ê\}}xÛt|i|ƒD]Æ\}}|i||GH||jo||jot|i||ƒdjorg}|i||ƒ}|i||GdG|GHx*|i||D]} |i| |ƒqóW||i||/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pytupdate_edge_scoresôs0     8  cCsœt}x|o‡g}t}xn|D]f}|ii|ƒo@xJ|i|D]*}||jo|i|ƒt}qJqJWq#|i|ƒq#W|}q W|S(sÒ As nodes are collapsed into others, they are replaced by the new node in the graph, but it's still necessary to keep track of what these original nodes were. This takes a list of node addresses and replaces any collapsed node addresses with their original addresses. @type new_address: A list of integers. @param new_addresses: A list of node addresses to check for subsumed nodes. (tTruetFalset inner_nodesthas_keyR (Rt new_indexestswappedt originalst new_indextold_val((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyRJs     cCsNd}xA|D]9}x0|i||D]}||jo |}q%q%Wq W|S(s When updating scores the score of the highest-weighted incoming arc is subtracted upon collapse. This returns the correct amount to subtract from that edge. @type column_index: integer. @param column_index: A index representing the column of incoming arcs to a particular node being updated @type cycle_indexes: A list of integers. @param cycle_indexes: Only arcs from cycle nodes are considered. This is a list of such nodes addresses. i`yþÿ(R=(Rt column_indext cycle_indexest max_scoret row_indexRO((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyRL-s  c Cs÷|i|gƒ}dG|GHt}t}xŽtt|iƒƒD]w}xntt|i|ƒƒD]S}||jo@|i|||jo(|i||}|}|GdG|GHq]q]Wq=W|GHx3|iD](}|i|}||jo|SqÇqÇW|S(sü Returns the source of the best incoming arc to the node with address: node_index @type node_index: integer. @param node_index: The address of the 'destination' node, the node that is arced to. s originals:t,(RJtNoneRRR=RU( Rt node_indexRYtmax_arcR^R_t col_indextkeytreplaced_nodes((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pytbest_incoming_arcBs&  %    cCs¸|i|gƒ}t}t}t}x‡tt|iƒƒD]p}xgtt|i|ƒƒD]L}||jo9|i|||jo!|i||}|}|}qZqZWq:W||gS(s ??? (RJRaRRR=(RRbRYRcR^tmax_origR_Rd((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pytoriginal_best_arc]s%cCs{h|_tƒ}x_t|ƒD]Q\}}|iih|d<||d<gd<dd<|dd<ƒqW|iƒtƒ}x_t|ƒD]Q\}}|iih|d<||d<gd<dd<|dd<ƒq”Wtƒ}g|_tƒ}g} t|ƒD]H\}}| h|d<||d<gd<dd<|dd<q~ |_|i|ƒ|iGHg} |iD]} | | dqŒ~ } t|ƒ} h}xôt| ƒdjoà| i dƒ}d G|GH|i |ƒ}d G|GH|i |ƒ}|i |ƒ||W|i||ƒ|iƒ}|oÚhdd<gd<dd<| dd<}|i |ƒ|i||ƒ|i|||||ƒx"|D]}|i|d|ƒqW||i|d<| id| dƒ| d7} x|D]}|i|ƒq[WndG|GHHdG|GHHdG|GHHdG|GHdG|iGHHq»WdG|iGHdGHx;tt|ƒd| dƒD]}|||||d tTEMPtNONEs g_graph: s b_graph: s c_graph: sBetas: sreplaced nodessFinal scores: sRecovering parse...sBetas: sDone.(RUtDependencyGraphRKRR t connect_graphR>R=RtpopRRgRitadd_nodetadd_arctcontains_cycleRRRHtinsertR@R(RttokensttagsRDtindexttokentoriginal_graphRERFt_[1]t_[2]tvertextunvisited_verticest nr_verticestbetastcurrent_vertext current_nodet best_in_edget new_vertexRCRBt cycle_indextcycle_node_addressR3t new_graphtnode((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pytparsensŽ   I   I   ^ (    5 .           $( RRRRR R>RHRRRJRLRgRiRˆ(((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyR:µs         tNonprojectiveDependencyParsercBs eZdZd„Zd„ZRS(sz A non-projective, rule-based, dependency parser. This parser will return the set of all possible non-projective parses based on the word-to-word relations defined in the parser's dependency grammar, and will allow the branches of the parse tree to cross in order to capture a variety of linguistic phenomena that a projective parser will not. cCs ||_dS(s¾ Creates a new C{NonprojectiveDependencyParser}. @param dependency_grammar: a grammar of word-to-word relations. @type depenedncy_grammar: C{DependencyGrammar} N(t_grammar(Rtdependency_grammar((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyRæscCs”tƒ|_g|i_xQt|ƒD]C\}}|iiih|d<gd<dd<|d<ƒq%Wx„|iiD]v}g}x]|iiD]O}|ii|d|dƒo+|d|dj o|i|dƒq’q’W||d/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyRˆïs¸   ;  4  #-  *        J(RRRRRˆ(((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyR‰Üs cCstƒtƒdS(N(tnonprojective_conll_parse_demotrule_based_demo(((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pytdemocscCsItƒ}|igtƒƒ|idddgdddgƒ}|GHdS(Ntv1tv2tv3(R:R R9RˆRa(tnppt parse_graph((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyt hall_demois $c Cs“g}tidƒD]}|o|t|ƒqq~}tƒ}|i|tƒƒ|idddddgddd d dd gƒ}|GHdS( Ns tCathytzagthentzwaaient.tNtVtProntAdjtPunc(t conll_data2tsplitRnR:R RRˆ(RztentryRR£R¤((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyRos 4 3c Csltdƒ}|GHt|ƒ}|iddddddddd d d g ƒ}d GHx|D] }|GHqYWdS( Ns¬ 'taught' -> 'play' | 'man' 'man' -> 'the' | 'in' 'in' -> 'corner' 'corner' -> 'the' 'play' -> 'golf' | 'dachshund' | 'to' 'dachshund' -> 'his' tthetmantintcornerttaughtthist dachshundttotplaytgolfsGraphs:(RR‰Rˆ(tgrammartndpRR ((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pyRžws  0t__main__(R0R"RtdependencygraphtobjectRRR9R:R‰RŸR¥RRžR(((s>/p/zhu/06/nlp/nltk/nltk/parse/nonprojectivedependencyparser.pys s  BOÿ(‡