³ò 4ÒÇIc@sdZddkZddkZddklZlZlZlZddkl Z ddk l Z l Z l Z lZddklZddkZ ddkTddkTdeeeddd „Zd efd „ƒYZd efd „ƒYZdefd„ƒYZdefd„ƒYZde fd„ƒYZ!defd„ƒYZ"de#fd„ƒYZ$de$fd„ƒYZ%defd„ƒYZ&e'dd„Z(d„Z)e*djoUe(ƒHed d!d"ƒZ+d#Z,e,i-ƒZ.e+i/e.ƒZ0xe0D] Z1e1GHqêWndS($s` Extension of chart parsing implementation to handle grammars with feature structures as nodes. iÿÿÿÿN(t FeatStructtunifytFeatStructParsertTYPE(tlogic(t Nonterminalt ProductiontContextFreeGrammart defaultdict(tFeatStructNonterminal(t*ic Cs@tii|d|d|d|d|ƒ}t|d|d|ƒS(s¾ Load a grammar from a file, and build an Earley feature parser based on that grammar. You can optionally specify a tracing level, for how much output you want to see: 0: No output. 1: Show edges from scanner and completer rules (not predictor). 2 (default): Show all edges as they are added to the chart. 3: Show all edges, plus the results of successful unifications. 4: Show all edges, plus the results of all attempted unifications. 5: Show all edges, plus the results of all attempted unifications, including those with cached results. If C{verbose} is set to C{True}, then more diagnostic information about grammar-loading is displayed. tcachetverboset logic_parsertfstruct_parserttracet chart_class(tnltktdatatloadtFeatureEarleyChartParser(tfilenameRR R RR Rtgrammar((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyt load_earleys  tFeatureTreeEdgecBsSeZdZded„Zd„Zd„Zd„Zd„Zd„Z d„Z RS( s A specialized tree edge that allows shared variable bindings between nonterminals on the left-hand side and right-hand side. Each C{FeatureTreeEdge} contains a set of C{bindings}, i.e., a dictionary mapping from variables to values. If the edge is not complete, then these bindings are simply stored. However, if the edge is complete, then the constructor applies these bindings to every nonterminal in the edge whose symbol implements the interface L{SubstituteBindingsI}. icCs |djo h}n|t|ƒjoP|oI|i||ƒ}g}|D]}||i||ƒqN~}h}nti|||||ƒ||_dS(su Construct a new edge. If the edge is incomplete (i.e., if C{dotlss%s %s(t is_completeRt__str__tjointsortedRtitems(RR$((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyR0hs  cCsd|i|ijodSnt|i|i|i|i|if|i|i|i|i|ifƒS(Niÿÿÿÿ(t __class__tcmpt_spant_lhst_rhst_dotR(Rtother((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyt__cmp__qs cCs4t|i|i|i|itt|iƒƒfƒS(N(thashR7R8R6R9ttupleR2R(R((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyt__hash__xs( t__name__t __module__t__doc__RRRR+R$R0R;R>(((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyR6s     tFeatureFundamentalRulecBseZdZd„ZRS(s¤ A specialized version of the fundamental rule that operates on nonterminals whose symbols are C{FeatStructNonterminal}s. Rather tha simply comparing the nonterminals for equality, they are unified. Variable bindings from these unifications are collected and stored in the chart using a L{FeatureTreeEdge}. When a complete edge is generated, these bindings are applied to all nonterminals in the edge. The fundamental rule states that: - [AS{->}S{alpha}*B1S{beta}][i:j] - [B2S{->}S{gamma}*][j:k] licenses the edge: - [AS{->}S{alpha}B3*S{beta}][i:j] assuming that B1 and B2 can be unified to generate B3. c csh|iƒ|iƒjoX|iƒoK|iƒo>t|tƒo.t|tƒo|iƒt|iƒtjpdSn|i ƒ}t |iƒ|iƒ|dt ƒ}|djodSnt d|iƒ|iƒfd|iƒd|iƒd|iƒdd|ƒ}t }x;|i|ƒD]*} |i|| |fƒo t}q&q&W|o |VndS(Nt rename_varsR R!R"R#iR$(tendtstartt is_incompleteR/R'RR*RR!R$RtFalseRRR"R#tchild_pointer_liststinserttTrue( RtchartRt left_edget right_edgeR$tresulttnew_edget changed_charttcpl1((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyt apply_iterŽs(  ! (R?R@RARR(((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRB}stFeatureTopDownExpandRulecBseZdZd„ZRS(sÎ A specialized version of the top down expand rule that operates on nonterminals whose symbols are C{FeatStructNonterminal}s. Rather tha simply comparing the nonterminals for equality, they are unified. The top down expand rule states that: - [AS{->}S{alpha}*B1S{beta}][i:j] licenses the edge: - [B2S{->}*S{gamma}][j:j] for each grammar production C{B2S{->}S{gamma}}, assuming that B1 and B2 can be unified. c cs |iƒodSnxð|iƒD]â}t|iƒƒdjo!t|iƒdtƒoq"n|iƒt|iƒtjot |iƒ|i ƒdt ƒo_t d|i ƒ|i ƒfd|iƒd|iƒddƒ}|i|dƒo |Vqq"q"WdS( NiiRCR R!R"R#((R/t productionsRR"R'tstrR!RR*RR+RJRRDRI(RRKRtedgetprodRO((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRR»s :!" (R?R@RARR(((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRS­s tFeatureCompleterRulecBs eZdZeƒZd„ZRS(s A specialized version of the completer rule that operates on nonterminals whose symbols are C{FeatStructNonterminal}s. Rather tha simply comparing the nonterminals for equality, they are unified. See L{CompleterRule} for more information. ccs^|i}xN|id|iƒdtƒD].}x%|i||||ƒD] }|VqGWq(WdS(NRDR/(t_fundamental_ruletselectRERGRR(RRKRtedge1tfrtedge2RO((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRRØs  (R?R@RARBRYRR(((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRXÏs tFeatureScannerRulecBseZd„ZRS(c cs2|iƒp|iƒ|iƒjodSn|iƒ}|i|ƒ}xâg}|id|ƒD]}||iƒqc~D]­}|t|iƒtjoŒt||i ƒdt ƒopt ||ƒ} |i | dƒo | Vnt ||df||gdƒ} |i | | fƒo | Vq*q}q}WdS(NR"RCi((R/RDt num_leavestleafRTR!RR*RR+RJtLeafEdgeRIR( RRKtgramarRVtindexR`R%RWtpost new_leaf_edget new_pos_edge((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRRßs. 4 (R?R@RR(((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyR^ÞstFeaturePredictorRulecBseZRS((R?R@(((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRgïstEarleyChartParsercBsbeZdZeZeZeZde d„Z d„Z dZ e ee d„Zd„Zd„ZRS(sƒ A chart parser implementing the Earley parsing algorithm: - For each index I{end} in [0, 1, ..., N]: - For each I{edge} s.t. I{edge}.end = I{end}: - If I{edge} is incomplete, and I{edge}.next is not a part of speech: - Apply PredictorRule to I{edge} - If I{edge} is incomplete, and I{edge}.next is a part of speech: - Apply ScannerRule to I{edge} - If I{edge} is complete: - Apply CompleterRule to I{edge} - Return any complete parses in the chart @ivar _predictor_class, _completer_class, _scanner_class: The classes that are used to implement the three rules used by the Earley algorithm, Replacement rules can be specified by subclasses (such as L{FeatureEarleyChartParser }). icCs?t|tƒotdƒ‚n||_||_||_dS(s7 Create a new Earley chart parser, that uses C{grammar} to parse texts. @type grammar: C{ContextFreeGrammar} @param grammar: The grammar used to parse texts. @type trace: C{int} @param trace: The level of tracing that should be used when parsing a text. C{0} will generate no tracing output; and higher numbers will produce more verbose tracing output. @param chart_class: The class that should be used to create the charts used by this parser. stEarley parser no longer takes a lexicon as a separate parameter; the lexicon is calculated from the grammar instead.N(R'tdictt ValueErrort_grammart_tracet _chart_class(RRRR((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRs   cCs|iS(N(Rk(R((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyR*si(c Cs£|dj o ||_nt|ƒ}|ii|ƒ|i|ƒ}|i}td|i|iƒdƒ}|idjod G|i |ƒGHn|i |i |i ƒƒd ƒ|i ƒ}|iƒ} |iƒ} x—t|iƒdƒD]} |idjo d| GHn|idjo!tiidƒtiiƒnx(|id| ƒD]} | iƒoMxJ| i||| ƒD]/} |idjodG|i| |ƒGHq~q~Wn| iƒoMxJ|i||| ƒD]/} |idjod G|i| |ƒGHqØqØWn| iƒoMxJ| i||| ƒD]/} |idjod G|i| |ƒGHq2q2WqUqUWqîW|idjoHn|i||i ƒ|ƒ| S( Niit i sProcessing queue %dt.RDt Completert Predictors Scanner s ((RRltlistRktcheck_coverageRmtmaxt_trace_chart_widthR_t pp_leavesRIt _starter_edgeREt_predictor_classt_completer_classt_scanner_classtrangetsyststdouttwritetflushRZR/RRtpp_edgeRFt_parses(Rttokenstnt tree_classRRKRtwt predictort completertscannerRDRVte((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyt nbest_parse1sF   &   ! ! ! )cCstdƒ}td||fƒS(s9Return a 'starter edge' that expands to the start symbol.s[INIT]i(ii(RR(Rt start_symtroot((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRw]s cCs|i|d|ƒS(s+Return a list of parses in the given chart.R„(tparses(RRKR‹R„((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRds(R?R@RAt PredictorRuleRxt CompleterRuleRyt ScannerRuleRztChartRRRuRtTreeRŠRwR(((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRhùs , RcBs8eZdZeZeZeZdZ d„Z d„Z RS(sÅ A chart parser implementing the Earley parsing algorithm, allowing nonterminals that have features (known as L{FeatStructNonterminal}s). See L{EarleyChartParser} for more details. i cCs"tdƒ}td||fdƒS(Ns[*type*="[INIT]"]i(ii(R R(RRERŒ((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRwss cCsžg}x‘|idd|iƒfƒD]q}t|tƒ oZ|iƒt|tjo?t|iƒ|dtƒo#||i|dtd|ƒ7}q%q%W|S(NR iRCtcompleteR„( RZR_R'RaR!RRRJttrees(RRKRER„R”RV((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRws( R?R@RARgRxRXRyR^RzRuRwR(((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRhs tInstantiateVarsChartcBs2eZdZd„Zd„Zd„Zd„ZRS(s> A specialized chart that 'instantiates' variables whose names start with '@', by replacing them with unique new variables. In particular, whenever a complete edge is added to the chart, any variables in the edge's C{lhs} whose names start with '@' will be replaced by unique new L{Variable}s. cCs ti||ƒtƒ|_dS(N(R‘Rtsett _instantiated(RR‚((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRŽscCs:||ijotSn|i|ƒ}ti|||ƒS(N(R—RGtinstantiate_edgeR‘RI(RRVtchild_pointer_list((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyRI’scCs¶t|tƒp|Sn|iƒp|Sn||ijo|Sn|i|ƒ}|p|Sn|ii|ƒ|iƒi|ƒ}t|i ƒ||i ƒ|i ƒ|i ƒƒS(N( R'RR/t _edge_to_cplst inst_varsR—taddR!R(R R"R#R$(RRVR›R!((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyR˜—scCs td„|iƒiƒDƒƒS(Ncss<x5|].}|iidƒo|tiƒfVqqWdS(t@N(tnamet startswithRtunique_variable(R-tvar((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pys ªs (RiR!t variables(RRV((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyR›©s (R?R@RARRIR˜R›(((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyR•†s    icCsbddk}ddk}tdƒ}tdƒ}tdƒ}tdƒ}tdƒ}tdƒ} tdƒ} td ƒ} td ƒ} td ƒ} td ƒ}td ƒ}tdƒ}t|||fƒt|| |fƒt|||fƒt|||fƒt|||fƒt||fƒt|||fƒt|| |fƒt|dƒt|dƒt| d ƒt| d!ƒt| d"ƒt|d#ƒt|d$ƒt|d%ƒt|d&ƒt| d'ƒt| d(ƒg}t||ƒ}|GHHd}dG|GHH|iƒ}|iƒ}t|d|ƒ}|i|ƒ}H|od|iƒ|GHnx|D] }|GHqOWdS()NiÿÿÿÿtStVPtNPtPPtVtNtPtNametDetsDet[-pl]sDet[+pl]sN[-pl]sN[+pl]tJohntItthetmytatdogtcookietatetsawtwithtunders$I saw John with a dog with my cookies Sentence:RsTime: %s(sJohn(R­(sthe(smy(R°(sdog(scookie(sate(ssaw(swith(sunder(R|ttimeR RRtsplitRRŠ(tshould_print_timesRR|R·R£R¤R¥R¦R§R¨R©RªR«tDetSgtDetPltNSgtNPlRTtearley_grammartsentR‚tttcpR”ttree((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pytdemo³sR             $$!   cCsyddk}|iddƒddk}|idƒ}|iƒiddƒidƒ|iƒiddƒidƒdS(Niÿÿÿÿsfor i in range(1): demo()s/tmp/profile.outR·tcumi<(tprofiletruntpstatstStatst strip_dirst sort_statst print_stats(RÅRÇtp((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyt run_profileås   t__main__s!grammars/book_grammars/feat0.fcfgRisKim likes children(2RAtyamlR|tnltk.featstructRRRRtnltk.semRRRRRRt nltk.grammarR t nltk.datatapiRKRGR‘RRRRtFundamentalRuleRBtTopDownExpandRuleRSRRXRR^RgtParserIRhRR•RJRÃRÍR?RÁR¿R¸R‚RŠR”RÂ(((s-/p/zhu/06/nlp/nltk/nltk/parse/featurechart.pyss>""    G0" o-2