4Ic@s(dZdZddkZddkZddkZddkZddkZddklZddk l Z ddk Tddk Zddk TddklZlZddklZlZlZdd klZlZlZd efd YZeZed d ZdefdYZdefdYZ defdYZ!de!fdYZ"de!fdYZ#de$e$dZ%dZ&dZ'de$e$dZ(dZ)dZ*de$e$de+d d!Z,de$e$d d"Z-d#efd$YZ.d%Z/e0d&jo e/ndS('s A classifier model based on maximum entropy modeling framework. This framework considers all of the probability distributions that are empirically consistant with the training data; and chooses the distribution with the highest entropy. A probability distribution is X{empirically consistant} with a set of training data if its estimated frequency with which a class and a feature vector value co-occur is equal to the actual frequency in the data. Terminology: 'feature' ====================== The term I{feature} is usually used to refer to some property of an unlabeled token. For example, when performing word sense disambiguation, we might define a C{'prevword'} feature whose value is the word preceeding the target word. However, in the context of maxent modeling, the term I{feature} is typically used to refer to a property of a X{labeled} token. In order to prevent confusion, we will introduce two distinct terms to disambiguate these two different concepts: - An X{input-feature} is a property of an unlabeled token. - A X{joint-feature} is a property of a labeled token. In the rest of the C{nltk.classify} module, the term X{features} is used to refer to what we will call X{input-features} in this module. In literature that describes and discusses maximum entropy models, input-features are typically called X{contexts}, and joint-features are simply referred to as X{features}. Converting Input-Features to Joint-Features ------------------------------------------- In maximum entropy models, joint-features are required to have numeric values. Typically, each input-feature C{input_feat} is mapped to a set of joint-features of the form:: joint_feat(token, label) = { 1 if input_feat(token) == feat_val { and label == some_label { { 0 otherwise For all values of C{feat_val} and C{some_label}. This mapping is performed by classes that implement the L{MaxentFeatureEncodingI} interface. s epytext eniN(t defaultdict(t OrderedDict(t*(tattested_labelst CutoffChecker(t call_megamtwrite_megam_filetparse_megam_weights(t call_tadmtwrite_tadm_filetparse_tadm_weightstMaxentClassifierc BseZdZedZdZdZdZdZdZ ddZ d d d Z d Z d ddddddddg Z eedeeeddZhdd<dd<dd<dd<ddss t-ics|\}}ti|S((tabsR (R-tfidt_(R(s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pytss and label is ii/i,s...t is TOTAL:c3s!x|]}d|VqWdS(s%8.3fN((R-R.(tsums(s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pys ss PROBS:c3s&x|]}di|VqWdS(s%8.3fN(tprob(R-R.(tpdist(s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pys sN(tstrRtsortedtsamplesR6R"tljusttjoinRRtintt enumerateR R tsortRR R1tdescribetsplit( RRtcolumnst descr_widthtTEMPLATERtiR$R%R'R(tscoretdescr((RR7R5s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pytexplains8!    %i tallc stttidfddt}|djo=g}|D]%}i|djo ||qHqH~}nK|djo=g}|D]%}i|djo ||qq~}nx4|| D](}di|ii|fGHqWdS( sT @param show: all, neg, or pos (for negative-only or positive-only) R*csti|S((R0R (R1(R(s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyR3sR+tpositnegs%8.3f %sN(R9trangeRR R"R R@(Rtntshowtfidst_[1]R1t_[2]((Rs*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pytshow_most_informative_featuress  = =  cCs&dt|ii|iifS(Ns:(RR RR(R((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyt__repr__stGIStIIStCGtBFGStPowelltLBFGSBs Nelder-MeadtMEGAMtTADMiic Ks|djo6yddk} d}WqCtj o d}qCXnx/|D]'} | djotd | qJqJW|i}|djot|||||Sn|djot|||||Sn||ijo*t|||||i||||Sn|djot ||||||Sn`|djoB|} || d<|| d<|| d<|| d}. (requires that C{megam} be installed.) The default algorithm is C{'CG'} if C{'scipy'} is installed; and C{'iis'} otherwise. @type trace: C{int} @param trace: The level of diagnostic tracing output to produce. Higher values produce more verbose output. @type encoding: L{MaxentFeatureEncodingI} @param encoding: A feature encoding, used to convert featuresets into feature vectors. If none is specified, then a L{BinaryMaxentFeatureEncoding} will be built based on the features that are attested in the training corpus. @type labels: C{list} of C{str} @param labels: The set of possible labels. If none is given, then the set of all labels attested in the training data will be used instead. @param sparse: If true, then use sparse matrices instead of dense matrices. Currently, this is only supported by the scipy (optimization method) algorithms. For other algorithms, its value is ignored. @param gaussian_prior_sigma: The sigma value for a gaussian prior on model weights. Currently, this is supported by the scipy (optimization method) algorithms and C{megam}. For other algorithms, its value is ignored. @param cutoffs: Arguments specifying various conditions under which the training should be halted. (Some of the cutoff conditions are not supported by some algorithms.) - C{max_iter=v}: Terminate after C{v} iterations. - C{min_ll=v}: Terminate after the negative average log-likelihood drops under C{v}. - C{min_lldelta=v}: Terminate if a single iteration improves log likelihood by less than C{v}. - C{tolerance=v}: Terminate a scipy optimization method when improvement drops below a tolerance level C{v}. The exact meaning of this tolerance depends on the scipy algorithm used. See C{scipy} documentation for more info. Default values: 1e-3 for CG, 1e-5 for LBFGSB, and 1e-4 for other algorithms. I{(C{scipy} only)} iNtcgtiistmax_itertmin_llt min_lldeltat tolerancetmax_acct min_accdeltat count_cutofftnormsUnexpected keyword arg %rtgistmegamttadmttraceRRtgaussian_prior_sigmasUnknown algorithm %s(smax_itersmin_lls min_lldeltas tolerancesmax_accs min_accdeltas count_cutoffsnorm( tNonetscipyt ImportErrort TypeErrortlowert train_maxent_classifier_with_iist train_maxent_classifier_with_gist _SCIPY_ALGSt"train_maxent_classifier_with_scipyt"train_maxent_classifier_with_megamtTadmMaxentClassifierttraint ValueError( tclst train_tokst algorithmRiRRtsparseRjtcutoffsRlR*tkwargs((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRvsHR                R\tbfgstpowelltlbfgsbs nelder-mead(t__name__t __module__t__doc__R"RRRRRRRHRRRSt ALGORITHMSt classmethodRkRvRr(((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyR Ms"      "  us$Use MaxentClassifier.train() insteadcOsti||S(N(R Rv(targsR}((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyttrain_maxent_classifierYstMaxentFeatureEncodingIcBs;eZdZdZdZdZdZdZRS(s A mapping that converts a set of input-feature values to a vector of joint-feature values, given a label. This conversion is necessary to translate featuresets into a format that can be used by maximum entropy models. The set of joint-features used by a given encoding is fixed, and each index in the generated joint-feature vectors corresponds to a single joint-feature. The length of the generated joint-feature vectors is therefore constant (for a given encoding). Because the joint-feature vectors generated by C{MaxentFeatureEncodingI} are typically very sparse, they are represented as a list of C{(index, value)} tuples, specifying the value of each non-zero joint-feature. Feature encodings are generally created using the L{train()} method, which generates an appropriate encoding based on the input-feature values and labels that are present in a given corpus. cCstddS(sS Given a (featureset, label) pair, return the corresponding vector of joint-feature values. This vector is represented as a list of C{(index, value)} tuples, specifying the value of each non-zero joint-feature. @type featureset: C{dict} @rtype: C{list} of C{(int, number)} sNot implementedN(R(RRR$((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyR ws cCstddS(s @return: The size of the fixed-length joint-feature vectors that are generated by this encoding. @rtype: C{int} sNot implementedN(R(R((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRscCstddS(s @return: A list of the "known labels" -- i.e., all labels C{l} such that C{self.encode(fs,l)} can be a nonzero joint-feature vector for some value of C{fs}. @rtype: C{list} sNot implementedN(R(R((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRscCstddS(s @return: A string describing the value of the joint-feature whose index in the generated feature vectors is C{fid}. @rtype: C{str} sNot implementedN(R(RR1((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyR@scCstddS(s Construct and return new feature encoding, based on a given training corpus C{train_toks}. @type train_toks: C{list} of C{tuples} of (C{dict}, C{str}) @param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. sNot implementedN(R(RxRy((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRvs (RRRR RRR@Rv(((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRas   t#FunctionBackedMaxentFeatureEncodingcBs;eZdZdZdZdZdZdZRS(s A feature encoding that calls a user-supplied function to map a given featureset/label pair to a sparse joint-feature vector. cCs||_||_||_dS(so Construct a new feature encoding based on the given function. @type func: (callable) @param func: A function that takes two arguments, a featureset and a label, and returns the sparse joint feature vector that encodes them: >>> func(featureset, label) -> feature_vector This sparse joint feature vector (C{feature_vector}) is a list of C{(index,value)} tuples. @type length: C{int} @param length: The size of the fixed-length joint-feature vectors that are generated by this encoding. @type labels: C{list} @param labels: A list of the "known labels" for this encoding -- i.e., all labels C{l} such that C{self.encode(fs,l)} can be a nonzero joint-feature vector for some value of C{fs}. N(t_lengtht_funct_labels(RtfuncRR((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRs  cCs|i||S(N(R(RRR$((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyR scCs|iS(N(R(R((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRscCs|iS(N(R(R((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRscCsdS(Nsno description available((RR1((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyR@s(RRRRR RRR@(((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRs     tBinaryMaxentFeatureEncodingcBsVeZdZeedZdZdZdZdZe de dZ RS(s A feature encoding that generates vectors containing a binary joint-features of the form:: joint_feat(fs, l) = { 1 if (fs[fname] == fval) and (l == label) { { 0 otherwise Where C{fname} is the name of an input-feature, C{fval} is a value for that input-feature, and C{label} is a label. Typically, these features are constructed based on a training corpus, using the L{train()} method. This method will create one feature for each combination of C{fname}, C{fval}, and C{label} that occurs at least once in the training corpus. The C{unseen_features} parameter can be used to add X{unseen-value features}, which are used whenever an input feature has a value that was not encountered in the training corpus. These features have the form:: joint_feat(fs, l) = { 1 if is_unseen(fname, fs[fname]) { and l == label { { 0 otherwise Where C{is_unseen(fname, fval)} is true if the encoding does not contain any joint features that are true when C{fs[fname]==fval}. The C{alwayson_features} parameter can be used to add X{always-on features}, which have the form:: joint_feat(fs, l) = { 1 if (l == label) { { 0 otherwise These always-on features allow the maxent model to directly model the prior probabilities of each label. c CsWt|ittt|jotdnt||_||_t||_d|_ d|_ |o_t g}t |D] \}}||||ifq~|_ |it|i 7_n|ortd|D}t g} t |D] \}} | | ||ifq ~ |_ |it|7_ndS(s @param labels: A list of the "known labels" for this encoding. @param mapping: A dictionary mapping from C{(fname,fval,label)} tuples to corresponding joint-feature indexes. These indexes must be the set of integers from 0...len(mapping). If C{mapping[fname,fval,label]=id}, then C{self.encode({..., fname:fval, ...}, label)[id]} is 1; otherwise, it is 0. @param unseen_features: If true, then include unseen value features in the generated joint-feature vectors. @param alwayson_features: If true, then include always-on features in the generated joint-feature vectors. sHMapping values must be exactly the set of integers from 0...len(mapping)css"x|]\}}}|VqWdS(N((R-tfnametfvalR$((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pys (sN(tsettvaluesRLRRwtlistRt_mappingRRkt _alwaysont_unseentdictR>( RRtmappingtunseen_featurestalwayson_featuresRPRER$tfnamesRQR((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRs +    9 9cCs g}x|iD]\}}|||f|ijo'|i|i|||fdfq|iodxa|iD]$}|||f|ijoPqsqsW||ijo|i|i|dfqqqW|io.||ijo|i|i|dfn|S(Ni(titemsRtappendRRR(RRR$RRRtlabel2((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyR -s '   &cCst|ttfptdny |iWnYtj oMdgt|i|_x/|iiD]\}}||i|} for a description of the joint-features that will be included in this encoding. @type train_toks: C{list} of C{tuples} of (C{dict}, C{str}) @param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. @type count_cutoff: C{int} @param count_cutoff: A cutoff value that is used to discard rare joint-features. If a joint-feature's value is 1 fewer than C{count_cutoff} times in the training corpus, then that joint-feature is not included in the generated encoding. @type labels: C{list} @param labels: A list of labels that should be used by the classifier. If not specified, then the set of labels attested in C{train_toks} will be used. @param options: Extra parameters for the constructor, such as C{unseen_features} and C{alwayson_features}. sUnexpected label %si(RRR=RwtaddRRRk( RxRyRdRtoptionsRt seen_labelstcountttokR$RR((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRvgs      )( RRRtFalseRR R@RRRRkRv(((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRs' /    t GISEncodingcBsPeZdZeeedZedddZdZdZ dZ RS(s A binary feature encoding which adds one new joint-feature to the joint-features defined by L{BinaryMaxentFeatureEncoding}: a correction feature, whose value is chosen to ensure that the sparse vector always sums to a constant non-negative number. This new feature is used to ensure two preconditions for the GIS training algorithm: - At least one feature vector index must be nonzero for every token. - The feature vector must sum to a constant non-negative number for every token. c Csqti||||||djo>ttg}|D]\}}} ||q7~d}n||_dS(s @param C: The correction constant. The value of the correction feature is based on this value. In particular, its value is C{C - sum([v for (f,v) in encoding])}. @seealso: L{BinaryMaxentFeatureEncoding.__init__} iN(RRRkRRt_C( RRRRRtCRPRRR$((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRs  > cCs|iS((R(R((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyR3stdocsX The non-negative constant that all encoded feature vectors will sum to.c Csti|||}ti|}tg}|D]\}}||q2~}||ijotdn|i||i|f|S(Ns&Correction feature is not high enough!(RR RtsumRRwR( RRR$Rt base_lengthRPtftvR&((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyR s-cCsti|dS(Ni(RR(R((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRscCs9|ti|jod|iSnti||SdS(NsCorrection feature (%s)(RRRR@(RR'((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyR@s( RRRRRkRtpropertyRR RR@(((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRs     tTadmEventMaxentFeatureEncodingcBsPeZeedZdZdZdZdZede dZ RS(cCs;t||_t|_ti|||i||dS(N(RRt_label_mappingRR(RRRRR((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRs  cCsg}x|iD]\}}||f|ijo t|i|i||f Training (%d iterations)is- Iteration Log Likelihood Accuracys- ---------------------------------------s %9d %14.5f %9.3fis* Training stopped: keyboard interrupts! Final %14.5f %9.3f(t setdefaultRRkRRvthasattrRnRtcalculate_empirical_fcountRtnumpytnonzerotzerosRtNINFt ConditionalExponentialClassifiertlog2R"tlltnltkRtutiltlog_likelihoodtacctaccuracytitertcalculate_estimated_fcountRRtchecktKeyboardInterrupt(RyRiRRR|t cutoffcheckertCinvtempirical_fcountt unattestedRR1t classifiertlog_empirical_fcounttll_oldtacc_oldRRtiternumtestimated_fcounttlog_estimated_fcount((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRqsf               cCsfti|id}xG|D]?\}}x0|i||D]\}}||c|7WqW|S(NR(RRRR (RyRtfcountRR$tindextval((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRps  c Csti|id}x|D]x\}}|i|}xZ|iD]L}|i|}x4|i||D] \}} ||c|| 7 Training (%d iterations)is- Iteration Log Likelihood Accuracys- ---------------------------------------s %9d %14.5f %9.3fs* Training stopped: keyboard interrupts! Final %14.5f %9.3f( RRRkRRvRRtcalculate_nfmapRtarrayR9t __getitem__treshapeRRRRRR"RRRRRRRRtcalculate_deltasRRRR(RyRiRRR|Rtempirical_ffreqtnfmaptnfarrayt nftransposeRRR1RRRRRRtdeltas((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRps^     !             c Cst}xk|D]c\}}xT|iD]F}|itg}|i||D]\}}||qO~q)WqWtg} t|D]\} } | | | fq~ S(s Construct a map that can be used to compress C{nf} (which is typically sparse). M{nf(feature_vector)} is the sum of the feature values for M{feature_vector}. This represents the number of features that are active for a given labeled text. This method finds all values of M{nf(t)} that are attested for at least one token in the given list of training tokens; and constructs a dictionary mapping these attested values to a continuous range M{0...N}. For example, if the only values of M{nf()} that were attested were 3, 5, and 7, then C{_nfmap} might return the dictionary {3:0, 5:1, 7:2}. @return: A map that can be used to compress C{nf} to a dense vector. @rtype: C{dictionary} from C{int} to C{int} (RRRRR RR>( RyRtnfsetRR2R$RPtidRRQREtnf((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRs   Hc Csd}d} ti|id} tit||ifd} x|D]\} } |i| }x|iD]} |i| | }tg}|D]\}}||q~}x;|D]3\}}| |||fc|i | |7RRR{t lil_matrixRRR t maxentropytconditionalmodeltsigma2RR"tverbosetmaxitertavegtoltmaxgtolttoltfittparamsRteR (RyRiRRRzR{RjR|RlRRPRER$tlabelnumt num_featurestnum_tokst num_labelsRtFtNttoknumRRR1RtmodelR((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRs~sT   9     ( .      c KsWt}|tjo4|idd}ti||d|dt}n|tj otdnyTtidddd \}} ti | d } t ||| d || i Wn.t t tfj o} td | nXg} | d ddg7} |o| dg7} n|od|d} nd} | dd| dg7} |djo| dg7} nd|jo| dd|dg7} nd|jo"| ddt|dg7} n| d| g7} | GHt| }yti| Wn*t t fj o} d| | fGHnXt||}|titi9}t||S(s Train a new C{ConditionalExponentialClassifier}, using the given training samples, using the external C{megam} library. This C{ConditionalExponentialClassifier} will encode the model that maximizes entropy from all the models that are empirically consistent with C{train_toks}. @see: L{train_maxent_classifier()} for parameter descriptions. @see: L{nltk.classify.megam} RdiRRs$Specify encoding or labels, not bothtprefixsnltk-tsuffixs.gztwbtexplicits,Error while creating megam training file: %ss-nobiass-repeatt10s -explicitg?is-lambdas%.2fs-tuneis-quietR^s-maxis%stll_deltas-dppt multiclasss Warning: unable to delete %s: %s(R"RktgetRRvRwttempfiletmkstemptgziptopenRtclosetOSErrortIOErrorR0RtostremoveRRRRR (RyRiRRRjR}RRdtfdttrainfile_namet trainfileRRt inv_variancetstdoutR((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRtsL       " RucBseZedZRS(cKsk|idd}|idd}|idd}|idd}|idd}|id d}|id } |id } |pti||d|}ntid d dd\} } tid d\} }ti| d}t||||i g}|i dg|i d|g|o|i dd|dgn| o|i dd| gn| o!|i ddt | gn|i d| g|i d|g|djo|i dgn|i dgt |t|d}t |}|i ti| ti||titi9}|||S(NRzttao_lmvmRiiRRRjiRdR^R`R snltk-tadm-events-R s.gzsnltk-tadm-weights-Rs-monitors-methods-l2s%.6fis-max_its%ds-fatols -events_ins -params_outs2>&1s-summarytrb(RRkRRvRRRRR RtextendR0RR RRRRR(RxRyR}RzRiRRtsigmaRdR^Rt trainfile_fdRt weightfile_fdtweightfile_nameRRt weightfileR((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRv+sN   !      (RRRRv(((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pyRu*scCs#ddkl}|ti}dS(Ni(t names_demo(tnltk.classify.utilR*R Rv(R*R((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pytdemogst__main__(1Rt __docformat__RttimeRRRRRt nltk.utilRtnltk.probabilityR+tapiRRRRgRRRRhRR R t ClassifierIR Rt deprecatedRtobjectRRRRRRkRqRRRpRRR"RsRtRuR,R(((s*/p/zhu/06/nlp/nltk/nltk/classify/maxent.pys6sP         G-:; `  Y  ~\M=