³ò ŽB_Kc @s¯dZddkZddkZddkTddklZlZlZlZl Z l Z l Z l Z ddk lZddklZddklZlZlZddkTedƒZd Zd Zd efd „ƒYZd efd„ƒYZdefd„ƒYZ de fd„ƒYZ!de fd„ƒYZ"d„Z#d„Z$d„Z%edƒe&d„ƒZ'd„Z(d„Z)d„Z*d„Z+e,djoe$ƒe(ƒe*ƒndS(s» Hidden Markov Models (HMMs) largely used to assign the correct label sequence to sequential data or assess the probability of a given label and data sequence. These models are finite state machines characterised by a number of states, transitions between these states, and output symbols emitted while in each state. The HMM is an extension to the Markov chain, where each state corresponds deterministically to a given event. In the HMM the observation is a probabilistic function of the state. HMMs share the Markov chain's assumption, being that the probability of transition from one state to another only depends on the current state - i.e. the series of states that led to the current state are not used. They are also time invariant. The HMM is a directed graph, with probability weighted edges (representing the probability of a transition between the source and sink states) where each vertex emits an output symbol when entered. The symbol (or observation) is non-deterministically generated. For this reason, knowing that a sequence of output observations was generated by a given HMM does not mean that the corresponding sequence of states (and what the current state is) is known. This is the 'hidden' in the hidden markov model. Formally, a HMM can be characterised by: - the output observation alphabet. This is the set of symbols which may be observed as output of the system. - the set of states. - the transition probabilities M{a_{ij} = P(s_t = j | s_{t-1} = i)}. These represent the probability of transition to each state from a given state. - the output probability matrix M{b_i(k) = P(X_t = o_k | s_t = i)}. These represent the probability of observing each symbol in a given state. - the initial state distribution. This gives the probability of starting in each state. To ground this discussion, take a common NLP application, part-of-speech (POS) tagging. An HMM is desirable for this task as the highest probability tag sequence can be calculated for a given sequence of word forms. This differs from other tagging techniques which often tag each word individually, seeking to optimise each individual tagging greedily without regard to the optimal combination of tags for a larger unit, such as a sentence. The HMM does this with the Viterbi algorithm, which efficiently computes the optimal path through the graph given the sequence of words forms. In POS tagging the states usually have a 1:1 correspondence with the tag alphabet - i.e. each state represents a single tag. The output observation alphabet is the set of word forms (the lexicon), and the remaining three parameters are derived by a training regime. With this information the probability of a given sentence can be easily derived, by simply summing the probability of each distinct path through the model. Similarly, the highest probability tagging sequence can be derived with the Viterbi algorithm, yielding a state sequence which can be mapped into a tag sequence. This discussion assumes that the HMM has been trained. This is probably the most difficult task with the model, and requires either MLE estimates of the parameters or unsupervised learning using the Baum-Welch algorithm, a variant of EM. iÿÿÿÿN(t*(tFreqDisttConditionalFreqDisttConditionalProbDisttDictionaryProbDisttDictionaryConditionalProbDisttLidstoneProbDisttMutableProbDistt MLEProbDist(t deprecated(taccuracy(tLazyMaptLazyConcatenationtLazyZips-1e300iitHiddenMarkovModelTaggercBsþeZdZd„Zeeed„ƒZeeed„ƒZd„Zd„Z d„Z d„Z d„Z d „Z d „Zd „Zd „Zd „Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„ZRS(s€ Hidden Markov model class, a generative model for labelling sequence data. These models define the joint probability of a sequence of symbols and their labels (state transitions) as the product of the starting state probability, the probability of each state transition, and the probability of each observation being generated from each state. This is described in more detail in the module documentation. This implementation is based on the HMM description in Chapter 8, Huang, Acero and Hon, Spoken Language Processing and includes an extension for training shallow HMM parsers or specializaed HMMs as in Molina et. al, 2002. A specialized HMM modifies training data by applying a specialization function to create a new training set that is more appropriate for sequential tagging with an HMM. A typical use case is chunking. cKs˜||_||_||_||_||_d|_|idtƒƒ|_ t |i t i ƒot |i ƒ|_ nt |i tƒp‚ndS(sP Creates a hidden markov model parametised by the the states, transition probabilities, output probabilities and priors. @param symbols: the set of output symbols (alphabet) @type symbols: seq of any @param states: a set of states representing state space @type states: seq of any @param transitions: transition probabilities; Pr(s_i | s_j) is the probability of transition from state i given the model is in state_j @type transitions: C{ConditionalProbDistI} @param outputs: output probabilities; Pr(o_k | s_i) is the probability of emitting symbol k when entering state i @type outputs: C{ConditionalProbDistI} @param priors: initial state distribution; Pr(s_i) is the probability of starting in state i @type priors: C{ProbDistI} @kwparam transform: an optional function for transforming training instances, defaults to the identity function. @type transform: C{function} or C{HiddenMarkovModelTaggerTransform} t transformN(t_statest _transitionst_symbolst_outputst_priorstNonet_cachetgettIdentityTransformt _transformt isinstancettypest FunctionTypetLambdaTransformt!HiddenMarkovModelTaggerTransformI(tselftsymbolststatest transitionstoutputstpriorstkwargs((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyt__init__js        c Ks|idtƒƒ}t|tiƒot|ƒ}nt|tƒp‚n|idd„ƒ}t|i|ƒ}t t d„|Dƒƒƒ}t t d„|Dƒƒƒ}t ||ƒ} | i |d|ƒ} || i | i| i| i| id|ƒ} |o#| i|d|idtƒƒn|o[|iddƒ} | i|d | d| ƒ} |o#| i|d|idtƒƒq‹n| S( NRt estimatorcSst|d|ƒS(gš™™™™™¹?(R(tfdtbins((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyt™scss0x)|]"}x|D]\}}|VqWqWdS(N((t.0tsenttwordttag((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pys scss0x)|]"}x|D]\}}|VqWqWdS(N((R+R,R-R.((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pys Ÿstverbosetmax_iterationsitmodel(RRRRRRRR RtlisttsettHiddenMarkovModelTrainerttrain_supervisedRRRRRttesttFalsettrain_unsupervised( tclstlabeled_sequencet test_sequencetunlabeled_sequenceR%RR'R ttag_setttrainerthmmR0((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyt_trains,# 'cKs|i||||S(s¬ Train a new C{HiddenMarkovModelTagger} using the given labeled and unlabeled training instances. Testing will be performed if test instances are provided. @return: a hidden markov model tagger @rtype: C{HiddenMarkovModelTagger} @param labeled_sequence: a sequence of labeled training instances, i.e. a list of sentences represented as tuples @type labeled_sequence: C{list} of C{list} @param test_sequence: a sequence of labeled test instances @type test_sequence: C{list} of C{list} @param unlabeled_sequence: a sequence of unlabeled training instances, i.e. a list of sentences represented as words @type unlabeled_sequence: C{list} of C{list} @kwparam transform: an optional function for transforming training instances, defaults to the identity function, see L{transform()} @type transform: C{function} @kwparam estimator: an optional function or class that maps a condition's frequency distribution to its probability distribution, defaults to a Lidstone distribution with gamma = 0.1 @type estimator: C{class} or C{function} @kwparam verbose: boolean flag indicating whether training should be verbose or include printed output @type verbose: C{bool} @kwparam max_iterations: number of Baum-Welch interations to perform @type max_iterations: C{int} (R@(R9R:R;R<R%((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyttrain³s cCsd|i|ii|ƒƒS(s Returns the probability of the given symbol sequence. If the sequence is labelled, then returns the joint probability of the symbol, state sequence. Otherwise, uses the forward algorithm to find the probability over all label sequences. @return: the probability of the sequence @rtype: float @param sequence: the sequence of symbols which must contain the TEXT property, and optionally the TAG property @type sequence: Token i(tlog_probabilityRR(Rtsequence((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyt probabilityÕs c Cs(|ii|ƒ}t|ƒ}t|iƒ}|djo·|dto¨|dt}|ii|ƒ|i|i|dtƒ}xat d|ƒD]P}||t}||i |i|ƒ|i|i||tƒ7}|}q•W|Sn4|i |ƒ}t ||ddd…fŒ}|SdS(s Returns the log-probability of the given symbol sequence. If the sequence is labelled, then returns the joint log-probability of the symbol, state sequence. Otherwise, uses the forward algorithm to find the log-probability over all label sequences. @return: the log-probability of the sequence @rtype: float @param sequence: the sequence of symbols which must contain the TEXT property, and optionally the TAG property @type sequence: Token iiN( RRtlenRt_TAGRtlogprobRt_TEXTtrangeRt_forward_probabilityt_log_add( RRCtTtNt last_statetptttstatetalpha((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyRBäs"     cCs|ii|ƒ}|i|ƒS(sD Tags the sequence with the highest probability state sequence. This uses the best_path method to find the Viterbi path. @return: a labelled sequence of symbols @rtype: list @param unlabeled_sequence: the sequence of unlabeled symbols @type unlabeled_sequence: list (RRt_tag(RR<((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyR.s cCs|i|ƒ}t||ƒS(N(t _best_pathtzip(RR<tpath((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyRSscCs|i|i|ƒS(s€ @return: the log probability of the symbol being observed in the given state @rtype: float (RRG(RRQtsymbol((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyt_output_logprobsc Csb|ipTt|iƒ}t|iƒ}t|tƒ}t||ftƒ}t||ftƒ}x³t|ƒD]¥}|i|}|ii|ƒ||t | |ƒD]-} |i | i |i| ƒ|| | f       85 8 (-cCs dt|iƒt|iƒfS(Ns9(RERR(R((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyt__repr__s(t__name__t __module__t__doc__R&t classmethodRR@RARDRBR.RSRXRdRiRjRTRxRwRƒR~RŽR˜R R¤RJR‹R6Rº(((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyRYs: %"  !   (     - % E  ' ' ! 6R4cBs>eZdZeed„Zeed„Zd„Zd„ZRS(s Algorithms for learning HMM parameters from training data. These include both supervised learning (MLE) and unsupervised learning (Baum-Welch). cCs>|o ||_n g|_|o ||_n g|_dS(s× Creates an HMM trainer to induce an HMM with the given states and output symbol alphabet. A supervised and unsupervised training method may be used. If either of the states or symbols are not given, these may be derived from supervised training. @param states: the set of state labels @type states: sequence of any @param symbols: the set of observation symbols @type symbols: sequence of any N(RR(RR!R ((s"/p/zhu/06/nlp/nltk/nltk/tag/hmm.pyR&$s    cKsn|p|pt‚t}|o|i||}n|o+|o||d