B_Kc @sdZddkZddkZddklZddklZddklZdd>Z dd>Z dd>Z dd >Z dd >Z dd >Ze e e Ze e eZhe d d f<e dd f<e dd f<e d df<e ddf<eddf %(NonWord)s # either other punctuation | \s+(?P\S+) # or whitespace and some other token ))cCs^y |iSWnLti|ih|id<|id<titiB|_|iSnXdS(sjCompiles and returns a regular expression to find contexts including possible sentence boundaries.Rt SentEndCharsN(R RRt_period_context_fmtRt_re_sent_end_charsRR(R ((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pytperiod_context_res   (s_re_period_contexts_re_word_tokenizer(RRR(t__name__t __module__t__doc__t __slots__R Rtsent_end_charsR$tinternal_punctuationRRt MULTILINEtre_boundary_realignmentRRRRRR!R#R%(((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRRs"      s[^\W\d]tPunktWordTokenizercBs eZedZdZRS(cCs ||_dS(N(t _lang_vars(R t lang_vars((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyt__init__scCs|ii|S(N(R/R!(R ttext((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyttokenizes(R&R'RR1R3(((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR.sccsIt|}|i}x|D]}||fV|}qW|dfVdS(s Yields pairs of tokens from the given iterator such that each input token will appear as the first element in a yielded tuple. The last pair will have None as its second element. N(titertnexttNone(tittprevtel((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyt _pair_iters    tPunktParameterscBsDeZdZdZdZdZdZdZdZRS(sCStores data used to perform sentence boundary detection with punkt.cCs7t|_t|_t|_tt|_dS(N(tsett abbrev_typest collocationst sent_startersRtintt ortho_context(R ((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR1s    cCst|_dS(N(R<R=(R ((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyt clear_abbrevsscCst|_dS(N(R<R>(R ((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pytclear_collocationsscCst|_dS(N(R<R?(R ((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pytclear_sent_startersscCstt|_dS(N(RR@RA(R ((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pytclear_ortho_context scCs|i|c|Os s %s(%s,%s %s)(RPRORttjoinRTt __class__R&(R ttypestrtpropvals((R s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyt__repr__|s  cCsU|i}|io|d7}n|io|d7}n|io|d7}n|S(sO A string representation akin to that used by Kiss and Strunk. sss(RORMRNRL(R tres((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyt__str__s    (R&R'R(RTR)R1RRRfR[RRkRmRRtpropertyR^R_RaRcReRhRjRlRnRqRzR|(((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRIs,     t_PunktBaseClasscBsAeZdZeeedZdZdZdZ RS(sP Includes common components of PunktTrainer and PunktSentenceTokenizer. cCs||_||_||_dS(N(t_paramsR/t_Token(R R0t token_clsRW((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR1s   ccst}x|idD]z}|ioat|ii|}|i|id|dtVt}x&|D]}|i|VqnWqt}qWdS(sB Divide the given text into tokens, using the punkt word segmentation regular expression, and generate the resulting list of tokens augmented as three-tuples with two boolean values for whether the given token occurs at the start of a paragraph or a new line, respectively. s RJRKN( tFalsetsplittstripR4R/R!RR5tTrue(R t plaintextRJtlinet line_tokstt((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyt_tokenize_wordss ccs'x |D]}|i||VqWdS(s Perform the first pass of annotation, which makes decisions based purely based on the word type of each word: - '?', '!', and '.' are marked as sentence breaks. - sequences of two or more periods are marked as ellipsis. - any word ending in '.' that's a known abbreviation is marked as an abbreviation. - any other word ending in '.' is marked as a sentence break. Return these annotations as a tuple of three sets: - sentbreak_toks: The indices of all sentence breaks. - abbrev_toks: The indices of all abbreviations. - ellipsis_toks: The indices of all ellipsis marks. N(t_first_pass_annotation(R ttokenstaug_tok((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyt_annotate_first_passs cCs|i}||iijo t|_n|io t|_n}|ior|id oa|d i |i i jp*|d i i dd|i i jo t|_ qt|_ndS(sC Performs type-based annotation on a single token. s..it-N(ROR/R*RRLRhRNRQRSRRR=RRM(R RRO((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRs    *  ( R&R'R(RRIR;R1RRR(((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR~s    t PunktTrainercBs(eZdZeeeedZdZdZ eZ dZ dZ dZ eZeZdZeedZeed Zd Zd Zed Zd d d d dZdZdZdZdZdZedZedZdZ dZ!dZ"dZ#dZ$RS(s<Learns parameters used in Punkt sentence boundary detection.cCs}ti|d|d|t|_d|_t|_t|_d|_t|_ |o|i ||dtndS(NR0Ritfinalize( R~R1Rt _type_fdistt_num_period_tokst_collocation_fdistt_sent_starter_fdistt_sentbreak_countRt _finalizedttrain(R t train_texttverboseR0R((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR1s     cCs|ip|in|iS(sl Calculates and returns parameters for sentence boundary detection as derived from training.(Rtfinalize_trainingR(R ((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyt get_params)s g333333?igQ@iicCs5|i|i|||o|i|ndS(s8 Collects training data from a given text. If finalize is True, it will determine all the parameters for sentence boundary detection. If not, this will be delayed until get_params() or finalize_training() is called. If verbose is True, abbreviations found will be listed. N(t _train_tokensRR(R R2RR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR^s cs<ifd|D||oi|ndS(sE Collects training data from a given list of tokens. c3s"x|]}i|VqWdS(N(R(RuR(R (s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pys osN(RR(R RRR((R s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyt train_tokensks c Cs"t|_t|}x>|D]6}|ii|i|io|id7_qqW|i|}x|i |D]\}}}||i jo<|o1|i i i ||od||fGHqqqu|p1|i i i||od||fGHqququWt|i|}|i||i|i|7_xt|D]\}} |i p| oqSn|i|| o1|i i i |i|od|iGHqn|i| |o|ii| in|i|| o |ii|i| ifqSqSWdS(Nis Abbreviation: [%6.4f] %ss" Removed abbreviation: [%6.4f] %ss Rare Abbrev: %s(RRtlistRtincRPRQRt _unique_typest_reclassify_abbrev_typestABBREVRR=taddtremoveRt_get_orthography_dataRt_get_sentbreak_countR:t_is_rare_abbrev_typeR^t_is_potential_sent_starterRt_is_potential_collocationRR_( R RRRt unique_typesRMtscoretis_addtaug_tok1taug_tok2((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRssJ      cCstd|DS(Ncssx|]}|iVqWdS(N(RP(RuR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pys s(R<(R R((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRscCs|iixG|iD]9\}}|iii||od||fGHqqW|iixV|iD]H\\}}}|iii||f|od|||fGHqqqqWt|_ dS(s~ Uses data that has been gathered in training to determine likely collocations and sentence starters. s Sent Starter: [%6.4f] %rs Collocation: [%6.4f] %r+%rN( RRDt_find_sent_startersR?RRCt_find_collocationsR>RR(R RRFtllttyp1ttyp2((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRs    icCs|djob|ii}|iixF|iiD]1\}}||jo|||ii| count_removed). ii(RRVRR6(R tfdistt thresholdR{t num_removedROR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRs   cCsd}t|}x|D]}|io|djo d}n|io|djo d}n|i}ti||ifd}|o|ii||n|i o(|i p|i p d}qd}q|i p |i o d}qd}qWdS(s Collect information about whether each token type occurs with different case patterns (i) overall, (ii) at sentence-initial positions, and (iii) at sentence-internal positions. RRRiN(RRJRKR_t _ORTHO_MAPtgetReRRHRLRjRlRNRM(R RtcontextRRFRG((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRs&        c cs^xW|D]O}ti| p |djoqn|ido.||iijoqn|d }t}n!||iijoqnt}|idd}t||d}|i |d}|i |}|i |||i ||i i }t i| } |} t|ipt i|| } || | | } || |fVqWdS(s (Re)classifies each given token if - it is period-final and not a known abbreviation; or - it is not period-final and is otherwise a known abbreviation by checking whether its previous classification still holds according to the heuristics of section 3. Yields triples (abbr, score, is_add) where abbr is the type in question, score is its log-likelihood with penalties applied, and is_add specifies whether the present type is a candidate for inclusion or exclusion as an abbreviation, such that: - (is_add and score >= 0.3) suggests a new abbreviation; and - (not is_add and score < 0.3) suggests excluding an abbreviation. s ##number##RiiN(RoRpRSRR=RRRR]Rt_dunning_log_likelihoodRtNtmathtexpR@tIGNORE_ABBREV_PENALTYtpow( R ttypesRFRt num_periodstnum_nonperiodstcount_with_periodtcount_without_periodRtf_lengtht f_periodst f_penaltyR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR&s4    cCsn|iid|iD}xG|i|D]6\}}}||ijo|iii|q0q0WdS(s Recalculates abbreviations given type frequencies, despite no prior determination of abbreviations. This fails to include abbreviations otherwise found as "rare". css4x-|]&}|o|ido |VqqWdS(RN(RS(RuRF((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pys msN(RRBRRRR=R(R RRMRR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pytfind_abbrev_typesfs  cCs|ip |i otSn|i}|i||i|d }||iijp||ijotSn|id |i i jot SnG|i o<|i}|ii |}|t@o|t@ ot SqndS(s A word type is counted as a rare abbreviation if... - it's not already marked as an abbreviation - it occurs fewer than ABBREV_BACKOFF times - either it is followed by a sentence-internal punctuation mark, *or* it is followed by a lower-case word that sometimes appears with upper case, but never occurs with lower case at the beginning of sentences. iiN(RMRLRR_RRR=tABBREV_BACKOFFROR/R+RRcRAt _ORTHO_BEG_UCt _ORTHO_MID_UC(R tcur_toktnext_tokRFRRttyp2ortho_context((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRus  #    c Cst||}d}t|ti|||tid|}t|ti|||tid|}||}d|S(s A function that calculates the modified Dunning log-likelihood ratio scores for abbreviation candidates. The details of how this works is available in the paper. gGz?g?g(tfloatRtlog( tcount_atcount_btcount_abRtp1tp2t null_hypotalt_hypot likelihood((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRs c CsNddk}d||}d||}d||||}||i||||id|}|||i||||||id|} ||jo d} n-||i||||id|} ||jo d} n9|||i||||||id|} || | | } d| S(s< A function that will just compute log-likelihood estimate, in the original paper it's decribed in algorithm 6 and 7. This *should* be the original Dunning log-likelihood values, unlike the previous log_l function where it used modified Dunning log-likelihood values iNg?ig(RR( RRRRRRXRRtsummand1tsummand2tsummand3tsummand4R((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyt_col_log_likelihoods$ $    $cCsM|ip2|io |ip|io%|ip |io|io|iS(st Returns True if the pair of tokens may form a collocation given log-likelihood statistics. (tINCLUDE_ALL_COLLOCStINCLUDE_ABBREV_COLLOCSRMRLRjRlRq(R RR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRs    c csWxP|iiD]?\}}y|\}}Wntj o qnX||iijoqn|i||i|d}|i||i|d}|djo|djo|i|jot||jnot|i||||ii }||i jo?t |ii |t ||jo||f|fVqOqqWdS(sI Generates likely collocations and their log-likelihood. RiN( RRVt TypeErrorRR?RtMIN_COLLOC_FREQtminRRt COLLOCATIONR(R Rt col_countRRt typ1_countt typ2_countR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRs& $ cCs&|io|ip|i o|iS(s Returns True given a token and the token that preceds it if it seems clear that the token is beginning a sentence. (RLRjRlRn(R Rtprev_tok((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR s ccsx|iiD]\}}|pqn|i||i|d}||joqn|i|i|||ii}||ijo<t|ii|it||jo||fVqqWdS(s~ Uses collocation heuristics for each candidate token to determine if it frequently starts sentences. RN(RRVRRRRt SENT_STARTERR(R RFttyp_at_break_countt typ_countR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRs  cCstd|DS(sj Returns the number of sentence breaks marked in a given set of augmented tokens. css'x |]}|io dVqqWdS(iN(RL(RuR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pys 5s(tsum(R R((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR0s(%R&R'R(R6RRRIR1RRRRRRRRRRRRRRRRRRRRRt staticmethodRRRRRRR(((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRs>)    >      0 @  /*  tPunktSentenceTokenizercBseZdZeeeedZedZedZ edZ dZ dZ dZ dZd Zd Zd Zd Zed ZdZdZdZRS(s' A sentence tokenizer which uses an unsupervised algorithm to build a model for abbreviation words, collocations, and words that start sentences; and then uses that model to find sentence boundaries. This approach has been shown to work well for many European languages. cCs=ti|d|d||o|i|||_ndS(s train_text can either be the sole training text for this sentence boundary detector, or can be a PunktParameters object. R0RN(R~R1RR(R RRR0R((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR1EscCsOt|tdtdfjo|Snt|d|id|iiS(s Derives parameters from a given training text, or uses the parameters given. Repeated calls to this method destroy previous parameters. For incremental training, instantiate a separate PunktTrainer instance. RruR0R(RPRR/RR(R RR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRQs%cCst|i||S(sM Given a text, returns a list of the sentences in that text. (Rtsentences_from_text(R R2trealign_boundaries((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR3`scCs-|i|}|o|i|}n|S(s Given a text, generates the sentences in that text by only testing candidate sentence breaks. If realign_boundaries is True, includes in the sentence closing punctuation that follows the period. (t_sentences_from_textt_realign_boundaries(R R2Rtsents((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRfsccsd}x|iii|D]t}|i|id}|i|oE|||i!V|ido|id}q|i}qqW||VdS(Nit after_tokR(R/R%tfinditertgroupttext_contains_sentbreaktendtstart(R R2t last_breakRgR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRrsccsd}xt|D]\}}||}|p|o |Vqqn|iii|}|o(||idiV|i}qd}|o |VqqWdS(s\ Attempts to realign punctuation that falls after the period but should otherwise be included in the same sentence. For example: "(Sent1.) Sent2." will otherwise be split as:: ["(Sent1.", ") Sent1."]. This method will produce:: ["(Sent1.)", "Sent2."]. iN(R:R/R-RgRRR(R Rtrealignts1ts2tm((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRs    cCsPt}xC|i|i|D])}|otSn|io t}qqWtS(sK Returns True if the given text includes a sentence break. (Rt_annotate_tokensRRRL(R R2tfoundR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRs cCs(|i|i|}|i||S(s Given a text, generates the sentences in that text. Annotates all tokens, rather than just those with possible sentence breaks. Should produce the same results as L{sentences_from_text}. (RRt_build_sentence_list(R R2R((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pytsentences_from_text_legacysc#sytifd|D}g}x7|D]/}|i|i|io|Vg}q2q2W|o |VndS(sw Given a sequence of tokens, generates lists of tokens, each list corresponding to a sentence. c3s"x|]}i|VqWdS(N(R(RuR(R (s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pys sN(R4RtappendRORL(R RtsentenceR((R s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pytsentences_from_tokenss% cCs"|i|}|i|}|S(s Given a set of tokens augmented with markers for line-start and paragraph-start, returns an iterator through those tokens with full annotation including predicted sentence breaks. (Rt_annotate_second_pass(R R((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyRsc csOd}tid}d}x|D]}|i}|i||i}|t|7}|||t|!|joOdid|D} ti| i||} | o| i}qn|||t|!|jpt|t|7}|o|||7}n ||7}|io|Vd}q"q"W|o |VndS(s Given the original text and the list of augmented word tokens, construct and return a tokenized list of sentence strings. is\s*Rrcss"x|]}ti|VqWdS(N(Rtescape(Rutc((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pys sN( RRRORgRR]RvtAssertionErrorRL( R R2Rtpost WS_REGEXPR RROtwstpatR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR s,  %  cCsdGHtdd}xd|D]\}|io|idn)|io|idn|id|it|qW|idS(Nswriting to /tmp/punkt.new...s/tmp/punkt.newtws s t (topenRJtwriteRKtstrtclose(R RtoutR((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pytdumps   s;:,.!?ccs6x/t|D]!\}}|i|||Vq WdS(s Performs a token-based classification (section 4) over the given tokens, making use of the orthographic heuristic (4.1.1), collocation heuristic (4.1.2) and frequent sentence starter heuristic (4.1.3). N(R:t_second_pass_annotation(R Rtt1tt2((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR&s c Cs|pdSn|i}|ipdSn|i}|i}|i}|i}||f|iijot|_t |_ dSn|i p |i og| o_|i |}|t jot |_dSn|i o$||iijot |_dSqn|p |djo|i |}|tjot|_t |_ dSn|djoA|o:|i o0|ii|t@ ot|_t |_ dSqndS(s Performs token-based classification over a pair of contiguous tokens returning an updated augmented token for the first of them. Ns ##number##R(RORQR^R_RlRR>RRLRRMRNt_ortho_heuristicRaR?RAt _ORTHO_LC( R RRRORFRtnext_typttok_is_initialtis_sent_starter((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR0sH                    cCs|i|ijotSn|ii|i}|io|t@o|t@ ot Sn|i o|t @p |t @ otSndS(sR Decide whether the given token is the first token in a sentence. R( ROt PUNCTUATIONRRRAR_RaR"RRRct _ORTHO_UCt _ORTHO_BEG_LC(R RRA((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR!s      (R&R'R(R6RRRIR1RR3RRRRR R RR RttupleR&RRR!(((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyR=s$       7   OcCsid}|}t|_|i|||i}x(|i|dtD]}||GHqPWdS(s4Builds a punkt model and applies it to the same textcSs+tidtiid|iddS(s (?:\r|^\s+)Rrs R(RRR,R\treplace(R ((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pytsRN(RRRRR(R2ttok_clst train_clstcleanupttrainertsbdtl((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pytmains    t__main__(#R(RRt nltk.compatRtnltk.probabilityRtapiRRRt _ORTHO_UNK_UCR(t _ORTHO_MID_LCt _ORTHO_UNK_LCR'R"RtobjectRRRRoR.R:R;RIR~RRR2R&tsyststdintread(((s)/p/zhu/06/nlp/nltk/nltk/tokenize/punkt.pyssH         l  *[Cb