³ò RÔjKc@sÝddkZddkZddkZddklZedƒ\ZZedƒ\ZZ edƒZ defd„ƒYZ de fd„ƒYZ d e fd „ƒYZd d d „Zed„Zedjo eƒndS(iÿÿÿÿN(t TokenizerIiitTextTilingTokenizerc BsƒeZdZddeeeddeed„ Zd„Z d„Z d„Z d „Z d „Z d „Zd „Zd „Zd„ZRS(sjA section tokenizer based on the TextTiling algorithm. The algorithm detects subtopic shifts based on the analysis of lexical co-occurence patterns. The process starts by tokenizing the text into pseudosentences of a fixed size w. Then, depending on the method used, similarity scores are assigned at sentence gaps. The algorithm proceeds by detecting the peak differences between these scores and marking them as boundaries. The boundaries are normalized to the closest paragraph break and the segmented text is returned. @type w: number @param w: Pseudosentence size @type k: number @param k: Size(in sentences) of the block used in the block comparison method @type similarity_method: constant @param similarity_method: The method used for determining similarity scores @type stopwords: list @param stopwords: A list of stopwords that are filtered out @type smoothing_method: constant @param smoothing_method: The method used for smoothing the score plot @type smoothing_width: number @param smoothing_width: The width of the window used by the smoothing method @type smoothing_rounds: number @param smoothing_rounds: The number of smoothing passes @type cutoff_policy: constant @param cutoff_policy: The policy used to determine the number of boundaries ii iic CsQ|djo#ddkl}|idƒ}n|iitƒƒ|id=dS(Niÿÿÿÿ(t stopwordstenglishtself(tNonet nltk.corpusRtwordst__dict__tupdatetlocals( Rtwtktsimilarity_methodRtsmoothing_methodtsmoothing_widthtsmoothing_roundst cutoff_policyt demo_mode((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyt__init__4s c sÐ|iƒ}ˆi|ƒ}t|ƒ}dig}|D]$}tid|ƒo ||q8q8~ƒ}ˆi|ƒ}ˆi|ƒ} x,| D]$} t‡fd†| iƒ| _qWˆi | |ƒ} ˆi t joˆi | | ƒ} n!ˆi t jotdƒ‚nˆitjoˆi| ƒ} nˆi| ƒ}ˆi|ƒ}ˆi|||ƒ}g}d}x<|D]4}|djoqvn|i|||!ƒ|}qvWˆio| | ||fSn|S(s0The main function. Follows a pipeline structure.ts [a-z\-' ]cs|dˆijS(i(R(twi(R(s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyt_ss'Vocabulary introduction not implementedi(tlowert_mark_paragraph_breakstlentjointretmatcht_divide_to_tokensequencestfiltert wrdindex_listt_create_token_tableR tBLOCK_COMPARISONt_block_comparisontVOCABULARY_INTRODUCTIONtNotImplementedErrorRtDEFAULT_SMOOTHINGt_smooth_scorest _depth_scorest_identify_boundariest_normalize_boundariestappendR(Rttexttlowercase_texttparagraph_breakst text_lengtht_[1]tct nopunct_texttnopunct_par_breaksttokseqsttst token_tablet gap_scorest smooth_scorest depth_scorestsegment_boundariestnormalized_boundariestsegmented_texttprevbtb((Rs./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyttokenizeFsD  *      cs³‡fd†}g}t|ƒd}x‡t|ƒD]y}d\}}} d} ||idjo|d} n,|||ijo||} n |i} g} ||| d|d!D]} | | iq¸~ }g}||d|| d!D]} || iqï~}xbˆiƒD]T}||||ƒ|||ƒ7}||||ƒd7}| |||ƒd7} qWy|ti|| ƒ} Wntj onX|i| ƒq2W|S(s&Implements the block comparison methodcsNt‡fd†ˆ|iƒ}tg}|D]}||dq-~ƒ}|S(Ncs|dˆjS(i((to(tblock(s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyR‡si(Rt ts_occurencestsum(ttokR@tts_occsR/ttsocctfreq(R5(R@s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pytblk_frq†s+igi(ggg( RtrangeR tindextkeystmathtsqrttZeroDivisionErrorR*(RR3R5RGR6tnumgapstcurr_gaptscore_dividendtscore_divisor_b1tscore_divisor_b2tscoret window_sizeR/R4tb1t_[2]tb2tt((R5s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyR"„s8  00   cCs'ttti|ƒd|idƒƒS(s1Wraps the smooth function from the SciPy Cookbookt window_leni(tlisttsmoothtnumpytarrayR(RR6((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyR&­scCs…d}tidƒ}|i|ƒ}d}dg}xK|D]C}|iƒ||joq:q:|i|iƒƒ|iƒ}q:W|S(sNIdentifies indented text or line breaks as the beginning of paragraphsids[ ]* [ ]* [ ]*i(RtcompiletfinditertstartR*(RR+t MIN_PARAGRAPHtpatterntmatchest last_breaktpbreakstpb((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyR²s c Csš|i}g}tid|ƒ}x-|D]%}|i|iƒ|iƒfƒq(Wg}tdt|ƒ|ƒD]%}|t||||||!ƒqn~S(s3Divides the text into pseudosentences of fixed sizes\w+i( R RR_R*tgroupR`RHRt TokenSequence(RR+R RRcRR/ti((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyRÅs #c Csáh}d}d}|iƒ}|iƒ}|djo6y|iƒ}Wqmtj otdƒ‚qmXnxm|D]e}xR|iD]G\} } y/x(| |jo|iƒ}|d7}q–WWntj onX| |iƒjo©|| id7_|| i|jo$||| _|| id7_n|| i |jo+||| _ || i i |dgƒqË|| i ddcd7R"R&RRR R(R'R)(((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyRs& > )   3  !RscBs#eZdZddded„ZRS(s[A field in the token table holding parameters for each token, used later in the processiicCs!|iitƒƒ|id=dS(NR(RR R (RRjRARkRlRmRn((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyR`s(RœRRžRR(((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyRs]s RhcBseZdZed„ZRS(s3A token list with its original length and its indexcCs4|p t|ƒ}|iitƒƒ|id=dS(NR(RRR R (RRIRtoriginal_length((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyRls(RœRRžRR(((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyRhjsi tflatcCs|idjo td‚n|i|jo td‚n|djo|Sn|djo td ‚ntid |d ||dd …|d |d |d | d …f}|djoti|dƒ}ntd|dƒ}ti||iƒ|ddƒ}||d| d!S(sÌsmooth the data using a window with requested size. This method is based on the convolution of a scaled window with the signal. The signal is prepared by introducing reflected copies of the signal (with the window size) in both ends so that transient parts are minimized in the begining and end part of the output signal. input: x: the input signal window_len: the dimension of the smoothing window; should be an odd integer window: the type of window from 'flat', 'hanning', 'hamming', 'bartlett', 'blackman' flat window will produce a moving average smoothing. output: the smoothed signal example: t=linspace(-2,2,0.1) x=sin(t)+randn(len(t))*0.1 y=smooth(x) see also: numpy.hanning, numpy.hamming, numpy.bartlett, numpy.blackman, numpy.convolve scipy.signal.lfilter TODO: the window parameter could be the window itself if an array instead of a string is'smooth only accepts 1 dimension arrays.s1Input vector needs to be bigger than window size.iR¡thanningthammingtbartletttblackmansDWindow is on of 'flat', 'hanning', 'hamming', 'bartlett', 'blackman'iiiÿÿÿÿtdsnumpy.s (window_len)tmodetsame(sflatshanningshammingsbartlettsblackman( tndimRrtsizeR\tr_tonestevaltconvolveRB(R{RYtwindowtsR ty((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pyR[ws     I "cCsddkl}ddk}tdtƒ}|djo|iƒd }n|i|ƒ\}}}}|idƒ|i dƒ|i t t |ƒƒ|ddƒ|i t t |ƒƒ|ddƒ|i t t |ƒƒ|dd ƒ|i t t |ƒƒ|ƒ|iƒ|iƒdS( Niÿÿÿÿ(tbrownRi'sSentence Gap indexs Gap ScorestlabelsSmoothed Gap scoress Depth scores(RR²tpylabRR’RtrawR>txlabeltylabeltplotRHRtstemtlegendtshow(R+R²R´tttR°tssR¦R=((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pytdemo²s !  """ t__main__(RRKR\tapiRRHR!R#RRŸR%RtobjectRsRhR[RR¾Rœ(((s./p/zhu/06/nlp/nltk/nltk/tokenize/texttiling.pys s    ÿJ  ;