B >-Me0 @sdZdZddlZddlZddlmZmZddlTddlm Z dd e e Dd d d gZ [ ye Wne k rzeZ YnXd dZddZddZddZdZdZeedeeeZdZdZdZdZeddZeeeeeZdZedd eeZd!eZeeeZed"ed#Z ee eeZ!d$Z"d%Z#d&Z$d'Z%d(Z&ee&d)e&d*Z'ee&d+e&d,Z(ed-d.d/d0d1d2d3d4d5 Z)d6Z*ed7d8d9Z+ee)e*e+Z,ee!e,e(eZ-ee-Z.ee&d:ed;de&d<ed=dZ/edee'Z0eee0e!e,e/eZ1e2ej3e.e1e$e%f\Z4Z5Z6Z7ed>d?d@dAed>d?dBdCBdDdEdFdGdHdIhBZ8e3e"e3e#e6e7dJdKdLe8DdMdLe8DdNdLe8DZ9d)d*hdOdPe8DBdQdPe8DBZ:d;d=hdRdPe8DBdSdPe8DBZ;dTZGdWdXdXe=Z?dYdZZ@e@fd[d ZAd\d]ZBGd^d_d_ZCe3d`ejDZEe3daejDZFdbdcZGdddeZHdfd ZIdgd ZJeKdhkrddlLZLeMeLjNdkreAeOeLjNdjPn eAeLjQjPdS)izKa-Ping Yee z@GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip MontanaroN)BOM_UTF8lookup)*)tokencCsg|]}|ddkr|qS)r_).0xrr9/opt/alt/python37/lib/python3.7/lib2to3/pgen2/tokenize.py %sr tokenizegenerate_tokens untokenizecGsdd|dS)N(|))join)choicesrrr group0rcGs t|dS)Nr)r)rrrr any1rrcGs t|dS)N?)r)rrrr maybe2rrcstfddDS)Nc3s4|],}dD]}||kr||VqqdS))N)casefold)r r y)lrr 5sz _combinations..)set)rr)rr _combinations3sr z[ \f\t]*z #[^\r\n]*z\\\r?\nz\w+z0[bB]_?[01]+(?:_[01]+)*z(0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?z0[oO]?_?[0-7]+(?:_[0-7]+)*[lL]?z[1-9]\d*(?:_\d+)*[lL]?z0[lL]?z[eE][-+]?\d+(?:_\d+)*z\d+(?:_\d+)*\.(?:\d+(?:_\d+)*)?z\.\d+(?:_\d+)*z \d+(?:_\d+)*z\d+(?:_\d+)*[jJ]z[jJ]z[^'\\]*(?:\\.[^'\\]*)*'z[^"\\]*(?:\\.[^"\\]*)*"z%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''z%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""z'(?:[uUrRbBfF]|[rR][fFbB]|[fFbBuU][rR])?z'''z"""z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"z\*\*=?z>>=?z<<=?z<>z!=z//=?z->z[+\-*/%&@|^=<>]=?~z[][(){}]z\r?\nz:=z[:;.,`@]z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"rRfFbBuUurZuRZUrZUR)r"r#z'''z"""cCsi|]}t|dqS)z''') single3prog)r prefixrrr ysr/cCsi|]}t|dqS)z""") double3prog)r r.rrr r/zscCsi|] }d|qS)Nr)r r.rrr r/{scCsh|]}|dqS)z'''r)r r.rrr sr1cCsh|]}|dqS)z"""r)r r.rrr r1scCsh|]}|dqS)r"r)r r.rrr r1scCsh|]}|dqS)r#r)r r.rrr r1sc@s eZdZdS) TokenErrorN)__name__ __module__ __qualname__rrrr r3sr3c@s eZdZdS)StopTokenizingN)r4r5r6rrrr r7sr7c Cs4|\}}|\}}td||||t|t|fdS)Nz%d,%d-%d,%d: %s %s)printtok_namerepr) typerZxxx_todo_changemeZxxx_todo_changeme1lineZsrowZscolZerowZecolrrr printtokensr=cCs(yt||Wntk r"YnXdS)N) tokenize_loopr7)readline tokeneaterrrr r s cCsxt|D] }||q WdS)N)r)r?r@Z token_inforrr r>sr>c@s,eZdZddZddZddZddZd S) UntokenizercCsg|_d|_d|_dS)Nrr)tokensprev_rowprev_col)selfrrr __init__szUntokenizer.__init__cCs*|\}}||j}|r&|jd|dS)N )rDrBappend)rEstartrowcol col_offsetrrr add_whitespaces zUntokenizer.add_whitespacecCsxv|D]n}t|dkr$|||P|\}}}}}|||j||\|_|_|ttfkr|jd7_d|_qWd |jS)Nrrr) lencompatrMrBrHrCrDNEWLINENLr)rEiterablettok_typerrIendr<rrr rs        zUntokenizer.untokenizec Csd}g}|jj}|\}}|ttfkr,|d7}|ttfkr|t kr| qBn*|ttfkrd}n|r|r||dd}||qBWdS)NFrGTrN) rBrHNAMENUMBERrQrRASYNCAWAITINDENTDEDENTpop) rErrS startlineindents toks_appendtoknumtokvaltokrrr rPs0      zUntokenizer.compatN)r4r5r6rFrMrrPrrrr rAsrAz&^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)s^[ \t\f]*(?:[#\r\n]|$)cCsH|dddd}|dks*|dr.dS|dks@|drDdS|S) N r-zutf-8zutf-8-)zlatin-1z iso-8859-1z iso-latin-1)zlatin-1-z iso-8859-1-z iso-latin-1-z iso-8859-1)lowerreplace startswith)orig_encencrrr _get_normal_names rlcsdd}d}fdd}fdd}|}|trHd|dd}d }|sT|gfS||}|rj||gfSt|s~||gfS|}|s||gfS||}|r|||gfS|||gfS) NFzutf-8cs"yStk rtSXdS)N) StopIterationbytesr)r?rr read_or_stopsz%detect_encoding..read_or_stopcsy|d}Wntk r"dSXt|}|s6dSt|d}y t|}Wn tk rptd|YnXr|j dkrtd|d7}|S)Nasciirzunknown encoding: zutf-8zencoding problem: utf-8z-sig) decodeUnicodeDecodeError cookie_rematchrlrr LookupError SyntaxErrorname)r< line_stringrtencodingcodec) bom_foundrr find_cookie s"   z$detect_encoding..find_cookieTz utf-8-sig)rirblank_rert)r?rydefaultror|firstsecondr)r{r?r detect_encodings0         rcCst}||S)N)rAr)rSutrrr r:sccsd}}}d\}}d}dg}d}d} d} d} xy |} Wntk rTd} YnX|d}dt| } }|r6| std||| }|r|d} }t|| d||||f|| fVd\}}d}nd|r | dddkr | d dd kr t|| ||t| f|fVd}d}q2n|| }|| }q2nD|dkrb|sb| sNPd}xf| |kr| | d krv|d}n6| | d kr|tdt}n| | d krd}nP| d} qTW| |krP|r|Vd}| | dkr| | dkrV| | dd}| t|}t ||| f|| t|f| fVt | |d||f|t| f| fVq2t t f| | dk| | d|| f|t| f| fVq2||dkr| |t | d| |df|| f| fVxt||dkr8||krt dd|| | f|dd}| r| |dkrd} d} d} td|| f|| f| fVqW| rz| rz| |dkrzd} d} d} n| svtd|dfd}x| |kr&t| | }|r|d\}}||f||f|}}} | ||| |}}|tjks|dkr |dkr t|||| fVq"|dkrVt}|dkr(t }n | r2d} |rB|Vd}||||| fVq"|dkr|rp|Vd}t |||| fVq"|tkrt|}|| | }|r|d} | || }|r|Vd}t|||| f| fVn||f}| |d}| }Pq"|tks0|ddtks0|ddtkr|ddkr||f}t|pht|dpht|d}| |dd}}| }Pn |r|Vd}t|||| fVq"|r~|dkr| r|dkrtnt|||| fVq~t|||| f}|dkr |s |}q~|dkrf|rf|dtkrf|ddkrfd} |d} t|d|d|d|dfVd}|rv|Vd}|Vnz|dkr|r|Vd}t |||| f| fVd}nF|dkr|d}n|d kr|d}|r|Vd}t|||| fVn(t| | || f|| df| fV| d} q~Wq2W|r<|Vd}x.|ddD]}td|df|dfdfVqJWtd|df|dfdfVdS)!Nr)rrFrrzEOF in multi-line stringz\ z\ rG  z# #z rWz3unindent does not match any outer indentation levelz zEOF in multi-line statement.TrNr} )asyncawaitrdef\z([{z)]})rmrOr3rtrVSTRING ERRORTOKENtabsizerstripCOMMENTrRrHr\IndentationErrorr] pseudoprogspanstringdigitsrYrQ triple_quotedendprogs single_quoted isidentifierrZr[rXOP ENDMARKER)r?lnumparenlev continuedcontstrneedcontcontliner`ZstashedZ async_defZasync_def_indentZ async_def_nlr<posmaxstrstartendprogendmatchrVcolumn comment_tokenZnl_pos pseudomatchrIsposeposrinitialnewlinerdindentrrr rOsn     *                             __main__)R __author__ __credits__rrecodecsrrZlib2to3.pgen2.tokenrrdir__all__rn NameErrorstrrrrr WhitespaceCommentIgnoreName Binnumber Hexnumber Octnumber Decnumber IntnumberExponent PointfloatExpfloat Floatnumber ImagnumberNumberSingleDoubleSingle3Double3Z _litprefixTripleStringOperatorBracketSpecialFunny PlainTokenTokenContStr PseudoExtras PseudoTokenmapcompileZ tokenprogrr-r0Z _strprefixesrrrr Exceptionr3r7r=r r>rAASCIIrsr~rlrrrr4sysrOargvopenr?stdinrrrr s               8 Ib