o hX@s:ddlmZddlZddlmZddlmZddlmZm Z m Z m Z ddl m Z mZmZmZddlmZdd lmZmZdd lmZmZmZmZmZmZmZed Ze Z!e!"e#d         d2d3d$d%Z$        d2d4d(d)Z%        d2d5d,d-Z&        d6d7d0d1Z'dS)8) annotationsN)PathLike)BinaryIO)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE) mess_ratio) CharsetMatchCharsetMatches)any_specified_encodingcut_sequence_chunks iana_nameidentify_sig_or_bom is_cp_similaris_multi_byte_encodingshould_strip_sig_or_bomcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s皙?TF皙? sequencesbytes | bytearraystepsint chunk_size thresholdfloat cp_isolationlist[str] | None cp_exclusionpreemptive_behaviourboolexplainlanguage_thresholdenable_fallbackreturnrc 2 Cs t|ttfstdt||rtj} tt t t t |} | dkrGt d|r;tt t | p9tjtt|dddgdgS|dur]tt d d |d d |D}ng}|durutt d d |dd |D}ng}| ||krtt d||| d}| }|dkr| ||krt| |}t |tk} t |tk} | rtt d| n | rtt d| g}|rt|nd}|dur||tt d|t}g}g}d}d}d}t}t}t|\}}|dur||tt dt |||dd|vr|d|tD]7}|r$||vr$q|r.||vr.q||vr5q||d}||k}|oFt|}|dvrX|sXtt d|q|dvri|sitt d|qzt|}Wnt t!fytt d|Yqwz9| r|durt"|dur|dtdn |t |td|dnt"|dur|n|t |d|d}Wn+t#t$fy}zt|t$stt d|t"|||WYd}~qd}~wwd} |D] }!t%||!rd} nq| rtt d||!qt&|sdnt || t| |}"|o&|duo&t || k}#|#r1tt d |tt |"d!}$t'|$d"}$d}%d}&g}'g}(zLt(|||"|||||| D]=})|'|)|(t)|)||duordt |kopd"kn|(d#|kr|%d7}%|%|$ks|r|durnqSWn!t#y}ztt d$|t"||$}%d}&WYd}~nd}~ww|&s| r|sz|td%dj*|d&d'Wn#t#y}ztt d(|t"|||WYd}~qd}~ww|(rt+|(t |(nd}*|*|ks|%|$krJ||tt d)||%t,|*d*d+d,| rH|dd|d-d.fvrH|&sHt||||g||d/}+||kr>|+}n |dkrF|+}n|+}qtt d0|t,|*d*d+d,|s`t-|},nt.|},|,rstt d1|t"|,g}-|dkr|'D]})t/|)||,rd2|,nd}.|-|.q|t0|-}/|/rtt d3|/|t|||*||/| dus||ddfvr|nd|d/}0||0||ddfvr|*d4kr|*dkrt d5|0j1|rtt t | t|0gS||0t |r-|dus||vr-d|vr-d|vr-|2}1t d5|1j1|r&tt t | t|1gS||krNt d6||rEtt t | t||gSqt |dkr|s`|s`|rftt d7|rvt d8|j1||n2|r~|dus|r|r|j3|j3ks|durt d9||n |rt d:|||rt d;|2j1t |dnt d<|rtt t | |S)=af Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain' toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging. Custom logging format and handler can be set manually. z3Expected object of type bytes or bytearray, got: {}rz[zfrom_bytes..zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.cSr/r0r1r2r5r5r6r7fr8z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.rz>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_16utf_32z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.>utf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.z2Encoding %s does not provide an IncrementalDecodergA)encodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sgj@strict)errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigitsr:r;)preemptive_declarationz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {},z We detected language {} using {}rz.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4 isinstance bytearraybytes TypeErrorformattypeloggerlevel addHandlerexplain_handlersetLevelr lendebug removeHandlerloggingWARNINGrrlogjoinr r r rappendsetrr addrrModuleNotFoundError ImportErrorstrUnicodeDecodeError LookupErrorrrangemaxrrdecodesumroundrrrr r=best fingerprint)2rrr!r"r$r&r'r)r*r+previous_logger_levellengthis_too_small_sequenceis_too_large_sequenceprioritized_encodingsspecified_encodingtestedtested_but_hard_failuretested_but_soft_failurefallback_ascii fallback_u8fallback_specifiedresultsearly_stop_results sig_encoding sig_payload encoding_ianadecoded_payloadbom_or_sig_availablestrip_sig_or_bomis_multi_byte_decoderesimilar_soft_failure_testencoding_soft_failedr_multi_byte_bonusmax_chunk_gave_upearly_stop_countlazy_str_hard_failure md_chunks md_ratioschunkmean_mess_ratiofallback_entrytarget_languages cd_ratioschunk_languagescd_ratios_merged current_matchprobable_resultr5r5r6 from_bytes!s$                            &                                      rfprc Cst|||||||||| S)z Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )rread) rrr!r"r$r&r'r)r*r+r5r5r6from_fp!srpathstr | bytes | PathLikec CsHt|d} t| ||||||||| WdS1swYdS)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openr) rrr!r"r$r&r'r)r*r+rr5r5r6 from_path?s $rfp_or_path_or_payload!PathLike | str | BinaryIO | bytesc Cst|ttfrt|||||||||| d } | St|ttfr0t|||||||||| d } | St|||||||||| d } | S)a) Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string. Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match are disabled to be stricter around ASCII-compatible but unlikely to be a string. ) rr!r"r$r&r'r)r*r+)rHr_rrrJrIrr) rrr!r"r$r&r'r)r*r+guessesr5r5r6 is_binary^s\- r) rrrNNTFrT)rrrr r!r r"r#r$r%r&r%r'r(r)r(r*r#r+r(r,r)rrrr r!r r"r#r$r%r&r%r'r(r)r(r*r#r+r(r,r)rrrr r!r r"r#r$r%r&r%r'r(r)r(r*r#r+r(r,r) rrrNNTFrF)rrrr r!r r"r#r$r%r&r%r'r(r)r(r*r#r+r(r,r()( __future__rrVosrtypingrcdrrrr constantr r r r mdrmodelsrrutilsrrrrrrr getLoggerrN StreamHandlerrQ setFormatter Formatterrrrrr5r5r5r6st    $     !