o g@sddlmZddlmZmZddlmZddlZddlmZddl Z ddl Z ddl m Z m Z mZddlZddlZddlmmZddlmZddlmZmZdd lmZdd lmZdd lmZm Z d Z!Gd ddeZ"Gdddej Z#Gddde"Z$dddZ%dddZ&dS)) annotations)abc defaultdict)copyN)StringIO) DefaultDictIteratorcast)FilePathOrBuffer)EmptyDataError ParserError) is_integer) is_dict_like) ParserBaseparser_defaultsuc@seZdZd9ddZddZd:dd Zd d Zd:d d ZddZddZ d;ddZ ddZ ddZ ddZ d d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2Zd3d4Zd5d6Zd:d7d8ZdS)< PythonParserfFilePathOrBuffer | listc s*t|d_g_d_d_|d_tjr!j_nfdd_t |d_ |d_ |d_ t j trEtj _ |d _|d _|d _|d _|d _|d_|dphd_d_d|vrv|d_|d_|d_t|d_|d_|d_|d_t |trtt t|_n/!||j"dusJt#j"j$dsJz %j"j$Wnt&j't(fy)wd_*z +\_,_-_.Wnt/t0fy)wt1j,dkr2j,j3j4\_,_3_4}t1j,_-nj,d_,tj,_5j6s87j,\}_5_,d_8j3dur8|_3j*durHtt9t1j,_*:j,d}j;r\<j*j,}|_=t1jdkrkt0dt>?j}jdur~d|d}nt>?j}d|d|d}t>@|_AdS) zN Workhorse function for processing nested list into DataFrame Nrskiprowscs |jvSN)r)xself]/home/ubuntu/cloudmapper/venv/lib/python3.10/site-packages/pandas/io/parsers/python_parser.py<s z'PythonParser.__init__.. skipfooter delimiter quotechar escapechar doublequoteskipinitialspacelineterminatorquotingskip_blank_linesnamesFhas_index_namesverbose convertersdtype thousandsdecimalcommentreadlineTz'Only length-1 decimal markers supportedz^[\-\+]?[0-9]*(z [0-9]*)?([0-9]?(E|e)\-?[0-9]+)?$z^[\-\+]?([0-9]+z |[0-9])*()Br__init__databufposline_posrcallableskipfunc_validate_skipfooter_argrrr isinstancestrrr r!r"r#r$ names_passedr&r'r(rr)r*r+r,listr r _open_handleshandleshasattrhandle _make_readercsvErrorUnicodeDecodeErrorclose _col_indices_infer_columnscolumnsnum_original_columns unnamed_cols TypeError ValueErrorlen_extract_multi_indexer_columns index_names col_names orig_names_has_complex_date_col_get_index_name_name_processedrange_validate_parse_dates_presence parse_dates_set_noconvert_dtype_columns_no_thousands_columnsreescapecompilenum) rrkwds_rMno_thousands_columnsr+regexr*rrrr/,s                               zPythonParser.__init__c s@jdus tdkrjrtdGfdddtj}|}dur)|_n_}|ggd}j s>|sZj d7_ }|ggd}j s>|r>|d}j d7_ j d7_ t |}|j|_tj t||d}jt|tj |dd}n fd d } | }|_dS) Nr.z.MyDialect N) __name__ __module__ __qualname__rrrr r!r#r"rrrr MyDialectsrdr)dialectT)restrictc3sB}t}||VD] }||VqdSr)r-rXrZsplitstrip)linepat)rseprr_reads z(PythonParser._make_reader.._read)rrKr"rJr@Dialectr-_check_commentsr5r2r3Sniffersniffreaderrr1extendr:r0) rrrddiarilinessniffedline_rdrrqrlr)rrrkrr?s8  zPythonParser._make_readerNc Csz||}Wnty|jrg}n|Ynwd|_t|j}t|sH||j}|||j |j |j \}}}| ||j }|||fSt|d}d}|jrc|t|krc|d}|dd}||} || \} }||| \}} || } || | ||\}}||| fS)NFrr.) _get_lines StopIteration _first_chunkrCr:rOrK_maybe_dedup_names_get_empty_meta index_colrMr)_maybe_make_multi_index_columnsrNcount_empty_valsr& _rows_to_cols_exclude_implicit_index_do_date_conversions _convert_data _make_index) rrowscontentrFr%indexcol_dictcount_empty_content_vals indexnamerowalldatar0rrrreads@          zPythonParser.readcsR||j}d|jrt|jt||fddt|D|fS)Nrcs&i|]\}}|kr||qSrr).0inamer len_alldataoffsetrr +s z8PythonParser._exclude_implicit_index..)rzrO_implicit_indexrKr|_check_data_length enumerate)rrr%rrrr s   z$PythonParser._exclude_implicit_indexcCs|dur|j}|j|dS)N)r) chunksizer)rsizerrr get_chunk0s zPythonParser.get_chunkc sfdd}|j}tjtsj}n|j}i}i}tjtrLjD]#}j|}j|} t|trB|jvrBj|}|||<| ||<q'nj}j}|||j ||S)Ncs@i}|D]\}}t|tr|jvrj|}|||<q|S)zconverts col numbers to names)itemsr7intrO)mappingcleancolvrrr_clean_mapping8s   z2PythonParser._convert_data.._clean_mapping) r(r7r)dict na_values na_fvaluesrrO_convert_to_ndarraysr') rr0r clean_conv clean_dtypesclean_na_valuesclean_na_fvaluesrna_value na_fvaluerrrr6s4          zPythonParser._convert_datac sDjd}d}t}jdurj}t|tttjfr0t|dk}|r/t||ddg}nd}|g}g}t |D]n\}}z } j |krS } j |ksJWnZt y} zNj |krptd|dj dd| |r|dkr|r||dgt|d|||fWYd} ~ Sjstd | jdd} WYd} ~ nd} ~ wwgg} t | D](\} } | d kr|rd | d |}nd | }| | |q| q|sOjrOtt}t D]^\} }|}||}|dkrC|dkr|d||<|d |}||}|dksjdurCtjrCj|durCj|durCj|j|i|| <|d||<qn?|r||dkrt}j}|durit|nd}t| }||kr{|||ks|dkrd}dg|jdg_||fdd| Dt|dkrt}q;|rdurtt|dkrtdt|dkrtdjdur߈||nt}jdurttjkrfddtjDg}ng}n||d|}nz } Wn!t y6} zs&td | dd} WYd} ~ nd} ~ wwt| }|}sbj rQfddt!|Dg}ntt!|g}||d|}n;jdusot|kr|g|}t}n!t"jsttjkrtdg|g}|}|||fS)NrTr.FzPassed header=z but only z lines in filezNo columns to parse from filez Unnamed: _level_.csh|]}|qSrrrr) this_columnsrr z.PythonParser._infer_columns..zHNumber of passed names did not match number of header fields in the filez*Cannot pass names with multi-index columnscsg|]}|qSrrr)r%rr rz/PythonParser._infer_columns..csg|] }j|qSr)prefixrrrrrs)#r%setheaderr7r:tuplenpndarrayrKr_buffered_liner3 _next_linerxrJ _clear_bufferappendr mangle_dupe_colsrrr)rgetupdater|r1rIusecols_handle_usecolsrDsortedrrSr4)rrG clear_bufferrHrhave_mi_columnsrFlevelhrrierrthis_unnamed_colsrccol_namecountsrold_col cur_countlcsicic unnamed_countncolsr)r%rrrrEcs               "           zPythonParser._infer_columnsrFlist[list[str | int | None]] usecols_keylist[str | int | None]rGrc s|jduryt|jr||j|nYtdd|jDrTt|dkr&tdg|jD]'}t|trMz | |Wq+tyL| |j|Yq+w|q+nfdd|jD}|rht j dt d d |jfd d|D}t|_|S) zb Sets self._col_indices usecols_key is used if there are string usecols. Ncss|]}t|tVqdSr)r7r8)rurrr z/PythonParser._handle_usecols..r.z4If using multiple headers, usecols must be integers.csg|]}|kr|qSrr)rr)rGrrr.sz0PythonParser._handle_usecols..zkDefining usecols with out of bounds indices is deprecated and will raise a ParserError in a future version.) stacklevelcs"g|] }fddt|DqS)cg|] \}}|vr|qSrr)rrn col_indicesrrr;z;PythonParser._handle_usecols...)r)rcolumnrrrr:s)rr4_evaluate_usecolsanyrKrJr7r8rr_validate_usecols_nameswarningswarn FutureWarningrrD)rrFrrGrmissing_usecolsr)rrGrrsB         zPythonParser._handle_usecolscCs t|jdkr |jdS|S)zH Return a line from buffer, filling buffer if required. r)rKr1rrrrrrAs zPythonParser._buffered_linecCs|s|St|dts |S|ds|S|dd}|tkr|S|d}t|dkrZ|d|jkrZd}|d}|dd|d}|||}t||dkrY|||dd7}n|dd}|g|ddS)a- Checks whether the file begins with the BOM character. If it does, remove it. In addition, if there is quoting in the field subsequent to the BOM, remove it as well because it technically takes place at the beginning of the name, not the middle of it. rr.N)r7r8_BOMrKrr)r first_row first_elt first_row_bomstartquoteendnew_rowrrr_check_for_bomJs(    zPythonParser._check_for_bomcCs| p tdd|DS)z Check if a line is empty or not. Parameters ---------- line : str, array-like The line of data to check. Returns ------- boolean : Whether or not the line is empty. css|]}| VqdSrrrrrrrrsz.PythonParser._is_line_empty..)all)rrirrr_is_line_empty}s zPythonParser._is_line_emptycCst|jtr^||jr|jd7_||js z8||j|jgd}|jd7_|js@||j|jds>|r@Wn|jrQ||g}|rQ|d}Wn Wn t y[t wqnV||jr}|jd7_|jdusrJt |j||jsd |j |jdd}|jd7_|dur||gd}|jr||g}|r|d}n n||s|rnq~|jdkr| |}|jd7_|j||S)Nr.Trrow_num)r7r0r:r5r2rnr$r_remove_empty_lines IndexErrorrxnext_next_iter_linerr3r1r)rriret orig_linerrrrsb            zPythonParser._next_linecCsL|j|jjkr t||j|jjkr$d|d}tj||ddSdS)a Alert a user about a malformed row, depending on value of `self.on_bad_lines` enum. If `self.on_bad_lines` is ERROR, the alert will be `ParserError`. If `self.on_bad_lines` is WARN, the alert will be printed out. Parameters ---------- msg : The error message to display. row_num : The row number where the parsing error occurred. Because this row number is displayed, we 1-index, even though we 0-index internally. zSkipping line z: r`N) on_bad_linesBadLineHandleMethodERRORr WARNsysstderrwrite)rmsgrbaserrr_alert_malformeds  zPythonParser._alert_malformedc Csz |jdusJt|jWStjyP}z6|j|jjks$|j|jjkrEt|}d|vs0d|vr2d}|j dkr?d}|d|7}| ||WYd}~dSd}~ww)aL Wrapper around iterating through `self.data` (CSV source). When a CSV error is raised, we check for specific error messages that allow us to customize the error message displayed to the user. Parameters ---------- row_num : The row number of the line being parsed. Nz NULL bytezline contains NULzNULL byte detected. This byte cannot be processed in Python's native csv library at the moment, so please pass in engine='c' insteadrzError could possibly be due to parsing errors in the skipped footer rows (the skipfooter keyword is only applied after Python's csv library has parsed all rows).. ) r0rr@rArrrrr8rr)rrerreasonrrrrs$     zPythonParser._next_iter_linecCs|jdur|Sg}|D]8}g}|D],}t|tr"|j|vs"||jvr(||q|d||j}t|dkr=||||q |S)Nr)r,r7r8rrfindrK)rrtrrirlrrrrrns"       zPythonParser._check_commentscCsNg}|D] }t|dkst|dkr$t|dtr|dr$||q|S)a} Iterate through the lines and remove any that are either empty or contain only one whitespace value Parameters ---------- lines : array-like The array of lines that we are to filter. Returns ------- filtered_lines : array-like The same array of lines with the "empty" ones removed. r.r)rKr7r8rhr)rrtrrirrrrs     z PythonParser._remove_empty_linescCs |jdur|S|j||jddS)Nrrtsearchreplace)r*_search_replace_num_columnsrrtrrr_check_thousands2s zPythonParser._check_thousandsc Csg}|D]:}g}t|D],\}}t|tr)||vs)|jr!||jvs)|j|s/||q ||||q ||q|Sr) rr7r8rWr[r rhrr) rrtr rrrir rrrrrr:s    z(PythonParser._search_replace_num_columnscCs$|jtdkr |S|j||jddS)Nr+rr )r+rrrrrr_check_decimalKs zPythonParser._check_decimalcCs g|_dSr)r1rrrrrSs zPythonParser._clear_bufferFc CsFt|}t|}z|}Wn tyd}Ynwz|}Wn ty+d}Ynwd}|dur{|j}|dur@t||j}|dur{t|t||jkr{ttt||_|jdd|_t|D]}| d|qdt|}t||_|||fS|dkrd|_ |jdurtt||_d}n | ||j|j \}} |_|||fS)a Try several cases to get lines: 0) There are headers on row 0 and row 1 and their total summed lengths equals the length of the next line. Treat row 0 as columns and row 1 as indices 1) Look for implicit index: there are more columns on row 1 than row 0. If this is true, assume that row 1 lists index columns and row 0 lists normal columns. 2) Get index from the columns if it was listed. NrFr.T) r:rrxr|rKrGrSr1reversedinsertr_clean_index_namesrH) rrFrOri next_lineimplicit_first_colsr|r index_namecolumns_rrrrQXsF           zPythonParser._get_index_namecsj}jr |tj7}tdd|D}||krjdurjdurjr*jnd}g}t|}t|}g}|D]9\}} t| } | |krnjj j ksTjj j krmj |||} | | | fjj j krmnq:| | q:|D]1\} } d|d| dd| } jrtjdkrjtjkrd } | d | 7} | | dqvttj||d j}jrۈjdusJjjrЇfd d t|D}|Sfdd t|D}|S)Ncss|]}t|VqdSr)rKrrowrrrrsz-PythonParser._rows_to_cols..Frz Expected z fields in line r.z, saw zXError could possibly be due to quotes being ignored when a multi-char delimiter is used.r) min_widthcs4g|]\}}|tjks|tjvr|qSr)rKr|rrarrrrrs z.PythonParser._rows_to_cols..crrrrrrrrs)rGrrKr|maxrrrrrrrr2rrr#r@ QUOTE_NONErr:libto_object_arrayTrD)rrcol_lenmax_lenfooters bad_lines iter_content content_lenrl actual_lenrrr zipped_contentrrrrsb         zPythonParser._rows_to_colscsj}d}|dur(tj|kr!jd|j|d}_n|tj8}|durtjtrujtjkr|durt |D]}jdusJ| t jq| |nd} j j|dd}|d7}|dur| |qWn#tyjrʇfddt|D}| |t|dkrւYnwjt|7_g_n|}jr|dj }|}jr|}|}|S)Nc$g|]\}}|js|qSrr5r2rrrrrrr z+PythonParser._get_lines..rTr.rcr.rr/r0rrrr'r1)r1rKr7r0r:r2rxrrrrrSrrrrrnr$rrr)rrrtnew_rowsnew_posr]rrrrrwsp"                 zPythonParser._get_lines)rrr)rFrrrrGr)rarbrcr/r?rrrrrErrrrrrrrnrrrrrrrQrrwrrrrr+s6  B, - .1 36,BSrc@s6eZdZdZd ddZd ddZddd Zd d ZdS)FixedWidthReaderz( A reader of fixed-width lines. NdcCs||_d|_|r d|nd|_||_|dkr|j||d|_n||_t|jttfs4t dt |j |jD]-}t|ttfr`t |dkr`t|dt tjt dfr`t|dt tjt dfsdt d q7dS) Nz z infer) infer_nrowsrz;column specifications must be a list or tuple, input was a rrr.zEEach column specification must be 2 element tuple or list of integers)rbufferrr,detect_colspecscolspecsr7rr:rItyperarKrrinteger)rrr:rr,rr7colspecrrrr/Ds8    zFixedWidthReader.__init__cCsd|durt}g}g}t|jD]\}}||vr||||t||kr*nqt||_|S)a Read rows from self.f, skipping as specified. We distinguish buffer_rows (the first <= infer_nrows lines) from the rows returned to detect_colspecs because it's simpler to leave the other locations with skiprows logic alone than to modify them to deal with the fact we skipped some rows here as well. Parameters ---------- infer_nrows : int Number of rows to read from self.f, not counting rows that are skipped. skiprows: set, optional Indices of rows to skip. Returns ------- detect_rows : list of str A list containing the rows to read. N)rrrrrKiterr8)rr7r buffer_rows detect_rowsrrrrrget_rowsbs    zFixedWidthReader.get_rowsc sdddjD}td|d}||}|s tdttt|}t j |dt d}j dur>fd d |D}|D]}| |D] } d|| | <qGq@t |d} d | d <t || Adkd } tt| ddd | ddd } | S) Nrcss|]}d|VqdS)\Nrrrrrrrz3FixedWidthReader.detect_colspecs..z([^z]+)z(No rows from which to infer column widthr.)r)csg|] }|jdqS)r) partitionr,rrrrrrz4FixedWidthReader.detect_colspecs..rr)joinrrXrZrAr r maprKrzerosrr,finditerrrrollwherer:zip) rr7r delimiterspatternrr&maskrmshiftededges edge_pairsrrrr9s$   "z FixedWidthReader.detect_colspecscs\jdurztjWntyd_tjYnwtjfddjDS)Ncs$g|]\}}||jqSr)rhr)rfrommtorirrrrs$z-FixedWidthReader.__next__..)r8rrxrr:rrrTr__next__s   zFixedWidthReader.__next__)Nr5r)r5N)rarbrc__doc__r/rAr9rUrrrrr4?s    & r4c@s*eZdZdZddZddZd dd Zd S) FixedWidthFieldParserzl Specialization that Converts fixed-width fields into DataFrames. See PythonParser for details. cKs0|d|_|d|_tj||fi|dS)Nr:r7)popr:r7rr/)rrr\rrrr/s  zFixedWidthFieldParser.__init__cCs"t||j|j|j|j|j|_dSr)r4r:rr,rr7r0)rrrrrr?s z"FixedWidthFieldParser._make_readerreturnr:cCsdd|DS)z Returns the list of lines without the empty ones. With fixed-width fields, empty lines become arrays of empty strings. See PythonParser._remove_empty_lines. cSs"g|] }tdd|Dr|qS)css$|] }t|t p |VqdSr)r7r8rh)rrrrrr"zGFixedWidthFieldParser._remove_empty_lines...)r)rrirrrrs z=FixedWidthFieldParser._remove_empty_lines..rrrrrrsz)FixedWidthFieldParser._remove_empty_linesN)rYr:)rarbrcrVr/r?rrrrrrWs  rWrYrcCstdd|DS)Ncss$|] }|dks |durdVqdS)rNr.r)rrrrrrrZz#count_empty_vals..)sum)valsrrrr~sr~rcCs$t|std|dkrtd|S)a Validate the 'skipfooter' parameter. Checks whether 'skipfooter' is a non-negative integer. Raises a ValueError if that is not the case. Parameters ---------- skipfooter : non-negative integer The number of rows to skip at the end of the file. Returns ------- validated_skipfooter : non-negative integer The original input if the validation succeeds. Raises ------ ValueError : 'skipfooter' was not a non-negative integer. zskipfooter must be an integerrzskipfooter cannot be negative)r rJ)rrrrr6s r6)rYr)rrrYr)' __future__r collectionsrrrr@iorrXrtypingrrr rnumpyrpandas._libs.lib_libsr"pandas._typingr pandas.errorsr r pandas.core.dtypes.commonr pandas.core.dtypes.inferencerpandas.io.parsers.base_parserrrrrr4rWr~r6rrrrs<       j $