o g, @sdZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z m Z ddlmZmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZddlZddlmZdd l m!Z!m"Z"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)dd l*m+Z+dZ,e-dZ.ede/Z0ede/Z1GdddeZ2edee0fZ3ee0dfZ4eee e fZ5GdddZ6ede6fZ7GdddZ8GdddZ9GdddeZ:ede1e1fZ;e,fddde/dee3e4ffd d!ZGd,d-d-eZ?Gd.d/d/Z@d0eeAd1eBd2eBd3eBd4eBdee2f d5d6ZCejDGd7ddZEd8d9iZFd:e)d;e)fdd?d?e!e@e"ZHd@dAZIdGdBdCZJdHdDdEZKeLdFkrbeKdSdS)Iaa similarities / code duplication command line tool and pylint checker The algorithm is based on comparing the hash value of n successive lines of a file. First the files are read and any line that doesn't fullfill requirement are removed (comments, docstrings...) Those stripped lines are stored in the LineSet class which gives access to them. Then each index of the stripped lines collection is associated with the hash of n successive entries of the stripped lines starting at the current index (n is the minimum common lines option). The common hashes between both linesets are then looked for. If there are matches, then the match indices in both linesets are stored and associated with the corresponding couples (start line number/end line number) in both files. This association is then postprocessed to handle the case of successive matches. For example if the minimum common lines setting is set to four, then the hashes are computed with four lines. If one of match indices couple (12, 34) is the successor of another one (11, 33) then it means that there are in fact five lines wich are common. Once postprocessed the values of association table are the result looked for, i.e start and end lines numbers of common lines in both files. N) defaultdict)getopt)BufferedIOBaseBufferedReaderBytesIO)chaingroupby) AnyDict FrozenSet GeneratorIterableList NamedTupleNewTypeOptionalSetTextIOTupleUnion)nodes) BaseCheckerMapReduceMixintable_lines_from_stats) IRawChecker)Table) CheckerStats)decoding_streamz.*\w+Index LineNumberc@seZdZUeed<eed<dS) LineSpecifs line_numbertextN)__name__ __module__ __qualname__r __annotations__strr)r)U/home/ubuntu/cloudmapper/venv/lib/python3.10/site-packages/pylint/checkers/similar.pyr![s  r! LinesChunkSuccessiveLinesLimitsc@s.eZdZdZdZdddddeddfd d ZdS) CplSuccessiveLinesLimitsz This class holds a couple of SuccessiveLinesLimits objects, one for each file compared, and a counter on the number of common lines between both stripped lines collections extracted from both files  first_file second_fileeffective_cmn_lines_nbr/r,r0r1returnNcCs||_||_||_dSNr.)selfr/r0r1r)r)r*__init__ts z!CplSuccessiveLinesLimits.__init__)r$r%r&__doc__ __slots__intr5r)r)r)r*r-ksr-LineSetStartCouplec@sneZdZdZdZdededeeddfdd Zd e de fd d Z defd dZ defddZ defddZdS)r+zm The LinesChunk object computes and stores the hash of some consecutive stripped lines of a lineset. _fileid_index_hashfileidnum_linelinesr2NcGs,||_ t||_ tdd|D|_dS)Ncs|]}t|VqdSr3)hash).0linr)r)r* z&LinesChunk.__init__..)r;rr<sumr=)r4r>r?r@r)r)r*r5s  zLinesChunk.__init__ocCst|tstS|j|jkSr3) isinstancer+NotImplementedr=)r4rHr)r)r*__eq__  zLinesChunk.__eq__cC|jSr3)r=r4r)r)r*__hash__szLinesChunk.__hash__cCsd|jd|jd|jdS)Nzr:rNr)r)r*__repr__szLinesChunk.__repr__cCsd|jd|jd|jS)NzLinesChunk object for file z, starting at line z Hash is r:rNr)r)r*__str__szLinesChunk.__str__)r$r%r&r6r7r(r8r r5r boolrKrOrPrQr)r)r)r*r+s c@steZdZdZdZdededdfddZedefd d Zedefd d Z e j d eddfdd Z de fddZ dS)r,z A class to handle the numbering of begin and end of successive lines. :note: Only the end line number can be updated. _start_endstartendr2NcCs||_||_dSr3rS)r4rVrWr)r)r*r5s zSuccessiveLinesLimits.__init__cCrMr3)rTrNr)r)r*rVzSuccessiveLinesLimits.startcCrMr3rUrNr)r)r*rWrXzSuccessiveLinesLimits.endvaluecCs ||_dSr3rYr4rZr)r)r*rW cCd|jd|jdS)Nz>rSrNr)r)r*rPszSuccessiveLinesLimits.__repr__) r$r%r&r6r7r r5propertyrVrWsetterr(rPr)r)r)r*r,sc@s^eZdZUdZeed<eed<defddZdefddZ de fd d Z d eddfd d Z dS)r9zN Indices in both linesets that mark the beginning of successive lines fst_lineset_indexsnd_lineset_indexr2cCr])Nz.csg|] }t|dqSr3)iter)rCir@r)r* sz hash_lineset..)rVrW)rlisttuplestripped_linesrange enumeratezipr" IndexErrorrr,r r+nameappend) rhri hash2index index2lines shifted_linesindex_i succ_linesstart_linenumberend_linenumberindexl_cr)rmr* hash_linesets,    r all_couplesc Cst|D]Q}g}|td}||vrB||jj||j_||jj||j_||jd7_|||td}||vs|D]}z| |WqDt yVYqDwqdS)a" Removes all successive entries in the dictionary in argument :param all_couples: collection that has to be cleaned up from successives entries. The keys are couples of indices that mark the beginning of common entries in both linesets. The values have two parts. The first one is the couple of starting and ending line numbers of common successives lines in the first file. The second part is the same for the second file. For example consider the following dict: >>> all_couples {(11, 34): ([5, 9], [27, 31]), (23, 79): ([15, 19], [45, 49]), (12, 35): ([6, 10], [28, 32])} There are two successives keys (11, 34) and (12, 35). It means there are two consecutive similar chunks of lines in both files. Thus remove last entry and update the last line numbers in the first entry >>> remove_successives(all_couples) >>> all_couples {(11, 34): ([5, 10], [27, 32]), (23, 79): ([15, 19], [45, 49])} rpN) rrkeysrfrr/rWr0r1rypopKeyError)rcouple to_removetesttargetr)r)r*remove_successivess$  rls_1 stindex_1ls_2 stindex_2common_lines_nbcCsPdd|j|||D}dd|j|||D}tddt||DS)az Return the effective number of common lines between lineset1 and lineset2 filtered from non code lines, that is to say the number of common successive stripped lines except those that do not contain code (for example a ligne with only an ending parathensis) :param ls_1: first lineset :param stindex_1: first lineset starting index :param ls_2: second lineset :param stindex_2: second lineset starting index :param common_lines_nb: number of common successive stripped lines before being filtered from non code lines :return: the number of common successives stripped lines that contain code cSg|] }t|jr|jqSr)REGEX_FOR_LINES_WITH_CONTENTmatchr#rClspecifr)r)r*rnQ  z(filter_noncode_lines..cSrr)rrr)r)r*rnVrcss|] \}}||kVqdSr3r))rCsline_1sline_2r)r)r*rE[sz'filter_noncode_lines..)rsrGrv)rrrrr stripped_l1 stripped_l2r)r)r*filter_noncode_lines>src@sFeZdZUeed<ded<eed<eed<ded<eed<eed<d S) Commonality cmn_lines_nbrgfst_lsetfst_file_start fst_file_endsnd_lsetsnd_file_start snd_file_endN)r$r%r&r8r'r r)r)r)r*r^s  rc @seZdZdZeddddfdedededededd f d d Z d%d ed e de edd fddZ d&ddZ de eeeeffddZde eeeefdd fddZde eeeefdefddZdddddeed d ffddZdeed d ffdd Zd!d"Zd#d$Zd S)'Similarz,finds copy-pasted lines of code in a projectF min_linesignore_commentsignore_docstringsignore_importsignore_signaturesr2NcCs(||_||_||_||_||_g|_dSr3)rrrrrlinesets)r4rrrrrr)r)r*r5ks  zSimilar.__init__streamidstreamencodingc Csjt|tr|dur tt||j}n|j}z|jt|||j|j |j |j WdSt y4YdSw)z(append a file to search for similaritiesN) rIr ValueErrorr readlinesrryrgrrrrUnicodeDecodeError)r4rrrrr)r)r* append_streamzs&  zSimilar.append_streamcCs |jdkrdS||dS)z.z TOTAL lines=z duplicates=z percent=Y@z.2f)rsortedrx _real_linesrstriprGr) r4rrduplicated_line_numbernumberr couples_lline_set start_lineend_linelinetotal_line_numberr)r)r*rs   "&zSimilar._get_similarity_reportrrgrc #s<t||j\}t||j\}}t}t|}t||@fddd}i} t|tddD].} t| || D]!} | d} | d} t t || t || |jd| t | | <q@q4t | | D]0\}}|j}|j}|j}t|||jj|jj||jj|jjd}t|||||}||jkr|Vqkd S) al Find similarities in the two given linesets. This the core of the algorithm. The idea is to compute the hashes of a minimal number of successive lines of each lineset and then compare the hashes. Every match of such comparison is stored in a dict that links the couple of starting indices in both linesets to the couple of corresponding starting and ending lines in both files. Last regroups all successive couples in a bigger one. It allows to take into account common chunk of lines that have more than the minimal number of successive lines required. cs |dSNrr))mhash_to_index_1r)r*s z&Similar._find_common..keyr<rrp)r1)rrrrrrrN)rr frozensetrroperator attrgetter itertoolsproductr-copyr9rrrbrcr1rr/rVrWr0r)r4rrindex_to_lines_1hash_to_index_2index_to_lines_2hash_1hash_2 common_hashesrc_hashindices_in_linesetsindex_1index_2cml_stripped_lcmn_l start_index_1 start_index_2nb_common_linescom eff_cmn_nbr)rr* _find_commonsZ        zSimilar._find_commonccsLt|jddD]\}}|j|ddD] }|||EdHqq dS)zWiterate on similarities among all files, by making a cartesian product Nrorp)rurr)r4idxrhrr)r)r*rs zSimilar._iter_simscCrM)zReturns the data we can use for a map/reduce process In this case we are returning this instance's Linesets, that is all file information that will later be used for vectorisation. rrNr)r)r* get_map_data'szSimilar.get_map_datacCsdd|D|_dS)oReduces and recombines data into a format that we can report on The partner function of get_map_data()cSsg|] }|D]}|qqSr)r))rCrhrr)r)r*rn3sz2Similar.combine_mapreduce_data..Nr)r4linesets_collectionr)r)r*combine_mapreduce_data/szSimilar.combine_mapreduce_datar3r2N)r$r%r&r6DEFAULT_MIN_SIMILARITY_LINEr8rRr5r( STREAM_TYPESrrrrrrLinesChunkLimits_Trrrr rrrrrr)r)r)r*rhsh  '    E rr@rrrrcs|s|r td|}|r%dd|jD}ddt|dddD}d }|rKd ttjd tjd ttjffd d g|} tt dd| D} g} d} t |ddD]s\} }| }|r| s| dsk| drx|dd} |dd}n| ds| dr|dd} |dd}| r| | rd} d}|r|| |}|rd}|r|ddd }|r| | vrd}|r| t|t| ddqU| S)as Return tuples of line/line number/line type with leading/trailing whitespace and any ignored code features removed :param lines: a collection of lines :param ignore_comments: if true, any comment in the lines collection is removed from the result :param ignore_docstrings: if true, any line that is a docstring is removed from the result :param ignore_imports: if true, any line that is an import is removed from the result :param ignore_signatures: if true, any line that is part of a function signature is removed from the result :return: the collection of line/line number/line type tuples rcss(|]}|jt|tjtjffVqdSr3)linenorIrImport ImportFrom)rCnoder)r)r*rEJs  z!stripped_lines..cSs$i|]\}}|tdd|DqS)css|]\}}|VqdSr3r))rC_ is_importr)r)r*rEOrFz,stripped_lines...)all)rCr node_is_import_groupr)r)r* Nsz"stripped_lines..cSs|dSrr))rjr)r)r*rQsz stripped_lines..rF functionstreer2csL|jD] }t|tjtjfr||t|tjtjtjfr#||q|S)zLRecursively get all functions including nested in the classes from the tree.)bodyrIr FunctionDefAsyncFunctionDefryClassDef)rrr _get_functionsr)r*rWs   z&stripped_lines.._get_functionscss4|]}t|j|jr|jdjn|jdVqdS)rrpN)rtr rtolineno)rCfuncr)r)r*rEks Nrp)rVz"""z'''zr"""zr'''r#r)r#r")astroidparsejoinrrrrNodeNGsetrrustrip startswithendswithgetsplitryr!r )r@rrrrrnode_is_import_by_linenoline_begins_importcurrent_line_is_importrsignature_lines strippedlines docstringr rr)rr*rs6sx       rsc@seZdZdZ    ddedeededededed d fd d Zd dZddZ ddZ ddZ ddZ ddZ eddZeddZd S)rgz Holds and indexes all the lines of a single source file. Allows for correspondance between real lines of the source file and stripped ones, which are the real ones from which undesired patterns have been removed. Frxr@rrrrr2NcCs"||_||_t||||||_dSr3)rxrrs_stripped_lines)r4rxr@rrrrr)r)r*r5s    zLineSet.__init__cCsd|jdS)Nz rxrNr)r)r*rQszLineSet.__str__cCs t|jSr3)rrrNr)r)r*__len__ zLineSet.__len__cCs |j|Sr3r/)r4rr)r)r* __getitem__r3zLineSet.__getitem__cCs |j|jkSr3r1rdr)r)r*__lt__s zLineSet.__lt__cCst|Sr3)idrNr)r)r*rOszLineSet.__hash__cCst|tsdS|j|jkS)NF)rIrg__dict__rdr)r)r*rKrLzLineSet.__eq__cCrMr3r4rNr)r)r*rsrXzLineSet.stripped_linescCrMr3)rrNr)r)r* real_linesrXzLineSet.real_lines)FFFF)r$r%r&r6r(rrRr5rQr2r5r6rOrKr`rsr9r)r)r)r*rgs>   R0801)zSimilar lines in %s files %szduplicate-codezIndicates that a set of similar lines has been detected among multiple file. This usually means that the code should be refactored to avoid this duplication.stats old_statscCs2gd}|t||d7}|t|dddddS)z/make a layout with some stats about duplication)rnowprevious differencenb_duplicated_linespercent_duplicated_linesrrp)childrencolsrheaderscheadersN)rryr)sectr;r<r@r)r)r*report_similaritiess rHc @seZdZdZefZdZeZde ddddfdd d d d dfd d d d ddfddd d ddfddd d ddffZ dde ffZ d'd(ddZ d)ddZddZdejddfdd Zd!d"Zd#d$Zd%d&ZdS)*SimilarCheckerzchecks for similarities and duplicated code. This computation may be memory / CPU intensive, so you should disable it if you experiment some problems. rmin-similarity-linesr8zz%Minimum lines number of a similarity.)defaulttypemetavarhelpignore-commentsTynzz4Comments are removed from the similarity computationignore-docstringsz6Docstrings are removed from the similarity computationignore-importsFz3Imports are removed from the similarity computationignore-signaturesz6Signatures are removed from the similarity computationRP0801 DuplicationNr2cCs@t||tj||jj|jj|jj|jj|jjdi|_ dS)N)rrrrr) rr5rconfigmin_similarity_linesrrrrr;)r4linterr)r)r*r5"s  zSimilarChecker.__init__cCst||||||dkr|jj|_dS|dkr|jj|_dS|dkr*|jj|_dS|dkr5|jj|_dS|dkr@|jj|_dSdS)zmethod called to set an option (registered in the options list) Overridden to report options setting to Similar rJrOrQrRrSN) r set_optionrVrWrrrrr)r4optnamerZactionoptdictr)r)r*rY.szSimilarChecker.set_optioncCsg|_|jjddd|_dS)z.rNrrr]r:r)argsrprArrB) rGrr;rryrxrr9r add_messagerr!) r4total duplicatedr;rrmsgrhrrrr)r)r*closePs  "zSimilarChecker.closecCs t|S)zPassthru override)rrrNr)r)r*rer\zSimilarChecker.get_map_datacCsRt|}|j|_|j|_|j|_|j|_|j|_|tj||d| dS)r)rN) rIrrrrrr^rrrh)r4rXdata recombinedr)r)r*reduce_map_datais zSimilarChecker.reduce_map_datar3r)NN)r$r%r&r6r__implements__rxMSGSmsgsroptionsrHreportsr5rYr^rModulerarhrrkr)r)r)r*rIsb     0    rIcCs|t|dS)z-required method to auto register this checkerN)register_checkerrI)rXr)r)r*registerxsrscCs$tdttdt|dS)z&display command line usage informationz*finds copy pasted blocks in a set of fileszUsage: symilar [-d|--duplicates min_duplicated_lines] [-i|--ignore-comments] [--ignore-docstrings] [--ignore-imports] [--ignore-signatures] file1...N)rsysexit)statusr)r)r*usage}s rwc Cs|dur tjdd}d}d}t}d}d}d}d}t|||\}} |D]0\} } | dvr0t| }q#| dvr8tq#| dvr?d }q#| d vrFd }q#| d vrMd }q#| d vrSd }q#| sZtdt|||||} | D]} t| d d}| | |Wdn1s}wYqd| t ddS)z$standalone command line access pointNrphdi)rNz duplicates=rOrRrQrSF)z-dz --duplicates)z-hz--help)z-iz--ignore-commentsT)z--ignore-docstrings)z--ignore-imports)z--ignore-signatureszutf-8)rr) rtargvrrr8rwrr^rrru)rys_optsl_optsrrrrroptsrcoptvalsimfilenamerr)r)r*RunsH   r__main__)rr3)Mr6r functoolsrrrert collectionsrriorrrrrtypingr r r r r rrrrrrrrrrpylint.checkersrrrpylint.interfacesrpylint.reporters.ureports.nodesr pylint.typingr pylint.utilsrrcompilerr8rr r! HashToIndex_TIndexToLines_Trr-CplIndexToCplLines_Tr+r,r9r rrrrrr(rRrstotal_orderingrgrmrHrIrsrwrr$r)r)r)r*s  <          %  *,   O `4    -