Find files with the same content in one folder (and all its sub-folders)












1














Say I have a folder named container and there are many files/sub-folders in it. I would like to find all the files with the same content (they can have different names but the content should be the same).



Is there any way I can do this on Mac?






macos file







share|improve this question
























  • I'd calculate the md5 checksum for all the files in the directory hierarchy and look for checksum duplicates.
    – Nimesh Neema
    1 hour ago
















1














Say I have a folder named container and there are many files/sub-folders in it. I would like to find all the files with the same content (they can have different names but the content should be the same).



Is there any way I can do this on Mac?






macos file







share|improve this question
























  • I'd calculate the md5 checksum for all the files in the directory hierarchy and look for checksum duplicates.
    – Nimesh Neema
    1 hour ago














1












1








1







Say I have a folder named container and there are many files/sub-folders in it. I would like to find all the files with the same content (they can have different names but the content should be the same).



Is there any way I can do this on Mac?






macos file







share|improve this question















Say I have a folder named container and there are many files/sub-folders in it. I would like to find all the files with the same content (they can have different names but the content should be the same).



Is there any way I can do this on Mac?






macos file




macos file






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 1 hour ago









Nimesh Neema

14.7k43871




14.7k43871










asked 2 hours ago









AGamePlayer

3681516




3681516












  • I'd calculate the md5 checksum for all the files in the directory hierarchy and look for checksum duplicates.
    – Nimesh Neema
    1 hour ago


















  • I'd calculate the md5 checksum for all the files in the directory hierarchy and look for checksum duplicates.
    – Nimesh Neema
    1 hour ago
















I'd calculate the md5 checksum for all the files in the directory hierarchy and look for checksum duplicates.
– Nimesh Neema
1 hour ago




I'd calculate the md5 checksum for all the files in the directory hierarchy and look for checksum duplicates.
– Nimesh Neema
1 hour ago










1 Answer
1






active

oldest

votes


















2














If you're comfortable using the Terminal, you can try rdfind which purpose is precisely to find duplicate files.



You'll need to install it with Homebrew first: brew install rdfind



Then, let's say you want to run a dry test on your ~/Download folder:



rdfind -n true -outputname result.log ~/Downloads/




  • -n true is for dry mode: no files will be affected


  • -outputname result.log will output the scanning into that file


  • ~/Downloads/ is the argument of the root folder you want to scan
    (you can pass more than one folder)


Running that command will look something like that:



$ rdfind -n true -outputname result.log ~/Downloads/

(DRYRUN MODE) Now scanning "/Users/ym/Downloads", found 20132 files.

(DRYRUN MODE) Now have 20132 files in total.

(DRYRUN MODE) Removed 0 files due to nonunique device and inode.

(DRYRUN MODE) Now removing files with zero size from list...removed 75 files

(DRYRUN MODE) Total size is 59782752628 bytes or 56 GiB

(DRYRUN MODE) Now sorting on size:removed 3795 files due to unique sizes from list.16262 files left.

(DRYRUN MODE) Now eliminating candidates based on first bytes:removed 579 files from list.15683 files left.

(DRYRUN MODE) Now eliminating candidates based on last bytes:removed 134 files from list.15549 files left.

(DRYRUN MODE) Now eliminating candidates based on md5 checksum:removed 94 files from list.15455 files left.

(DRYRUN MODE) It seems like you have 15455 files that are not unique

(DRYRUN MODE) Totally, 324 MiB can be reduced.

(DRYRUN MODE) Now making results file result.log



Or, you can use an application with a graphical interface like dupeguru:



choose folder to scan



scan result






share|improve this answer










New contributor




Yoric is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.


















    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "118"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fapple.stackexchange.com%2fquestions%2f347217%2ffind-files-with-the-same-content-in-one-folder-and-all-its-sub-folders%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    2














    If you're comfortable using the Terminal, you can try rdfind which purpose is precisely to find duplicate files.



    You'll need to install it with Homebrew first: brew install rdfind



    Then, let's say you want to run a dry test on your ~/Download folder:



    rdfind -n true -outputname result.log ~/Downloads/




    • -n true is for dry mode: no files will be affected


    • -outputname result.log will output the scanning into that file


    • ~/Downloads/ is the argument of the root folder you want to scan
      (you can pass more than one folder)


    Running that command will look something like that:



    $ rdfind -n true -outputname result.log ~/Downloads/
    
(DRYRUN MODE) Now scanning "/Users/ym/Downloads", found 20132 files.
    
(DRYRUN MODE) Now have 20132 files in total.
    
(DRYRUN MODE) Removed 0 files due to nonunique device and inode.
    
(DRYRUN MODE) Now removing files with zero size from list...removed 75 files
    
(DRYRUN MODE) Total size is 59782752628 bytes or 56 GiB
    
(DRYRUN MODE) Now sorting on size:removed 3795 files due to unique sizes from list.16262 files left.
    
(DRYRUN MODE) Now eliminating candidates based on first bytes:removed 579 files from list.15683 files left.
    
(DRYRUN MODE) Now eliminating candidates based on last bytes:removed 134 files from list.15549 files left.
    
(DRYRUN MODE) Now eliminating candidates based on md5 checksum:removed 94 files from list.15455 files left.
    
(DRYRUN MODE) It seems like you have 15455 files that are not unique
    
(DRYRUN MODE) Totally, 324 MiB can be reduced.
    
(DRYRUN MODE) Now making results file result.log



    Or, you can use an application with a graphical interface like dupeguru:



    choose folder to scan



    scan result






    share|improve this answer










    New contributor




    Yoric is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.























      2














      If you're comfortable using the Terminal, you can try rdfind which purpose is precisely to find duplicate files.



      You'll need to install it with Homebrew first: brew install rdfind



      Then, let's say you want to run a dry test on your ~/Download folder:



      rdfind -n true -outputname result.log ~/Downloads/




      • -n true is for dry mode: no files will be affected


      • -outputname result.log will output the scanning into that file


      • ~/Downloads/ is the argument of the root folder you want to scan
        (you can pass more than one folder)


      Running that command will look something like that:



      $ rdfind -n true -outputname result.log ~/Downloads/
      
(DRYRUN MODE) Now scanning "/Users/ym/Downloads", found 20132 files.
      
(DRYRUN MODE) Now have 20132 files in total.
      
(DRYRUN MODE) Removed 0 files due to nonunique device and inode.
      
(DRYRUN MODE) Now removing files with zero size from list...removed 75 files
      
(DRYRUN MODE) Total size is 59782752628 bytes or 56 GiB
      
(DRYRUN MODE) Now sorting on size:removed 3795 files due to unique sizes from list.16262 files left.
      
(DRYRUN MODE) Now eliminating candidates based on first bytes:removed 579 files from list.15683 files left.
      
(DRYRUN MODE) Now eliminating candidates based on last bytes:removed 134 files from list.15549 files left.
      
(DRYRUN MODE) Now eliminating candidates based on md5 checksum:removed 94 files from list.15455 files left.
      
(DRYRUN MODE) It seems like you have 15455 files that are not unique
      
(DRYRUN MODE) Totally, 324 MiB can be reduced.
      
(DRYRUN MODE) Now making results file result.log



      Or, you can use an application with a graphical interface like dupeguru:



      choose folder to scan



      scan result






      share|improve this answer










      New contributor




      Yoric is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





















        2












        2








        2






        If you're comfortable using the Terminal, you can try rdfind which purpose is precisely to find duplicate files.



        You'll need to install it with Homebrew first: brew install rdfind



        Then, let's say you want to run a dry test on your ~/Download folder:



        rdfind -n true -outputname result.log ~/Downloads/




        • -n true is for dry mode: no files will be affected


        • -outputname result.log will output the scanning into that file


        • ~/Downloads/ is the argument of the root folder you want to scan
          (you can pass more than one folder)


        Running that command will look something like that:



        $ rdfind -n true -outputname result.log ~/Downloads/
        
(DRYRUN MODE) Now scanning "/Users/ym/Downloads", found 20132 files.
        
(DRYRUN MODE) Now have 20132 files in total.
        
(DRYRUN MODE) Removed 0 files due to nonunique device and inode.
        
(DRYRUN MODE) Now removing files with zero size from list...removed 75 files
        
(DRYRUN MODE) Total size is 59782752628 bytes or 56 GiB
        
(DRYRUN MODE) Now sorting on size:removed 3795 files due to unique sizes from list.16262 files left.
        
(DRYRUN MODE) Now eliminating candidates based on first bytes:removed 579 files from list.15683 files left.
        
(DRYRUN MODE) Now eliminating candidates based on last bytes:removed 134 files from list.15549 files left.
        
(DRYRUN MODE) Now eliminating candidates based on md5 checksum:removed 94 files from list.15455 files left.
        
(DRYRUN MODE) It seems like you have 15455 files that are not unique
        
(DRYRUN MODE) Totally, 324 MiB can be reduced.
        
(DRYRUN MODE) Now making results file result.log



        Or, you can use an application with a graphical interface like dupeguru:



        choose folder to scan



        scan result






        share|improve this answer










        New contributor




        Yoric is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.









        If you're comfortable using the Terminal, you can try rdfind which purpose is precisely to find duplicate files.



        You'll need to install it with Homebrew first: brew install rdfind



        Then, let's say you want to run a dry test on your ~/Download folder:



        rdfind -n true -outputname result.log ~/Downloads/




        • -n true is for dry mode: no files will be affected


        • -outputname result.log will output the scanning into that file


        • ~/Downloads/ is the argument of the root folder you want to scan
          (you can pass more than one folder)


        Running that command will look something like that:



        $ rdfind -n true -outputname result.log ~/Downloads/
        
(DRYRUN MODE) Now scanning "/Users/ym/Downloads", found 20132 files.
        
(DRYRUN MODE) Now have 20132 files in total.
        
(DRYRUN MODE) Removed 0 files due to nonunique device and inode.
        
(DRYRUN MODE) Now removing files with zero size from list...removed 75 files
        
(DRYRUN MODE) Total size is 59782752628 bytes or 56 GiB
        
(DRYRUN MODE) Now sorting on size:removed 3795 files due to unique sizes from list.16262 files left.
        
(DRYRUN MODE) Now eliminating candidates based on first bytes:removed 579 files from list.15683 files left.
        
(DRYRUN MODE) Now eliminating candidates based on last bytes:removed 134 files from list.15549 files left.
        
(DRYRUN MODE) Now eliminating candidates based on md5 checksum:removed 94 files from list.15455 files left.
        
(DRYRUN MODE) It seems like you have 15455 files that are not unique
        
(DRYRUN MODE) Totally, 324 MiB can be reduced.
        
(DRYRUN MODE) Now making results file result.log



        Or, you can use an application with a graphical interface like dupeguru:



        choose folder to scan



        scan result







        share|improve this answer










        New contributor




        Yoric is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.









        share|improve this answer



        share|improve this answer








        edited 3 mins ago





















        New contributor




        Yoric is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.









        answered 1 hour ago









        Yoric

        2335




        2335




        New contributor




        Yoric is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.





        New contributor





        Yoric is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.






        Yoric is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Ask Different!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.





            Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


            Please pay close attention to the following guidance:


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fapple.stackexchange.com%2fquestions%2f347217%2ffind-files-with-the-same-content-in-one-folder-and-all-its-sub-folders%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            -

            Popular posts from this blog

            What visual should I use to simply compare current year value vs last year in Power BI desktop

            Image
            0 I have column SubToBind that shows current year ratio and column PY SubToBind which shows previous year ratio. How can I simply create a card that would be displaying them both in a nice way? Something like that: I cannot achieve that with regular card, I also tried Card with States by OKViz but also unable to see both values. In KPI visual - I dont like the word "Goal", it will confuse end users. Are there any cards or other visuals that would do the job? Thanks powerbi dax powerbi-embedded share | improve this question edited Jun 11 '18 at 20:11 Oleg ...
            Read more

            How to ignore python UserWarning in pytest?

            Image
            up vote 1 down vote favorite I am using openpyxl for parsing .xlsm files, and pytest for testing. When I open a file, I get the: OpenPyxl -> UserWarning: Data Validation extension is not supported and will be removed That is not really a problem bcs. program works and I can't change the .xlsm file to fix that. But... When I run pytest with something like: def test_wrong_file_format(): assert check_excel(open_excel('file.xlsm')) == True I will get the warning i mentioned altought check_excel(open_excel('file.xlsm')) returns True and the test should suceed... Is there a nice way to tell the pytest that "It's not a bug it's a feature" and tests should pass even when they get this warning? Is there other way than using something like: with pytest.warns(UserWarning): ...
            Read more

            Alexandru Averescu

            Image
            Alexandre Averescu Alexandru Averescu Naissance 3 avril 1859 Babele, Ismail  (ro) près de Izmaïl (principauté de Moldavie) Décès 2 octobre 1938 (à 79 ans) [ 1 ] Bucarest Origine   Roumanie Arme cavalerie Grade Maréchal Années de service 1901-1917 Commandement chef d'état-major général 2 e  armée 3 e  armée Groupe d'armée sud armée du Danube Conflits Guerre russo-turque de 1877-1878 Deuxième guerre balkanique Première Guerre mondiale Faits d'armes Bataille de Turtucaia/Tutrakan Bataille de Mărăști Distinctions Autres fonctions 3 fois président du Conseil des ministres ministre des Affaires étrangères ministre sans portefeuille conseiller de la Couronne modifier   Alexandre Averescu (en roumain : Alexandru Averescu .mw-parser-output .prononciation>a{background:url("//upload.wikimedia.org/wikipedia/commons/thumb/8/8a/Loudspeaker.svg/11px-Loudspeaker.svg.png")center ...
            Read more