User Tools

Site Tools


howto:duplicate

Find Duplicates

There are various tools and commands to find duplicate files (i.e. find command), but those usually find exact matches which does not help with media files (music,video,images) that have been resized,reencoded,contain watermarks, etc). The following tools help find those duplicates.

Video Duplicate Finder (Win/Linux) finds Videos and Images based on similarity:

https://github.com/0x90d/videoduplicatefinder/releases

wget "https://github.com/0x90d/videoduplicatefinder/releases/download/2.0.7/VDF.Linux-x64.tar.gz"
tar -xvzf VDF.Linux-x64.tar.gz
rm VDF.Linux-x64.tar.gz
cd VDF.Linux-x64
./VideoDuplicateFinderLinux

Dupeguru (Win/Linux/Mac) finds Music, Images and general files:

https://dupeguru.voltaicideas.net/

sudo apt-add-repository ppa:dupeguru/ppa
sudo apt-get update
sudo apt-get install dupeguru

For ModuleNotFoundError: No module named 'core.pe._block' error, link to the proper libraries:

python3 --version
#note the version number and add links. Replace '38' with major and minor version on the system. replace 36 if necessary.

sudo ln /usr/share/dupeguru/core/pe/_cache.cpython-36m-x86_64-linux-gnu.so /usr/share/dupeguru/core/pe/_cache.cpython-38-x86_64-linux-gnu.so
sudo ln /usr/share/dupeguru/core/pe/_block.cpython-36m-x86_64-linux-gnu.so /usr/share/dupeguru/core/pe/_block.cpython-38-x86_64-linux-gnu.so
sudo ln /usr/share/dupeguru/qt/pe/_block_qt.cpython-36m-x86_64-linux-gnu.so /usr/share/dupeguru/qt/pe/_block_qt.cpython-38-x86_64-linux-gnu.so

FSlint is a find frontend and finds exact matches, temp files, bad symlinks etc on linux

sudo apt-get install fslint
howto/duplicate.txt · Last modified: 2023/05/29 11:55 by 127.0.0.1