Field note · Files
Why exact duplicate detection starts with size and ends with a full hash
Published: September 9, 2026 · Maintained by: Jisung Kim · AI assistance and release-check practices are disclosed in the editorial policy. This note documents WEBBE-B tool behavior, design choices and verification practice.
Duplicate cleanup sounds simple until two files have the same name but different bytes, or the same bytes under completely different names. WEBBE-B’s ExactDupe deliberately separates candidate finding from identity checking. That distinction matters because a fast filter can reduce work, but it should not silently become the proof that two files are interchangeable.
File names are labels, not identity
A camera can create IMG_0001.JPG in January and create another IMG_0001.JPG after its counter resets. A download folder can contain report.pdf, report (1).pdf and final-report.pdf even when two are byte-for-byte identical. Names are useful for navigation, but they are weak evidence for duplicate cleanup.
Modification time is similar. Copying a file may preserve the original timestamp, replace it, or create a new one depending on the operating system and application. Two identical files can therefore have different dates, while two different files can share the same date.
Size is an efficient first filter
Exact duplicates must contain the same number of bytes. That gives a cheap elimination rule: if a file size appears only once in the selected set, that file cannot have an exact duplicate in that set. ExactDupe groups files by byte size first and only hashes groups with at least two members. On a large folder this can avoid reading the entire contents of many unique files.
This is a performance decision, not a correctness shortcut. Equal size does not mean equal content. A 4 MB photograph and a 4 MB archive can share exactly the same byte count while containing unrelated data. Size tells the tool which files deserve a stronger comparison.
The hash covers the full candidate file
For the remaining candidates, ExactDupe computes a SHA-256 digest from the full file rather than hashing only a prefix. A prefix check can be useful as an additional performance stage, but it is not enough for an “exact duplicate” label because different files can share the same beginning. Full-file hashing makes the classification depend on all selected bytes.
The practical condition used by the tool is therefore: same byte size plus the same full SHA-256 digest. SHA-256 is designed so that different inputs overwhelmingly produce different digests, but a hash is still a finite fingerprint rather than a mathematical byte-by-byte proof. For ordinary personal file cleanup, the residual collision risk is extremely small. For evidence handling, archival preservation or another context that demands formal byte equality, a separate direct comparison can be added outside the tool.
A worked folder example
Imagine a backup folder containing 12,000 files. Ten thousand file sizes occur only once. Those 10,000 files can be excluded from hashing immediately. The remaining 2,000 files fall into repeated-size groups. ExactDupe reads those candidate files in chunks, calculates their hashes, and then groups matching digests. The resulting report might show three copies of one video and two copies of a project archive even though the filenames differ.
The “potential space” number should then be read as the storage represented by extra copies, not as an instruction to delete them. One copy may live in a backup location that exists for a reason. The program intentionally does not delete anything.
Why the keep suggestion is only a suggestion
A duplicate report knows bytes and paths. It does not know which folder is your authoritative archive, which disk is temporary, or which copy is referenced by another application. That is why a keeper label is organisational guidance rather than an automatic action. A safe cleanup sequence is: identify the group, choose the copy that belongs in the right location, confirm backup or recovery options, and delete manually only after that review.
What changes with very large libraries
Browser file APIs require the user to choose the files or folder. A website cannot silently crawl an entire drive. Very large selections also create memory and interface pressure even before hashing starts. ExactDupe’s large-library path enumerates gradually and limits expensive work to size-matched candidates. The design goal is to make a browser tool useful for real collections without pretending the browser has the same privileges as an installed disk utility.
The verification rule
The useful takeaway is not “hash everything.” It is “use a cheap property to narrow the set, then use a content-derived fingerprint for the identity decision, and keep deletion outside the automated result.” That order is what turns duplicate detection from filename guesswork into a reviewable workflow.
Related: FolderHeat helps find where storage is concentrated before duplicate analysis, while the file-cleanup guide explains the wider inventory → inspect → verify → act sequence.