Deduplicating 400,000 Files Without Deleting One

**Series: Multi-Cloud Tiered Storage (MCTS)** 1. Part 1: The NAS That Wasn't 2. Part 2: MCTS Architecture 3. Part 3: The Great Deduplication *(Current)* 4. Epilogue: The 400 GB Shortcut That Ran for Zero Seconds

Once our Multi-Cloud Tiered Storage architecture was established (Part 2: MCTS Architecture), we faced the reality of our raw data.

Ten years of fragmented backups across multiple drives and cloud accounts had accumulated into a mountain of **445,586 files**.

Before we could declare the hot master canonical, we had to clean house.

---

The Anatomy of Waste

We built an indexing scanner that generated deterministic MD5 hashes across every file in the three realms using `rclone lsf --format psh --hash md5`.

The numbers revealed the scale of ten years of drift:

• **56,655 duplicate clusters**
• **241.3 GiB of redundant storage**
• Files cloned across multiple project folders, "backup of backup" folders, and unorganized media dumps.

---

The Rule of Non-Destructive Quarantine

When dealing with a decade of creative work, running an automated deletion script is reckless. A path might look like a duplicate, but context (folder structure, association with nearby notes) matters.

Instead of deleting, we enforced the **Rule of Non-Destructive Quarantine**:

1. **Deterministic Path Scoring**: Every duplicate group was evaluated by an automated scoring model:
   • Shared Studio assets outranked individual folders.
   • Deeply organized project paths outranked root-level temporary folders.
   • Clean filenames outranked auto-incremented names like `IMG_001 (1).jpg`.
2. **Atomic Server-Side Isolation**: The losing duplicates (87,784 files, 213.75 GiB) were not deleted; they were moved into an isolated `_dedup_quarantine/` folder via server-side API calls in seconds.

Nothing was destroyed. The active tree became clean immediately, while the safety net remained 100% intact.

---

Auditing the Cold Archive Divergence

With the hot master purified, we ran a cross-system hash comparison between our Hot Master (Drive) and our historical Cold Archive (a Google Cloud Storage Archive bucket holding 409,237 objects).

The result was a wake-up call:

• **252,281 files (61.6%)** were already safely present in both places.
• **156,956 files (421.9 GiB)** existed *only* in the cold bucket—orphaned historical assets that had never made it to the modern drive hierarchy.

To make the Hot Master truly complete, we needed to backfill those 421.9 GiB back into the three realms before re-mirroring the entire structure.

How we navigated the 400 GB migration—and the lessons learned when attempting to use ephemeral cloud sandboxes—is documented in Epilogue: The 400 GB Shortcut That Ran for Zero Seconds.