The 400 GB Shortcut That Ran for Zero Seconds

**Series: Multi-Cloud Tiered Storage (MCTS)** 1. Part 1: The NAS That Wasn't 2. Part 2: MCTS Architecture 3. Part 3: The Great Deduplication 4. Epilogue: The 400 GB Shortcut That Ran for Zero Seconds *(Current)*

We had 422 gigabytes across 156,956 files sitting in an archival cloud storage bucket that needed to be consolidated into an active drive hierarchy.

Because both endpoints were inside the same cloud provider, the intuitive assumption was that moving the data would be instantaneous. But cross-API streaming doesn't support server-side reparenting: an engine has to download each chunk over HTTP and upload it to the target endpoint.

Doing that from a local machine on a metered mobile connection meant a 48-hour continuous stream. Naturally, we looked for a way to let the cloud do the heavy lifting.

---

The Allure of the Zero-Cost Cloud Shell

The immediate thought was simple: spin up a temporary browser-based Cloud Shell. It runs inside Google's datacenter backbone, costs nothing, and has gigabit pipes directly into both storage layers.

The logic seemed airtight. We packaged the `rclone` binary, OAuth tokens, and file manifests into a bundle, sent it to the cloud environment, and launched the job inside `tmux`.

Then we waited.

---

The Inactivity Trap

Two hours later, we checked the destination drive. Not a single byte had moved from the cloud instance.

The reason is a classic confusion between a **persistent virtual machine** and an **ephemeral interactive container**:

1. **The Watchdog Ignores CPU**: In a standard persistent Linux server (like an EC2 instance or GCE VM), `tmux` or `nohup` detaches your process from the SSH session. The kernel keeps executing regardless of client connectivity.
2. **WebSocket Lifeline**: Browser-based cloud shells are not headless servers. They are ephemeral sandboxes tethered strictly to active user interaction in the web UI.
3. **The Silent Reaper**: Google's container lifecycle policy dictates that after 20 to 60 minutes without keyboard or mouse input on the webpage, the entire container is terminated and reclaimed. Background jobs—`tmux` sessions, compilation pipelines, or storage transfers—are killed instantaneously with the container.

The "fast cloud shortcut" had run for a brief burst of directory scanning, noticed no humans typing in the browser tab, and was promptly recycled into the void.

---

The Unsung Hero: Idempotent Protocols

While the cloud experiment stalled at zero progress, the local MacBook—chugging along over days of mobile hotspot hiccups, DNS timeouts, and system reboots—had already quietly migrated **139.2 GiB (38,854 files)**.

What made switching back and forth completely painless was not the hardware, but protocol design:

rclone copy --files-from manifest.txt --retries 10 --drive-chunk-size 128M ...

Because the sync protocol is strictly **idempotent**:

• Every batch verifies destination state before writing.
• Network disconnects trigger linear backoffs without corrupting partially written objects.
• Resuming a transfer after a reboot or a detour costs zero duplicate bandwidth—it scans the hash set and resumes at the exact missing byte.

Checks:         30459 / 30459, 100%
Transferred:        0 B / 0 B
Elapsed time:   9m0.0s

When we pointed the local engine back at the source, it verified 30,000 existing files in nine minutes and continued chewing through the remaining queue.

---

Takeaway

When designing long-running pipelines:

1. **Never mistake an interactive sandbox for background infrastructure.** If a container's lifecycle is tied to a browser tab, any job longer than your attention span will die silently.
2. **Idempotence is the ultimate safety net.** When human and agent explore shortcuts, failures and detours will happen. A strictly resumable protocol ensures that no mistake ever costs you the work you already finished.