Dropbox Enhances Storage Efficiency in Magic Pocket via Adaptive Compaction
Dropbox has successfully improved storage efficiency within Magic Pocket, its core exabyte-scale immutable blob storage system. The initiative was driven by increased fragmentation and storage overhead resulting from a previous service update that reduced write amplification but inadvertently left many volumes under-filled. To address this, Dropbox engineering teams transformed the compaction process into a layered, adaptive pipeline. This new strategy allows for more effective reclamation of unused space caused by data immutability, where deleted or updated files leave behind residual data until physically compacted. By strengthening monitoring controls and optimizing how live blobs are gathered and rewritten into new volumes, the team significantly reduced storage overhead, bringing it below previous baselines. This improvement is critical for managing infrastructure costs at scale, as even modest increases in overhead translate to substantial expenses. The article details the technical challenges of managing an immutable store, the mechanics of garbage collection versus compaction, and the multi-strategy approach employed to ensure the system remains resilient to workload changes while maintaining durability through erasure coding.
Wire timeline
Dropbox Enhances Storage Efficiency in Magic Pocket via Adaptive Compaction
Dropbox has successfully improved storage efficiency within Magic Pocket, its core exabyte-scale immutable blob storage system. The initiative was driven by increased fragmentation and storage overhead resulting from a previous service update that reduced write amplification but inadvertently left many volumes under-filled. To address this, Dropbox engineering teams transformed the compaction process into a layered, adaptive pipeline. This new strategy allows for more effective reclamation of unused space caused by data immutability, where deleted or updated files leave behind residual data until physically compacted. By strengthening monitoring controls and optimizing how live blobs are gathered and rewritten into new volumes, the team significantly reduced storage overhead, bringing it below previous baselines. This improvement is critical for managing infrastructure costs at scale, as even modest increases in overhead translate to substantial expenses. The article details the technical challenges of managing an immutable store, the mechanics of garbage collection versus compaction, and the multi-strategy approach employed to ensure the system remains resilient to workload changes while maintaining durability through erasure coding.
Dropbox Tech Blog