Separating Storage from Metadata

When we added Docker support to Flux, we ran into a problem with the storage architecture. The way Flux stored artifacts wasn’t going to work for Docker images.

The original design was simple: each repository had its own storage config, and artifacts read their data from the repository’s storage using path conventions. A gem artifact knew it lived at name/version/name-version.gem. The registry code constructed the path, called storage.read(path), and got the bytes back. Straightforward, and perfectly adequate for RubyGems where each artifact is a single file.

Docker images are different. An image isn’t a file. It’s a manifest that references a config blob and a set of layer blobs. Those blobs are content-addressed: identified by their SHA256 hash, shared across images. Push ten images that use the same Alpine base layer, and that layer should be stored once, not ten times. The old “each artifact owns its file at a known path” model couldn’t express this.

So we needed a refactor. While actively shipping features. In production.

The four-layer model

What we ended up with:

BlobStore: the storage backend. Configurable (local filesystem or S3), shareable across multiple repositories, with quota management. Instead of every repository having its own storage config, repositories point to a shared BlobStore. This means gems and Docker images can share the same storage infrastructure, the same quotas, the same cleanup jobs.

Blob: a content-addressed, immutable data record. Identified by SHA256 hash, nothing else. No filename, no path, no content type. Just bytes and their hash. Two identical files, even from different repositories and different artifact types, result in one Blob on disk.

Asset: a named reference that gives a Blob meaning within a repository. An asset has a virtual_path (like gems/mygem-1.0.0.gem or blobs/sha256:abc123), a content type, and a security status. Multiple assets can point to the same blob. This is where deduplication becomes visible: ten Docker images sharing a base layer have ten asset references but one blob on disk.

ArtifactAsset: a join table connecting artifacts to their assets. A gem artifact links to its .gem file and its .gemspec.rz. A Docker tag links to its manifest, config blob, and every layer blob. Simple join, but it’s what makes the many-to-many relationship between tags and layers work.

Content-addressing in practice

When data enters the system via BlobStore#store(data), the store writes it to a temp file, computes the SHA256 as it streams, then saves it at a path derived from the hash: a1/b2/a1b2c3d4e5f6.... Two-level directory sharding to avoid filesystem issues from too many files in a single directory (256 × 256 = 65,536 possible directories).

The key operation is find_or_initialize_by(sha256: digest). If a blob with this hash already exists, no new record is created and no new file is written. The existing blob is returned. Deduplication happens automatically as a consequence of content addressing. Nobody has to check for duplicates or manage a dedup index.

Blob equality is defined by SHA256. Two Blob objects with the same hash are ==, eql?, and produce the same hash value. They work correctly in Sets and as Hash keys. This sounds like a small detail but it matters when you’re comparing blob sets across artifacts to figure out which ones are shared.

What broke

The refactor landed across about a week of commits, most of them marked WIP. Here’s what went wrong:

Index generation broke. Both the compact index (used by Bundler) and the modern index (used by gem) had to be rewritten to work with the new asset/blob model. The old code read files directly from storage using constructed paths. The new code had to go through asset lookups. The compact index broke first, then the modern index, and I spent two commits getting them both working again.

Migration ordering. The initial blob and asset migration files were timestamped for when I wrote them (May 13-14), but they needed to run before other migrations that referenced the new tables. I had to renumber them to slot earlier in the migration sequence. The kind of thing you only notice when you try to set up a fresh database.

Security scanning moved. The old design hung security scans off artifacts. The new design hangs them off assets, because you’re scanning individual files, not abstract package concepts. A Docker image has layer blobs that might be insecure individually. Moving the has_many :security_scans association from Artifact to Asset touched every part of the scanning pipeline.

The cleanup job was a stub. I’d written CleanupBlobsJob as a placeholder during the initial refactor. It became real when I realized chunked Docker uploads create intermediate blobs that become orphans after the upload finalizes. The implementation is simple: blobs.where.missing(:assets) finds blobs with no asset references, and the job deletes them from both storage and the database. But getting the safety guards right (validate the blob has no references, is locally loaded, and is owned by this store) took a couple of iterations.

The Docker registry broke after the second refactor. The first refactor introduced Blob and Asset. The second refactor extracted BlobStore from the old StorageConfig model. The Docker registry had been written between the two, so it was targeting an intermediate API that no longer existed. One more WIP commit to update it.

The S3 backend inconsistency

One thing I still haven’t fully cleaned up: the S3StorageService constructor still accepts a repository and calls repository.storage_config, rather than accepting a BlobStore directly. The LocalStorageService was updated to accept a BlobStore, but the S3 service still has the old interface. It works because the repository has a blob_store association that happens to provide the same credentials, but the code paths are asymmetric. On the list to fix.

What we gained

The numbers tell the story. Before the refactor, our test Flux instance with a dozen RubyGems repos and some Docker images was using about 2.3GB of storage. After enabling deduplication, it dropped to 1.4GB. Most of the savings came from Docker layers. The same Alpine and Debian base layers were stored once instead of per-image.

The shared BlobStore model also simplified operations. Instead of configuring storage for each repository individually, we configure one or two blob stores and point repositories at them. Quota management is per-store, so we can set a global storage limit instead of guessing per-repository limits. The cleanup job runs against the store, not against individual repositories, so orphaned blobs are found and removed regardless of which repository created them.

Refactoring storage in a system that’s actively being used and actively gaining new features was uncomfortable. The WIP commits, the broken indices, the two-phase refactor that left the Docker registry temporarily broken. None of that was fun. But the alternative was building Docker support on top of a storage model that couldn’t express shared blobs, which would have been worse in every way that matters.