Implementing the Docker Registry Protocol

rubytoolingbuilding-in-public

Flux started as a RubyGems registry. This week we added Docker support by implementing the OCI Distribution Spec, the protocol that docker push and docker pull speak. It’s the same protocol that Docker Hub, GitHub Container Registry, and every other Docker registry implements.

The spec itself is surprisingly well-designed. A handful of REST endpoints, content-addressable storage, and a clear separation between blobs (the actual data) and manifests (the metadata describing an image). But implementing it revealed edge cases and protocol quirks that aren’t obvious from reading the spec.

What docker push actually does

When you run docker push myregistry.io/myapp:latest, here’s what happens at the HTTP level:

1. Version check. The client hits GET /v2/. If the server responds with 200 and a Docker-Distribution-Api-Version: registry/2.0 header, the client knows it’s talking to a v2 registry. If it gets a 401, it reads the WWW-Authenticate challenge header and follows the auth flow: either Basic auth or Bearer token exchange.

2. Check which blobs already exist. Before uploading anything, the client sends HEAD /v2/<name>/blobs/sha256:<digest> for every layer and the config blob. If the server responds with 200, the client skips that blob entirely. This is the main optimization: if you’re pushing a new tag of an image that shares most layers with a previous push, only the changed layers get uploaded.

3. Upload blobs. For each blob that doesn’t exist, the client has two options:

Monolithic upload: send the entire blob in a single POST /v2/<name>/blobs/uploads/?digest=sha256:... request. Simple, but only works for smaller blobs.

Chunked upload, three steps:

4. Upload manifest. PUT /v2/<name>/manifests/<tag> with the JSON manifest body. The manifest references the config blob and layer blobs by digest. The server stores it, computes its own digest, and returns Docker-Content-Digest in the response header.

docker pull is the reverse: resolve the manifest by tag, then download each referenced blob.

The protocol surprises

The client drops auth headers during chunked uploads

This one cost me time. The Docker client authenticates the initial POST to start an upload session, but then does NOT send the Authorization header on subsequent PATCH, GET, or PUT requests for that same upload. It considers the session authenticated by the initial request.

This means the server has to skip authentication checks for upload continuation endpoints. If you enforce auth on every request (which seems like the obvious secure default), chunked uploads silently fail. The client gets a 401 on the first PATCH and gives up.

The finalize PUT can carry data

PUT /v2/<name>/blobs/uploads/<uuid>?digest=... isn’t just a “seal the upload” command. It can include a request body with the final chunk of data, or even the entire blob if the client decides to use the session flow but send everything in one shot. You have to check for Content-Length and Content-Type on the finalize request and write any included data before computing the hash.

Tags are mutable, digests are not

Tags like latest or v1.0 can be reassigned to point to different images over time. Digests (sha256:abc123...) are permanent. This means pushing myapp:latest twice with different content is valid. The second push replaces the first. Our artifact model handles this by looking up artifacts by (name, version) at the repository level and destroying and recreating all the asset links when a tag is re-pushed.

Config digest determines image identity, not manifest digest

This was a subtle design decision. Two manifests can have different byte representations (different whitespace, different field ordering) but reference the same config and the same layers. They’re functionally the same image. Docker itself uses the config digest, the SHA256 of the image configuration JSON, as the identity. So our Component model is keyed by config digest, not manifest digest. Multiple tags can point to different manifests that all share the same component.

Route format detection and dots in tags

Tags can contain dots: v1.0.0, 3.14-alpine. Rails interprets a dot in the URL path as a format separator, so /v2/myapp/manifests/v1.0.0 would try to render v1.0.0 as format 0 and reference v1.0. We had to disable format detection on the manifest routes with format: false.

Content-addressable storage

The storage model maps cleanly to the Docker world:

A Blob is identified by its SHA256 hash. Two identical blobs, even from different images and different repositories, are stored once. The storage backend uses two-level directory sharding (ab/cd/abcdef1234...) to avoid filesystem issues from too many files in one directory.

An Asset is a named reference to a Blob within a repository. The manifest lives at <name>/manifests/sha256:<digest>. Layer blobs live at blobs/sha256:<digest>. Upload sessions use temporary assets at uploads/<uuid>/data and uploads/<uuid>/info.json.

An Artifact is a tag: latest, v1.0, alpine. It links to a set of assets (the manifest, the config blob, each layer blob) through a join table.

The deduplication pays off immediately. Push ten images that share a common base layer, and that layer is stored once. This was one of our main complaints about Nexus. Docker image layers ate disk space because there was no meaningful deduplication.

The append problem

Chunked uploads revealed an interesting tension with content-addressable storage. When a client sends chunks, we need to append data to a temporary blob. But our blobs are immutable, identified by their SHA256, which changes when the content changes. So every PATCH (chunk upload) actually creates a new blob: read the old data, append the new chunk, compute a new hash, store the new blob. The old intermediate blob becomes an orphan.

This preserves the immutability invariant (no blob is ever modified in place) but it means chunked uploads create garbage. The cleanup job finds blobs with no asset references and removes them. It’s the right trade-off: immutable storage with periodic cleanup is simpler and more reliable than mutable storage with locking.

First successful docker push

It works. docker push flux.kuy.io/sirius2:latest pushes layers, uploads the manifest, and the image shows up in the Flux UI with its tag, layer count, and total size. docker pull brings it back. The integration with our existing blob store means Docker images and Ruby gems share the same storage infrastructure, the same quota management, the same cleanup jobs.

The Nexus replacement now handles both of our primary artifact types. The migration is getting closer.