How RubyGems' Compact Index Actually Works

When you run bundle install, Bundler talks to rubygems.org using a protocol that most Ruby developers never think about. I certainly didn’t. Not until we started building Flux and had to implement the server side from scratch.

Turns out there are actually two protocols, and the edge cases are where things get interesting.

Two protocols, one server

The RubyGems ecosystem has two index formats that a server needs to support:

The compact index is what modern Bundler uses. It’s a plain-text format with three endpoint paths: /names, /versions, and /info/<gem_name>. Bundler hits these in sequence to resolve dependencies. It’s efficient because it only downloads the metadata it actually needs.

The modern index (confusingly named, because it’s actually the older format) is what the gem CLI uses directly. It’s three gzipped Marshal-dumped files: specs.4.8.gz, latest_specs.4.8.gz, and prerelease_specs.4.8.gz. The 4.8 refers to Ruby’s Marshal version. Binary format, loaded all at once.

Bundler tries the compact index first. If the server responds to GET /versions, Bundler uses the compact protocol exclusively. If it gets a 404, it falls back to the Marshal protocol. So a registry server needs to serve both, but the compact index is the one that matters for day-to-day bundle install.

The compact index: three files

/names: the gem list

The simplest file. A header line with a timestamp, a --- separator, then one gem name per line, alphabetically sorted:

created_at: 2025-04-01T00:00:00Z
---
actionpack
activerecord
rails
sirius2

No incremental update strategy. This file gets regenerated from scratch every time a gem is added or removed. It’s small enough that this doesn’t matter.

/versions: the master index

This is where the interesting stuff happens. Each line has a gem name, a comma-separated list of versions, and an MD5 digest:

created_at: 2025-04-01T00:00:00Z
---
actionpack 7.0.0,7.1.0,7.1.1-java a1b2c3d4e5f6
rails 7.0.0,7.1.0 f6e5d4c3b2a1

The MD5 is a hash of the corresponding /info/<gem_name> file’s content. Bundler uses it to decide whether its locally cached copy of the info file is still valid. If the MD5 matches, Bundler skips the request. This is the main performance optimization in the protocol. For a typical bundle install, Bundler only downloads info files for gems whose versions have changed.

Platform variants get appended with a hyphen: 7.1.1-java, 1.16.0-x86_64-linux. The ruby platform is the default and gets omitted, so 7.0.0 means 7.0.0 on the ruby platform.

/info/<gem_name>: per-gem details

One line per version, with dependencies and requirements:

created_at: 2025-04-01T00:00:00Z
---
1.0.0 dep1:>= 1.0&< 2.0,dep2:~> 3.0|checksum:abc123,ruby:>= 2.7.0
1.0.0-java dep1:>= 1.0&< 2.0|checksum:def456
1.1.0 |checksum:789abc

Each line is VERSION DEPENDENCIES|REQUIREMENTS. Dependencies are comma-separated name:constraint pairs. The tricky bit: when a dependency has multiple constraints (like >= 1.0, < 2.0 in a gemspec), the comma is replaced with &. This matters because commas separate different dependencies, so you can’t use them within a single dependency’s constraint. Getting this substitution wrong means Bundler misparses the dependency tree.

Requirements after the | pipe include the SHA-256 checksum of the gem file, and optionally Ruby and RubyGems version requirements. The >= 0 default requirements are omitted to save space.

The gotchas

Negative versions (yanking)

This one surprised me. When a gem version is yanked (removed), the versions file doesn’t get regenerated. Instead, a new line is appended with a minus-prefixed version:

mygem -1.0.0 newmd5hash

The minus sign is a sentinel: it tells Bundler “remove version 1.0.0 from your local cache.” The versions file is append-only. It can only grow, never shrink. This is efficient for the protocol (clients just need to read from where they left off), but it means your versions file accumulates history over time.

In our virtual registry, this created a subtle bug. When merging versions from multiple sources, we have to track which versions have been “negated” so they don’t get re-added from a different source. A version prefixed with - marks it as seen but doesn’t include it in the output.

Compression format mismatch

Here’s one that will waste your afternoon if you don’t know about it. The specs.4.8.gz files use gzip compression (GzipWriter/GzipReader). But individual gemspec files under quick/Marshal.4.8/ use raw Zlib deflate (Deflate/Inflate), with the .rz extension. They are not the same thing. Gzip has headers and checksums wrapping the deflated data. Raw deflate is just the compressed bytes.

Mix these up and you get deserialization errors that look like corrupted data. The Marshal payload is fine. It’s the decompression that’s wrong.

Platform grouping in latest specs

The latest_specs.4.8.gz file should contain only the latest version of each gem. Sounds simple until you consider platforms. nokogiri 1.16.0 might be the latest for the ruby platform, but nokogiri 1.15.0-java might be the latest for java. The latest specs need to group by platform and pick the max version per group, not just the global max.

The info file must update before the versions file

When a new gem is published, the info file gets a new line appended, and the versions file gets updated with the new version list and a new MD5. The order matters: the versions file includes the MD5 of the info file, so the info file must be updated first. Get the order wrong and Bundler sees a new entry in the versions file, fetches the info file, but the MD5 doesn’t match because the info file hasn’t been updated yet. Bundler interprets this as corruption and re-fetches, causing unnecessary network traffic.

Namespace confusion protection

This isn’t a protocol issue. It’s a security concern that the protocol doesn’t address. If you run a private registry behind a virtual endpoint that merges private and upstream sources, an attacker can publish a malicious gem on rubygems.org with the same name as your internal gem but a higher version number. Bundler’s version resolution will prefer the higher version, and suddenly your build is pulling a compromised package from a public source.

Our virtual registry handles this by checking: if a gem name exists in any private repository, all remote versions of that name are excluded entirely. This is the right default for a private registry, and it’s something no artifact server we’ve used handles well out of the box.

Adding Rubygems support to Flux

Flux serves both protocols simultaneously. The compact index uses append-only updates for the versions and info files, with full regeneration available via a reindex command. The modern index loads, modifies, re-marshals, and re-gzips on every update, because there’s no way to append to a Marshal dump.

The versions file for the virtual registry is generated in parallel across four threads, because resolving checksums for thousands of gem names against multiple backend repositories takes time.

The protocol itself is simple once you understand the format. The edge cases are where the real learning happened: negative versions, compression formats, platform grouping, update ordering. None of this is well-documented. We figured most of it out by reading the rubygems.org source code and the Bundler compact index client.