The Encoding Wall

Wave CI hit its first real wall this week and it was string encoding.

The task execution engine uses PTY.spawn to run commands in a pseudo-terminal and stream output to the browser in real time. This has been working great for our Rails projects: rspec output, rubocop results, Docker build logs, all clean UTF-8. Then someone pointed Wave CI at a project that shells out to a C library during its test suite, and the output contained Latin-1 encoded characters.

PTY.spawn hung. The IO read loop hit a byte sequence that wasn’t valid UTF-8, Ruby’s string handling choked, and the task just… stopped. No error, no timeout, no output. Just a spinner in the browser that never resolved.

The fix was force_encoding('UTF-8') with .scrub to replace invalid byte sequences. Simple once you know what’s happening, maddening to debug when all you see is a task that silently stops producing output. I ended up adding the same fix to the Record update path too. Anywhere build output gets persisted needs to handle encoding defensively.

This is the kind of thing you never think about when you’re building a CI server. Build output is just text, right? Until it isn’t.

Chunked output

The other big change this week: chunked output storage. Up until now, build output was stored as a single text column that grew as the task ran. For short builds, fine. For a Docker multi-stage build that produces tens of thousands of lines, the database row was getting enormous and the Turbo Stream updates were pushing the full content on every append.

Now output is stored as OutputChunk records. Each chunk is a segment of the output stream. The browser receives chunks incrementally. When the task completes, the chunks get rolled up into a single record for archival. Much better for both database performance and real-time streaming latency.

NPM discovery

Also added NPM project discovery this week. Wave CI now detects package.json, checks for yarn.lock or bun.lockb, and runs the appropriate package manager. The discovery plugin order matters. NPM runs early in the chain so dependency installation happens before build steps that might need those packages.

The auto-discovery system is up to eight plugins now: Bundler, NPM, Capistrano, RubyGems, RSpec, RuboCop, Dockerfile, Makefile. For most projects, this means zero configuration. Wave CI figures out what to do by looking at what’s in the repo. That’s still the thing I’m happiest about with this tool.