Catching Secrets Before They Land

rubysecuritytoolingbuilding-in-public

Someone will eventually push a secret to a repository. An AWS access key in a config file. A Stripe API key in a test fixture. A private key someone generated for local development and forgot to gitignore. It happens. The question is whether you catch it before it hits the remote or after it’s baked into the git history where removing it is a painful, incomplete process.

This is one of the reasons we’re building Codex. Not just git hosting, but git hosting with policy enforcement at the push level.

How push policies work

When someone pushes to a Codex repository, three hooks fire in sequence:

pre-receive runs before any refs are updated. It reads the incoming ref updates (old commit, new commit, branch name) and sends them to Codex’s internal API. Codex runs every configured policy against the push. If any policy fails, the entire push is rejected. The developer sees the error message in their terminal, and nothing is written to the remote repository.

update runs once per ref being updated. This is where per-branch policies live: no force push to main, no tag deletion, no branch deletion on protected branches.

post-receive runs after refs are updated. It never blocks the push. It triggers async work like signature verification, activity feed updates, and PR status checks.

The pre-receive hook is where secret detection lives, because that’s the only point where we can reject a push before the data enters the repository.

The secrets detector

The detector uses two complementary approaches:

Pattern matching catches known credential formats. AWS access keys start with AKIA. Google API keys start with AIza. Slack tokens start with xox. Stripe keys start with sk_live_ or pk_live_. PEM-encoded private keys have BEGIN RSA PRIVATE KEY or BEGIN EC PRIVATE KEY headers. JWT tokens have the characteristic eyJ base64 prefix. Password assignments in code match patterns like password = "..." or PASSWORD: "...".

We have about a dozen pattern categories. Each is a regex, and each produces a specific error message so the developer knows exactly what was flagged and why.

Shannon entropy catches secrets that don’t match known patterns. A random 40-character API key doesn’t look like an AWS key or a Stripe key, but it has high entropy, meaning high randomness per character. Normal code, comments, and identifiers have lower entropy. We compute Shannon entropy for each token on each added line and flag anything above a configurable threshold (default: 4.5 bits per character).

The entropy approach has false positives. Base64-encoded content, UUIDs, and long hash strings all have high entropy. But as a second line of defense behind pattern matching, it catches credentials that wouldn’t match any specific pattern.

What we scan

Only added lines in the diff. We compute the diff between the old and new commit for each ref update, extract the + lines (additions), and scan those. Unchanged and deleted lines are ignored because we’re looking for secrets being introduced, not secrets that already exist in history.

This is a deliberate scope decision. Scanning the entire repository on every push would be expensive and would block pushes for problems that predate the current push. The pre-receive hook needs to be fast because developers are waiting for it to complete.

The full policy set

Secret detection is one of ten pre-receive policies Codex supports:

Plus five update-level policies for ref management: no force push, no tag modification, no tag deletion, no branch deletion, no branch creation.

Each policy is configured per-repository with a ref pattern, so you can enforce signed commits on main and release/* without requiring them on feature branches. Entities with admin permission or explicit bypass permission can skip policy checks.

The debugging experience

Getting hooks right was its own journey. The hook scripts are bash, generated by Codex and placed in the bare repository’s hooks/ directory. They curl back to localhost with the push data and the identity of the pusher (passed through CGI environment variables from the HTTP backend).

Debugging a curl command inside a bash script that’s invoked by a CGI process that’s piped through a Rails controller is… not great. The failure modes are silent. A malformed curl command means the hook exits with an error, git reports “remote rejected,” and the developer has no idea why. Error messages need to be precise because the developer only sees what git prints to their terminal.

We ended up logging every hook invocation with the full request and response, which made debugging manageable. But the first few iterations of the hook scripts had typos, wrong variable names, and malformed JSON payloads that took longer to find than they should have.

Why this matters

Git history is append-only by design. Once a secret is pushed to a remote, it’s in the object store. You can remove it from the current tree, but it’s still reachable through the reflog, through old commits, through tags. git filter-branch or git filter-repo can rewrite history, but that’s destructive, requires force-pushing, and doesn’t guarantee the secret hasn’t been cloned or cached somewhere.

Catching secrets in pre-receive is the only clean solution. The push is rejected. The data never enters the repository. The developer fixes it locally and pushes again. No history rewriting, no panic, no incident.

That’s worth the investment in getting the hooks right.