TextMate Grammars Are Regex All the Way Down

I already had a tree-sitter grammar for ORB, but most Rails developers use VS Code, and VS Code still uses TextMate grammars for highlighting. Tree-sitter support there is experimental. So I needed to describe the same language a second time, in a completely different formalism.

TextMate grammars are JSON files full of regex patterns. Each pattern matches some syntactic construct and assigns a scope name that color themes map to colors. That’s it. No parse tree, no AST. Just patterns evaluated against each line of text.

For simple languages this works fine. For a template language that nests Ruby expressions inside HTML attributes inside component tags. It’s a fight with regex, and regex doesn’t fight fair.

From Bitbucket to GitHub in half an hour

The extension actually started on our internal Bitbucket server as vscode-ruby-orb. I’d scaffolded it from the VS Code hello-world sample template and built it out privately. When I went to open-source it, I pushed the initial commit to GitHub and immediately noticed the package.json still listed the Microsoft sample repo as the repository URL. Classic. Fixed that, updated the logo, pushed again. Three commits in about half an hour.

The Makefile still has the internal Bitbucket URL in it. I keep meaning to fix that.

I renamed the public repo to just vscode-orb, which felt cleaner, but the language ID stays ruby-orb internally. Changing language IDs breaks anyone who’s already configured editor settings for the language, so you’re kind of stuck with whatever you pick first.

Riding on HTML

The best decision I made was registering ORB as an HTML language participant. One line in the manifest, and VS Code’s HTML language server gives you auto-closing tags, IntelliSense, all the standard HTML features. The TextMate grammar’s root scope inherits from text.html.basic, so all of HTML’s patterns come for free. I just layer the ORB-specific stuff on top.

This makes sense because most of an ORB template is just HTML. The interesting parts, components, expressions, blocks, are the bits that need custom patterns. Everything else is standard tags and attributes.

Nested braces in a world without nesting

TextMate grammars can’t do balanced matching. You can’t say “match {, then anything including more {} pairs, then the matching }.” You can fake it one level deep with a begin/end pattern that allows {...} in the interior, but real Ruby code nests braces as deep as it wants.

The trick, and it is a trick, not a clean solution, is a recursive #expression rule that includes itself. TextMate supports recursive includes inside begin/end patterns. It’s not real parsing and it has depth limits, but in practice it covers the nesting that shows up in actual templates.

The failure mode when this breaks is annoying: highlighting looks fine until you hit a certain depth, then a closing brace gets eaten by the wrong rule, and everything after it is the wrong color. Debugging this means staring at scope inspector output and adjusting pattern precedence until the expression rule’s begin fires before the text rule tries to grab the {. I spent more time on this than anything else in the grammar.

The absurd Unicode regex

The HTML spec defines valid attribute names as “basically any character except some control characters and non-character code points.” To express that as a TextMate regex, you have to explicitly exclude non-character code points across all seventeen Unicode planes. The resulting character class is comically long.

Is anyone going to put a Unicode non-character in an ORB attribute name? No. But the spec says they’re invalid, and accepting them in the grammar would be telling people “this is fine” when some downstream tool will choke on it. I left it in.

cycleTags

The extension has one custom command. Select an expression, hit Cmd+Shift+/, and it cycles through the delimiter types: {{ }} → {% %} → {!-- --} → back to {{ }}. It also converts ERB tags to ORB equivalents.

This came straight out of how I actually edit templates. I’m always changing my mind about whether an expression should produce output or run silently. In ERB you’d manually change <%= %> to <% %>. In ORB you hit a shortcut and it rotates. Small thing, I use it constantly.

Heredoc injection

One thing I want to add soon: highlighting for ORB templates inside Ruby heredocs. ORB supports inline templates: you can define a component’s markup right in the Ruby class with <<~ORB. Without heredoc injection, that inline template is just a grey string blob. With it, you’d get full ORB highlighting embedded in your Ruby file.

The implementation should be a second TextMate grammar that injects into Ruby source files. I’ve seen other extensions do this for SQL and HTML heredocs. It’s on the list.

Two grammars, zero shared code

So now I maintain two completely separate grammars for the same language. Tree-sitter for Zed and Neovim, TextMate for VS Code. They share nothing. Every time I change ORB’s syntax, I update both and check highlighting in both editors.

It’s tedious. It’s also the cost of supporting the editor ecosystem as it actually exists, not as I wish it existed. VS Code will probably switch to tree-sitter eventually, and then I can drop one. Until then, it’s just part of shipping a new language.