Use a flat Python AST in Ruff and ty - #29817

Summary

Ruff and ty parse Python directly into a table of 24-byte AST rows. Generated typed accessors resolve four-byte child handles and borrow fields from the parsed module. Expression and statement consumers pass View<Expr> and View<Stmt> by value and call classify() to match individual variants. Ruff lint rules can materialize a subtree when constructing replacement syntax. ty compacts the backing storage before caching a module and resolves its saved node references directly by arena row. Python AST storage and traversal

Reviewing the change

Start with the architecture guide in commit 2, then review the schema in commit 1 and parser construction in commit 3.

1. Review the field layout and both Python generators. All three generated Rust files, the ty rule reference, and the Cargo lockfile are confined to this commit. Schema and generators
2. Read the architecture guide and follow how concrete nodes widen to expression or statement views, how visitors traverse them, and how field accessors read a row. AST storage and traversal
3. Follow a node from parser construction into a cached module. Parser and cached modules
4. Review how the checker resolves names and how code generation and edits use borrowed nodes. Ruff checker and semantic model
5. Lint rules read typed fields. PT008, NumPy, and Airflow use descendant searches for expressions in a lambda or a try body. Ruff lint rules
6. Formatting rules, comment placement, and range formatting borrow nodes from the parsed module. One AsFormat implementation covers every formattable AST view. Use the flat AST in Ruff’s formatter
7. Follow definitions, scopes, and module resolution through indexed node references. ty semantic index
8. See how type inference and IDE queries read the indexed syntax. ty inference and IDE queries
9. Review the row-indexed binding table and the return and yield walks that can be skipped using the arena. Lint performance
10. An invalid nested pattern reuses the original expression handles so each literal is stored once. Parser error recovery
11. Follow how ty resolves saved row handles after cache eviction, looks up annotations by source position, and preserves dynamic class identities across edits. Remove ty’s node-ID pass

The parser and AST crates build at commit 3, Ruff’s linter at commit 5, and its formatter at commit 6. The full workspace builds from commit 8.

Performance

CodSpeed compared 310009a with 3b5d4a6 and reported 169 improved, 1 regressed, 114 unchanged, and 14 skipped. Pre-parsed linting with all preview rules on numpy/globals.py had 4.17% lower simulated efficiency. Parser peak heap on large/dataset.py fell from 645.3 KB to 513.3 KB (20.5%). Benchmark report

In the earlier comparison below, the local CodSpeed runner executed 1,050,949 instructions on 86b7fd1 and 1,013,610 on d22dfa8 (3.5% fewer). Simulated L1 instruction-cache misses rose from 24,982 to 28,561 (14.3%), while L1 data-read misses fell from 2,631 to 2,236. The two functions that dispatch expression and statement rules grew from 45.5 to 65.8 KiB and 40.9 to 51.8 KiB in these instrumented binaries. Most of the instruction-cache difference was present before the ty node-ID change. Its parent recorded 28,264 misses.

Native Criterion timings for pre-parsed preview lint on one core were inconclusive against the earlier base. Across 24 alternating pairs, d22dfa8 took a median 0.45% less time than 86b7fd1 (95% bootstrap interval, 0.70% less time to 0.13% more). Against the boxed-AST revision 52cca7e, it took 0.68% less time across 18 pairs (95% interval, 0.37% to 1.85% less).

Same-machine CLI measurements

These runs compare d22dfa8 and 52cca7e on the Python 3.12 standard library with four workers pinned to the same four cores. Each command produced identical output before timing. The time change is the median of 33 alternating pairs. The interval is a 95% bootstrap interval over those pairs.

Stripped binary sizes for these builds were Ruff +1.91% (27.50 to 28.02 MiB) and ty -0.49% (35.16 to 34.99 MiB).

Linting an already-parsed module

The repository’s linter benchmark parses the module before the timed batch. Each cell shows the median elapsed-time change over eight alternating pairs of 52cca7e and a4ee55d on one CPU core. Negative values mean less time. Linter benchmark

A separate 24-block run repeated numpy/globals.py on the same core with balanced execution order. The paired time changes and 95% bootstrap intervals were -3.22% (-4.12% to -1.99%) for default rules, +1.53% (+0.66% to +2.13%) for all stable rules, -0.33% (-0.74% to +0.44%) for all rules with preview.

Callgrind counted 0.66% to 2.60% fewer instructions in the pre-parsed, all-stable lint kernel across the five fixtures. On numpy/globals.py, it counted 1.16% fewer instructions and 9.48% more simulated L1 instruction-cache misses. The cache counts come from simulation, while the elapsed times above come from native execution.

ty module cache

Caching an ordinary module trims the parser's buffers and uses the existing arena rows as node IDs. The two source-order lookup tables took 77,580 bytes on the 9,698-row large/dataset.py fixture. The complete cached parse now takes 432,147 bytes, down from 509,719 bytes. Over 60 samples, median cache preparation time on that fixture fell from 117.33 µs to 1.25 µs. Callgrind counted 1,015,995 instructions per call to prepare the arena on a4ee55d and 7,668 on d22dfa8. Node identity

Isolating the node-ID change (a4ee55d to d22dfa8) changed end-to-end ty check time on the standard library by -6.64% (95% interval -10.40% to -3.30%) and peak RSS by -3.36 MiB over 33 pairs. In 32 alternating pairs per project, Pygments 2.17.2 took 7.46% less time (95% interval, 2.49% to 15.96% less). Jinja2 3.1.2 took 1.96% more time (95% interval, 2.09% less to 8.96% more). Median peak RSS fell by 2.55 MiB and 1.79 MiB, respectively. Diagnostics matched in every pair.

Test plan

All 9,464 selected workspace tests passed with the file-watching test binary excluded. Strict workspace Clippy passed across all targets and features. The tests cover resolving handles after cache eviction, quoted annotation key limits, IDE annotation lookup, and dynamic class identity across edits. The recovery change produced identical ASTs and parser errors on 562 inputs, including all 561 Python parser fixtures.