Use parse_lex instead of parse for Ruby and ERB documents

[vinistock]

Motivation

This is part of #1849, which is moving forward now with the immense help of @koic.

This PR switches the LSP from using parse to using parse_lex for Ruby and ERB documents. The difference is that parse only returns the AST nodes, whereas parse_lex returns both the AST + tokenization information. That information is required for RuboCop to use the Prism AST.

Implementation

The LSP itself doesn't need the lex part of the information. We will only need it for RuboCop related things.

So I created an ast method to return the same thing as parse does and fixed all of the call sites.

Note

Parse lex is actually quite a bit slower than regular parse. However, the performance gains we will get from avoiding the double-parse for every diagnostic computation will more than compensate for this price.

We'll monitor our performance telemetry to see if there's any meaningful degradation, but I don't expect that to be the case.

Automated Tests

Updated accordingly.

[vinistock]

• Use parse_lex instead of parse for Ruby and ERB documents #3252 👈 (View in Graphite)
• main

---

How to use the Graphite Merge Queue

Add the label graphite-merge to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.