Ruby - Feature #21795: Methods for retrieving ASTs</h1> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2025-12-19T09:52:08Z</p> <ul></ul><p>I anticipated that we would consider this eventually, but incorporating it into the core presents significant challenges.</p> <p>Here are two major issues regarding feasibility.</p> <p>(Based on chats with <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/17">@ko1 (Koichi Sasada)</a>, <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/10330">@tompng (tomoya ishida)</a>, and <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/7872">@yui-knk (Kaneko Yuichiro)</a>, though these are my personal views.)</p> <a name="The-Implementation-Approach"></a> <h2 >The Implementation Approach<a href="#The-Implementation-Approach" class="wiki-anchor">¶</a></h2> <p>CRuby currently discards source code and ASTs after ISeq generation. The proposed <code>#ast</code> method would have to re-read and re-parse the source, which causes two problems:</p> <ol> <li>If the file is modified after loading, <code>#ast</code> may return the wrong node.</li> <li>It does not work for <code>eval</code> strings.</li> </ol> <p><code>error_highlight</code> accepts this fragility because it displays just "hints". But I don't think that it is allowed for a built-in method. At least, we must avoid returning an incorrect node, and clarify when failures occur.</p> <p>I propose two approaches:</p> <ol> <li>Keep loaded source in memory (e.g., <code>RubyVM.keep_script_lines = true</code> by default). This supports <code>eval</code> but increase memory usage.</li> <li>Validate source hash. Store a hash in the ISeq and check it to ensure the file hasn't changed.</li> </ol> <a name="The-Parser-Switching-Problem"></a> <h2 >The Parser Switching Problem<a href="#The-Parser-Switching-Problem" class="wiki-anchor">¶</a></h2> <p>What is the node definition returned by <code>#ast</code>?</p> <p>As noted in <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: Allow to use the build-in prism version to parse code (Closed)" href="https://bugs.ruby-lang.org/issues/21618">#21618</a>, built-in Prism is not exposed as a Ruby API. If <code>Gemfile.lock</code> specifies an older version of prism gem, even <code>require "prism"</code> won't provide the expected definition.</p> <p>IMO, it would be good to have a node definition that does not depend on prism gem (maybe <code>Ruby::Node</code>?). I am not sure how much effort is needed for this. We would also need to consider where to place what in the ruby/prism and ruby/ruby repositories for development.</p> <p>We also need to decide if <code>#ast</code> should return <code>RubyVM::AST::Node</code> when <code>--parser=parse.y</code> is specified.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-01-06T21:07:55Z</p> <ul></ul><p>I think this would be great to have, and abstract over implementation details like <code>node_id</code>.<br> It's also very powerful as e.g. <code>Thread::Backtrace::Location#ast</code> would be able to return a <code>Prism::CallNode</code> with all the relevant information, which <code>error_highlight</code> and others could then use very conveniently.</p> <p>mame (Yusuke Endoh) wrote in <a href="#note-1">#note-1</a>:</p> <blockquote> <a name="The-Implementation-Approach"></a> <h2 >The Implementation Approach<a href="#The-Implementation-Approach" class="wiki-anchor">¶</a></h2> </blockquote> <blockquote> <p><code>error_highlight</code> accepts this fragility because it displays just "hints".<br> But I don't think that it is allowed for a built-in method. At least, we must avoid returning an incorrect node, and clarify when failures occur.</p> </blockquote> <p>I think a built-in method doesn't imply it must work perfectly, it's e.g. not really possible to succeed when the file is modified (without keeping the source in memory).</p> <blockquote> <p>I propose two approaches:</p> <ol> <li>Keep loaded source in memory (e.g., <code>RubyVM.keep_script_lines = true</code> by default). This supports <code>eval</code> but increase memory usage.</li> </ol> </blockquote> <p>I think we could keep only <code>eval</code> code (IOW, non-file-based code) in memory, then the memory increase wouldn't be so big, and it would work as long as the files aren't modified.<br> Keeping all code in memory would be convenient, and interesting to measure how much an overhead it is (source code is often quite a compact representation actually), but I suspect given the general focus of CRuby on memory footprint it would be considered only as last resort.</p> <blockquote> <ol start="2"> <li>Validate source hash. Store a hash in the ISeq and check it to ensure the file hasn't changed.</li> </ol> </blockquote> <p>This is a great idea, it should be reasonably fast and avoid the pitfall about modified files.<br> Although modified files are probably very rare in practice, so I'm not sure how much this matters, but it does seem nicer to fail properly than potentially return the wrong node.</p> <blockquote> <a name="The-Parser-Switching-Problem"></a> <h2 >The Parser Switching Problem<a href="#The-Parser-Switching-Problem" class="wiki-anchor">¶</a></h2> <p>What is the node definition returned by <code>#ast</code>?</p> </blockquote> <p>A <code>Prism::Node</code> because <a href="https://railsatscale.com/2024-04-16-prism-in-2024/" class="external">Matz has agreed that going forward the official parser API for Ruby will be the Prism API.</a></p> <p>Actually more specific nodes where known:</p> <ul> <li> <code>Proc#ast</code>: <code>LambdaNode | CallNode | ForNode | DefNode</code> (<code>DefNode</code> because <code>Method#to_proc</code>)</li> <li> <code>Method#ast</code> & <code>UnboundMethod#ast</code>: <code>DefNode | LambdaNode | CallNode | ForNode</code> (block-related nodes because <code>define_method(&Proc)</code>)</li> <li> <code>Thread::Backtrace::Location#ast</code> & <code>TracePoint#ast</code>: <code>Call*Node | Index*Node | YieldNode</code>, and maybe a few more.</li> </ul> <blockquote> <p>As noted in <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: Allow to use the build-in prism version to parse code (Closed)" href="https://bugs.ruby-lang.org/issues/21618">#21618</a>, built-in Prism is not exposed as a Ruby API. If <code>Gemfile.lock</code> specifies an older version of prism gem, even <code>require "prism"</code> won't provide the expected definition.</p> </blockquote> <p>This is basically a solved problem, as discussed there.<br> In that case, <code>Prism.parse(foo, version: "current")</code> fails with a clear exception explaining one needs to use a newer prism gem.<br> This only happens if one explicitly downgrades the Prism gem, which is expected to be pretty rare.</p> <blockquote> <p>IMO, it would be good to have a node definition that does not depend on prism gem (maybe <code>Ruby::Node</code>?). I am not sure how much effort is needed for this. We would also need to consider where to place what in the ruby/prism and ruby/ruby repositories for development.</p> </blockquote> <p>IMO there should only be <code>Prism::Node</code>, otherwise tools would have to switch between two APIs whether they want to use the current-running syntax or another syntax.<br> From the discussion in <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: Allow to use the build-in prism version to parse code (Closed)" href="https://bugs.ruby-lang.org/issues/21618">#21618</a> my take away is it's unnecessary to have a different API.</p> <blockquote> <p>We also need to decide if <code>#ast</code> should return <code>RubyVM::AST::Node</code> when <code>--parser=parse.y</code> is specified.</p> </blockquote> <p>It must not, the users of these new methods expect a <code>Prism::Node</code>.<br> Matz has said the official parser API for Ruby is the Prism API, so it doesn't make sense to return <code>RubyVM::AST::Node</code> there.<br> Also that wouldn't be reasonable when considering alternative Ruby implementations.</p> <p>This does mean these methods wouldn't work with <code>--parser=parse.y</code>, until <code>parse.y</code> can be used to create a <code>Prism::Node</code> AST.<br> Since that's the official Ruby parsing API, it's already a goal anyway for <code>parse.y</code> to do that (<a class="issue tracker-5 status-6 priority-4 priority-default closed" title="Misc: Status of the universal parser implementing the Prism API (Rejected)" href="https://bugs.ruby-lang.org/issues/21825">#21825</a>), so that shouldn't be a blocker.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-01-07T11:41:09Z</p> <ul></ul><p>One idea to make it work with <code>--parser=parse.y</code> until universal parser supports the Prism API (<a class="issue tracker-5 status-6 priority-4 priority-default closed" title="Misc: Status of the universal parser implementing the Prism API (Rejected)" href="https://bugs.ruby-lang.org/issues/21825">#21825</a>) would be to:</p> <ul> <li>Get the <code>RubyVM::AST::Node</code> of that object, then extract the start/end line & start/end columns. Or do the same internally without needing a <code>RubyVM::AST::Node</code>, it's just converting from parse.y node_id to "bounds".</li> <li>With those values, use something similar to <a href="https://github.com/ruby/prism/pull/3808" class="external">Prism.node_for</a> to find a node base on those bounds.</li> <li>There should be a single node matching those bounds because we are only looking for specific nodes.</li> </ul> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-01-07T15:24:51Z</p> <ul></ul><p>Thanks <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/18">@mame (Yusuke Endoh)</a> for the detailed reply! I appreciate your thoughtfulness here.</p> <p>With regard to the implementation approach problem, I love your solution of keeping a source hash on the iseq. I think that makes a lot of sense, and could be used in error highlight today as well. That could potentially even be used by other tools in the case of code reloading. I think we <em>could</em> potentially store the code for eval, but I would be tempted to say let us not change anything for now and return <code>nil</code> or raise an error in that case. (In the same way we would need to return <code>nil</code> or raise an error for C methods.)</p> <p>For the parser switching problem, I think I would like to introduce a Prism ABI version (alongside the Prism gem version). I would update this version whenever a structural change is made (field added/renamed/removed/etc.). Then, if we could store the Prism ABI version on the ISEQ as well, we could require prism and check if the ABI version matches before attempting to re-parse. We could be clear through the error message that the Prism ABI version is a mismatch and therefore we cannot re-parse.</p> <p>I am not sure if we should return RubyVM::AST nodes in the case the ISEQ was compiled with parse.y/compile.c, but I am okay with it if that's the direction you would like to go.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-01-08T21:01:04Z</p> <ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-1 priority-4 priority-default" href="https://bugs.ruby-lang.org/issues/21826">Feature #21826</a>: Deprecating RubyVM::AbstractSyntaxTree</i> added</li></ul> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-01-14T09:30:49Z</p> <ul><li><strong>Related to</strong> deleted (<i><a class="issue tracker-2 status-1 priority-4 priority-default" href="https://bugs.ruby-lang.org/issues/21826">Feature #21826</a>: Deprecating RubyVM::AbstractSyntaxTree</i>)</li></ul> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-01-14T09:30:58Z</p> <ul><li><strong>Blocks</strong> <i><a class="issue tracker-2 status-1 priority-4 priority-default" href="https://bugs.ruby-lang.org/issues/21826">Feature #21826</a>: Deprecating RubyVM::AbstractSyntaxTree</i> added</li></ul> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-01-16T09:14:11Z</p> <ul></ul><p>Rails' <code>_callable_to_source_string</code> would be a good use case for this, see <a href="https://github.com/rails/rails/pull/56624" class="external">https://github.com/rails/rails/pull/56624</a></p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-02-13T05:15:19Z</p> <ul></ul><p>Eregon (Benoit Daloze) wrote in <a href="#note-2">#note-2</a>:</p> <blockquote> <blockquote> <p>As noted in <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: Allow to use the build-in prism version to parse code (Closed)" href="https://bugs.ruby-lang.org/issues/21618">#21618</a>, built-in Prism is not exposed as a Ruby API. If <code>Gemfile.lock</code> specifies an older version of prism gem, even <code>require "prism"</code> won't provide the expected definition.</p> </blockquote> <p>This is basically a solved problem, as discussed there.<br> In that case, <code>Prism.parse(foo, version: "current")</code> fails with a clear exception explaining one needs to use a newer prism gem.</p> </blockquote> <p>I believe <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: Allow to use the build-in prism version to parse code (Closed)" href="https://bugs.ruby-lang.org/issues/21618">#21618</a> primarily discusses released Ruby versions. My concern is specifically about the behavior on the master branch.</p> <p>When new syntax is introduced to the Ruby master branch, the built-in <code>prism.c</code> is updated immediately. In this scenario, if we attempt to retrieve <code>#ast</code> using the node definitions from a released prism gem, I am concerned that we will not get a correct AST due to the node definition mismatch.</p> <p>kddnewton (Kevin Newton) wrote in <a href="#note-4">#note-4</a>:</p> <blockquote> <p>For the parser switching problem, I think I would like to introduce a Prism ABI version (alongside the Prism gem version). I would update this version whenever a structural change is made (field added/renamed/removed/etc.). Then, if we could store the Prism ABI version on the ISEQ as well, we could require prism and check if the ABI version matches before attempting to re-parse. We could be clear through the error message that the Prism ABI version is a mismatch and therefore we cannot re-parse.</p> </blockquote> <p>While this is certainly a feasible solution, I don't feel it is the optimal one.<br> I acknowledge the engineering challenges involved, but ideally, I believe having a built-in node definition (like <code>Ruby::Node</code>) within Ruby core itself would be the simplest and best approach.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-02-14T17:58:04Z</p> <ul></ul><p>Would a Ruby::Node be the same thing as a Prism::Node? As in, would it basically be a Ruby API that duplicates the Prism interface?</p> <p>I'm not sure about how to maintain it. For example, if we add more features to Prism's Ruby API (for example the work we've been doing on the translation layers to Ripper recently) would we also duplicate it to the various live branches of the Ruby::Node API? Or would it just be a trimmed down version? Either way, I'm not sure when I would recommend using Ruby::Node, because it seems like it would always be an out-of-date version of Prism::Node.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-03-17T07:43:05Z</p> <ul></ul><p>I have two concerns before we move forward.</p> <p>On the name AST</p> <p>I'm not sure <code>ast</code> is the right name. The nodes returned by Prism retain concrete information such as positions, whitespace, and comments, making them closer to a Concrete Syntax Tree than an Abstract Syntax Tree. A name like <code>node</code> or <code>syntax_tree</code> might be more accurate.<br> On the ABI version approach.</p> <p>Embedding a Prism ABI version in the ISeq sounds reasonable at first, but I'm worried it would make these methods reliably broken during active development on master — any time prism.c is updated ahead of a released gem, callers would get nil or an exception as a matter of course. That's a poor developer experience for people working on master. This concern points back to the suggestion from <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/18">@mame (Yusuke Endoh)</a>: perhaps we need the built-in Prism to be exposed as a gem-independent API first, before we can ship these methods in a stable way.</p> <p>I'm positive about the overall direction. I just want to make sure we resolve these two points before committing to the API shape.</p> <p>Matz.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-03-17T09:46:32Z</p> <ul></ul><p>mame (Yusuke Endoh) wrote in <a href="#note-9">#note-9</a>:</p> <blockquote> <p>When new syntax is introduced to the Ruby master branch, the built-in <code>prism.c</code> is updated immediately. In this scenario, if we attempt to retrieve <code>#ast</code> using the node definitions from a released prism gem, I am concerned that we will not get a correct AST due to the node definition mismatch.</p> </blockquote> <p>AFAIK the parser and node definitions always match.<br> If the node definitions need changes, they would be updated at the same time than <code>prism.c</code>.<br> If updating node definitions is somehow forgotten, then it needs to be fixed anyway (regardless of this issue), but it will only result in e.g. not having a new node field yet, not a big deal.<br> As such there will never be "an incorrect AST".</p> <p>The scenario you mention would <em>only</em> be an issue when all of these are the case:</p> <ul> <li>Using ruby-master and not a release</li> <li>Using <code>#ast</code> on a file which uses new syntax not in the latest prism release (very rare already)</li> <li>Using Bundler (just using RubyGems would pick the prism default gem which has the latest syntax changes)</li> <li>In Gemfile, depending on a release version of prism and not using <code>bundle install --prefer-local</code> (which would pick the prism default gem)</li> </ul> <p>This seems such a rare case, and there are solutions like <code>bundle install --prefer-local</code> for those cases, or releasing prism (e.g. if new syntax is being adopted quickly and widely).<br> In such a case, it wouldn't return an incorrect AST, it would raise a SyntaxError for the new syntax being used and not being recognized (or a <code>Prism::CurrentVersionError</code> if using an old Prism release).<br> Isn't that good enough?</p> <p>I think it's worth highlighting that users of Ruby releases wouldn't have this problem at all, they would just need to depend on a recent enough <code>prism</code>, which is already a requirement and is fine for the many existing usages of Prism.</p> <hr> <p>If we do want to raise when using an older Prism, one idea here is: <code>Method#ast</code> would do <code>require "prism"</code> and after that <code>require</code> it would check that <code>Prism::VERSION >= EMBEDDED_PRISM_VERSION</code> (i.e. the version of Prism used by the interpreter to parse). If not, raise an exception.<br> It's similar to the Prism ABI idea but simpler and doesn't require to maintain such an ABI version manually.</p> <p>One change we would need is to bump to the next version immediately in ruby/prism whenever doing a release, so e.g. on master the prism gem would be reported as 1.10.0 (or 1.10.0.dev to be more explicit), and not as 1.9.0 (which is the current latest release). That way the last release would be considered incompatible since there might be syntax changes since then.</p> <hr> <p>matz (Yukihiro Matsumoto) wrote in <a href="#note-11">#note-11</a>:</p> <blockquote> <p>I'm not sure <code>ast</code> is the right name. The nodes returned by Prism retain concrete information such as positions, whitespace, and comments, making them closer to a Concrete Syntax Tree than an Abstract Syntax Tree. A name like <code>node</code> or <code>syntax_tree</code> might be more accurate.</p> </blockquote> <p>The <code>parser</code> and <code>ast</code> gems and <code>RubyVM::AbstractSyntaxTree</code> call them "AST" and they also have positions.<br> I'm not sure what is meant by whitespace, Prism itself doesn't return objects for whitespace (even in the lexer).<br> Regarding comments, those would be ignored for these new methods as they would return a <code>Prism::Node</code>, not a <code>Prism::ParseResult</code>, but even then I think it's a minor detail.<br> Prism is also not a pure Concrete Syntax Tree as e.g. postfix-if and regular <code>if</code> are both <code>Prism::IfNode</code>.<br> And it's clearly used successfully as an abstract syntax tree in Ruby implementations.</p> <p>I think <code>ast</code> is the name most Rubyists would expect.</p> <p><code>syntax_tree</code> sounds fine to me too if <code>ast</code> is not acceptable.<br> <code>node</code> sounds rather ambiguous to me.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-04-05T13:46:20Z</p> <ul></ul><p>I thought more about <code>node_id</code> and I found at least one case where it is problematic with different versions of Prism.<br> The problem is the <code>node_id</code> from the bytecode is computed by the builtin Prism parser used by <code>prism_compile.c</code>, while the usage of e.g. <code>Prism.find</code> might use a more recent Prism.<br> Here is a reproduction showing the problem:</p> <pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">if</span> <span class="kp">false</span> <span class="c1"># A code snippet which generates a different number of nodes on Prism 1.2.0 and 1.9.0</span> <span class="k">case</span> <span class="mi">1</span> <span class="k">in</span> <span class="mi">2</span> <span class="no">A</span><span class="p">.</span><span class="nf">print</span> <span class="ss">message: </span><span class="k">in</span> <span class="mi">3</span> <span class="no">A</span><span class="p">.</span><span class="nf">print</span> <span class="ss">message: </span><span class="k">end</span> <span class="k">end</span> <span class="k">def</span> <span class="nf">a</span> <span class="k">end</span> <span class="k">def</span> <span class="nf">b</span> <span class="k">end</span> <span class="nb">require</span> <span class="s2">"prism"</span> <span class="nb">p</span> <span class="no">Prism</span><span class="p">.</span><span class="nf">find</span><span class="p">(</span><span class="nb">method</span><span class="p">(</span><span class="ss">:b</span><span class="p">))</span> </code></pre> <p>ruby master is fine:</p> <pre><code>$ ruby -v find_check.rb ruby 4.1.0dev (2026-03-27T16:16:27Z revert-source_loca.. f510d4103e) +PRISM [x86_64-linux] @ DefNode (location: (14,0)-(15,3)) ├── flags: newline ├── name: :b </code></pre> <p>But on Ruby 3.4.5 it's broken:</p> <pre><code># Use latest prism, there is no prism release with Prism.find yet: $ cd prism $ bundle exec rake compile $ bundle exec ruby find_check.rb @ DefNode (location: (11,0)-(12,3)) ├── flags: newline ├── name: :a </code></pre> <p>This returns method <code>a</code> and not <code>b</code>!</p> <p>And if I change <code>p Prism.find(method(:b))</code> to <code>p Prism.find(method(:a))</code>,<br> then ruby-master is correct, but 3.4.5 returns an <code>IfNode</code>.</p> <p>IOW, 3.4.5 returns the wrong node.</p> <p>I think this illustrates well that <code>node_id</code> is brittle, it depends on the number of nodes before the node of interest.</p> <p>OTOH, start line/column + end line/column (or equivalently, start & end offsets) is far more robust, because it represents an actual position in the source file, independent of the Prism version.<br> The only confusion there would be if there are nodes with exactly the same start & end offsets, and they can't be differentiated based on the input.<br> That's not a problem for the 5 methods proposed in this issue:</p> <p>This relates to the discussion <a href="https://bugs.ruby-lang.org/issues/6012#note-47" class="external">here</a> with <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/18">@mame (Yusuke Endoh)</a> about whether <code>node_id</code> is better, but from this finding it's clear <code>node_id</code> is worse if the version isn't fixed.<br> I'll quote here the relevant part about <code>source_location</code> being able to locate the right node:</p> <blockquote> <blockquote> <p>However, <code>source_location</code> is not an appropriate key to look up the AST subtree corresponding to a Ruby object.</p> </blockquote> </blockquote> <blockquote> <p>I believe it is though, with the knowledge of what kind of node we are looking for.<br> For example in <code>def foo; bar; end</code>, <code>bar</code> in the AST is covered exactly by both a <code>StatementsNode</code> and a <code>CallNode</code>.<br> If we are using <code>Thread::Backtrace::Location#ast</code> we'd want the location of the call to <code>bar</code>, so we know we want the <code>CallNode</code>, not the <code>StatementsNode</code> and there is no ambiguity.<br> I believe the same holds for all 5 methods proposed in <a class="issue tracker-2 status-1 priority-4 priority-default" title="Feature: Methods for retrieving ASTs (Open)" href="https://bugs.ruby-lang.org/issues/21795">#21795</a>.<br> My intuition there is all nodes listed in <a href="https://bugs.ruby-lang.org/issues/21795#note-2" class="external">https://bugs.ruby-lang.org/issues/21795#note-2</a> cannot have the exact same <code>source_location</code> (e.g. we cannot have code with the same starting and ending position that is two of <code>DefNode</code>, <code>LambdaNode</code>, <code>ForNode</code>, <code>Call*Node</code>, <code>Index*Node</code>, <code>YieldNode</code>).<br> Do you have a counter-example where this wouldn't hold?</p> </blockquote> <p>No counter-example has been found.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-04-05T13:51:17Z</p> <ul></ul><p>The implications of that are, assuming this proposal is implemented based on start line/column + end line/column (or equivalently, start & end offsets):</p> <ul> <li>No need for <code>node_id</code> to implement this feature</li> <li>This feature works with both <code>--parser=prism</code> and <code>--parser=parse.y</code> </li> <li>This feature works for Ruby implementations which do not have <code>node_id</code> (e.g. TruffleRuby)</li> <li>The Prism version is not a concern, the start/end offsets of such node are stable enough</li> </ul> <p>This makes this proposal simpler, more portable and safer, so I suggest we go this way.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-04-14T05:48:42Z</p> <ul></ul><p>As matz pointed out in <a href="#note-11">#note-11</a>, the ABI versioning approach would leave master in a routinely broken state. As a maintainer of error_highlight, I cannot accept this. Not being able to verify error_highlight's behavior against code using new syntax until the next Prism release would be a serious problem for me.</p> <p>My position is that the underlying assumption itself is untenable: that the parser and node definitions used by the interpreter, and those used by <code>#ast</code>, may legitimately differ. Eregon's example in <a href="#note-13">#note-13</a> is presented as evidence of <code>node_id</code>'s fragility, but I read it instead as a signal that this assumption should be reconsidered.</p> <p>The correct fix, I believe, is to make such divergence structurally impossible. Concretely, this means either integrating the Prism repository into ruby/ruby, or keeping the Prism repository separate but synchronizing the node definitions themselves into Ruby core. When I mentioned <code>Ruby::Node</code> in <a href="#note-9">#note-9</a>, I had the latter in mind.</p> <p>To add a personal note, I find it structurally unnatural that the parser, the component that defines the language's syntax, is primarily developed outside ruby/ruby. Don't get me wrong, I have great respect for Kevin and the Prism team's work. But I believe that unless we take one of the two forms above, it will be difficult to settle the design of <code>#ast</code>.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-04-14T06:12:52Z</p> <ul></ul><p>matz (Yukihiro Matsumoto) wrote in <a href="#note-11">#note-11</a>:</p> <blockquote> <p>I'm not sure ast is the right name. The nodes returned by Prism retain concrete information such as positions, whitespace, and comments, making them closer to a Concrete Syntax Tree than an Abstract Syntax Tree. A name like node or syntax_tree might be more accurate.</p> </blockquote> <p>Might I recommend <code>to_ast</code>? It would be more in line with common Ruby coercion patterns, and would very quickly indicate that it is a coercion method from <code>Object</code> to an AST variation. I fear that <code>node</code> would be overloaded with various graph and tree-like algorithms. Likewise <code>to_syntax_tree</code> may be workable, but I'm still more partial towards <code>to_ast</code>.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-04-14T09:16:54Z</p> <ul></ul><blockquote> <p>Concretely, this means either integrating the Prism repository into ruby/ruby</p> </blockquote> <p>I don't think that would be a very good solution, prism is not only the parser as used by CRuby. It has bindings to other languages (rust, javascript, java). Also the various translators for previous ruby syntax parser gems. Then there's integration with other other runtimes like jruby and truffleruby that also live in prism. These are all equally tied to the prism version, same as CRuby.</p> <p>As an example the java integration with jruby/truffleruby has seen much activity recently. It is good that this is not happening in ruby/ruby since it's not relevant and also would make it more difficult for them.</p> <blockquote> <p>To add a personal note, I find it structurally unnatural that the parser, the component that defines the language's syntax, is primarily developed outside ruby/ruby</p> </blockquote> <p>The C library does not have much connection to ruby (if you ignore the one big point that it is a parser for the syntax) and can exist without it. It's one of the main reasons why it sees such big adoption.</p> <p>It's true that they are tightly integrated and don't make sense in isolation but it's not necessary to drive development exclusively in ruby/prism. For me it is a preference since I have no permissions on ruby/ruby but it is also more managable since it's a smaller project overall. Ruby also does not run all the tests, either because they rely on external gems like <code>parser</code> or because they just aren't synced (intentionally).</p> <p>But especially changes that tweak behaviour in some way tend to be done in ruby/ruby first and later synced back instead of the other way around (failing syntax tests, prism_compile.c changes, etc.)</p> <hr> <p>Anyways, I don't think moving prism entirely into ruby/ruby would really change anything. The main problem is the mismatch between gem and standard version. You can move development into ruby/ruby but the code is already synced anyways and users can still make use of prism the gem, which gives you the same problem. As long as the prism version that ruby ships with is not used (or any other of the proposed solutions), you do not gain much from it. And solving it that way doesn't require such a drastic change. In the end you need to integrate <em>something</em> but there's nothing stopping you from doing that today already.</p> <p>It would only really work if there is no prism gem that users can cause a mismatch with, so to me it sounds more like you are arguing for <code>Ruby::Node</code> instead.</p> <blockquote> <p>Either way, I'm not sure when I would recommend using Ruby::Node, because it seems like it would always be an out-of-date version of Prism::Node.</p> </blockquote> <p>It would fill the gap where <code>ripper</code> is currently used. It's always exactly what ruby uses and it's clear that there's use-cases for it, especially in connection with <code>node_id</code>. Not many would need it, prism the gem is plenty for majority of the cases but it would indeed be very helpful for usage and exposure in ruby itself.</p> <p>It goes back to <a href="https://bugs.ruby-lang.org/issues/21618" class="external">https://bugs.ruby-lang.org/issues/21618</a> where I was happy with how it is handled when using the gem but with ruby internals it is not always good enough.</p> <hr> <p>More generally about the <code>node_id</code> mismatch from <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/772">@Eregon (Benoit Daloze)</a>. I haven't checked all the cases but it looks like they are all the result of one bug or another in prism where it misparsed the input (some examples are also syntax invalid in earlier prism versions, so I don't think they should be part of the list). I'm not saying that <code>node_id</code> should be considered stable or anything. Just practically it rarely happens, even less so as prism continues to mature and never on code that people actually write. Of course, <code>node_id</code> cannot change for any other reason or risk breaking things.</p> </article> <article> <h1>Ruby - Feature #21795: Methods for retrieving ASTs</h1> <p>2026-04-14T20:00:29Z</p> <ul></ul><p><a class="user active user-mention" href="https://bugs.ruby-lang.org/users/18">@mame (Yusuke Endoh)</a> What do you think about my idea to use start line/column + end line/column (or equivalently, start & end offsets)?<br> AFAIK it solves all problems around this area, it's reliable, works across Prism versions, etc.<br> We could even use <code>RubyVM::AbstractSyntaxTree</code> to get this line & column data on older Ruby versions, so it would work there too.</p> <p>Adding <code>Ruby::Node</code> seems not great to me, notably because it wouldn't be usable for gems needing to support anything older than Ruby 4.1.<br> Also the <code>Ruby::Node</code> API would change without any control from the gem to say e.g. which major version of Prism it wants (well, it could use <code>required_ruby_version</code> but that seems very inconvenient for this purpose).<br> With a dependency on the <code>prism</code> it lets Bundler resolve the version compatible with the various usages (or error if there isn't one).</p> </article> </main></body></html>