Skip to content

Incremental parser#3011

Open
dantleech wants to merge 10 commits intomasterfrom
incremental-parser
Open

Incremental parser#3011
dantleech wants to merge 10 commits intomasterfrom
incremental-parser

Conversation

@dantleech
Copy link
Collaborator

@dantleech dantleech commented Dec 30, 2025

This PR attempts to introduce a level of incremental parsing for the Tolerant PHP Parser.

Currently:

  • The user types $, a, b and c.
  • IDE sends 4 completion requests to Phpactor
  • Phpactor will re-prase the document for each request

With this change two strategies are employed to avoid reparsing:

In Token Changes

  • The user types $, a, b and c
  • Phactor sees that the c edit is within a VariableName token and that
    when the edit is applied to the previous token we go from $ab to $abc
    and $abc is also a variable name token so we can just update it.

Compound Node Edits

  • User changes a statement within a compound node (i.e. within { } braces)
  • Phactor sees this is a change to a statement
  • Extracts the text from where the edited statement starts
  • Transposes the text edit to the extracted text
  • Applies the text edit
  • Grafts the newly parsed AST into the old tree
  • Reindexes the tokens.

Affect

In certain situations this can result in a 2x or more speed increase, especially in large, complex, files because:

  • Parsing time is 10x faster if an incremental update can be found.
  • The unchanged node instances remain the same, meaning the cache can kick-in for type infrormation.

Things become more difficult if more context is required (e.g. public function foo is meaningless without the surrounding class construct) but in those cases they could, I suppose, be artificially supported.


TODO:

  • Document cache no longer cleared on update - document cache is used for diagnostics, so need to use separate caches.
  • WorkspaceListner
  • Diagnostics are no longer cleared on update (they were cleared via. the document cache clearing)
  • Document cache not purged on goto definition (... ? e.g. after generaitng a missing method and jumping to it).
  • Diagnostics are not updated n realtime (event is not captured - text is not available)

@dantleech dantleech marked this pull request as draft December 30, 2025 14:35
@dantleech dantleech changed the base branch from master to incremental-updates December 30, 2025 14:35
@dantleech dantleech changed the base branch from incremental-updates to master January 1, 2026 19:33
@dantleech dantleech force-pushed the incremental-parser branch 2 times, most recently from a113f6a to 7a4931e Compare January 3, 2026 16:37
@dantleech dantleech marked this pull request as ready for review January 4, 2026 11:46
$updatedSource = TextEdits::one($edit)->apply($this->node->getFileContents());

try {
$reason = $this->doApply($node, $edit);
Copy link
Collaborator Author

@dantleech dantleech Jan 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add a Result object here that can both indicate failure and success. i.e. the log should show:

PARS incremetal parse used (compound statement node)

the V/O should also make debug messages better.

$newToken = $newTokens[0];

// WHY?
if (strlen($editedTokenText) !== $newToken->length) {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why indeed

if (null === $extractStartPosition) {
$extractStartPosition = $compoundNode->openBrace->getStartPosition() + 1;
$statementNb = 0;
}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comment here - can any of this code be optimised?

use Phpactor\WorseReflection\Core\AstProvider;
use Throwable;

class IncrementalAstUpdater
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be beneficial to add a benchmark

Introduces an opt-in to enable experimental incremental parser updates.

When enabled the IDE will send text edits for each change to the LS
instead of the full text.

These text edits will be applied directly to the AST if possible and if
not will cause a full reparsing.
@dantleech
Copy link
Collaborator Author

[WR     ][INFO][135748.796360] PARS 0.0964 incremental update with "token" strategy
[WR     ][INFO][135753.272761] PARS 0.6905 incremental update with "token" strategy
[WR     ][INFO][135760.249353] PARS 0.0948 incremental update with "token" strategy

It's not clear how the token strategy can take half a second in some cases (35k lines of code)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant