Perf: Benchmark Xdebug performance #1029

carlos-granados · 2025-07-26T09:12:09Z

We want to be able to measure Xdebug's performance to see how different PRs affect it. To do this we will use Valgrind to count the number of instructions executed when running different php processes with Xdebug in any of its different modes. This is much more precise than measuring time as this can be affected by a number of external elements. Measuring the number of executed instructions should give us a very good approximation to execution time. The usage of Valgrind to measure the number of instructions is inspired by the benchmarking that is done on the php-src using similar techniques and some of the code here has been directly derived from that implementation.

We run three different php processes which represent a number of different circumstances:

The first one is an artificial benchmark which runs a number of algorithms. This process is the one that will be more affected by Xdebug as there is very little compilation (the process is just a single file) and lots of function calls.
The second one is running Rector with a small set of rules over a single file. This process represents a typical CLI tool run and is some middle ground in respect to Xdebug's influence. The number of compiled files is not too big and the number of function calls is not big either
The third one involves doing a request on a Symfony endpoint. This process is the one less influenced by Xdebug because there is a huge number of files involved so that a lot of time is spent compiling/reading them.
We run these processes without Xdebug being loaded and then with it being loaded and with all its different modes activated. This allows us to understand the effect that Xdebug has on the execution time of these processed depending on these modes.

This benchmark is run in GitHub's CI infrastructure and it will be triggered in three different ways:

It will run weekly on schedule
It can be triggered manually
It will run for any PR where the title of the PR starts with a specific prefix
The result of all these executions is summarised in a summary page which can be accessed through the Actions tab here in GitHub. You just need to open the corresponding workflow. Here is an example of this summary:

https://github.com/xdebug/xdebug/actions/runs/16538303240

carlos-granados

@derickr, in the end I have preferred to create a new PR instead of reopening the old one. This just a draft with a proposal, please let me know your thoughts, there are many things which could be adjusted if you feel that something would be better in some other way.

carlos-granados · 2025-07-26T09:12:26Z

.github/benchmark-files/bench.php

@@ -0,0 +1,353 @@
+<?php
+if (function_exists("date_default_timezone_set")) {


This file has been copied from the php-src project. It is just a php process that runs some fairly standard algorithms and which serves as an artificial benchmark. I removed a couple of functions which were taking long to run and which made these benchmarks too slow

carlos-granados · 2025-07-26T09:12:47Z

.github/benchmark-files/benchmark.ini

@@ -0,0 +1,11 @@
+max_execution_time=0


Needed so that the php scripts won't time out while being run

carlos-granados · 2025-07-26T09:12:58Z

.github/benchmark-files/benchmark.ini

@@ -0,0 +1,11 @@
+max_execution_time=0
+error_reporting=0


We remove all error handling, printing and displaying so that it does not interfere with the tests

carlos-granados · 2025-07-26T09:13:09Z

.github/benchmark-files/benchmark.ini

+error_reporting=0
+display_errors=off
+log_errors=off
+opcache.enable=1


We enable the opcache both for cli and cgi modes, disable the jit compiler and disable opcache optimization (so that the case where Xdebug is not loaded matches the case where it is loaded as Xdebug disables this optimization) We also set opcache to use a file cache so that it can be shared between the two runs that we do of the Symfony code

carlos-granados · 2025-07-26T09:13:20Z

.github/benchmark-files/rector.php

@@ -0,0 +1,15 @@
+<?php
+
+declare(strict_types=1);


This file is used to run Rector to measure the performance of PHP for CLI tools. It just runs a small set of rules against a single file in order to make the test reasonably fast for our benchmark

carlos-granados · 2025-07-26T09:18:01Z

.github/workflows/benchmark.yml

+                    php_versions=$(cut -d',' -f1 unique-matrix-values.txt | sort | uniq)
+                    xdebug_modes=$(cut -d',' -f3 unique-matrix-values.txt | sort | uniq)
+
+                    for command in $commands; do


And we loop over all these versions building a summary with tables of all the values

carlos-granados · 2025-07-26T09:18:15Z

.github/workflows/benchmark.yml

+                            echo "| Xdebug | Instructions | Slowdown |" >> summary.md
+                            echo "|--------|-------------:|---------:|" >> summary.md
+
+                            base_file="merged/php-${php}_cmd-${command}_xdebug-no.txt"


To calculate the slowdown, we read a base value which is taken from the data where Xdebug mode is "no" (it was not loaded)

carlos-granados · 2025-07-26T09:18:28Z

.github/workflows/benchmark.yml

+                                    else
+                                        slowdown=$(awk -v v=$value -v b=$base_value 'BEGIN { printf "%.1f%%", ((v - b) * 100) / b }')
+                                    fi
+                                    formatted_value=$(printf "%'d" "$value")


The number of instructions is formatted with thousands separators

carlos-granados · 2025-07-26T09:18:40Z

.github/workflows/benchmark.yml

+                    done
+
+            -   name: print summary
+                run: cat summary.md >> $GITHUB_STEP_SUMMARY


This generated summary file is copied into a special file which GitHub has defined in this special env variable and it will use it to print a summary for the workflow

carlos-granados · 2025-07-26T09:18:50Z

.github/workflows/benchmark.yml

+            -   name: print summary
+                run: cat summary.md >> $GITHUB_STEP_SUMMARY
+
+            -   name: delete intermediary artifacts


The intermediate files that we generated are not needed and can be deleted, helping us keep the summary page clean

derickr

This looks pretty good! Thanks for working on this.

I have a few nits (missing newlines), and one suggestion for working around having to remove/change the error_reporting() calls — Xdebug has a setting for this.

I would also not like to see having to use a commit message prefix to start these, but suggested using a label, like php.net uses for website change previews — I will create the "performance-check" label for that in a minute.

I like how you added these comments as a review, but I think that some of these can/should be a comment in the actual file(s) — I'll let it up to you which ones make sense or not.

.github/benchmark-files/benchmark.ini

.github/benchmark-files/rector.php

.github/benchmark-files/xdebug-benchmark.ini

derickr · 2025-07-31T12:40:36Z

.github/workflows/benchmark.yml

+        if: |
+            github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' ||
+            (
+                github.event_name == 'pull_request' &&


Can we change this so that it only runs when a label on the PR is set? I don't like clutter in my commit messages, and it's something that PHP also uses for the previews of changes to the website: https://github.com/php/web-php/blob/master/.github/workflows/pr-preview.yml#L3

derickr · 2025-07-31T12:43:05Z

.github/workflows/benchmark.yml

+                    git clone -q --branch "$version" --depth 1 https://github.com/symfony/demo.git ./symfony-demo
+                    cd symfony-demo
+                    composer install 
+                    find . -type f -name "*.php" -exec sed -i.bak -E 's/error_reporting\(.*\);/error_reporting(0);/g' {} +


You can work around changing files, by using some Xdebug settings instead:

https://xdebug.org/docs/all_settings#force_display_errors

https://xdebug.org/docs/all_settings#force_error_reporting

carlos-granados · 2025-08-11T09:18:40Z

I added the missing end of file newlines
Added many comments
Changed the workflow so that it would be activated with a PR label instead of the commit prefix
I could not use force_error_reporting because that config is ORed with the existing error_reporting and in this way you can use it to force some errors to be shown but cannot use it to force some error not to be shown
Compilation was failing for PHP 8.5 so I added a few temporary fixes. I was not attempting to provide the correct fix, I just modified the code enough so that it would compile. Once the PHP 8.5 issues are fixed on the main branch, I will rebase my branch and remove these temporary fixes
Tests are failing for PHP 8.5, I have not attempted to fix that as that is not needed for the performance tests to run
You will need to add the performance-check label to the PR to trigger the performance tests. I tested this in my fork and seemed to work ok, please add the label and let me know if anything does not work as expected

derickr · 2025-08-21T15:43:04Z

Just a note that I am not forgetting about it. I've been working/am working on making the tests pass on PHP 8.5 first.

# Conflicts: # .gitignore

carlos-granados · 2025-10-07T19:14:26Z

@derickr now that the PHP 8.5 issues are fixed in the master branch, I rebased this code and removed my temporary fixes. I tested it in my fork, works fine for all versions, see https://github.com/carlos-granados/xdebug/actions/runs/18322794670

jakoch · 2025-11-06T16:36:58Z

Hi!

I checked the PR run and the benchmark report looks great.

To make the performance impact easier to read, i'd suggest to add a fourth column: “× Times Slower”, alongside the current Slowdown (%). This would complement the percentage with a more intuitive multiplicative factor.
So, instead of just 1434.6%, readers would also see 15.35×, which is usually easier to interpret at a glance.

You could update the summary generation like this:

echo "| Xdebug | Instructions | Slowdown | x Times Slower |" >> summary.md
echo "|--------|--------------|----------|----------------|" >> summary.md

awk -v v="$value" -v b="$base_value" 'BEGIN {
  slowdown = ((v - b) * 100) / b
  times = v / b
  printf "| Enabled | %\047d | %.1f%% | %.2fx |\n", v, slowdown, times
}' >> summary.md

Just my two cents, best regards.

carlos-granados · 2025-11-24T12:23:15Z

@derickr any chance to get this reviewed at some point? It's been pending for several months now...

* pr/1029: Reflow some comments Add comments, add end of file newlines, fix php 8.5 compilation Benchmark Xdebug performance

derickr · 2025-12-10T17:02:12Z

I have merged this (finally). Thanks! Now we'll have to see which performance benefits we'll get.

carlos-granados commented Jul 26, 2025

View reviewed changes

carlos-granados marked this pull request as draft July 26, 2025 09:20

derickr requested changes Jul 31, 2025

View reviewed changes

carlos-granados added 2 commits October 7, 2025 20:38

Benchmark Xdebug performance

a8aec26

# Conflicts: # .gitignore

Add comments, add end of file newlines, fix php 8.5 compilation

b3c7019

carlos-granados force-pushed the benchmark-performance branch from f0dce68 to b3c7019 Compare October 7, 2025 18:40

derickr added a commit that referenced this pull request Dec 10, 2025

Merged pull request #1029

4a470b3

* pr/1029: Reflow some comments Add comments, add end of file newlines, fix php 8.5 compilation Benchmark Xdebug performance

derickr marked this pull request as ready for review December 10, 2025 17:01

		@@ -0,0 +1,353 @@
		<?php
		if (function_exists("date_default_timezone_set")) {

		@@ -0,0 +1,11 @@
		max_execution_time=0
		error_reporting=0

Uh oh!

Perf: Benchmark Xdebug performance #1029

Are you sure you want to change the base?

Perf: Benchmark Xdebug performance #1029

Uh oh!

Conversation

carlos-granados commented Jul 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlos-granados left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derickr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlos-granados commented Aug 11, 2025

Uh oh!

derickr commented Aug 21, 2025

Uh oh!

carlos-granados commented Oct 7, 2025

Uh oh!

jakoch commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlos-granados commented Nov 24, 2025

Uh oh!

derickr commented Dec 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

carlos-granados commented Jul 26, 2025 •

edited

Loading

jakoch commented Nov 6, 2025 •

edited

Loading