Skip to content

Performance: Add comment with perf changes in PRs #56899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
WunderBart wants to merge 10 commits into from
Closed

Performance: Add comment with perf changes in PRs #56899

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3fd0fe2
Introduce performance metrics comparison step
WunderBart Dec 8, 2023
3e498ba
tmp: simulate regression
WunderBart Dec 8, 2023
616bae4
Collect all regressed metrics
WunderBart Dec 8, 2023
74417ca
Rename to check-performance-regression
WunderBart Dec 8, 2023
3960e67
Print regression percentage
WunderBart Dec 8, 2023
eaed08c
Add a fail-safe for missing result files
WunderBart Dec 8, 2023
4425e8c
Fix file filter
WunderBart Dec 8, 2023
b992876
Ensure results json has correct format
WunderBart Dec 8, 2023
6616ca9
Revert "tmp: simulate regression"
WunderBart Dec 8, 2023
2eb27e9
Try 30% max regression allowed
WunderBart Dec 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add a fail-safe for missing result files
  • Loading branch information
@WunderBart
WunderBart committed Dec 8, 2023
commit eaed08ccccab02207c54a571174aabd3f3b7a4c6
58 changes: 32 additions & 26 deletions bin/check-performance-regression.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ const fs = require( 'fs' );
const path = require( 'path' );

const resultsPath = process.env.WP_ARTIFACTS_PATH;
const maxRegressionFactor = 1.1; // fail on 10% regression
const maxRegressionFactor = 1.1; // fail on >10% regression
const regressedMetrics = [];

fs.readdir( resultsPath, ( err, files ) => {
Expand All @@ -16,35 +16,41 @@ fs.readdir( resultsPath, ( err, files ) => {
process.exit( 1 );
}

files
.filter(
( file ) =>
file.endsWith( '.performance-results.json' ) &&
! file.includes( '_round-' )
)
.forEach( ( file ) => {
const data = JSON.parse(
fs.readFileSync( path.join( resultsPath, file ), 'utf8' )
);
// Only get the curated results.
files.filter(
( file ) =>
file.endsWith( '.performance-results.json' ) &&
! file.includes( '_round-' )
);

const trunkMetrics = data.trunk;
const branchMetrics =
data[ Object.keys( data ).find( ( key ) => key !== 'trunk' ) ];
if ( files.length === 0 ) {
console.error( 'No performance results found.' );
process.exit( 1 );
}

files.forEach( ( file ) => {
const data = JSON.parse(
fs.readFileSync( path.join( resultsPath, file ), 'utf8' )
);

const trunkMetrics = data.trunk;
const branchMetrics =
data[ Object.keys( data ).find( ( key ) => key !== 'trunk' ) ];

Object.keys( trunkMetrics ).forEach( ( metric ) => {
const trunkValue = trunkMetrics[ metric ];
const branchValue = branchMetrics[ metric ];
Object.keys( trunkMetrics ).forEach( ( metric ) => {
const trunkValue = trunkMetrics[ metric ];
const branchValue = branchMetrics[ metric ];

if ( branchValue > trunkValue * maxRegressionFactor ) {
regressedMetrics.push( {
file,
metric,
branchValue,
trunkValue,
} );
}
} );
if ( branchValue > trunkValue * maxRegressionFactor ) {
regressedMetrics.push( {
file,
metric,
branchValue,
trunkValue,
} );
}
} );
} );

if ( regressedMetrics.length > 0 ) {
console.log( 'Performance regression detected:' );
Expand Down