Add ability to prime URL metrics across a site to avoid needing to collect from visitors

Feature Description

When activating Optimization Detective, it starts gathering URL metrics from visitors to the site. However, it has a storage locking mechanism to prevent a user (by hashed IP address) from submitting URL metrics more than once every 60 seconds. ~~Additionally, URL metrics are not collected for administrator users since pages often include additional elements for admins which will make the URL metrics not applicable to non-admins.~~ URL Metrics are now collected from admins as of #1788, and storage locking was disabled for administrators in #1835, allowing URL Metrics to be more rapidly collected as an admin browses the site.

If a site relies exclusively on user visits to populate a site with URL Metrics, then a site which does not get much traffic it can take a few days (or longer) to gather enough URL metrics to apply the optimizations. We could speed up this process by providing a way for an administrator to initiate a site-wide collection of URL metrics. A new query parameter could be introduced which (1) loads the page as a non-admin and (2) disables any storage locking for URL metrics. Then given a list of URLs on the site (e.g. from the sitemaps), each one could be loaded up in an iframe for each breakpoint. A progress bar could also be shown. The process could be interrupted at any time, and if the user wants to start again they could either initiate collection only for URLs for which there are no URL metrics gathered or else to start again from scratch.

Additionally, as discussed in #1850 (comment), there should also be a way to facilitate a Puppeteer script to automatically collect URL Metrics for a site on an ongoing basis using system cron (or in response to a post being updated).

Being able to turn off URL Metric collection from unauthenticated visitors in favor of an admin process or an automated system process could go a long way to making Optimization Detective feasible for higher traffic sites or sites that are at higher risk of scaling issues or bad actors attempting to submit bogus URL Metrics (e.g. which could erroneously report that the LCP element is in the footer, for example). See also: #1655.

This would be cool to implement as a CLI tool as well, using Puppeteer.

This would make sense to do in a hidden iframe in the post editor screen whenever a post has been updated. This would not require any new UI (which we've avoided until now) and it would ensure that posts get fresh URL metrics immediately.

This would address the issue where someone blocks the REST API for authenticated requests.

For very high traffic sites, this could be an alternative to collecting URL Metrics from visitors. See #1655.

@westonruter

@westonruter

I've been working on a proof of concept (POC) for priming URL metrics and wanted to share an update and gather your thoughts on it.

Current Implementation

The POC uses URLs from WordPress sitemaps, fetched in batches. Breakpoints are calculated on the backend with simple logic: the height for each breakpoint is based on its width and the min/max aspect ratios.

The minimum aspect ratio is capped at 1 to avoid very small heights for large breakpoints.
The maximum is capped at 2 to prevent overly tall small breakpoints.
To accommodate n+1 breakpoints, an additional one is added by increasing the last breakpoint’s width by 400.

These are placeholder settings for now and can be adjusted as needed.

The detect.js is also modified to address two limitations:

Data is now sent immediately, regardless of tab visibility.
Requests are made using fetch instead of navigator.beacon when od_prime=1 is present, as fetch provides better feedback on request success.

Parent-iframe communication uses postMessage. When the iframe completes a fetch request, it notifies the parent to move to the next breakpoint or URL.

Handling URLs

The od_prime=1 parameter has been added to iframe URLs. This flag:

Allows detect.js in Optimization Detective to skip the storage lock and other restrictions, enabling bulk requests.
Is also passed to the REST endpoint to bypass its storage lock.

While this works for now, maybe a nonce would be a more secure approach.

On the backend, the od_prime=1 parameter is removed from URLs before hashing.

Demo

od-demo.mp4

Note: To aid debugging, the current status of the processing URL, breakpoint dimensions, and the iframe are currently visible. These could be replaced with a progress bar later.

This feature can be accessed through a new settings page in the tools. However, we need to discuss the UI design further. @ShyamGadde is also looking into adding a resume feature to skip URLs with completed metrics after a page refresh.

@b1ink0

@b1ink0 awesome!

The POC uses URLs from WordPress sitemaps, fetched in batches.

For initial collection of URL Metrics this makes sense, but for incremental updates in the course of normal operations of a site, it would probably make more sense to prime URL metrics in the background on the post editor screen just for the current post's URL.

1. Data is now sent immediately, regardless of tab visibility. [///]
Parent-iframe communication uses postMessage. When the iframe completes a fetch request, it notifies the parent to move to the next breakpoint or URL.

Is that necessary? Wouldn't the data be sent when navigating away anyway? For example, once the iframe's load event fires, you could navigate it to about:blank. Or rather, when in this priming mode, the JS in the page could do postMessage right here:

performance/plugins/optimization-detective/detect.js

Line 568 in 6ca5c4b

// Wait for the page to be hidden.

At that point, the parent frame would know it is free to continue on to the next URL, allowing it to proceed with sending the metrics.

2. Requests are made using fetch instead of navigator.beacon when od_prime=1 is present, as fetch provides better feedback on request success.

What feedback is needed?

While this works for now, maybe a nonce would be a more secure approach.

Yes, a nonce will be important as otherwise malicious actors could bypass the safeguards.

Note: To aid debugging, the current status of the processing URL, breakpoint dimensions, and the iframe are currently visible. These could be replaced with a progress bar later.

Since desktop viewport size can be too large to display in the admin, you can consider using a transform to shrink it down to fit. I implemented this in https://googlechromelabs.github.io/layout-shift-terminator/

Related: ampproject/amp-wp#4729 and the work that @thelovekesh did in ampproject/amp-wp#4729 (comment) (note the post editor integration, although for the actual implementation no button should be required as it should happen automatically when saving a post)

Note also that Optimization Detective is a spiritual descendant of Layout Shift Terminator!

@westonruter

@westonruter

What feedback is needed?

I was thinking of handling of request failure which could be only done if we use fetch as we can't use .then or .catch on the navigator.beacon. In #1850 I have used navigator.beacon and it is somewhat faster than the fetch approach. If we don't have to worry about the failure case then navigator.beacon is better approach.

Well, it turns out that fetch() can now be used as a replacement for navigator.sendBeacon() when the the keepalive property is set (supported by all browsers). It seems that uBlock may block navigator.sendBeacon() for the sake of privacy, so using fetch() may be more reliable.

According to the spec, I'm just seeing that such keepalive requests have a POST body have a size limit of 64 KiB (65,536 bytes):

If the sum of contentLength and inflightKeepaliveBytes is greater than 64 kibibytes, then return a network error.

This is an aggregated limit, so if multiple keepalive fetch() requests are being made, then as soon as one is added which goes over the limit, then a network error will occur. Apparently for navigator.sendBeacon() there is also a limit of 64 KiB for all pending beacons on the page, just for one beacon.

Aside: I also just learned about fetchLater() which may be an option in the future.

All this to say, it seems that either navigator.sendBeacon() or keepalive fetch() will work.

In #1850 I have used navigator.beacon and it is somewhat faster than the fetch approach.

How do you mean it is somewhat faster?

How do you mean it is somewhat faster?

I forgot to clarify I was taking about the speed of moving to next URL or breakpoint after receiving a postMessage from IFRAME not the speed of fetch or navigator.sendBeacon().

By somewhat, I meant that in my previous implementation, when fetch was used, the postMessage was sent to the parent inside the .then or .catch handler of fetch, which introduced a delay. In contrast, with navigator.sendBeacon(), the postMessage is sent before navigator.sendBeacon() sends a request, making it faster as it does not wait for the request to finish.

Aside: I also just learned about fetchLater() which may be an option in the future.

Now intended to ship.

Aside: I also just learned about fetchLater() which may be an option in the future.

Now intended to ship.

I've opened #1970 to explore this.

Navigation Menu

Repository navigation

Add ability to prime URL metrics across a site to avoid needing to collect from visitors #1311

Feature Description

Current Implementation

Handling URLs

Demo

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add ability to prime URL metrics across a site to avoid needing to collect from visitors #1311

Description

Feature Description

Activity

westonruter commented on Aug 5, 2024

westonruter commented on Oct 11, 2024

westonruter commented on Dec 10, 2024

westonruter commented on Dec 10, 2024

b1ink0 commented on Jan 14, 2025

Current Implementation

Handling URLs

Demo

westonruter commented on Jan 14, 2025

b1ink0 commented on Feb 5, 2025

westonruter commented on Feb 5, 2025

b1ink0 commented on Feb 6, 2025

westonruter commented on Feb 12, 2025

westonruter commented on Apr 3, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions