Fix Algolia search crawler #14163

Activity

1. mergeable commented on Feb 19, 2026

   

   mergeableBot

   on Feb 19, 2026 – with Mergeable

   More actions

   This bug report is missing a link to reproduction at phpstan.org/try.

   It will most likely be closed after manual review.
2. phpstan-bot commented on Feb 19, 2026

   

   phpstan-bot

   on Feb 19, 2026

   Last edited by phpstan-bot

   Contributor

   More actions

   Claude finished @ondrejmirtes's task in 12m 37s —— View job

   ---

   Algolia Search Crawler Analysis

   • Explore website structure and build configuration
   • Build the website to inspect dist/ output
   • Analyze the current crawler config against actual site structure
   • Identify issues causing crawler timeout
   • Provide improvement recommendations

   ---

   Website Scale Summary

   Section	Page Count
   User Guide	15 pages
   Developing Extensions	27 pages
   Writing PHP Code	4 pages
   Config Reference	1 page
   Blog posts	53 posts + 3 category pages
   Error identifier detail pages	1,075 pages
   Error identifiers index (JS-rendered)	1 page
   Root pages (home, try, sponsor, merch, etc.)	~11 pages
   phpstan.org total	~1,190 pages
   API Reference (apiref.phpstan.org)	Hundreds of class/interface/namespace pages per version

   ---

   Issues Found (Likely Causing Crawler Failure)

   1. API Reference discovery pattern is unbounded (CRITICAL)

   discoveryPatterns: [
     "https://apiref.phpstan.org/**",  // ← crawls ALL versions
   ],

   The discovery pattern https://apiref.phpstan.org/** makes the crawler discover and attempt to fetch pages across all available API reference versions (e.g., 1.x, 2.0.x, 2.1.x). Each version contains hundreds of class, interface, and namespace pages. But the action's pathsToMatch only indexes 2.0.x:

   pathsToMatch: ["https://apiref.phpstan.org/2.0.x/**"],

   This means potentially thousands of pages are crawled but never indexed — pure waste of crawl budget and the most likely reason the crawler times out.

   Fix: Restrict discovery to the version you actually want to index:

   discoveryPatterns: [
     "https://phpstan.org/**",
     "https://apiref.phpstan.org/2.1.x/**",  // only current version
   ],

   2. API Reference version is outdated (BUG)

   The action indexes 2.0.x but all documentation links point to 2.1.x:

   https://apiref.phpstan.org/2.1.x/PHPStan.Rules.Rule.html  ← used in docs
   https://apiref.phpstan.org/2.0.x/**                         ← indexed by crawler
   

   Fix: Update the action's pathsToMatch to 2.1.x:

   pathsToMatch: ["https://apiref.phpstan.org/2.1.x/**"],

   3. No sitemaps configured (EFFICIENCY)

   sitemaps: [],  // ← empty!

   The website generates a sitemap at https://phpstan.org/sitemap.xml listing all ~1,190 pages. Using this would let the crawler know exactly which pages exist instead of discovering them through link-following (which is slower and can miss pages or discover duplicates).

   Fix:

   sitemaps: ["https://phpstan.org/sitemap.xml"],

   4. Incomplete action for /error-identifiers index page (BUG)

   {
     indexName: "phpstan",
     pathsToMatch: ["https://phpstan.org/error-identifiers"],
     // ← NO recordExtractor!
   },

   This action has no recordExtractor, meaning it either does nothing or uses a default that produces poor results. The error-identifiers index page is JavaScript-rendered (Knockout.js) and generates ~1,075 headings dynamically. Without a proper extractor, this page contributes nothing to the index but still costs a JavaScript render.

   Fix: Either remove this action entirely (the individual detail pages are already indexed by the action below it), or add a proper recordExtractor if you want the index page itself in search results.

   5. Missing exclusion patterns (EFFICIENCY)

   Several pages have no useful search content but are still crawled:

   exclusionPatterns: [
     "https://phpstan.org/merch*",
     "https://apiref.phpstan.org/*/source-*",
     // Missing:
     // "https://phpstan.org/try*",              ← playground (JS app, no static text)
     // "https://phpstan.org/sponsor*",           ← sponsor page
     // "https://phpstan.org/merch-privacy*",     ← merch privacy policy
     // "https://phpstan.org/merch-terms*",       ← merch terms
     // "https://apiref.phpstan.org/1.*/**",      ← old API versions (if they exist)
   ],

   Fix: Add exclusions for non-content pages and old API reference versions.

   6. Error identifiers index page triggers massive discovery burst

   The JS-rendered /error-identifiers page contains 1,075 links to individual detail pages. When renderJavaScript processes this page, it discovers all 1,075 URLs at once. Combined with the API reference pages, this creates a huge crawl queue. Using the sitemap (fix #3) and removing the JS render for this page (since the noscript fallback also has all 1,075 links) would reduce this cost.

   ---

   Recommended Improved Configuration

   Here's the optimized crawler config addressing all issues above:

   new Crawler({
     rateLimit: 8,
     startUrls: ["https://phpstan.org/", "https://apiref.phpstan.org/2.1.x/"],
     renderJavaScript: false,  // Remove JS rendering — noscript fallback has all links
     sitemaps: ["https://phpstan.org/sitemap.xml"],
     ignoreCanonicalTo: false,
     discoveryPatterns: [
       "https://phpstan.org/**",
       "https://apiref.phpstan.org/2.1.x/**",  // Only current version
     ],
     exclusionPatterns: [
       "https://phpstan.org/merch*",
       "https://phpstan.org/try*",
       "https://phpstan.org/sponsor*",
       "https://phpstan.org/rss*",
       "https://phpstan.org/sitemap*",
       "https://phpstan.org/llms*",
       "https://apiref.phpstan.org/*/source-*",
     ],
     schedule: "at 12:05 PM on Monday",
     actions: [
       // Documentation (User Guide, Developing Extensions, Writing PHP Code, Config Reference)
       {
         indexName: "phpstan",
         pathsToMatch: [
           "https://phpstan.org/**",
           "!https://phpstan.org/blog/**",
           "!https://phpstan.org/error-identifiers",
           "!https://phpstan.org/error-identifiers/**",
         ],
         recordExtractor: ({ helpers }) => {
           return helpers.docsearch({
             recordProps: {
               lvl1: ["header h1", "article h1", "main h1", "h1", "head > title"],
               content: ["article p, article li", "main p, main li", "p, li"],
               lvl0: {
                 selectors: "#algoliaSectionTitle",
                 defaultValue: "Documentation",
               },
               lvl2: ["article h2", "main h2", "h2"],
               lvl3: ["article h3", "main h3", "h3"],
               lvl4: ["article h4", "main h4", "h4"],
               lvl5: ["article h5", "main h5", "h5"],
               lvl6: ["article h6", "main h6", "h6"],
               pageRank: "90",
             },
             aggregateContent: true,
           });
         },
       },
       // Error identifier detail pages (removed the empty /error-identifiers action)
       {
         indexName: "phpstan",
         pathsToMatch: [
           "https://phpstan.org/error-identifiers/**",
         ],
         recordExtractor: ({ helpers }) => {
           return helpers.docsearch({
             recordProps: {
               lvl1: ["header h1", "article h1", "main h1", "h1", "head > title"],
               content: ["article p, article li", "main p, main li", "p, li"],
               lvl0: {
                 selectors: "#algoliaSectionTitle",
                 defaultValue: "Error Identifiers",
               },
               lvl2: ["article h2", "main h2", "h2"],
               lvl3: ["article h3", "main h3", "h3"],
               lvl4: ["article h4", "main h4", "h4"],
               lvl5: ["article h5", "main h5", "h5"],
               lvl6: ["article h6", "main h6", "h6"],
               pageRank: "70",
             },
             aggregateContent: true,
           });
         },
       },
       // Blog posts
       {
         indexName: "phpstan",
         pathsToMatch: ["https://phpstan.org/blog/**"],
         recordExtractor: ({ helpers }) => {
           return helpers.docsearch({
             recordProps: {
               lvl1: ["header h1", "article h1", "main h1", "h1", "head > title"],
               content: ["article p, article li", "main p, main li", "p, li"],
               lvl0: {
                 selectors: "#algoliaSectionTitle",
                 defaultValue: "Blog",
               },
               lvl2: ["article h2", "main h2", "h2"],
               lvl3: ["article h3", "main h3", "h3"],
               lvl4: ["article h4", "main h4", "h4"],
               lvl5: ["article h5", "main h5", "h5"],
               lvl6: ["article h6", "main h6", "h6"],
               pageRank: "50",
             },
             aggregateContent: true,
           });
         },
       },
       // API Reference (updated to 2.1.x)
       {
         indexName: "phpstan",
         pathsToMatch: ["https://apiref.phpstan.org/2.1.x/**"],
         recordExtractor: ({ helpers }) => {
           return helpers.docsearch({
             recordProps: {
               lvl1: ["header h1", "article h1", "main h1", "h1", "head > title"],
               content: ["article p, article li", "main p, main li", "p, li"],
               lvl0: {
                 selectors: "#sectionTitle",
                 defaultValue: "API Reference",
               },
               lvl2: ["article h2", "main h2", "h2"],
               lvl3: ["article h3", "main h3", "h3"],
               lvl4: ["article h4", "main h4", "h4"],
               lvl5: ["article h5", "main h5", "h5"],
               lvl6: ["article h6", "main h6", "h6"],
               pageRank: "10",
             },
             aggregateContent: true,
           });
         },
       },
     ],
     // initialIndexSettings unchanged
   });

   Summary of Changes

   Change	Impact
   Restrict API ref discovery to 2.1.x only	Eliminates potentially thousands of wasted page crawls — likely the #1 cause of timeout
   Update API ref version 2.0.x → 2.1.x	Fixes version mismatch with documentation links
   Add sitemap https://phpstan.org/sitemap.xml	Faster, more complete page discovery
   Remove renderJavaScript for error-identifiers	Saves expensive JS render; noscript fallback already has all 1,075 links for discovery
   Remove empty /error-identifiers action	Eliminates no-op action with no recordExtractor
   Simplify error-identifiers pathsToMatch	Removes unnecessary ! negation for the index page (which is no longer matched)
   Add exclusions for /try*, /sponsor*, /rss*, /sitemap*, /llms*	Reduces crawl waste on non-searchable pages
   Update startUrls to apiref.phpstan.org/2.1.x/	Starts API reference crawling from the correct version

   The most impactful fix is #1 (restricting API ref discovery). If apiref.phpstan.org hosts multiple versions and each has hundreds of class pages, the current ** discovery pattern could be causing the crawler to attempt crawling thousands of pages that are never indexed, exhausting the crawl budget/time limit before it can finish.
3. ondrejmirtes commented on Feb 19, 2026

   

   ondrejmirtes

   on Feb 19, 2026

   MemberAuthor

   More actions

   @phpstan-bot Is it possible to update the crawler programatically and have it versioned here? Ideally from GitHub actions. Also it'd be nice to launch the crawler after each website deploy if it's a good idea.
4. phpstan-bot commented on Feb 19, 2026

   

   phpstan-bot

   on Feb 19, 2026

   Last edited by phpstan-bot

   Contributor

   More actions

   Claude finished @ondrejmirtes's task in 7m 1s —— View job • claude/issue-14163-20260219-1037 • Create PR ➔

   ---

   Algolia Crawler: Programmatic Updates & CI Integration

   • Research Algolia Crawler API for programmatic configuration updates
   • Check existing website deploy workflow for integration points
   • Create versioned crawler config file in the repository
   • Provide recommendations with implementation details

   ---

   Short answer: Yes to all three.

   The Algolia Crawler has a REST API at https://crawler.algolia.com/api/1/ that supports both configuration updates and triggering reindexes. Here's the full breakdown:

   ---

   1. Versioning the crawler config in this repo

   I've added website/algolia-crawler-config.js with the optimized config from the previous analysis. The config contains JavaScript functions (recordExtractor), so it must be a .js file rather than JSON.

   The file includes the appId (which is public — it's used client-side for search) but uses "xxx" as a placeholder for apiKey, which should remain a secret.

   2. Updating the config programmatically from GitHub Actions

   The Crawler API accepts the config as a JavaScript string:

   curl -X PATCH "https://crawler.algolia.com/api/1/crawlers/${CRAWLER_ID}/config" \
     -H "Content-Type: application/javascript" \
     -u "${CRAWLER_USER_ID}:${CRAWLER_API_KEY}" \
     --data-binary @website/algolia-crawler-config.js

   Authentication uses HTTP Basic Auth with Crawler-specific credentials (not the regular Search API keys). You'll find these in the Algolia Dashboard under Data Sources > Crawler > Settings > API. Three secrets are needed:

   Secret	Where to find it
   ALGOLIA_CRAWLER_ID	Crawler dashboard URL or API
   ALGOLIA_CRAWLER_USER_ID	Crawler API settings
   ALGOLIA_CRAWLER_API_KEY	Crawler API settings

   3. Triggering the crawler after each website deploy

   To trigger a reindex (using the already-saved config):

   curl -X POST "https://crawler.algolia.com/api/1/crawlers/${CRAWLER_ID}/reindex" \
     -H "Content-Type: application/json" \
     -u "${CRAWLER_USER_ID}:${CRAWLER_API_KEY}"

   Is it a good idea? Yes, with the optimized config it should be fine:

   • The website only deploys on pushes to 2.2.x that change website/** — this isn't very frequent
   • With the config fixes (restricted API ref discovery, sitemap usage, no JS rendering), the crawl should complete quickly
   • Content changes will be reflected in search results promptly instead of waiting up to a week

   Proposed workflow changes

   Add these steps to the deploy job in .github/workflows/website.yml, after the CloudFront invalidation step (I'm unable to modify workflow files directly due to permissions):

         - name: "Update Algolia Crawler config"
           run: |
             # Replace the placeholder apiKey with the real one before pushing
             sed 's/apiKey: "xxx"/apiKey: "${{ secrets.ALGOLIA_SEARCH_ADMIN_API_KEY }}"/' \
               dist/../website/algolia-crawler-config.js | \
             curl -X PATCH \
               "https://crawler.algolia.com/api/1/crawlers/${{ secrets.ALGOLIA_CRAWLER_ID }}/config" \
               -H "Content-Type: application/javascript" \
               -u "${{ secrets.ALGOLIA_CRAWLER_USER_ID }}:${{ secrets.ALGOLIA_CRAWLER_API_KEY }}" \
               --data-binary @-
   
         - name: "Trigger Algolia Crawler reindex"
           run: |
             curl -X POST \
               "https://crawler.algolia.com/api/1/crawlers/${{ secrets.ALGOLIA_CRAWLER_ID }}/reindex" \
               -H "Content-Type: application/json" \
               -u "${{ secrets.ALGOLIA_CRAWLER_USER_ID }}:${{ secrets.ALGOLIA_CRAWLER_API_KEY }}"

   Alternatively, if you don't want to push config changes on every deploy (to avoid unnecessary API calls when the config hasn't changed), you could split this into two approaches:

   1. Always trigger reindex after deploy (add just the reindex curl step)
   2. Update config only when it changes (use a separate workflow triggered on changes to website/algolia-crawler-config.js)

   The simpler option is to just always do both — the config update is idempotent and the API calls are lightweight.

   Note about the deprecated GitHub Action

   Algolia had an official GitHub Action (algolia/algoliasearch-crawler-github-actions) but it was archived/deprecated. The direct curl approach above is more reliable and doesn't depend on a maintained action.

   Summary of secrets to add

   Add these to the repository secrets for the workflow:

   • ALGOLIA_CRAWLER_ID — The crawler's ID
   • ALGOLIA_CRAWLER_USER_ID — Crawler API user ID
   • ALGOLIA_CRAWLER_API_KEY — Crawler API key
   • ALGOLIA_SEARCH_ADMIN_API_KEY — The Algolia admin API key (for the apiKey field in the config, which the crawler uses to write to the index)

   ---

   |
5. phpstan-bot

   added a commit that references this issue on Feb 19, 2026

   Add versioned Algolia Crawler configuration

   ...

   This issue will close once commit 13d4fd4 is merged into the '2.3.x' branch.

   13d4fd4
6. ondrejmirtes

   closed this as completedin #14164on Feb 19, 2026
7. ondrejmirtes

   reopened this on Feb 19, 2026
8. ondrejmirtes commented on Feb 19, 2026

   

   ondrejmirtes

   on Feb 19, 2026

   MemberAuthor

   More actions

   @phpstan-bot The workflow you added to reindex after deploy works! But the workflow to update the crawler config does not. The curl ends with exit code 22.

   Since the reindex job works, it cannot be due to credentials, but some other problem with the API call.
9. phpstan-bot commented on Feb 19, 2026

   

   phpstan-bot

   on Feb 19, 2026

   Last edited by phpstan-bot

   Contributor

   More actions

   Claude finished @ondrejmirtes's task in 4m 42s —— View job

   ---

   Fixing Algolia Crawler Config Update Workflow

   • Investigate the crawler config file and workflow setup
   • Research Algolia Crawler API requirements for config updates
   • Identify the issue causing curl exit code 22
   • Provide corrected workflow steps

   ---

   Root Cause

   The failing step in .github/workflows/algolia-crawler-config.yml (line 22-25) sends the config as raw JavaScript with Content-Type: application/javascript:

   curl -s -f -X PATCH "https://crawler.algolia.com/api/1/crawlers/$CRAWLER_ID/config" \
     -H "Content-Type: application/javascript" \
     -u "$USER_ID:$API_KEY" \
     --data-binary @website/algolia-crawler-config.js

   The Algolia Crawler API's PATCH /config endpoint expects Content-Type: application/json with the JavaScript config wrapped in a JSON object:

   {"config": "new Crawler({...the entire JS source...})"}

   Sending raw JavaScript with the wrong Content-Type results in an HTTP 4xx error, which curl reports as exit code 22 (from the -f flag).

   Fix

   The workflow step needs to:

   1. JSON-encode the JavaScript file content (escape quotes, newlines, etc.)
   2. Wrap it in a {"config": "..."} JSON object
   3. Send with Content-Type: application/json

   I'm unable to modify workflow files directly due to GitHub App permissions. Here's the corrected step to replace the current "Update Crawler Config" step:

         - name: "Update Crawler Config"
           run: |
             sed -i 's/apiKey: "xxx"/apiKey: "${{ secrets.ALGOLIA_API_KEY }}"/' website/algolia-crawler-config.js
             jq -Rs '{"config": .}' website/algolia-crawler-config.js | \
             curl -s -f -X PATCH "https://crawler.algolia.com/api/1/crawlers/${{ secrets.ALGOLIA_CRAWLER_ID }}/config" \
               -H "Content-Type: application/json" \
               -u "${{ secrets.ALGOLIA_CRAWLER_USER_ID }}:${{ secrets.ALGOLIA_CRAWLER_API_KEY }}" \
               -d @-

   The key change is piping through jq -Rs '{"config": .}' which:

   • -R reads the file as raw text (not JSON)
   • -s slurps all input into a single string
   • '{"config": .}' wraps it in the JSON envelope the API expects

   jq is pre-installed on ubuntu-latest runners.

   View job · claude/issue-14163-20260219-1403
10. ondrejmirtes commented on Feb 19, 2026

    

    ondrejmirtes

    on Feb 19, 2026

    MemberAuthor

    More actions

    @phpstan-bot Change the config file in the repo from .js to .json, it will be easier to call this API.
11. phpstan-bot commented on Feb 19, 2026

    

    phpstan-bot

    on Feb 19, 2026

    Last edited by phpstan-bot

    Contributor

    More actions

    Claude finished @ondrejmirtes's task in 4m 51s —— View job • claude/issue-14163-20260219-1413

    ---

    Converting Algolia Crawler Config from JS to JSON

    • Read the current .js config file
    • Convert config to .json format
    • Delete the old .js config file
    • Commit and push changes
    • Note required workflow changes

    ---

    Converted website/algolia-crawler-config.js to website/algolia-crawler-config.json. The JSON file contains the same configuration with recordExtractor functions stored as string values.

    The workflow file .github/workflows/algolia-crawler-config.yml needs to be updated to reference the new .json file. The required changes:

    1. Update the paths trigger from website/algolia-crawler-config.js to website/algolia-crawler-config.json
    2. Update the "Update Crawler Config" step — with JSON, the API call becomes simpler:

          - name: "Update Crawler Config"
            run: |
              jq --arg key "${{ secrets.ALGOLIA_API_KEY }}" '.apiKey = $key' website/algolia-crawler-config.json | \
              curl -s -f -X PATCH "https://crawler.algolia.com/api/1/crawlers/${{ secrets.ALGOLIA_CRAWLER_ID }}/config" \
                -H "Content-Type: application/json" \
                -u "${{ secrets.ALGOLIA_CRAWLER_USER_ID }}:${{ secrets.ALGOLIA_CRAWLER_API_KEY }}" \
                -d @-

    I'm unable to modify workflow files due to GitHub App permissions, so that change needs to be applied manually.

    Create PR

    View job · claude/issue-14163-20260219-1413
12. ondrejmirtes

    closed this as completedin #14167on Feb 19, 2026
13. ondrejmirtes

    reopened this on Feb 19, 2026
14. ondrejmirtes commented on Feb 19, 2026

    

    ondrejmirtes

    on Feb 19, 2026

    MemberAuthor

    More actions

    @phpstan-bot With the latest change we forgot the JSON has to be in "config" key, right? Also make the curl call verbose so we actually see the HTTP error.
15. phpstan-bot commented on Feb 19, 2026

    

    phpstan-bot

    on Feb 19, 2026

    Last edited by phpstan-bot

    Contributor

    More actions

    Claude finished @ondrejmirtes's task in 1m 3s —— View job

    ---

    Fixing Algolia Crawler Config API Call

    • Inspect current config file and workflow
    • Provide corrected workflow steps

    ---

    You're right — the PATCH /config endpoint expects the config wrapped in a {"config": ...} JSON envelope. The current workflow sends the raw config object directly, which the API rejects.

    I'm unable to modify workflow files due to GitHub App permissions. Here's the corrected "Update Crawler Config" step for .github/workflows/algolia-crawler-config.yml:

          - name: "Update Crawler Config"
            run: |
              jq --arg key "${{ secrets.ALGOLIA_API_KEY }}" '(.apiKey = $key) | {config: .}' website/algolia-crawler-config.json | \
              curl -v -f -X PATCH "https://crawler.algolia.com/api/1/crawlers/${{ secrets.ALGOLIA_CRAWLER_ID }}/config" \
                -H "Content-Type: application/json" \
                -u "${{ secrets.ALGOLIA_CRAWLER_USER_ID }}:${{ secrets.ALGOLIA_CRAWLER_API_KEY }}" \
                -d @-

    Two changes from the current version:

    1. Wrap in config key: (.apiKey = $key) | {config: .} — parentheses are needed because jq's = has lower precedence than |, so without them .apiKey = $key | {config: .} would be parsed as .apiKey = ($key | {config: .}) which is wrong
    2. Verbose curl: Changed -s (silent) to -v (verbose) so HTTP request/response headers and error details are visible in the workflow logs

    View job · claude/issue-14163-20260219-1428
16. ondrejmirtes

    closed this as completedon Feb 19, 2026
17. github-actions commented on Mar 23, 2026

    

    github-actionsBot

    on Mar 23, 2026 – with GitHub Actions

    More actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
18. github-actions

    locked as resolved and limited conversation to collaborators on Mar 23, 2026