Edge returns a 403 JavaScript crawler challenge to search-engine bots — hosted sites can't be crawled, indexed, or previewed #75

Summary

Every hostname I serve through Spacefast (apex domain, the app subdomain, and 1,300+ pair subdomains of migrateme.tech) returns HTTP 403 with a JavaScript challenge page to crawler user-agents, while the same URLs return 200 to browsers. The challenge page's title is Checking search engine crawler... and it requires JavaScript, which search crawlers don't run in this form — so the sites are effectively invisible to search engines and link-preview services.

This looks like the crawler-blocking behaviour I raised on 2026-10-14; it is still reproducing on 2026-10-18, so I'm opening it here with fresh evidence.

Evidence (2026-10-18, ~22:28–22:37 CDT)

Same URLs, two user-agents:

Other crawler user-agents tested against https://migrateme.tech/ — all 403:

• Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
• Googlebot-Image/1.0
• facebookexternalhit/1.1 (link previews)

Response body for crawler UAs begins:

<!DOCTYPE html><title>Checking search engine crawler...</title>
<noscript><p>Javascript required</p></noscript>

Impact

• Google Search Console (domain property migrateme.tech, verified via DNS on 2026-10-18) accepted the sitemap submission but reports "Couldn't fetch" for https://migrateme.tech/sitemap.xml, with 0 discovered pages — consistent with the 403 challenge.
• Because even robots.txt is challenged, crawlers can't reliably fetch crawl rules or sitemaps at all.
• Facebook's crawler is also blocked, so Open Graph link previews fail.

Expected

Verified search-engine crawlers (at minimum Googlebot and Bingbot) should receive the real content (200) for public spaces, the way browsers do. If the bot check is an intentional anti-scraping feature, a per-space or per-domain setting to allow search crawlers — or to disable the challenge — would resolve it.

Context

• Spaces involved: migrateme-tech (apex site), migrateme-universal (app + all pair subdomains), migrateme-from-wp-sandbox.
• Everything above was verified from multiple networks/clients on 2026-10-18; browser traffic is unaffected and the sites themselves are healthy.

Happy to provide full request/response captures or test again after any change — I can re-run the same UA matrix and the Search Console fetch on request.