Now that we're on the bleeding edge with #962 we should try and run this via cron but _not_ for PRs since then it will be flaky for contributors