Reject crawlers by default - #734

Hi @benrubson,

Thanks for bringing this to my attention and creating this pull request, Never really thought about the included robots.txt file before.

I completely agree with rejecting by default but thinking we should achieve this with a configurable solution. As it stands, If the robots.txt is altered it will throw a conflict upon update.

Thinking we could have this toggle-able via the .env and either add a robots.txt endpoint or possibly set a robots meta tag instead.
I'm siding on a robots.txt endpoint as we could configure the file as a template which then could be overridden by the user, to extend or set additional rules, via theming if they wanted.

Hi @ssddanbrown,

Thank you very much for your answer 👍

I agree, would be nice it if could be user-configurable.
Regarding the method, I don't really know what would be best.
Perhaps a simple on/off switch (reject/accept crawlers) in admin panel would be sufficient ?

Thank you !

I'm OK with allowing users to configure, but would also like to have it turned off by default.

Thanks again @benrubson for raising this and providing a potential solution.

I have now implemented a configurable solution as discussed above. Details of implementation and use can be seen in #779. This will be in the next release.

I will therefore go ahead and close this pull request.

Perfect thank you Dan 👍