Bots are crawling everything on the web, even articles with an expired date.
Fortunately, the "robots" meta tag offers a good and simple solution to tell such a situation (no more article available) to bots:
<meta name="robots" content="unavailable_after:25-Aug-2007 15:00:00 EST">.
Because it's pretty difficult to integrate this into the core, maybe a 301 header would be simplier?
or x-robots-tag. i.e.
HTTP/1.1 200 Ok
Date: Tue, 25 May 2010 21:42:43 GMT
...
X-Robots-Tags: unavailable_after:25-Aug-2007 15:00:00 EST
...
Bots are crawling everything on the web, even articles with an expired date.
Fortunately, the "robots" meta tag offers a good and simple solution to tell such a situation (no more article available) to bots:
<meta name="robots" content="unavailable_after:25-Aug-2007 15:00:00 EST">.Because it's pretty difficult to integrate this into the core, maybe a 301 header would be simplier?
or
x-robots-tag. i.e.HTTP/1.1 200 Ok
Date: Tue, 25 May 2010 21:42:43 GMT
...
X-Robots-Tags: unavailable_after:25-Aug-2007 15:00:00 EST
...