Tuesday, October 27, 2009

Google News Crawl Problem...

Posted on 1:58 PM by programlover









Hi, I submitted my site: www.jazzou.com to be reviewed by Google News. I already have a website listed with them so everything is made to fit the requirements however I have got this message back:



'Thank you for your note. In order to for us to evaluate your site articles

we need to be able to access the content on your site. Currently, crawlers

cannot fill out registration forms, nor do they support cookies. Given

that, in order to successfully crawl your site, we need to be able to

circumvent your registration page. When we tried to access

http://jazzou.com/index.php?option=com_content we got the following

message:

"You are not authorised to view this resource. You need to login"



The easiest way to do this is to configure your webservers to not serve

the registration when our crawlers visit your pages (when the User-Agent

is "Googlebot"). You can verify that the request is actually from our

robot by making sure the IP address is within the range of 66.249.64.0/20.

It is equally important that your robots.txt file allows access by

Googlebot.'



The thing is I sent this url which is not restricted: http://jazzou.com/index.php?option=c...id=0&Itemid=40



I'm a bit baffled, any help would be appreciated



Thanks