500 Internal Server Error Googlebot
Contents |
from GoogleSign inHidden fieldsSearch for groups or messages
Part 1, then check out Part 2 & Part 3). Gateway timeout... File not found. Authorization Required. Forbidden! INTERNAL SERVER ERROR! If search bots are spending time on your site just running
How To Fix Crawl Errors In Google Webmaster Tools
into walls and spinning around in circles, that's time they won't be spending crawling error 500 your carefully-optimized content, and as we know from part 1 of this series , the more often Googlebot crawls a page, the better that page will rank. But persistent HTTP errors and other bot-confounding behaviors just squander your crawl budget, limiting the potential for improvement elsewhere. In this installment, we'll look at the different types of crawl errors, what they http://productforums.google.com/d/topic/webmasters/gJKtR1vWQgE mean, and how to discover and monitor them on your site. Broken links Broken links just happen. Writers mangle links while editing content. Developers mangle links when the logic to auto-generate those links fails to account for some potential condition (or if they're just ignorant of basic on-page technical SEO). With multi-user CMS environments, people fat-finger the delete button or change settings that can cause important pages to suddenly go missing and return "404 Not http://www.dirigodev.com/blog/seo-web-best-practices/crawl-errors-increasing-googlebot-crawl-budget-part-4 Found" errors. The bots will follow these links repeatedly. 404s are okay in and of themselves; they're not going to hurt your rankings unless more of your indexed pages are returning 404 than not (e.g., high-turnover content like individual URLs of for-sale listings, or large-catalog e-commerce with fluctuating availability). Outside of such extreme cases, Google knows that accidents happen and will re-crawl previously-good URLs that have gone 404 just in case they've come back. That's a great service, but your site's crawl budget is still eroded by every request that Googlebot spends handling an error response instead of discovering valuable content, so it's obviously important to monitor 400-level errors and resolve whatever you can so they stop happening, whether that's restoring content on that URL so it can return 200, or 301-redirecting it to a functioning URL with similar content in order to preserve the page authority that the now-404ing URL had accumulated. Google does a decent job of detecting and reporting "soft 404s", where the site responds to a request for a non-existent URL with a temporary redirect (302 instead of 301) to a page that reads like a 404 page but returns 200 instead. Many CMSes do this automatically, and it's especially common on IIS web servers and .NET-based web applications, but it's an insane, zero-sum ritual to put
here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss http://stackoverflow.com/questions/13249258/in-webmaster-tools-googlebot-is-getting-a-crawl-error-500-from-the-server the workings and policies of this site About Us Learn more http://stackoverflow.com/questions/10521244/googlebot-crawl-error-500-and-php-error-reporting-with-a-strange-solution about Stack Overflow the company Business Learn more about hiring developers or posting ads with us Stack Overflow Questions Jobs Documentation Tags Users Badges Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 4.7 million programmers, just like you, helping internal server each other. Join them; it only takes a minute: Sign up In Webmaster tools googlebot is getting a crawl error 500 from the server up vote 0 down vote favorite I've noticed my sites are not ranking as well as they did before and when I checked Webmaster tools I see that gooblebot cannot crawl pages that I internal server error can perfectly crawl with my browser and I'm getting an 500 error. The websites are not wordpress and use php. What can be causing this problem? This is the actual error in WMT HTTP/1.1 500 Internal Server Error Date: Tue, 06 Nov 2012 21:04:38 GMT Server: Apache Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Set-Cookie: PHPSESSID=blkss9toirna36p2mjl44htv01; path=/ Vary: Accept-Encoding Content-Encoding: gzip Content-Length: 3840 Connection: close Content-Type: text/html php error-handling googlebot share|improve this question edited Nov 18 '15 at 1:46 pnuts 33.7k63569 asked Nov 6 '12 at 10:41 user1678673 43 have you tried to do any configuration..may be with .htaccess..?? –Rishi Kalia Nov 6 '12 at 10:49 addtionaly check these links, if they help: stackoverflow.com/questions/10521244/… and webmasters.stackexchange.com/questions/28257/… –Rishi Kalia Nov 6 '12 at 10:52 add a comment| 3 Answers 3 active oldest votes up vote 0 down vote You may be blocking Googlebot with .htaccess, robots.txt or by some other means (maybe firewall settings?) a. this is not good b. yo
here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss the workings and policies of this site About Us Learn more about Stack Overflow the company Business Learn more about hiring developers or posting ads with us Stack Overflow Questions Jobs Documentation Tags Users Badges Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 4.7 million programmers, just like you, helping each other. Join them; it only takes a minute: Sign up Googlebot Crawl Error 500 and PHP Error reporting (with a strange solution) up vote 1 down vote favorite So Google wouldn't crawl anywhere on my live site other than some simple first pages - instead just giving me 500 errors. Fetching as Google in the Webmaster tools showed that it would return the full html output with the header "HTTP/1.0 500 Internal Server Error". I work locally in xampp with display_errors turned on but couldn't see any problems there so I checked the error_log on the live server - nothing there either. Eventually I decided to switch on display_errors on the server - I don't have php.ini access so I just added "php_flag display_errors on" to the htaccess file. When I did this Google no longer returned a 500 internet server error and will now index the page! The weirdest thing is no new PHP errors are being displayed! I just don't get it. So basically: PHP output causes 500 error for Googlebot until I put "php_flag display_errors on" in .htaccess even though no errors are displayed. Any help? I obviously would rather have display_errors turned off in the long run but I need to get rid of these 500 errors. I just don't know how to find out what is causing them. php error-handling http-headers googlebot share|improve this question edited Nov 18 '15 at 11:38 pnuts 33.7k63569 asked May 9 '12 at 17:39 Duncan 208 view the source and double check that the error isn't in a tag that doesn't get displayed such as style, script, head...etc. Also if you are logging php errors, check the php error log to see what error you are getting. Sometimes it is easier to view the errors without all the html surrounding it. –Jonathan Kuhn May 9 '12 at 17:48 add a comment| 1 Answe