Error 404 Robots.txt
only in small numbers, so I wasn't too worried. This month, the number of 404 errors have increased considerably. In most cases, the 404 figure is 25-50% of the successful download figure; on one site, the errors are 1.5 times as numerous as the successes. Other sites continue to show no 404 errors for robots.txt. I haven't changed any of the robots.txt files over this period. Google isn't reporting any errors and I can access the files at domain/robots.txt every time I try. Please can somebody explain what's going on? (Apologies if this is the wrong forum for the question.) TechnoBear 2011-09-01 08:34:35 UTC #2 fastreplies said: 404 might be generated by old cache as for example G. is reporting 404 for dead links/pages I am well aware of that and my question is not about 404s for non-existent pages, which I understand perfectly well. I am asking why some sites are suddenly producing hundreds of 404 errors for a page which does exist, which has not been modified in months and which I and most other users can access without problem. Stevie_D 2011-08-31 17:37:40 UTC #3 Are the errors on /robots.txt or /someotherfolder/robots.txt? Can you see in the logs the user-agent that caused these errors? All in all, if the only unexplained 404s on your site are on robots.txt, you've almost certainly got bigger things to worry about (if not, I want your life!) ... so many sites don't have that file at all, that no search engine is going to slap down your site for not having it. TechnoBear 2011-08-31 19:03:14 UTC #4 The errors are on /robots.txt. I hadn't thought to look for the user-agent, but I've checked the logs and they are almost all discoveryengine.com (discobot). I've checked the logs for the sites where I'm not getting 404s for the robots.txt and the same agent is visiting those, from the same IP, without any problem. There are the odd one or two 404s for other existing pages, but nothing of any significance. I'm not worried about this from an SEO point of view (hence I wasn't sure this was the right place to post) - it's just my usual paranoid panic when I find something happening on my sites that I don't understand and can't explain.:confused: fastreplies 2011-08-31 20:12:16 UTC #5 404 might be generated by old cache as for example G. is reporting 404 for dead links/pages we're deleting from our directory by dozens every week but it doesn't mean we have problems. You might experiencing the same condition and if you know there is nothing wrong with your site's files, then stop worry, get yourself cup of coffee and relax. fastreplies Mittineague 2014-09-30 23:46:12 UTC #6 Home Categories FAQ/Guidel
von GoogleAnmeldenAusgeblendete FelderNach Gruppen oder Nachrichten suchen
von GoogleAnmeldenAusgeblendete FelderNach Gruppen oder Nachrichten suchen
Start here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss the workings and policies of this site About Us Learn more about Stack Overflow the company Business Learn more about hiring developers or posting ads with us Server Fault Questions Tags Users Badges Unanswered Ask Question _ Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute: Sign up Here's how it works: Anybody can ask a question Anybody can answer The best answers are voted up and rise to the top robots.txt and other .txt returning 404 on IIS? up vote 1 down vote favorite 2 We have an IIS site running Dotnetnuke that we took over from another group. We have added a robots.txt file to the root but it returns a 404. Actually any txt file in the root seems to return 404. I can't seem to spot where they may have blocked this. Any suggestions? EDIT: We have a duplciate of the site files and database setup on a clean IIS server and it works fine there, so it seems to be at the IIS level or somethign else outside of the web.config or similar. EDIT2: Turns out it was a bug within a module that the developer had been using causing redirects for txt files to be messed up. Thanks everyone that replied. iis http-status-code-404 robots.txt share|improve this question edited Aug 31 '09 at 17:04 asked Aug 31 '09 at 14:54 schooner2000 14615 Which version of IIS? –Massimo Aug 31 '09 at 15:38 Verify that server root is actually configured to the folder you think it is. –Chris Nava Aug 31 '09 at 15:39 IIS 6. Root is where we expected. We cannot browse to a txt file regardless of where we place it it would appear. Tried ones uploaded via DNN and uploads file and downlaod link works but cannot browse any txt file directly via URL in browser. At a total loss as to why. –schooner2000 Aug 31 '09 at 15:42 Which 404 is returned? 404.2 or 404.3 -- or another? –Chris_K Aug 31 '09 at 16:44 add a comment| 3 Answers 3 active oldest votes up vote 6 down vote Another (small) possibility: did you check that the file is really robots.txt, not robots.txt.txt - as in, did you change your Windows Explorer settings? share|improve this answer answere