Disk Error Exception Could Not Find Tasktracker
here for a hadoop could not find any valid local directory for output quick overview of the site Help Center Detailed org apache hadoop util diskchecker diskerrorexception could not find any valid local directory for answers to any questions you might have Meta Discuss the workings and policies of this site About Us Learn more about Stack Overflow the company Business Learn more about hiring developers or posting ads with us Stack Overflow Questions Jobs Documentation Tags Users Badges Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 4.7 million programmers, just like you, helping each other. Join them; it only takes a minute: Sign up keep failing in running hadoop distributed mode up vote 0 down vote favorite 1 I'm stuck on this problem for a very long time. I try to run something in distibuted node. I have 2 datanodes and a master with namenode and jobtracker. I keep getting the following error in tasktracker.log of each of the nodes < 2012-01-03 08:48:30,910 WARN mortbay.log - /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201201031846_0001/attempt_201201031846_0001_m_000000_1/output/file.out.index in any of the configured local directories 2012-01-03 08:48:40,927 WARN mapred.TaskTracker - getMapOutput(attempt_201201031846_0001_m_000000_2,0) failed : org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201201031846_0001/attempt_201201031846_0001_m_000000_2/output/file.out.index in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webap
Classic List Threaded ♦ ♦ Locked 8 messages igors Reply | Threaded Open this post in threaded view ♦ ♦ | Report Content as Inappropriate ♦ ♦ org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_local.. Hi to all, We're having trouble with nutch when trying to crawl. Nutch version 1.4, Hadoop 0.20.2. (working in local mode). After 2 days of crawling we've got: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_local_0015/attempt_local_0015_m_000000_0/output/spill0.out in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389) http://stackoverflow.com/questions/8717049/keep-failing-in-running-hadoop-distributed-mode at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138) at org.apache.hadoop.mapred.MapOutputFile.getSpillFile(MapOutputFile.java:94) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1443) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1154) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:359) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) We've looked at mailing list archives but I'm not sure if exact thing is mentioned. Tried to upgrade to hadoop-core-0.20.203.0.jar but then this is thrown: Exception in http://lucene.472066.n3.nabble.com/org-apache-hadoop-util-DiskChecker-DiskErrorException-Could-not-find-taskTracker-jobcache-job-local-td3950733.html thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration Can someone, please, shed some light on this? Thanks. Igor Adriana Farina Reply | Threaded Open this post in threaded view ♦ ♦ | Report Content as Inappropriate ♦ ♦ Re: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_local.. Hello! I had the same kind of problem. In my case this was caused by one of the node of my cluster with full memory, so to solve the priblem I simply freed up memory on that node. Check if all of the nodes of your cluster have free memory. As for the second error, it seems you're missing some library: try adding it to hadoop. Inviato da iPhone Il giorno 30/apr/2012, alle ore 15:15, Igor Salma <[hidden email]> ha scritto: > Hi to all, > > We're having trouble with nutch when trying to crawl. Nutch version 1.4, > Hadoop 0.20.2. (working in local mode). After 2 days of crawling we've got: > org
at 1:16 pm Hi to all,We're having trouble with nutch when trying to crawl. Nutch version 1.4,Hadoop 0.20.2. (working in local mode). After http://grokbase.com/t/nutch/user/124yz5h82e/org-apache-hadoop-util-diskchecker-diskerrorexception-could-not-find-tasktracker-jobcache-job-local 2 days of crawling we've got:org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not findtaskTracker/jobcache/job_local_0015/attempt_local_0015_m_000000_0/output/spill0.outin any of the configured local directoriesatorg.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)atorg.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)atorg.apache.hadoop.mapred.MapOutputFile.getSpillFile(MapOutputFile.java:94)atorg.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1443)atorg.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1154)at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:359)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)atorg.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)We've looked at mailing list archives but https://www.mail-archive.com/nutch-user@lucene.apache.org/msg16308.html I'm not sure if exact thing ismentioned. Tried to upgrade to hadoop-core-0.20.203.0.jar but then this isthrown:Exception in thread "main" java.lang.NoClassDefFoundError:org/apache/commons/configuration/ConfigurationCan someone, please, shed could not some light on this?Thanks.Igor reply Tweet Search Discussions Search All Groups user 3 responses Oldest Nested Adriana Farina Hello! I had the same kind of problem. In my case this was caused by one of the node of my cluster with full memory, so to solve the could not find priblem I simply freed up memory on that node. Check if all of the nodes of your cluster have free memory. As for the second error, it seems you're missing some library: try adding it to hadoop. Inviato da iPhone Il giorno 30/apr/2012, alle ore 15:15, Igor Salma ha scritto: Adriana Farina at Apr 30, 2012 at 1:34 pm ⇧ Hello!I had the same kind of problem. In my case this was caused by one of the node of my cluster with full memory, so to solve the priblem I simply freed up memory on that node. Check if all of the nodes of your cluster have free memory.As for the second error, it seems you're missing some library: try adding it to hadoop.Inviato da iPhoneIl giorno 30/apr/2012, alle ore 15:15, Igor Salma
= 10 depth = 5 Injector: starting Injector: crawlDb: cmrolg-even/crawl/crawldb. Injector: urlDir: /projects/events/search/nutch-1.0/cmrolg-even/urls Injector: Converting injected urls to crawl db entries. Injector: Merging injected urls into crawl db. Injector: done Generator: Selecting best-scoring urls due for fetch. Generator: starting Generator: segment: cmrolg-even/crawl/segments/20100420175131 Generator: filtering: true Generator: jobtracker is 'local', generating exactly one partition. Generator: Partitioning selected urls by host, for politeness. Generator: done. Fetcher: starting Fetcher: segment: cmrolg-even/crawl/segments/20100420175131 Fetcher: threads: 10 QueueFeeder finished: total 1 records. fetching http:// [...] -finishing thread FetcherThread, activeThreads=1 -finishing thread FetcherThread, activeThreads=1 -finishing thread FetcherThread, activeThreads=1 -finishing thread FetcherThread, activeThreads=1 -finishing thread FetcherThread, activeThreads=1 -finishing thread FetcherThread, activeThreads=1 -finishing thread FetcherThread, activeThreads=1 -finishing thread FetcherThread, activeThreads=1 -finishing thread FetcherThread, activeThreads=1 -finishing thread FetcherThread, activeThreads=0 -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0 -activeThreads=0, Exception in thread "main" java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:969): at org.apache.nutch.crawl.Crawl.main(Crawl.java:122) Elapsed time: 16 (So yes, 16 seconds total) 2010-04-20 17:51:36,994 INFO fetcher.Fetcher - fetching http:// [...] 2010-04-20 17:51:37,006 INFO http.Http - http.proxy.host = null 2010-04-20 17:51:37,007 INFO http.Http - http.proxy.port = 8080 2010-04-20 17:51:37,007 INFO http.Http - http.timeout = 10000 2010-04-20 17:51:37,007 INFO http.Http - http.content.limit = -1 2010-04-20 17:51:37,007 INFO http.Http - http.agent = Nutch/Nutch (webmaster@ [...] ) 2010-04-20 17:51:37,007 INFO http.Http - protocol.plugin.check.blocking = false...] 2010-04-20 17:51:37,007 INFO http.Http - protocol.plugin.check.robots = false 2010-04-20 17:51:37,025 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=1 2010-04-20 17:51:37,027 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=1 2010-04-20 17:51:37,028 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=1 2010-04-20 17:51:37,030 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=1 2010-04-20 17:51:37,031 INFO fetcher.Fetcher - -finish