Jsoup Http Error Fetching Url. Status=500
Contents |
here for a quick overview of the site Help Center Detailed answers jsoup http error fetching url. status=403 to any questions you might have Meta Discuss the workings jsoup httpstatusexception and policies of this site About Us Learn more about Stack Overflow the company Business org.jsoup.httpstatusexception: http error fetching url. status=503 Learn more about hiring developers or posting ads with us Stack Overflow Questions Jobs Documentation Tags Users Badges Ask Question x Dismiss Join the org.jsoup.httpstatusexception: http error fetching url. status=404 Stack Overflow Community Stack Overflow is a community of 6.2 million programmers, just like you, helping each other. Join them; it only takes a minute: Sign up how to fix HTTP error fetching URL. Status=500 in java while crawling? up vote 7 down vote favorite I am trying to crawl
Org.jsoup.httpstatusexception 403
the user's ratings of cinema movies of imdb from the review page: (number of movies in my database is about 600,000). I used jsoup to parse pages as below: (sorry, I didn't write the whole code here since it is too long) try { //connecting to mysql db ResultSet res = st .executeQuery("SELECT id, title, production_year " + "FROM title " + "WHERE kind_id =1 " + "LIMIT 0 , 100000"); while (res.next()){ ....... ....... String baseUrl = "http://www.imdb.com/search/title?release_date=" + ""+year+","+year+"&title="+movieName+"" + "&title_type=feature,short,documentary,unknown"; Document doc = Jsoup.connect(baseUrl) .userAgent("Mozilla") .timeout(0).get(); ..... ..... //insert ratings into database ... I tested it for the first 100, then first 500 and also for the first 2000 movies in my db and it worked well. But the problem is that when I tested for 100,000 movies I got this error: org.jsoup.HttpStatusException: HTTP error fetching URL. Status=500, URL=http://www.imdb.com/search/title?release_date=1899,1899&title='Columbia'%20Close%20to%20the%20Wind&title_type=feature,short,documentary,unknown at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:449) at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:424) at org.jsoup.helper.HttpConnection.execute(Ht
here for a quick overview of the site Help Center Detailed answers to any questions you might
Http Error Fetching Url. Status=400
have Meta Discuss the workings and policies of this site About http error fetching url. status=503 jsoup Us Learn more about Stack Overflow the company Business Learn more about hiring developers or posting http error fetching url. status=999 ads with us Stack Overflow Questions Jobs Documentation Tags Users Badges Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 6.2 http://stackoverflow.com/questions/21858701/how-to-fix-http-error-fetching-url-status-500-in-java-while-crawling million programmers, just like you, helping each other. Join them; it only takes a minute: Sign up HTTP Error fetching URL. Status=400 up vote 0 down vote favorite JSoup seems to work with just about all URLs I've tried but this one gives me the 400 error. String url = "http://localad.walmart.com?storeref=3008&forceview=y"; Response response = Jsoup.connect(url.replaceAll(" http://stackoverflow.com/questions/23966210/http-error-fetching-url-status-400 ", "%20")) .method(Method.GET) .userAgent("Mozilla") .followRedirects(false) .timeout(5000) .data("pragma", "no-cache") .execute(); Error I get is: Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=400, URL=http://localad.walmart.com?storeref=3008&forceview=y&pragma=no-cache at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:449) at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:424) at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:178) jsoup share|improve this question edited May 31 '14 at 3:55 zishe 6,788103272 asked May 31 '14 at 3:50 user2117997 11 add a comment| 2 Answers 2 active oldest votes up vote 0 down vote 400 is Bad Request. You should try URLEncoder.encode(url, "UTF-8") instead of using replaceAll. share|improve this answer answered May 31 '14 at 3:57 J. Titus 3,9781921 I did try that but because the URL has no protocol that throws a MalformedURLException –user2117997 May 31 '14 at 4:33 add a comment| up vote 0 down vote Moved on without relying on JSoup for resolving the intermediate (redirected URL). I needed the final redirect URL (which JSoup had no problems working with), so used the following code to get that. import java.net.URI; import org.apache.http.HttpRequest; import org.apache.http.HttpResponse; import org.apache.http.ProtocolException; import or
here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss http://stackoverflow.com/questions/13558448/jsoup-http-error-fetching-url the workings and policies of this site About Us Learn more about Stack Overflow the company Business Learn more about hiring developers or posting ads with us Stack Overflow Questions Jobs Documentation Tags Users Badges Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 6.2 million programmers, just like you, helping http error each other. Join them; it only takes a minute: Sign up Jsoup http error fetching url up vote 7 down vote favorite I just download the latest version of jsoup(1.7.1) and follow the official code(changed the url). Then i got "http error fetching url" public class MainActivity extends Activity { @Override public void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); http error fetching try { loadData(); } catch (IOException e) { Log.i("error",e.getMessage()); } } @Override public boolean onCreateOptionsMenu(Menu menu) { getMenuInflater().inflate(R.menu.activity_main, menu); return true; } public void loadData() throws IOException { Document doc = Jsoup.connect("http://forum.mtr.com.hk/search.php?station=30&cat=&x=25&y=2").get(); String title = doc.title(); Log.i("title",title); }} What's the problem of my code? It seems the error just happend in Android Project since i do the same thing in a Java Project that works fine. Notes: - I haved added the Internet permission android jsoup share|improve this question edited Nov 26 '12 at 4:16 Mayur Birari 5,01282657 asked Nov 26 '12 at 3:55 Hekensi 4215 add a comment| 1 Answer 1 active oldest votes up vote 4 down vote I had a similar problem. Could be that your device connection is too slow and the connect() function times-out too soon, or maybe it is failing with some harmless HTTP error status. Also, my Jsoup connection worked for most pages, but for some I was getting "HTTP error fetching URL. Status=307". Turns out this is really a redirect request from the