Http Error 429 Unknown
Contents |
here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss the workings and policies of this site About Us Learn more about Stack Overflow the company Business Learn more about hiring developers or posting python requests 429 ads with us Stack Overflow Questions Jobs Documentation Tags Users Badges Ask Question x Dismiss Join the geopy http error 429: too many requests Stack Overflow Community Stack Overflow is a community of 4.7 million programmers, just like you, helping each other. Join them; it only takes a minute: error: too many requests Sign up urllib2 HTTP error 429 up vote 7 down vote favorite 2 So I have a list of sub-reddits and I'm using urllib to open them. As I go through them eventually urllib fails with: urllib2.HTTPError: HTTP Error 429: Unknown
429 Too Many Requests Squarespace
Doing some research I found that reddit limits the ammount of requests to their servers by IP: Make no more than one request every two seconds. There's some allowance for bursts of requests, but keep it sane. In general, keep it to no more than 30 requests in a minute. So I figured I'd use time.sleep() to limit my requests to one page each 10 seconds. This ends up failing just as well. The quote above is grabbed from the reddit API http 429 retry after page. I am not using the reddit API. At this point I'm thinking two things. Either that limit applies only to the reddit API or urllib also has a limit. Does anyone know which one of these two things it is? Or how I could go around this issue? python urllib2 reddit share|improve this question asked Nov 3 '12 at 20:10 Florin Stingaciu 3,8721133 3 There's no limit in urllib2, as you could have found out by trying some other webpage. They might have blocked your IP from API access, try sending them an email. –Fred Foo Nov 3 '12 at 20:28 @larsmans The reqeusts go through randomly. I get some then it fails for a while and then it works again. Also they could not have blocked me from their API as I am not using their API. –Florin Stingaciu Nov 3 '12 at 20:43 add a comment| 2 Answers 2 active oldest votes up vote 15 down vote accepted From https://github.com/reddit/reddit/wiki/API: Many default User-Agents (like "Python/urllib" or "Java") are drastically limited to encourage unique and descriptive user-agent strings. This applies to regular requests as well. You need to supply your own user agent header when making the request. #TODO: change user agent string hdr = { 'User-Agent' : 'super happy flair bot by /u/spladug' } req = urllib2.Request(url, headers=hdr) html = urllib2.urlopen(req).read() However, this will create a new connection for every request. I suggest using another library that is capable of
Sign in Pricing Blog Support Search GitHub This 429 error dropbox repository Watch 57 Star 1,068 Fork 220 praw-dev/praw Code
Error 429 Too Many Requests - Please Slow Down
Issues 28 Pull requests 0 Projects 0 Pulse Graphs New issue HTTP Error
Error Code 429 Activex Component Can't Create Object
429: Unknown when fetching submissions #63 Closed mpenkov opened this Issue Feb 16, 2012 · 9 comments Projects None yet Labels http://stackoverflow.com/questions/13213048/urllib2-http-error-429 None yet Milestone No milestone Assignees No one assigned 2 participants mpenkov commented Feb 16, 2012 Hi, I am the developer of http://rvytpl.appspot.com/. First of all, thank you for providing the Python library for the Reddit API. It's very convenient and easy https://github.com/praw-dev/praw/issues/63 to use. I'm encountering a problem relatively frequently in this code: reddit_api = reddit.Reddit(user_agent=USER_AGENT) subreddit = reddit_api.get_subreddit('videos').get_top(limit=100) new_entries = { } for rank in range(100): if rank and (rank % 25 == 0): logging.info('Fetched %d entries, sleeping' % rank) time.sleep(60) entry = subreddit.next() new_entries[entry.id] = (rank, entry) I'm trying to fetch the top 100 entries of the videos subreddit. I get this exception: HTTP Error 429: Unknown Traceback (most recent call last): File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__ handler.post(*groups) File "/base/data/home/apps/s~rvytpl/1.356858274957999053/admin.py", line 62, in post entry = subreddit.next() File "/base/data/home/apps/s~rvytpl/1.356858274957999053/reddit/__init__.py", line 231, in get_content page_data = self.request_json(page_url, url_data=url_data) File "/base/data/home/apps/s~rvytpl/1.356858274957999053/reddit/decorators.py", line 110, in error_checked_func return_value = func(self, *args, **kwargs) File "/base/data/home/apps/s~rvytpl/1.356858274957999053/reddit/__init__.py", line 265, in request_json response = self._request(page_url, params, url_data) File "/base/data/home/apps/s~rvytpl/1.356858274957999053/reddit/__init__.py", line 165, in _request url_data) File "/base/data/home/apps/s~rvytpl/1.356858274957999053/reddit/util.py", line 4
»learnpythoncommentsWant to join? Log in or sign up in seconds.|Englishlimit my search to /r/learnpythonuse the following search parameters to narrow your results:subreddit:subredditfind submissions in "subreddit"author:usernamefind submissions by "username"site:example.comfind submissions from "example.com"url:textsearch for "text" in urlselftext:textsearch https://www.reddit.com/r/learnpython/comments/1q6xpk/i_cant_figure_out_why_urllib2_will_load/ for "text" in self post contentsself:yes (or self:no)include (or exclude) self postsnsfw:yes (or nsfw:no)include (or exclude) results marked as https://forum.sinusbot.com/threads/error-unable-to-download-webpage-http-error-429.1072/ NSFWe.g. subreddit:aww site:imgur.com dogsee the search faq for details.advanced search: by author, subreddit...this post was submitted on 08 Nov 201310 points (82% too many upvoted)shortlink: remember mereset passwordloginSubmit a new text postlearnpythonsubscribeunsubscribe66,572 readers~98 users here now Welcome to /r/LearnPython! Please read the rules and guidelines below and search before posting. Learning resources All learning resources are in the wiki: /r/learnpython/w/index Frequently Asked Questions: too many requests /r/learnpython/w/FAQ IRC Chat Join us in the IRC channel: ##learnpython on irc.freenode.net Webchat link: webchat.freenode.net Guide on how to join and different IRC clients: /wiki/IRC Rules General Rules Posting only assignment/project goal is not allowed. Read posting guidelines. Easily googleable questions are not allowed. Posting screenshot of the code is (generally) not allowed. Read posting guidelines. Insulting will not be tolerated. Reddit rules These apply also on this subreddit. Breaking these rules may result in post removal and/or ban from this subreddit. Guidelines Commenting Try to guide OP to a solution instead of providing one directly. Provide links to related resources. Answer the question and highlight side-issues if any exist. Do
!! Here are my logs : ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. youtube-dl -U = youtube-dl is up-to-date (2016.01.15). youtube-dl --verbose : [debug] System config: [u'--extract-audio', u'--no-mtime'] [debug] User config: [] [debug] Command-line args: [u'--verbose', u' [debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968 [debug] youtube-dl version 2016.01.15 [debug] Python version 2.7.3 - Linux-2.6.32-openvz-042stab106.4-amd64-x86_64-with-debian-7.9 [debug] exe versions: none [debug] Proxy map: {} [youtube] TMSIR210mRg: Downloading webpage ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 351, in _request_webpage return self._downloader.urlopen(url_or_request) File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1890, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python2.7/urllib2.py", line 407, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 520, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 445, in error return self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 528, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) Can you help me please ? Thank you for help. Ps: sorry for my english , I'm french. Grepher76, Jan 20, 2016 #1 Xuxe encounters broken stuff Staff Member V.I.P. is awesome! Insider Grepher76 said: ↑ ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and in