Memory Error Django
here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss the workings and policies of this site About Us Learn more about Stack Overflow the company Business Learn more about hiring developers or posting ads with us Stack Overflow Questions Jobs Documentation Tags Users Badges Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 6.2 million programmers, just like you, helping each other. Join them; it only takes a minute: Sign up MemoryError with django up vote 1 down vote favorite 1 I have a django app that loads a lot of data into a sqlite3 database file. We are talking about millions of entries, entered in one function that at some point of its execution is throwing: File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 219, in inner self.__exit__(*sys.exc_info()) File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 207, in __exit__ self.exiting(exc_value, self.using) File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 286, in exiting leave_transaction_management(using=using) File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 56, in leave_transaction_management connection.leave_transaction_management() File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/backends/__init__.py", line 114, in leave_transaction_management self.rollback() MemoryError I examined my function and it is NOT leaking memory. What else can be done here? django share|improve this question edited Jan 11 '12 at 10:04 Daniel Roseman 322k24405490 asked Jan 11 '12 at 8:25 user1094786 1,55931631 2 Do you have DEBUG set to True, by any chance? –Daniel Roseman Jan 11 '12 at 10:04 add a comment| 3 Answers 3 active oldest votes up vote 1 down vote accepted The function may not be, but if your realing with such a huge database, the sqlite library may be consuming it in order to update the table. If you need to store such huge amounts of data you may have to move away from SQLite to a fully featured database server such as MySQL. share|improve this answer answered Jan 11 '12 at 8:27 Geoffrey 3,9661030 add a
- How-to work with largedatabases If you're working with resource intensive operations on large databases in Django, chances are you'll encounter a "Memory Error". I've seen this error numerous times over the last couple of weeks as part of a large data conversion project and wanted to share some the tips I learned. Number one, DON'T PANIC!!! It's not Django's fault, Django's just doing some caching for you to speed things up. This is a good thing, seriously… you want Django optimized for performance by default since most resource intense operations are one-off tasks, like data imports, bulk updates, special ad-hoc reports, etc. In these types of tasks all we need to do is slow Django down a bit http://stackoverflow.com/questions/8816238/memoryerror-with-django and ease up on the memory, here's how… Use MyModel.objects.all().iterator() instead of MyModel.objects.all() If you don't need all of the fields in a query use values_list Use the Memory efficient Django Queryset Iterator, written by Thierry Schellenbach set DEBUG = False in your settings.py file. If DEBUG is True, then Django saves a copy of every SQL statement it has executed. See Django FAQ: Why is Django leaking memory? (spoiler: Django's not really leaking memory). Ranked https://travelingfrontiers.wordpress.com/2010/06/26/django-memory-error-how-to-work-with-large-databases/ in the order of highest performance, but also in the order of most memory intensive. MyModel.objects.all() - fastest, for standard queries MyModel.objects.all().iterator() - medium, for large queries queryset_iterator(MyModel.objects.all()) - slowest, for monster queries One size doesn't fit all, use the best method for your particular task. By the way, the Django Snippet: "Memory Efficient Django Queryset Iterator" is solid. I've been using it a lot over the past couple of weeks and been validating the results along the way -- works like a champ🙂 Share this:EmailPrintTwitterLike this:Like Loading... Related Tags: Django This entry was posted on June 26, 2010 at 04:13 and is filed under Django. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site. Leave a Reply Cancel reply Enter your comment here... Fill in your details below or click an icon to log in: Email (required) (Address never made public) Name (required) Website You are commenting using your WordPress.com account. (LogOut/Change) You are commenting using your Twitter account. (LogOut/Change) You are commenting using your Facebook account. (LogOut/Change) You are commenting using your Google+ account. (LogOut/Change) Cancel Connecting to %s Notify me of new comments via email.
Previous TicketNext Ticket → Opened 5 years ago Closed 5 years ago Last modified 5 years ago #17346 closed https://code.djangoproject.com/ticket/17346 Uncategorized (invalid) MemoryError Reported by: anonymous Owned by: nobody Component: Uncategorized Version: 1.3 Severity: Normal Keywords: Cc: Triage Stage: Unreviewed Has patch: https://groups.google.com/d/topic/django-developers/c6ra9weXcl0 no Needs documentation: no Needs tests: no Patch needs improvement: no Easy pickings: no UI/UX: no Description (last modified by Karen Tracey) memory error I have my django application on a apache server with mod_wsgi When i start it with python manage.py runserver, it works perfectly, but if I try to use it over apache/mod_wsgi it causes major Error (MemoryError at /index) Any ideas? Description of problem: MemoryError memory error django at /index No exception supplied Request Method: GET Request URL: http://127.0.0.1/index Django Version: 1.3.1 Exception Type: MemoryError Exception Location: /usr/lib/python2.7/ctypes/__init__.py in
van GoogleInloggenVerborgen veldenZoeken naar groepen of berichten