The wisdom of DEBUG=False when loading LOTS of data.

For a site that I’ve been working on, I’ve enabled a load up of data all in a python script. Darned handy. The magic method that does all that nice reseting bits is:

from django.core import management
from django.contrib.auth.create_superuser import createsuperuser
from django.db import connection
cursor = connection.cursor()
cursor.execute(“select tablename from pg_tables where schemaname=’public'”)
tables = cursor.fetchall()
for table in tables:
cursor.execute(‘drop table “%s” cascade’ % table[0])
management.syncdb()

And then I go on to run a bunch of other methods to do all that cool initial data loading goodness. Only we ran into a problem:

Traceback (most recent call last):
File “initial_data_load.py”, line 641, in ?
create_geography_table()
File “initial_data_load.py”, line 474, in create_geography_table
latitude = radians(float(lat)),)
File “/users/home/joseph/local/Django-0.95/django/db/models/manager.py”, line 73, in create
return self.get_query_set().create(**kwargs)
File “/users/home/joseph/local/Django-0.95/django/db/models/query.py”, line 223, in create
obj.save()
File “/users/home/joseph/local/Django-0.95/django/db/models/base.py”, line 203, in save
‘,’.join(placeholders)), db_values)
File “/users/home/joseph/local/Django-0.95/django/db/backends/util.py”, line 19, in execute
self.db.queries.append({
MemoryError

I couldn’t figure out what was happening for a while, since it was only happening on one system (and annoyingly – not my laptop…). Turns out the detail is in that last line of the traceback. We had “DEBUG=True” enabled in settings.py, and that there Geography table – well, it’s a big’un. We blew out the memory for the python process because when DEBUG=True, the database cursor does a really nice little thing – it keeps all your SQL queries for you. Which is great until you hit that per-process memory limit and your script terminates unexpectedly.

Switching the settings.py to DEBUG=False stopped keeping those queries in memory – and the memory exception went away. Yeah for the wisdom of DEBUG=False!

4 thoughts on “The wisdom of DEBUG=False when loading LOTS of data.

  1. We could have used straight SQL, but we were using the Django ORM setup because it was easier. The only thing we hadn’t counted on was how many SQL statements would get made when we were “bulk loading” using it with DEBUG=True, the cursor object keeping a copy of all those statements in memory…

    Like

  2. Thank you so much, this is exactly the problem I had. Django ORM worked fine for importing the smaller (legacy) tables into my application, but ran out of memory on importing the 225K rows of data in my history table.

    Thanks again for the solution.

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s