Databases¶

PostgreSQL 8.2 to 8.2.4¶

The implementation of the population statistics aggregates STDDEV_POP and VAR_POP that shipped with PostgreSQL 8.2 to 8.2.4 are known to be faulty. Users of these releases of PostgreSQL are advised to upgrade to Release 8.2.5 or later. Django will raise a NotImplementedError if you attempt to use the StdDev(sample=False) or Variance(sample=False) aggregate with a database backend that falls within the affected release range.

PostgreSQL connection settings¶

See HOST for details.

Optimizing PostgreSQL’s configuration¶

Django needs the following parameters for its database connections:

• client_encoding: 'UTF8',
• default_transaction_isolation: 'read committed',
• timezone: 'UTC' when USE_TZ is True, value of TIME_ZONE otherwise.

If these parameters already have the correct values, Django won’t set them for every new connection, which improves performance slightly. You can configure them directly in postgresql.conf or more conveniently per database user with ALTER ROLE.

Django will work just fine without this optimization, but each new connection will do some additional queries to set these parameters.

Transaction handling¶

By default, Django runs with an open transaction which it commits automatically when any built-in, data-altering model function is called. The PostgreSQL backends normally operate the same as any other Django backend in this respect.

'OPTIONS': {
    'autocommit': True,
}

Version support¶

Django supports MySQL 5.0.3 and higher.

MySQL 5.0 adds the information_schema database, which contains detailed data on all database schema. Django’s inspectdb feature uses it.

Django expects the database to support Unicode (UTF-8 encoding) and delegates to it the task of enforcing transactions and referential integrity. It is important to be aware of the fact that the two latter ones aren’t actually enforced by MySQL when using the MyISAM storage engine, see the next section.

Storage engines¶

MySQL has several storage engines (previously called table types). You can change the default storage engine in the server configuration.

Until MySQL 5.5.4, the default engine was MyISAM [1]. The main drawbacks of MyISAM are that it doesn’t support transactions or enforce foreign-key constraints. On the plus side, it’s currently the only engine that supports full-text indexing and searching.

Since MySQL 5.5.5, the default storage engine is InnoDB. This engine is fully transactional and supports foreign key references. It’s probably the best choice at this point.

If you upgrade an existing project to MySQL 5.5.5 and subsequently add some tables, ensure that your tables are using the same storage engine (i.e. MyISAM vs. InnoDB). Specifically, if tables that have a ForeignKey between them use different storage engines, you may see an error like the following when running syncdb:

_mysql_exceptions.OperationalError: (
    1005, "Can't create table '\\db_name\\.#sql-4a8_ab' (errno: 150)"
)

In previous versions of Django, fixtures with forward references (i.e. relations to rows that have not yet been inserted into the database) would fail to load when using the InnoDB storage engine. This was due to the fact that InnoDB deviates from the SQL standard by checking foreign key constraints immediately instead of deferring the check until the transaction is committed. This problem has been resolved in Django 1.4. Fixture data is now loaded with foreign key checks turned off; foreign key checks are then re-enabled when the data has finished loading, at which point the entire table is checked for invalid foreign key references and an IntegrityError is raised if any are found.

MySQLdb¶

MySQLdb is the Python interface to MySQL. Version 1.2.1p2 or later is required for full MySQL support in Django.

Creating your database¶

You can create your database using the command-line tools and this SQL:

CREATE DATABASE <dbname> CHARACTER SET utf8;

This ensures all tables and columns will use UTF-8 by default.

Connecting to the database¶

Refer to the settings documentation.

Connection settings are used in this order:

1. OPTIONS.
2. NAME, USER, PASSWORD, HOST, PORT
3. MySQL option files.

In other words, if you set the name of the database in OPTIONS, this will take precedence over NAME, which would override anything in a MySQL option file.

Here’s a sample configuration which uses a MySQL option file:

# settings.py
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'OPTIONS': {
            'read_default_file': '/path/to/my.cnf',
        },
    }
}


# my.cnf
[client]
database = NAME
user = USER
password = PASSWORD
default-character-set = utf8

Several other MySQLdb connection options may be useful, such as ssl, use_unicode, init_command, and sql_mode. Consult the MySQLdb documentation for more details.

Creating your tables¶

When Django generates the schema, it doesn’t specify a storage engine, so tables will be created with whatever default storage engine your database server is configured for. The easiest solution is to set your database server’s default storage engine to the desired engine.

If you’re using a hosting service and can’t change your server’s default storage engine, you have a couple of options.

• After the tables are created, execute an ALTER TABLE statement to convert a table to a new storage engine (such as InnoDB):

  ALTER TABLE <tablename> ENGINE=INNODB;
  

  This can be tedious if you have a lot of tables.

• Another option is to use the init_command option for MySQLdb prior to creating your tables:

  'OPTIONS': {
     'init_command': 'SET storage_engine=INNODB',
  }
  

  This sets the default storage engine upon connecting to the database. After your tables have been created, you should remove this option as it adds a query that is only needed during table creation to each database connection.

• Another method for changing the storage engine is described in AlterModelOnSyncDB.

Table names¶

There are known issues in even the latest versions of MySQL that can cause the case of a table name to be altered when certain SQL statements are executed under certain conditions. It is recommended that you use lowercase table names, if possible, to avoid any problems that might arise from this behavior. Django uses lowercase table names when it auto-generates table names from models, so this is mainly a consideration if you are overriding the table name via the db_table parameter.

Savepoints¶

Both the Django ORM and MySQL (when using the InnoDB storage engine) support database savepoints, but this feature wasn’t available in Django until version 1.4 when such supports was added.

If you use the MyISAM storage engine please be aware of the fact that you will receive database-generated errors if you try to use the savepoint-related methods of the transactions API. The reason for this is that detecting the storage engine of a MySQL database/table is an expensive operation so it was decided it isn’t worth to dynamically convert these methods in no-op’s based in the results of such detection.

Notes on specific fields¶

Row locking with QuerySet.select_for_update()¶

MySQL does not support the NOWAIT option to the SELECT ... FOR UPDATE statement. If select_for_update() is used with nowait=True then a DatabaseError will be raised.

Substring matching and case sensitivity¶

For all SQLite versions, there is some slightly counter-intuitive behavior when attempting to match some types of strings. These are triggered when using the iexact or contains filters in Querysets. The behavior splits into two cases:

1. For substring matching, all matches are done case-insensitively. That is a filter such as filter(name__contains="aa") will match a name of "Aabb".

2. For strings containing characters outside the ASCII range, all exact string matches are performed case-sensitively, even when the case-insensitive options are passed into the query. So the iexact filter will behave exactly the same as the exact filter in these cases.

Some possible workarounds for this are documented at sqlite.org, but they aren’t utilised by the default SQLite backend in Django, as incorporating them would be fairly difficult to do robustly. Thus, Django exposes the default SQLite behavior and you should be aware of this when doing case-insensitive or substring filtering.

SQLite 3.3.6 or newer strongly recommended¶

Versions of SQLite 3.3.5 and older contains the following bugs:

• A bug when handling ORDER BY parameters. This can cause problems when you use the select parameter for the extra() QuerySet method. The bug can be identified by the error message OperationalError: ORDER BY terms must not be non-integer constants.
• A bug when handling aggregation together with DateFields and DecimalFields.

SQLite 3.3.6 was released in April 2006, so most current binary distributions for different platforms include newer version of SQLite usable from Python through either the pysqlite2 or the sqlite3 modules.

Version 3.5.9¶

The Ubuntu “Intrepid Ibex” (8.10) SQLite 3.5.9-3 package contains a bug that causes problems with the evaluation of query expressions. If you are using Ubuntu “Intrepid Ibex”, you will need to update the package to version 3.5.9-3ubuntu1 or newer (recommended) or find an alternate source for SQLite packages, or install SQLite from source.

At one time, Debian Lenny shipped with the same malfunctioning SQLite 3.5.9-3 package. However the Debian project has subsequently issued updated versions of the SQLite package that correct these bugs. If you find you are getting unexpected results under Debian, ensure you have updated your SQLite package to 3.5.9-5 or later.

The problem does not appear to exist with other versions of SQLite packaged with other operating systems.

Version 3.6.2¶

SQLite version 3.6.2 (released August 30, 2008) introduced a bug into SELECT DISTINCT handling that is triggered by, amongst other things, Django’s DateQuerySet (returned by the dates() method on a queryset).

You should avoid using this version of SQLite with Django. Either upgrade to 3.6.3 (released September 22, 2008) or later, or downgrade to an earlier version of SQLite.

Using newer versions of the SQLite DB-API 2.0 driver¶

For versions of Python 2.5 or newer that include sqlite3 in the standard library Django will now use a pysqlite2 interface in preference to sqlite3 if it finds one is available.

This provides the ability to upgrade both the DB-API 2.0 interface or SQLite 3 itself to versions newer than the ones included with your particular Python binary distribution, if needed.

“Database is locked” errors¶

SQLite is meant to be a lightweight database, and thus can’t support a high level of concurrency. OperationalError: database is locked errors indicate that your application is experiencing more concurrency than sqlite can handle in default configuration. This error means that one thread or process has an exclusive lock on the database connection and another thread timed out waiting for the lock the be released.

Python’s SQLite wrapper has a default timeout value that determines how long the second thread is allowed to wait on the lock before it times out and raises the OperationalError: database is locked error.

If you’re getting this error, you can solve it by:

• Switching to another database backend. At a certain point SQLite becomes too “lite” for real-world applications, and these sorts of concurrency errors indicate you’ve reached that point.

• Rewriting your code to reduce concurrency and ensure that database transactions are short-lived.

• Increase the default timeout value by setting the timeout database option option:

  'OPTIONS': {
      # ...
      'timeout': 20,
      # ...
  }
  

  This will simply make SQLite wait a bit longer before throwing “database is locked” errors; it won’t really do anything to solve them.

QuerySet.select_for_update() not supported¶

SQLite does not support the SELECT ... FOR UPDATE syntax. Calling it will have no effect.

Parameters not quoted in connection.queries¶

sqlite3 does not provide a way to retrieve the SQL after quoting and substituting the parameters. Instead, the SQL in connection.queries is rebuilt with a simple string interpolation. It may be incorrect. Make sure you add quotes where necessary before copying a query into a SQLite shell.

Connecting to the database¶

Your Django settings.py file should look something like this for Oracle:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.oracle',
        'NAME': 'xe',
        'USER': 'a_user',
        'PASSWORD': 'a_password',
        'HOST': '',
        'PORT': '',
    }
}

If you don’t use a tnsnames.ora file or a similar naming method that recognizes the SID (“xe” in this example), then fill in both HOST and PORT like so:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.oracle',
        'NAME': 'xe',
        'USER': 'a_user',
        'PASSWORD': 'a_password',
        'HOST': 'dbprod01ned.mycompany.com',
        'PORT': '1540',
    }
}

You should supply both HOST and PORT, or leave both as empty strings.

Threaded option¶

If you plan to run Django in a multithreaded environment (e.g. Apache in Windows using the default MPM module), then you must set the threaded option of your Oracle database configuration to True:

'OPTIONS': {
    'threaded': True,
},

Failure to do this may result in crashes and other odd behavior.

INSERT ... RETURNING INTO¶

By default, the Oracle backend uses a RETURNING INTO clause to efficiently retrieve the value of an AutoField when inserting new rows. This behavior may result in a DatabaseError in certain unusual setups, such as when inserting into a remote table, or into a view with an INSTEAD OF trigger. The RETURNING INTO clause can be disabled by setting the use_returning_into option of the database configuration to False:

'OPTIONS': {
    'use_returning_into': False,
},

In this case, the Oracle backend will use a separate SELECT query to retrieve AutoField values.

Naming issues¶

Oracle imposes a name length limit of 30 characters. To accommodate this, the backend truncates database identifiers to fit, replacing the final four characters of the truncated name with a repeatable MD5 hash value.

When running syncdb, an ORA-06552 error may be encountered if certain Oracle keywords are used as the name of a model field or the value of a db_column option. Django quotes all identifiers used in queries to prevent most such problems, but this error can still occur when an Oracle datatype is used as a column name. In particular, take care to avoid using the names date, timestamp, number or float as a field name.

NULL and empty strings¶

Django generally prefers to use the empty string (‘’) rather than NULL, but Oracle treats both identically. To get around this, the Oracle backend ignores an explicit null option on fields that have the empty string as a possible value and generates DDL as if null=True. When fetching from the database, it is assumed that a NULL value in one of these fields really means the empty string, and the data is silently converted to reflect this assumption.

TextField limitations¶

The Oracle backend stores TextFields as NCLOB columns. Oracle imposes some limitations on the usage of such LOB columns in general:

• LOB columns may not be used as primary keys.
• LOB columns may not be used in indexes.
• LOB columns may not be used in a SELECT DISTINCT list. This means that attempting to use the QuerySet.distinct method on a model that includes TextField columns will result in an error when run against Oracle. As a workaround, use the QuerySet.defer method in conjunction with distinct() to prevent TextField columns from being included in the SELECT DISTINCT list.