Some tips for django

Posted on 2018-05-21 in Trucs et astuces

View last executed query

Run (the query will be as executed by the database with values correctly replaced and escaped):

from django.db import connection
print(connection.queries[-1])

Logging

Queries

You can configure your logger to view all requests made to the database with:

LOGGING = {
    # ...
    'loggers': {
        # ...
        'django.db': {
            'handlers': ['console'],
            'level': 'DEBUG',
        },
    },
}

Source: Surviving Django (if you care about databases) under Another random bit of advice.

Migrations

Checks

You can use this check to verify your migrations during CI (no dependency issue and no creation):

python manage.py makemigrations
if [[ $(git status --porcelain | grep migrations | wc -l) -gt 0 ]]; then
    echo 'New migrations were created in the project. Please fix that.' >&2
    exit 1
fi

If you want to use it in a git hook, you should only consider untracked migrations:

python manage.py migrate
migs_exit_code=$?

if [[ "${migs_exit_code}" -ne 0 ]]; then
    exit 1
fi

if [[ $(git status --porcelain | grep migrations | grep .py | grep '??' | wc -l) -gt 0 ]]; then
    echo 'New migrations were created in the project. Please fix that.' >&2
    exit 1
fi

Or you can just use python manage.py makemigrations --dry-run --check which I discovered recently.

Fake all pending migrations

#!/usr/bin/env bash

set -eu
set -o pipefail

# This will contain a list like this:
#wagtailusers
# [ ] 0001_initial
# [ ] 0002_add_verbose_name_on_userprofile
# [ ] 0003_add_verbose_names
# [ ] 0004_capitalizeverbose
# [ ] 0005_make_related_name_wagtail_specific
# [ ] 0006_userprofile_prefered_language
# [ ] 0007_userprofile_current_time_zone
# [ ] 0008_userprofile_avatar
# [ ] 0009_userprofile_verbose_name_plural

pending_migrations="$(python manage.py showmigrations | grep --color '\[ \]\|^[a-z]' | grep --color '[  ]' -B 1)"

declare -A app_to_last_migration

# Make sure we loop over lines, not words.
IFS=$'\n'
# Array must not be quoted to correctly loop over each line.
# shellcheck disable=SC2068
for line in ${pending_migrations[@]}; do
    if [[ "$line" =~ ^[a-z]+ ]]; then
        app_name="${line}"
    elif [[ "$line" =~ \[ ]]; then
        # Capture the last migration of the app to pass it and all migrations before it.
        migration_name=$(echo "${line}" | awk '{print $3}')
        app_to_last_migration["${app_name}"]="${migration_name}"
    fi
done

for app_name in "${!app_to_last_migration[@]}"; do
    python manage.py migrate ${app_name} ${app_to_last_migration[${app_name}]} --fake
done

Model proxies

They are useful to change the behavior of a model (based on a type column for instance) or to work on a subset of a table (based on a type column for instance). See the documentation for more details.

from django.db import models

class Person(models.Model):
    first_name = models.CharField(max_length=30)
    last_name = models.CharField(max_length=30)

class MyPerson(Person):
    class Meta:
        proxy = True

    def do_something(self):
        # ...
        pass

class OrderedPerson(Person):
    class Meta:
        ordering = ["last_name"]
        proxy = True

Easily increment migrations numbers

Here is a small bash script to increment the numbers of your migrations. Put it into your ~/.profile or ~/.bashrc. Use it like: inc-migrations PROJECT/apps/APP_NAME/migrations 0012

It will rename all migrations from 0012 (ie transform 0012_mig.py into 0013_mig.py and so on) and replace all occurrences of the previous name (eg 0012_mig.py) with the new one (eg 0013_mig.py).

function inc-migrations() {
    local folder="$1"
    local number="$2"
    local start_file
    local new_number
    local next_file

    if ! [[ "$(pwd)" =~ "${folder}$" ]]; then
        cd "${folder}"
    fi

    if [[ -f "${number}" ]]; then
        start_file="${number}"
        number=$(echo "${number}" | cut -d _ -f 1)
    else
        start_file=$(ls ${number}* 2> /dev/null)
    fi

    if [[ ! -f "${start_file}" ]]; then
        echo "${start_file} doesn't exits" >&2
        return 1
    fi
    let "new_number = ${number} + 1"
    new_number=$(printf "%04d\n" "${new_number}")
    next_file=$(ls ${new_number}* 2> /dev/null)

    new_file_name="${start_file/${number}/${new_number}}"
    git mv "${start_file}" "${new_file_name}"
    sed -i "s/${start_file/.py/}/${new_file_name/.py/}/g" *.py

    if [[ -f "${next_file:-}" ]]; then
        inc-migrations "${folder}" "${next_file}"
    fi
}

Create a widget with custom display

class BonusTimeWidget(AdminIntegerFieldWidget):
    UNITS_TO_TRANSLATIONS = {
        'month': partial(ungettext_lazy, '%(count)d month', '%(count)d free months'),
    }

    def __init__(self, *args, unit='month', **kwargs):
        super().__init__(*args, **kwargs)
        if unit not in self.UNITS_TO_TRANSLATIONS:
            supported_units = ','.join(self.UNITS_TO_TRANSLATIONS.keys())
            raise ValueError(
                f'{unit} is not a supported unit. Supported units are: {supported_units}',
            )

        self.bonus_translation = self.UNITS_TO_TRANSLATIONS[unit]

    def render(self, name, value, attrs=None):
        """Render the value in a custom span."""
        if value == 0:
            return ''

        text = self.bonus_translation(value) % {'count': value}
        return mark_safe(f'<span>+ {text}</span>')

Use a nginx reverse proxy in dev

The django web server works well but can be slow when it needs to handle many requests (to load many images for instance). One way to solve this is to use a production web server (nginx in this case) to handle most of the work (ie everything but dynamically generated pages).

Prerequisites:

  1. Install nginx
  2. Add the domain you want to use in your /etc/hosts file. For instance 127.0.0.1 myproject.localhost.
  3. Make sure nginx has access to the files of your project. Most of the time a chmod 755 /PATH/TO/PROJECT will do (repeat on each subdirectory nginx need to pass to access to your files). If you are using a shared computer, you may need to think about a more secure way to allow nginx to access the file (ACL may help you).
  4. If you are using SELinux, don't forget to add the proper context to the files. For instance, do something like:
    1. Add these files to the proper SELinux context by copying the one from the default web folder: semanage fcontext --add --equal /var/www/html /PATH/TO/PROJECT
    2. Restore the context of the files: restorecon -R /home/jenselme/Work/bureauxlocaux
    3. Check that the context is correct: ls -Z The output should contain something like system_u:object_r:httpd_sys_content_t:s0.

Here is the nginx configuration to put in /etc/nginx/conf.d (or /etc/nginx/sites-enabled):

server {
    listen 80;
    # Make sure this host is in the ALLOWED_HOSTS variable in the settings.
    server_name PROJECT.localhost;
    root /PATH/TO/PROJECT;

    # Prevent access to pyc and py files.
    location ~ .*\.pyc? {
        return 404;
    }

    # Search for files in the media folder. Change this if you configured Django to store your uploaded files elsewhere.
    location ~ ^/files/(.*) {
        try_files /media/$1 =404;
    }

    # Look for static files in the static folder or at the root of the project.
    location ~ ^/static {
        # Look both in the production static folder at the root of you project and in PROJECT
        # (where you have the apps directory and the static directory you use in dev).
        try_files /PROJECT/$uri /$uri $uri;
    }

    # Relay everything else to the django web server.
    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://127.0.0.1:8000;
    }
}

To add a connect timeout (eg to mimic Heroku's timeout) to the Django dev server, add the lines below in the @django block:

proxy_connect_timeout   30;
proxy_send_timeout      30;
proxy_read_timeout      30;
send_timeout            30;

Checking PO files

#!/usr/bin/env bash

set -eu

for file in "$@"; do
  msgfmt -v --check "${file}"
done

Testing

Inserting pytest fixtures in a test case

class SpamTest(unittest.TestCase):

    @pytest.fixture(autouse=True)
    def inject_fixtures(self, caplog):
        self._caplog = caplog

    def test_eggs(self):
        with self._caplog.at_level(logging.INFO):
            spam.eggs()
            assert self._caplog.records[0].message == 'bacon'

Source: https://stackoverflow.com/a/50375022

Reset PK sequence

def reset_database_sequences(*models_or_factories):
    models = [
        model_or_factory._meta.model for model_or_factory in models_or_factories
    ]
    sequence_sql = connection.ops.sequence_reset_sql(no_style(), models)
    with connection.cursor() as cursor:
        for sql in sequence_sql:
            cursor.execute(sql)

Source: https://stackoverflow.com/a/50275965/3900519

Reset factoryboy sequence

def reset_sequences(*factories):
    reset_database_sequences(*factories)
    for factory_cls in factories:
        factory_cls.reset_sequence()