Django ORM Basics Part 2

Last updated on July 27, 2020

In the previous lesson, we have covered all the basics of interacting with the database using Django ORM. We have created, modified and deleted many objects. The type of the objects we have handled till now were simple objects that can exist on their own. In this lesson, we will learn how to insert and access related data, but first, we will populate some data in our tables.

If you are following lessons closely, at this point, all the tables of djangobin app should be empty. Before we proceed, let's add some Author and Language objects to djangobin_author and djangobin_language tables, respectively. Open Django shell by executing the shell command and import all the models from the djangobin app.

$ ./manage.py shell
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> 
>>> from djangobin.models import *
>>>

Copy and paste the following code into the Django shell to create some authors and languages.

Author.objects.bulk_create([
    Author(name='tom', email='tom@mail.com'),
    Author(name='jerry', email='jerry@mail.com'),
    Author(name='spike', email='spike@mail.com'),
    Author(name='tyke', email='tyke@mail.com'),
])

Language.objects.bulk_create([
    Language(name='Python', lang_code='python', slug='python', mime='text/x-cython', file_extension=".py"),
    Language(name='PHP', lang_code='php', slug='php', mime='text/x-php', file_extension=".php"),
    Language(name='Java', lang_code='java', slug='java', mime='text/x-java', file_extension=".java"),
    Language(name='JavaScript', lang_code='js', slug='js', mime='application/javascript', file_extension=".js"),
])

>>>
>>>
>>> Author.objects.bulk_create([
...     Author(name='tom', email='tom@mail.com'),
...     Author(name='jerry', email='jerry@mail.com'),
...     Author(name='spike', email='spike@mail.com'),
...     Author(name='tyke', email='tyke@mail.com'),
... ])
[<Author: tom : tom@mail.com>, <Author: jerry : jerry@mail.com>, <Author: spike : spike@mail.com>, <Author: tyke : tyke@mail.com>]
>>> 
>>> Language.objects.bulk_create([
...     Language(name='Python', lang_code='python', slug='python', mime='text/x-cython', file_extension=".py"),
...     Language(name='PHP', lang_code='php', slug='php', mime='text/x-php', file_extension=".php"),
...     Language(name='Java', lang_code='java', slug='java', mime='text/x-java', file_extension=".java"),
...     Language(name='JavaScript', lang_code='js', slug='js', mime='application/javascript', file_extension=".js"),
... ])
[<Language: Python>, <Language: PHP>, <Language: Java>, <Language: JavaScript>]
>>> 
>>>

Let's now shift our attention to the Snippet model. Here is how the Snippet model is defined:

djangobin/django_project/djangobin/models.py

#...

class Snippet(models.Model):
    title = models.CharField(max_length=200, blank=True)
    original_code = models.TextField()
    highlighted_code = models.TextField()
    expiration = models.CharField(max_length=10, choices=Pref.expiration_choices)
    exposure = models.CharField(max_length=10, choices=Pref.exposure_choices)
    hits = models.IntegerField(default=0)
    slug = models.SlugField()
    created_on = models.DateTimeField(auto_now_add=True)

    language = models.ForeignKey(Language, on_delete=models.CASCADE)
    author = models.ForeignKey(Author, on_delete=models.CASCADE)
    tag = models.ManyToManyField('Tag')

    class Meta:
        ordering = ['-created_on']

#...

And this is how djangobin_snippet table looks like:

The Snippet model has two one-to-many relationships. The first with the Author model and second with the Language model.

The data for one-to-many relationships will be stored in the djangobin_snippet table. Furthermore, the author and language fields of the Snippet model do not accept NULL values, as a result, a Snippet object must be associated with an Author and a Language object. Trying to create Snippet object without an Author or Language object will throw an IntegrityError exception.

The Snippet model also has a many-to-many relationship with the Tag model. But since the data for a many-to-many relationship is stored in a separate table (djangobin_snippet_tags), we are not required to provide this data at the time of creating Snippet object. Once we have created a Snippet object, we can add tags to it later. We will see how to do that later in this chapter.

Let's try creating a Snippet object without providing an author and language and see what happens:

>>>
>>>
>>> import time
>>>
>>> s = Snippet(
...     original_code = 'some code',
...     highlighted_code = '<p>some code</p>',
...     expiration="never",
...     exposure = "public",
...     slug=str(time.time()).replace(".", "")
... )
>>> 
>>>
>>> s.save()
>>>
Traceback (most recent call last):
...
django.db.utils.IntegrityError: NOT NULL constraint failed: djangobin_snippet.author_id

>>>

As expected, calling save() method raises an exception of type IntegrityError. You would get the same exception, if you try to create a Snippet object using the create() method of the objects manager.

The error is telling us that the author_id column has NOT NULL constraint on it. Consequently, It can't accept NULL values.

The reason why we are getting IntegrityError on author_id column instead of language_id column is that the author_id appears before the language_id column in the djangobin_snippet table.

We can allow author field to accept NULL values by passing null=True to the field constructor in the model class. For now, there is no reason to do that.

Okay, let's try creating Snippet object again, but this time we will provide an Author and a Language object.

>>>
>>> 
>>> Author.objects.values_list("id", "name")
<QuerySet [(7, 'tom'), (8, 'jerry'), (9, 'spike'), (10, 'tyke')]>
>>> 
>>> Language.objects.values_list("id", "name")
<QuerySet [(3, 'Java'), (4, 'JavaScript'), (2, 'PHP'), (1, 'Python')]>
>>> 
>>>
>>> a1 = Author.objects.get(pk=7)
>>>
>>> a1
<Author: tom : tom@mail.com>
>>>
>>> a2 = Author.objects.get(pk=8) 
>>>
>>> a2
<Author: jerry : jerry@mail.com>
>>>
>>> l1 = Language.objects.get(pk=1)
>>>
>>> l1
<Language: Python>
>>> 
>>> 
>>> s1 = Snippet(
...     original_code = 'some code',
...     highlighted_code = '<p>some code</p>',
...     expiration="never",
...     exposure = "public",
...     slug=str(time.time()).replace(".", ""),
...     author = a1,
...     language = l1
... )
>>> 
>>> s1
<Snippet: Untitled - Python>
>>>  
>>> s1.save()
>>>

As you can see, this time operation succeeded. Instead of calling Snippet() constructor you could also use create() method of the objects manager to create a Snippet object.

>>>
>>> s2 = Snippet.objects.create(
...     original_code = 'another snippet',
...     highlighted_code = '<p>another snippet</p>',
...     expiration="never",
...     exposure = "public",
...     slug=str(time.time()).replace(".", ""),
...     author = a1,
...     language = l1
... )
>>> 
>>> s2
<Snippet: Untitled - Python>
>>>

We can now use s1 or s2 variable to get the information about the Snippet object as well as about the Author and Language object to which it belongs to using the dot(.) operator.

>>>
>>>
>>> s1.title
''
>>> 
>>> s1.original_code
'some code'
>>> 
>>> s1.highlighted_code
'<p>some code</p>'
>>> 
>>> s1.expiration
'never'
>>> 
>>> s1.created_on
datetime.datetime(2018, 3, 26, 8, 19, 40, 824422, tzinfo=<UTC>)
>>> 
>>> s1.slug
'1522052373360689' 
>>>
>>> s1.language     # get the Language instance attached to s1
<Language: Python>
>>> 
>>> s1.language.slug    
'python'
>>> 
>>> s1.author       # get the Author instance attached to s1 
<Author: tom : tom@mail.com>
>>> 
>>> s1.author.email
'tom@mail.com'
>>> 
>>>

Notice how we were able to access data stored in different tables without writing SQL JOIN query. That's the power of the Django ORM.

Instead of passing Author object while creating Snippet object, you could also pass the primary key of the author, but then you would have to assign it to the author_id keyword argument instead of author. In the same way, we can pass Language object's primary key to the language_id keyword argument.

>>>
>>> 
>>> Author.objects.values_list('id', 'name')
<QuerySet [(7, 'tom'), (8, 'jerry'), (9, 'spike'), (10, 'tyke')]>
>>> 
>>> Language.objects.values_list('id', 'name')
<QuerySet [(3, 'Java'), (4, 'JavaScript'), (2, 'PHP'), (1, 'Python')]>
>>> 
>>>
>>> s3 = Snippet.objects.create(
...     original_code = 'cool snippet',
...     highlighted_code = '<p>cool snippet</p>',
...     expiration="never",
...     exposure = "public",
...     slug=str(time.time()),
...     author_id = 7,
...     language_id = 2
... )
>>>
>>>

Before we visit next section, let's add some Tag objects to our database.

>>>
>>> t1 = Tag.objects.create(name="django", slug="django")
>>>
>>> t2 = Tag.objects.create(name="flask", slug="flask")
>>>

Additional Managers #

The objects is not the only the manager available in Django. It turns out that when dealing with many-to-many relationship Django uses a manager called related manager to connect data. In the Snippet model tags field is one such manager. You can verify this by typing the following code in the Django shell.

>>>
>>> type(s1.tags)
<django.db.models.fields.related_descriptors.create_forward_many_to_many_manager.<locals>.ManyRelatedManager object at 0x7fc981e36e10>
>>>

As tags is a manager you can use all manager methods on it that we have learned in the lesson Django ORM Basics Part 1. For example, to view all the tags associated with the s1 and s2 posts type the following code:

>>>
>>> s1.tags.all()
<QuerySet []>
>>>
>>> 
>>> s2.tags.all()
<QuerySet []>
>>>

Currently, snippet s1 and s2 are not associated with any tags that's why an empty QuerySet is returned.

So how do we add tags to existing Snippet objects?

All relationship managers come with add() method which can be used to connect objects.

>>>
>>> t1
<Tag: django>
>>>
>>> s1.tags.add(t1)
>>>

Now snippet s1 is associated with django tag. The preceding code adds the following record in the djangobin_snippet_tags table.

>>>
>>> s1.tags.all()
<QuerySet [<Tag: django>]>
>>>

You can also associate multiple tags with a Snippet object by passing multiple arguments to the add() method.

>>>
>>> s2.tags.add(t1, t2)
>>> 
>>> s2.tags.all()
<QuerySet [<Tag: django>, <Tag: flask>]>
>>> 
>>> s2.tags.order_by("-name")
<QuerySet [<Tag: flask>, <Tag: django>]>
>>> 
>>>

Reverse Access #

When you define relationships in the model class using fields like ForeignKey, ManyToManyField etc. Every instance of that model will have attributes to access related objects. For example, given a Snippet object s1, we can access its author using s1.author, its language using s1.language, and its tags using s1.tags.

But, How do we access data other way around? Simply put, How do we access snippets from an Author instance or Tag instance?

Django automatically adds a relation manager of the form
<related_model>_set on the other side of the relationship. More Concretely, given a Author object a1, the snippets associated with this author is given by a1.snippet_set. The snippets_set attribute is an instance of type ManyRelatedManager, which inherits from django.db.models.manager.Manager, as a result, most of the manager methods we discussed in the chapter Django ORM basics Part 1 are available to it. For example:

>>>
>>> a1.snippet_set.all()
<QuerySet [<Snippet: Untitled - PHP>, <Snippet: Untitled - Python>, <Snippet: Untitled - Python>]>
>>> 
>>> a1.snippet_set.order_by("-created_on")
<QuerySet [<Snippet: Untitled - PHP>, <Snippet: Untitled - Python>, <Snippet: Untitled - Python>]>
>>> 
>>> a1.snippet_set.order_by("-created_on").values_list('id', 'created_on')
<QuerySet [(4, datetime.datetime(2018, 3, 26, 8, 48, 48, 956850, tzinfo=<UTC>)), (3, datetime.datetime(2018, 3, 26, 8, 23, 23, 534384, tzinfo=<UTC>)), (2, datetime.datetime(2018, 3, 26, 8, 19, 40, 824422, tzinfo=<UTC>))]>
>>> 
>>>

Similarly, given a Tag instance t1, the snippets associated with this tag is given by t1.snippet_set.

>>> 
>>> t1.snippet_set.all()
<QuerySet [<Snippet: Untitled - Python>, <Snippet: Untitled - Python>]>
>>> 
>>> 
>>> t1.snippet_set.filter(exposure='public')
<QuerySet [<Snippet: Untitled - Python>, <Snippet: Untitled - Python>]>
>>> 
>>>

If for some reason <related_model>_set attribute doesn't appear to be intuitive or readable. We can change it by passing related_name parameter to the ForeignKey or ManyToManyField constructor in the model class. For example:

class Snippet(models.Model):
    #...

    language = models.ForeignKey(Language, on_delete=models.CASCADE, )
    author = models.ForeignKey(Author, on_delete=models.CASCADE, related_name='snippets')
    tags = models.ManyToManyField('Tag', related_name='snippets')

This code will change the relation name from snippet_set to snippets for Author and Tag instance. Given a Tag instance t1, we can now access associated snippets with t1.snippets. Similarly, given an Author instance a1, we can access associated snippets with a1.snippets.

Lookups Spanning Relationship #

Up until now, keyword arguments (with or without lookups) we have been passing to the lookup functions (e.g. filter(), exclude(), get() etc; ) were restricted to the model class we are currently operating on.

It turns out that we can also pass the field name of the related model class to lookup functions. To do so, type model name in lower case, followed by two underscores, which is then followed by the field name. Here is the general format:

Model.objects.filter(related_model__field=some_value)

This spanning can be deep as you would like. Further, you can also use all the lookups we have learned in the lesson Django ORM Basics Part 1. This technique reduces the ad-hoc queries required to perform a given task. For example, let's say we want to find snippets only from authors whose name ends with om. Here is the long way to find all such snippets.

>>>
>>> al = Author.objects.filter(name__iendswith="om")
>>>
>>> al
<QuerySet [<Author: tom : tom@mail.com>]>
>>> 
>>> sl = Snippet.objects.filter(author__in=al)  # using in lookup
>>>
>>> sl
<QuerySet [<Snippet: Untitled - PHP>, <Snippet: Untitled - Python>, <Snippet: Untitled - Python>]>
>>> 
>>> sl.count()
3
>>>
>>>

Using lookups that can span relationship, we can code preceding query in a single line.

>>>
>>> sl = Snippet.objects.filter(author__name__iendswith="om")
>>> 
>>> sl
<QuerySet [<Snippet: Untitled - PHP>, <Snippet: Untitled - Python>, <Snippet: Untitled - Python>]>
>>> 
>>> sl.count()
3
>>>

This technique works in both directions. For example, the following query retrieves all the authors who have created snippets in the year 2018.

>>> 
>>> Author.objects.filter(snippet__created_on__year=2018)
<QuerySet [<Author: tom : tom@mail.com>, <Author: tom : tom@mail.com>, <Author: tom : tom@mail.com>]>
>>>

Notice that the preceding query returns three instances of the same Author, this is because in my SQLite database tom has created three snippets. To filter out the duplicate result we can use distinct() method. The distinct() method returns a QuerySet after removing all the duplicates.

>>>
>>> Author.objects.filter(snippet__created_on__year=2018).distinct()
<QuerySet [<Author: tom : tom@mail.com>]>
>>>

get_object_or_404() function #

Most of the time our views function goes like this:

Code try and except block.
Query the database in the try block.
If an exception is thrown, catch the exception in the except block and show a 404 page.

This pattern is so common that Django a provides a shortcurt function called
get_object_or_404(). Its syntax is:

get_object_or_404(klass, **kwargs)

The klass can be a model, a manager or a queryset.

The **kwargs represents all keyword arguments as well as the lookup parameters that we have been using with the lookup functions ( get(), filter(), exclude() etc).

On success, it returns a single object of the given model, if it can't find any records then it raises a Http404 exception.

Internally this method calls get() method of the objects manager, so you must always use this method to get a single object.

To use get_object_or_404() first import it from django.shortcuts using the following code.

from django.shortcuts import get_object_or_404

The following examples shows how to use get_object_or_404() function with models, queryset and managers. It also shows that when a matching record is not found, get_object_or_404() raises a Http404 exception.

Example 1: where klass is model

>>>
>>> from django.shortcuts import get_object_or_404
>>> 
>>> s1 = get_object_or_404(Snippet, pk=1)
>>>
>>> s1
<Snippet: Untitled - Python>
>>> 
>>>
>>> s1 = get_object_or_404(Snippet, pk=200)
Traceback (most recent call last):
  ...    
django.http.response.Http404: No Snippet matches the given query.
>>>

This code is equivalent to:

>>>
>>> try:
...   s1 = Snippet.objects.get(pk=1)
... except Snippet.DoesNotExist:
...   raise Http404("Post not found")
...
>>>

Example 2: where klass is queryset

>>> 
>>> queryset = Author.objects.filter(active=False)
>>> 
>>> get_object_or_404(queryset, email="tom@mail.com")
<Author: tom : tom@mail.com>
>>> 
>>> get_object_or_404(queryset, email="bob@mail.com")
...  
django.http.response.Http404: No Author matches the given query.
>>> 
>>>
Traceback (most recent call last):

Example 3: where klass is a manager

>>> 
>>> s2 = Snippet.objects.get(id=2)
>>> 
>>> s2.tags.all()
<QuerySet [<Tag: django>, <Tag: flask>]>
>>>
>>> type(s1.tags)
<class 'django.db.models.fields.related_descriptors.create_forward_many_to_many_manager.<locals>.ManyRelatedManager'>
>>> 
>>> get_object_or_404(s2.tags, name='flask')
<Tag: flask>
>>> 
>>>

The get_list_or_404() method #

The get_list_or_404() works just like get_object_or_404() function, but instead of returning a single model instance, it returns a Queryset. If no matching results found, it raises a Http404 exception.

>>>
>>> from django.shortcuts import get_list_or_404
>>> 
>>> queryset = Author.objects.filter(active=False)
>>> 
>>> get_list_or_404(queryset, name__icontains="ke")
[<Author: spike : spike@mail.com>, <Author: tyke : tyke@mail.com>]
>>>  
>>> get_list_or_404(queryset, name__icontains="jo")
Traceback (most recent call last):
...
django.http.response.Http404: No Author matches the given query.
>>>

Load Comments