Add caching to speed up your site: Even if you think it isn’t needed

In this post, we will explore solving the n+1 problem using caching. The tools used are Django ORM with which makes the problem becomes common, and the Cachalot library.

Writing systems at a start-up

As Donald Knuth famously stated,

Premature optimization is the root of all evil.

Along these lines, a startup has daily changing business requirements, so it is pointless to optimize your code today, which might be rewritten tomorrow.(sometimes completely)

Reason for this blog post

This blog post is written with the aim of exploring whether a plug and play caching solution should be used and can it prove to be of some optimization advantage when still at a nascent stage in product development.
My first thoughts in the context of LeanAgri were, will caching even make sense? To give an idea, LeanAgri’s backend is modelled as a normalized MySQL DB in order to have data consistency. Further, it is a permissioned model where users can only view and modify their own data. My idea was that it would be fairly pointless to think about caching in such a scenario where data is sparsely read.
As we go ahead, we will realise how data proved that I was wrong.

Badly written queries

What I didn’t realize in the above assumption was that our code was written badly. We wanted to get off the ground as quickly as possible, so it was never thought about to optimize the DB queries that we make.
We are hitting the n+1 problem at various places. Let’s try to understand with an example what the n+1 problem is. We have the following models where Schedule model is related to Crop and Fertilizer models.

class Fertilizer(Model):
    pass
class Schedule(Model)
    crop = ForeignKey(Crop)
    fertilizers = ManyToManyField(Fertilizer)

With the above models, we want to get all related data for fertilizers for all schedules. How can we do that? In Django ORMs syntax, we simply do(the code should be fairly similar for any other ORM as well):

for schedule in Schedule.objects.all():
    for fertilizer in schedule.fertilizer.all():
        print fertilizer.get_data()

Now let’s try to analyze the above code by putting our “Optimization hats” on:

• We make 1 query to get all schedules
• We make 1 query per fertilizer to get data of all fertilizers for that schedule!!!

Thus, getting data for each schedule is going to be extremely costly for our Database.

The correct way to fix it: Throw away ORMs

ORMs are often considered leaky abstractions but really they help speed up the entire process of software development which is extremely crucial for a startup like us.
Fixing the above example using JOINs is fairly easy but non-intuitive way.

for schedule in Schedule.objects.select_related('fertilizers').all():
    for fertilizer in schedule.fertilizers.all():
        print fertilizer.get_data()

The above takes just 1 query to fetch everything.
There are various advantages of using the above:

• Get all data in a single query. Relational DBs are extremely efficient in doing this work.
• Duplicate queries don’t get executed, For example: Different schedules can have the same fertilizers and it is possible to remove any duplicate queries using JOINs.

Are we going to throw away our ORM?

Absolutely not. Atleast at this stage where we are rapidly creating more and more business logic, it makes no sense to sit and optimize each and every query when it is practically going to add very little value.
Is the above Bad code? Yes it is
Is it inevitable? Again yes it is. If you are working with something like Django Rest framework to create CRUD APIs from ORM models, it is going to be generate queries like the above everywhere.
Important disclaimer: This is not an excuse to write bad SQL queries, but it does become hard to optimize everything when you are working on a lean development cycle.

Solving the n+1 problem using Caching

Now we want to analyze if we can optimize the above queries with a simple plug and play caching solution. The aim of such a solution is to add nothing but a middle layer between our application and Database.
We chose the Cachalot library based on the interesting detail of research done by the author in the documentation.
A caching system would perform great in our case specially with: