Jekyll2022-11-02T01:53:39+00:00/feed.xmlRefactored NoiseSoftware Engineer. Musician. Minimalist.Time Blocking With A $39 Casio Watch2022-09-28T00:00:00+00:002022-09-28T00:00:00+00:00/2022/09/28/staying-productive-with-casio-time-boxing<p>I have been (unsuccessfully) trying to use the Time Blocking <a href="https://www.timeblockplanner.com/#timemethod">technique</a> endorsed by Cal Newport.</p>
<p>Time Blocking is a work productivity technique that involves slicing your day into blocks. These blocks prioritize your most important work for the day. Each block involves deep focus and is basically treated like a meeting for your self.</p>
<p>I love the spirit of this idea but I have found that the execution has been tricky.</p>
<h2 id="where-i-stumbled">Where I stumbled</h2>
<p>Initially I turned to a template that I could generate each day in Notion. It was a simple table of 1 hour blocks that looked a little something like this.</p>
<table>
<thead>
<tr>
<th>Block</th>
<th>Task</th>
</tr>
</thead>
<tbody>
<tr>
<td>9 - 10</td>
<td>scope project</td>
</tr>
<tr>
<td>10 - 11</td>
<td>scope project continued</td>
</tr>
<tr>
<td>11 - 12</td>
<td>review bug list</td>
</tr>
<tr>
<td>12 - 1</td>
<td>lunch</td>
</tr>
<tr>
<td>1 - 2</td>
<td>deep work on project</td>
</tr>
<tr>
<td>2 - 3</td>
<td>deep work on project continued</td>
</tr>
<tr>
<td>3 - 4</td>
<td>meetings</td>
</tr>
<tr>
<td>4 - 5</td>
<td>email and shutdown</td>
</tr>
</tbody>
</table>
<p>This was fine for a while. Each day I would generate a new template and pencil in my priorities. Unlike a todo list this chart allowed me to think about my priorities through the constraint of time.</p>
<p>In reality it’s easy to lose track of this list during the day. My most common pitfall was getting absorbed in the task and not finishing within the time box. It started to feel even more stressful when I couldn’t keep up with my own schedule.</p>
<p>My next step was to do the same thing in my calendar. This was a little more fluid. I could schedule blocks of varying sizes around meetings. I also received alerts letting me know when I entered a new time block. Unfortunately I couldn’t get an alert telling me when a meeting ends (why is this not a thing?).</p>
<h2 id="how-a-39-watch-helped-me-figure-it-out">How a $39 watch helped me figure it out</h2>
<p>In the end the answer to time boxing came from an unexpected vintage piece of digital hardware. The casio digital watch.</p>
<p>In particular I started out with the <a href="https://www.casio.com/us/watches/casio/product.DW-290-1V/">DW-290</a>. This watch is a 90s throwback made famous in the film Mission Impossible.</p>
<p><img src="https://www.watch-id.com/sites/default/files/upload/sighting/casio-dw290-1v-tom-cruise-mission-impossible.jpg" alt="" /></p>
<p>Unlike modern smart watches this watch has minimal features. The DW-290 has a single alarm, a countdown timer, an hourly beep and a stop watch. Probably the best feature of all is the lack of connectivity to the internet. There are no notifications or distractions with this watch. Just useful tools that help me to stay focused and on task.</p>
<p><img src="https://www.casio.com/content/dam/casio/product-info/locales/us/en/timepiece/product/watch/D/DW/DW2/DW-290-1V/us-assets/DW290-1V.png" alt="Casio DW-290 Watch" /></p>
<p>I still make a list of priorities for the day in my calendar. But now I don’t rely on notifications from my calendar. Instead I begin a bock by setting the alarm for the time that the block ends. I also have the hourly beep sound enabled so that I know where I am in the day. Think of this beep as a mileage marker for the day. It’s helpful to know when I am at the top of the hour.</p>
<p>Other times I like to use the countdown timer to block time. It’s helpful to occasionally glance at the countdown timer and know how much time I have left on a task. It’s also incredibly easy to set the timer and this is often my default over using the alarm.</p>
<p>Some work I do requires small chunks of repetitive time boxing. When I review a bug list or need to get caught up on Basecamp messages I like to spend no more than 10 minutes on each item. For these tasks I will set my timer to 10 minutes with the auto repeat function enabled.</p>
<p>Finally, there are times when I want to estimate the time it will take me to complete some work (if that’s possible). If the work consists of some repeatable tasks (like a refactor) then I will reach for the stop watch function. I can set the stop watch to see how long it takes for me to complete a single task. I might even do a few and average the times. This gives me a good approximation of the total time it will take me to complete a project of repeated tasks.</p>
<p>There are no shortage of uses for this watch. The Casio DW-290 has no opinions that get in the way.</p>I have been (unsuccessfully) trying to use the Time Blocking technique endorsed by Cal Newport.Integration Testing in Python Django2021-03-08T00:00:00+00:002021-03-08T00:00:00+00:00/2021/03/08/integration-testing-in-python-django<p>We all know that we should write tests to validate the functionality of our Django applications. However, it can be easy to fall into the trap of writing too many tests at the unit or integration level. Having thousands of unit tests is not helpful when the interactions between those units are not tested. To make matters worse, Django does not currently offer an opinionated way to write true integration tests that span multiple view requests. Fortunately, there is a way to write classic integration tests using the building blocks of the Django testing framework.</p>
<p>Typical tests in a Django project will evaluate models, views, serializers, and perhaps some utility functions. In this setup there are no integration tests that validate the functionality of a complete feature. Instead many unit tests are written with assumptions about how the state will be when the respective code is called. Naturally, it’s difficult to predict all of the edge cases and potential problems that will happen once various components of your code base are used together.</p>
<p>I would like to show how you can write integration tests in Django. I believe that integration tests are a good place to start when writing a new feature based on user stories. It’s easier to write tests at this level that closely approximate the interactions of the user even when your app is only an API backend for a front end application built in a JS framework.</p>
<h2 id="the-testing-pyramid">The Testing Pyramid</h2>
<p>The layers that make up testing responsibilities takes on a shape resembling a pyramid. There are lots of books and articles written about this topic. Pictured below is the testing pyramid featured in Martin Fowler’s blog about testing. The 3 layers of tests (UI, Service, and Unit tests) increase in cost and speed at the top of the pyramid. It’s important to know when to choose the best layer to focus on for testing. It’s better to have fewer tests at the UI level and many more tests at the unit level.</p>
<p><img src="https://martinfowler.com/bliki/images/testPyramid/test-pyramid.png" alt="Martin Fowler's Testing Pyramid" /></p>
<p>At the top of the pyramid we have the most expensive tests. These are the types of tests that QA will typically perform either manually or with an automated testing tool. These tests evaluate the entire stack. This includes front end client code and its interaction with the back end. It’s not atypical to have 3rd party services (payment gateways) included in this level of testing. This is the closest approximation to what users will see in production and it’s also the most expensive form of testing.</p>
<p>The service or integration layer sits below UI tests. At this level the server’s API is tested directly. Writing tests at this layer will be the focus of this article. At this level only 3rd party services are mocked when performing broad integration tests. Narrow integration tests (a single endpoint) can also be written at this layer to test the responses to typical inputs for the respective endpoint.</p>
<p>At the bottom of the pyramid there are unit tests. This should be the largest part of a test suite. With that said it shouldn’t be the only part of the test suite. Unit tests should be efficient and only test a small part of code with the dependencies mocked. At this level many lower level edge cases can be rapidly tested at a low cost. A very specific bug can be replicated in a unit test. Given the low cost of these tests there really is little penalty to writing many niche tests at this level.</p>
<h2 id="integration-testing">Integration Testing</h2>
<p>All test layers are important but for this article we will focus on writing integration tests in Django. Some people might make the mistake of describing a View test as an integration test. This is partially true. Django is a Model-View-Template framework. A view test will interact with all 3 components of the framework. However, this is an example of a ‘narrow integration test’ in that it doesn’t test a broader user story or interaction that spans multiple requests. To serve this need we will need to write tests that support multiple requests. Fortunately this is easy enough to achieve in Django and does not require an additional testing library.</p>
<p>I like to separate multi-request view tests from the single request views. When testing multiple requests I find that it’s helpful to create a base class that provides some convenient wrapper functions for quickly generating requests with some sensible defaults. I like to think of this base class as an endpoint factory.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">BaseRequestTestCase</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">user</span> <span class="o">=</span> <span class="n">factories</span><span class="p">.</span><span class="n">UserFactory</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">login</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">user</span><span class="p">.</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="o">=</span><span class="s">'test'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">generate_fake_person_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="p">{</span>
<span class="s">'first_name'</span><span class="p">:</span> <span class="n">fake</span><span class="p">.</span><span class="n">first_name</span><span class="p">(),</span>
<span class="s">'last_name'</span><span class="p">:</span> <span class="n">fake</span><span class="p">.</span><span class="n">last_name</span><span class="p">(),</span>
<span class="s">'email'</span><span class="p">:</span> <span class="n">fake</span><span class="p">.</span><span class="n">email</span><span class="p">(),</span>
<span class="p">}</span>
<span class="k">def</span> <span class="nf">create_student</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">student_data</span><span class="p">):</span>
<span class="n">url</span> <span class="o">=</span> <span class="n">reverse</span><span class="p">(</span><span class="s">'create-student'</span><span class="p">,</span> <span class="n">kwargs</span><span class="o">=</span><span class="p">{</span><span class="s">'role'</span><span class="p">:</span> <span class="n">user_role</span><span class="p">})</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">student_data</span><span class="p">),</span> <span class="n">content_type</span><span class="o">=</span><span class="s">'application/json'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">register_student</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">student_data</span><span class="p">):</span>
<span class="n">url</span> <span class="o">=</span> <span class="n">reverse</span><span class="p">(</span><span class="s">'register'</span><span class="p">)</span>
<span class="n">birthday</span> <span class="o">=</span> <span class="p">(</span><span class="n">datetime</span><span class="p">.</span><span class="n">today</span><span class="p">()</span> <span class="o">-</span> <span class="n">relativedelta</span><span class="p">(</span><span class="n">years</span><span class="o">=</span><span class="mi">20</span><span class="p">)).</span><span class="n">date</span><span class="p">()</span>
<span class="n">data</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'phone_number'</span><span class="p">:</span> <span class="n">fake</span><span class="p">.</span><span class="n">msisdn</span><span class="p">()[:</span><span class="mi">10</span><span class="p">],</span>
<span class="s">"password"</span><span class="p">:</span> <span class="s">"password"</span><span class="p">,</span>
<span class="s">"birthday"</span><span class="p">:</span> <span class="n">birthday</span><span class="p">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">'%Y-%m-%d'</span><span class="p">),</span>
<span class="p">}</span>
<span class="n">data</span><span class="p">.</span><span class="n">update</span><span class="p">(</span><span class="n">student_data</span><span class="p">)</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'content_type'</span><span class="p">:</span> <span class="s">'application/json'</span><span class="p">,</span>
<span class="s">'HTTP_USER_AGENT'</span><span class="p">:</span> <span class="s">'Firefox'</span><span class="p">,</span>
<span class="p">}</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="o">**</span><span class="n">headers</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">remove_student</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">student_id</span><span class="p">):</span>
<span class="n">url</span> <span class="o">=</span> <span class="n">reverse</span><span class="p">(</span><span class="s">'remove-student'</span><span class="p">,</span> <span class="n">kwargs</span><span class="o">=</span><span class="p">{</span>
<span class="s">'student_id'</span><span class="p">:</span> <span class="n">student_id</span><span class="p">,</span>
<span class="p">})</span>
<span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">delete</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="k">return</span> <span class="n">response</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">BaseRequestTestCase</code> can then be used to write new TestCase classes that inherit from this class. URLs can easily be generated to support multiple requests and test an entire user story.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">TestUserRegistersStudent</span><span class="p">(</span><span class="n">BaseRequestTestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">classroom_settings</span> <span class="o">=</span> <span class="n">factories</span><span class="p">.</span><span class="n">ClassroomSettingsFactory</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">user</span> <span class="o">=</span> <span class="n">factories</span><span class="p">.</span><span class="n">TeacherFactory</span><span class="p">(</span>
<span class="n">profile__classroom_settings</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">classroom_settings</span><span class="p">,</span>
<span class="n">profile__role__can_view_students</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="n">profile__role__can_remove_students</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">login</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">user</span><span class="p">.</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="o">=</span><span class="s">'test'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_adding_and_deleting_students</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">student_data</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'first_name'</span><span class="p">:</span> <span class="s">'smartie'</span><span class="p">,</span>
<span class="s">'last_name'</span><span class="p">:</span> <span class="s">'pants'</span>
<span class="p">}</span>
<span class="bp">self</span><span class="p">.</span><span class="n">register_student</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">Student</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="n">count</span><span class="p">(),</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">student</span> <span class="o">=</span> <span class="n">Student</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="n">last</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">student</span><span class="p">.</span><span class="n">name</span><span class="p">,</span> <span class="s">'Smartie Pants'</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">remove_student</span><span class="p">(</span><span class="n">student</span><span class="p">.</span><span class="n">pk</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">Student</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="n">count</span><span class="p">(),</span> <span class="mi">0</span><span class="p">)</span></code></pre></figure>
<p>Imagine that there is a user story, however unlikely, that calls for the addition and deletion of a student all in one single step. This test will evaluate the entire stack of the Django application for this particular feature. Often times there could be side effects that result from creating an object via an endpoint that are not present or maintained in a model factory.</p>
<p>The main takeaway is that we can create convenient base class methods for an application API that can be used in multiple request tests.</p>
<h2 id="when-unit-tests-are-more-appropriate">When Unit Tests are More Appropriate</h2>
<p>There are times where unit tests are more appropriate. I generally write unit tests for any low level changes to a model, utility function or schema. As mentioned previously, specific fail cases for bugs are often helpful to write at this level. I would rather pay as little cost as I can to verify an edge case while still including the test in my test suite.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Integration tests are possible in Django even though it’s not immediately obvious. I shared how I like to write integration tests that simulate user interactions that span multiple API requests. Tests at this level should be used sparingly given the higher cost to maintain and run. However, a test suite is not complete without tests that test high level user features.</p>We all know that we should write tests to validate the functionality of our Django applications. However, it can be easy to fall into the trap of writing too many tests at the unit or integration level. Having thousands of unit tests is not helpful when the interactions between those units are not tested. To make matters worse, Django does not currently offer an opinionated way to write true integration tests that span multiple view requests. Fortunately, there is a way to write classic integration tests using the building blocks of the Django testing framework.Mocking My Way Through Python Tests2021-01-30T00:00:00+00:002021-01-30T00:00:00+00:00/2021/01/30/mocking-my-way-through-python-tests<p>Mocking is a powerful tool that I like to turn to while unit testing. Python ships with a flexible mocking package in the <code class="language-plaintext highlighter-rouge">unittest</code> library called <code class="language-plaintext highlighter-rouge">unittest.mock</code>. Mocks can help simplify test setup, allow you to test code that depends on features not yet implemented, and minimize undesired side effects in our test suite. I would like to show you how I use mocks in Python Django. We will cover the main reasons you would consider mocks in your unit tests, different styles of writing mocks, and common mocking anti-patterns.</p>
<h2 id="why-mock-test-dependencies">Why Mock Test Dependencies</h2>
<p>Mocks allow us to get to the point of our tests. Some parts of our codebase might be tricky to setup without introducing a lot of dependent objects. It also makes sense to mock services that we are not unit testing. Some services might have undesired consequences in a test suite. This is common when interacting with 3rd party services such as payment gateways and social media applications. In most cases we would prefer our test suite to be resilient to 3rd party service outages or dependencies. This makes tests run faster and easier to deploy to CI environments.</p>
<p>I also like to use mocks when practicing TDD. In these cases my mocks might even be temporary stubs that I use while the dependent service is in development. This can be extremely useful when working with other developers on the same feature. Usually you already have a good idea of what a response or input for a required service is. In those cases you can quickly mock that service and get on with writing unit tests for code that would otherwise be blocked.</p>
<h2 id="various-flavors-of-mocks">Various Flavors of Mocks</h2>
<p>Unittest mocks come in a variety of flavors. It can be a little daunting when visiting the <a href="https://docs.python.org/3/library/unittest.mock.html">documentation</a>. Let’s take a look at the 4 ways I like to write mocks. For each example we will be looking at implementing a feature in a Django blog app that will allow us to post status update about a blog article to Twitter. Our code that we will be testing looks like this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">TwitterClient</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">client</span> <span class="o">=</span> <span class="n">twitter</span><span class="p">.</span><span class="n">Api</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">post_update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">PostUpdate</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Post</span><span class="p">(</span><span class="n">models</span><span class="p">.</span><span class="n">Model</span><span class="p">):</span>
<span class="p">....</span>
<span class="k">def</span> <span class="nf">post_to_twitter</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">client</span><span class="p">):</span>
<span class="n">resp</span> <span class="o">=</span> <span class="n">client</span><span class="p">.</span><span class="n">post_update</span><span class="p">(</span><span class="s">'Check out my new article: {}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">title</span><span class="p">))</span>
<span class="k">if</span> <span class="n">resp</span> <span class="o">==</span> <span class="mi">200</span><span class="p">:</span>
<span class="k">return</span> <span class="s">'Posted {}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">title</span><span class="p">)</span>
<span class="k">return</span> <span class="s">'Unsuccessful. Try again...'</span></code></pre></figure>
<h4 id="1-mock-class">1. Mock Class</h4>
<p>hen I first approached writing mocks in Python I went for what I thought was the most obvious pattern. Creating instances of Mocks with the generic Mock class is straightforward. In this example we create an instance of Mock and assign it to <code class="language-plaintext highlighter-rouge">twitter_api</code>. We can pass <code class="language-plaintext highlighter-rouge">twitter_api</code> to Post’s <code class="language-plaintext highlighter-rouge">post_to_twitter</code> method and assert that the <code class="language-plaintext highlighter-rouge">post_update</code> method is called with a notification about about the Post’s title.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">mock</span> <span class="kn">import</span> <span class="n">Mock</span>
<span class="k">class</span> <span class="nc">PostTests</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">post</span> <span class="o">=</span> <span class="n">factories</span><span class="p">.</span><span class="n">PostFactory</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">test_post_to_twitter</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">twitter_api</span> <span class="o">=</span> <span class="n">Mock</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">post_to_twitter</span><span class="p">(</span><span class="n">client</span><span class="o">=</span><span class="n">twitter_api</span><span class="p">)</span>
<span class="n">twitter_api</span><span class="p">.</span><span class="n">post_update</span><span class="p">.</span><span class="n">assert_called_with</span><span class="p">(</span>
<span class="s">'Check out my new article: {}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">title</span><span class="p">)</span>
<span class="p">)</span></code></pre></figure>
<p>In theory this looks good but there is a very dangerous pitfall lurking here. Mock is a little too good at faking its role. The <code class="language-plaintext highlighter-rouge">post_update</code> method does not even need to exist on the instance of the <code class="language-plaintext highlighter-rouge">TwitterClient</code> class that we created for this test to pass.</p>
<p>For this reason, I only like to use Mock in cases where I need a true test double. Usually this is early in the process of developing a new feature where the service I need is not yet available.</p>
<h4 id="2-create-autospec">2. Create Autospec</h4>
<p>We can get a little better if we use <code class="language-plaintext highlighter-rouge">mock.create_autospec</code> method. This ensures that we can only call methods that are part of the Class API specification. In this example we a mock of our <code class="language-plaintext highlighter-rouge">TwitterClient</code> class and then assign an instance of that class to <code class="language-plaintext highlighter-rouge">client</code>. The rest of the test is pretty similar as before. However, this time the test will fail if <code class="language-plaintext highlighter-rouge">TwitterClient</code> does not respond to <code class="language-plaintext highlighter-rouge">post_update</code>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">mock</span> <span class="kn">import</span> <span class="n">create_autospec</span>
<span class="kn">from</span> <span class="nn">mysite.blog.models</span> <span class="kn">import</span> <span class="n">TwitterClient</span>
<span class="k">class</span> <span class="nc">PostTests</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">post</span> <span class="o">=</span> <span class="n">factories</span><span class="p">.</span><span class="n">PostFactory</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">test_post_to_twitter</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">MockTwitterClient</span> <span class="o">=</span> <span class="n">create_autospec</span><span class="p">(</span><span class="n">TwitterClient</span><span class="p">,</span> <span class="n">spec_set</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">MockTwitterClient</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">post_to_twitter</span><span class="p">(</span><span class="n">client</span><span class="p">)</span>
<span class="n">client</span><span class="p">.</span><span class="n">post_update</span><span class="p">.</span><span class="n">assert_called_with</span><span class="p">(</span>
<span class="s">'Check out my new article: {}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">title</span><span class="p">)</span>
<span class="p">)</span></code></pre></figure>
<h4 id="3-mock-patch">3. Mock Patch</h4>
<p>The method I turn to most often when writing mocks is <code class="language-plaintext highlighter-rouge">mock.patch</code>. This is a Python decorator method that can be added to each test. In this example (without even importing TwitterClient) we create a <code class="language-plaintext highlighter-rouge">mock_twitter_client</code> by attaching <code class="language-plaintext highlighter-rouge">mock.patch</code> with the full module path of <code class="language-plaintext highlighter-rouge">TwitterClient</code>. We can then create an instance of the mock twitter client and test as before. At this stage it might also make sense to write an assertion on the return value.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">mock</span>
<span class="k">class</span> <span class="nc">PostTests</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">post</span> <span class="o">=</span> <span class="n">factories</span><span class="p">.</span><span class="n">PostFactory</span><span class="p">()</span>
<span class="o">@</span><span class="n">mock</span><span class="p">.</span><span class="n">patch</span><span class="p">(</span><span class="s">'mysite.blog.models.TwitterClient'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_post_to_twitter</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">mock_twitter_client</span><span class="p">):</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">mock_twitter_client</span><span class="p">()</span>
<span class="n">client</span><span class="p">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="mi">200</span>
<span class="n">reponse</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">post_to_twitter</span><span class="p">(</span><span class="n">client</span><span class="p">)</span>
<span class="n">client</span><span class="p">.</span><span class="n">post_update</span><span class="p">.</span><span class="n">assert_called_with</span><span class="p">(</span>
<span class="s">'Check out my new article: {}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">title</span><span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="s">'Posted {}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">title</span><span class="p">),</span> <span class="n">response</span><span class="p">)</span></code></pre></figure>
<h4 id="4-mock-patch-object">4. Mock Patch Object</h4>
<p>This example is almost identical to <code class="language-plaintext highlighter-rouge">mock.patch</code>. However, by calling <code class="language-plaintext highlighter-rouge">object</code> on patch we can also include the method and the return value that we are mocking. One thing to keep in mind here is that we still need to import TwitterClient into the test. With a little cut and paste it’s easy to also test what happens when the response code is something other than 200.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">mock</span>
<span class="kn">from</span> <span class="nn">mysite.blog.models</span> <span class="kn">import</span> <span class="n">TwitterClient</span>
<span class="k">class</span> <span class="nc">PostTests</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">post</span> <span class="o">=</span> <span class="n">factories</span><span class="p">.</span><span class="n">PostFactory</span><span class="p">()</span>
<span class="o">@</span><span class="n">mock</span><span class="p">.</span><span class="n">patch</span><span class="p">.</span><span class="nb">object</span><span class="p">(</span><span class="n">TwitterClient</span><span class="p">,</span> <span class="s">'post_update'</span><span class="p">,</span> <span class="n">return_value</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_post_to_twitter</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">mock_post_update</span><span class="p">):</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">TwitterClient</span><span class="p">()</span>
<span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">post_to_twitter</span><span class="p">(</span><span class="n">client</span><span class="p">)</span>
<span class="n">mock_post_update</span><span class="p">.</span><span class="n">assert_called_with</span><span class="p">(</span>
<span class="s">'Check out my new article: {}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">title</span><span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="s">'Posted {}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">title</span><span class="p">),</span> <span class="n">response</span><span class="p">)</span>
<span class="o">@</span><span class="n">mock</span><span class="p">.</span><span class="n">patch</span><span class="p">.</span><span class="nb">object</span><span class="p">(</span><span class="n">TwitterClient</span><span class="p">,</span> <span class="s">'post_update'</span><span class="p">,</span> <span class="n">return_value</span><span class="o">=</span><span class="mi">400</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_post_to_twitter</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">mock_post_update</span><span class="p">):</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">TwitterClient</span><span class="p">()</span>
<span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">post_to_twitter</span><span class="p">(</span><span class="n">client</span><span class="p">)</span>
<span class="n">mock_post_update</span><span class="p">.</span><span class="n">assert_called_with</span><span class="p">(</span>
<span class="s">'Check out my new article: {}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">post</span><span class="p">.</span><span class="n">title</span><span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="s">'Unsuccessful. Try again...'</span><span class="p">,</span> <span class="n">response</span><span class="p">)</span></code></pre></figure>
<h2 id="mocking-anti-patterns">Mocking Anti-patterns</h2>
<p>Mocks are a useful tool to have while unit testing. However, like most things they are not bullet proof.</p>
<p>When testing with mocks it’s important that you test the code that you are trying to write tests for. It seems obvious but it’s entirely possible to write a test with so many mocks that the only thing left are some meaningless assertions about code that doesn’t exist in your project.</p>
<p>Mocks are not substitutes for factories. When possible always use real data dependencies. Mocks really should be a last resort for times when it would be tricky to create the specific state or dependent objects that you need to write a test.</p>
<p>Mocks are useful for testing 3rd party services like we did throughout the examples. However, there are tools that can make testing 3rd party libraries even better. One tool I would like to explore more is <a href="https://vcrpy.readthedocs.io/en/latest/">VCR</a>. The VCR library allows you to record 3rd party service responses as ‘cassettes’. From there you can interact with the service pretty much as you would in production. For each request you will received a the cached recorded response captured in your cassettes. If the API ever updates, it’s just as easy to rerecord a ‘cassette’.</p>
<p>I prefer not to use Mocks in my integration tests. The whole point of these types of tests is to test your entire stack and mocking any part of the stack would not be a true integration test. The only exception to this would be any interaction with a 3rd party service.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Mocking is a great tool to use in situations where supporting dependencies in a test have either undesired side effects that should be faked or when it is complicated to setup supporting dependencies in a test. Even so the need for mocking can sometimes be an indicator that the current implementation is too complex. Also, it is entirely possible to write mocks that can mislead your tests. Mocks are best used sparingly but can be a powerful tool when practicing TDD.</p>Mocking is a powerful tool that I like to turn to while unit testing. Python ships with a flexible mocking package in the unittest library called unittest.mock. Mocks can help simplify test setup, allow you to test code that depends on features not yet implemented, and minimize undesired side effects in our test suite. I would like to show you how I use mocks in Python Django. We will cover the main reasons you would consider mocks in your unit tests, different styles of writing mocks, and common mocking anti-patterns.Writing a Chatbot in Django: Part 22021-01-02T00:00:00+00:002021-01-02T00:00:00+00:00/2021/01/02/writing-a-chatbot-in-django-part-2<p>Last time we managed to create a terminal based chatbot with Chatterbot that can understand basic scheduling instructions. Today, we will take that simple terminal application and drop it into a Django application. This will allow us to wrap our chatbot’s functionality within a full fledged web server that can receive RESTful requests. It will also allow us to create a separate service that can schedule real appointments.</p>
<h2 id="getting-started">Getting Started</h2>
<p>Be sure to install Django in your local environment and then freeze your dependencies to your requirements.txt file.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ pip install django && pip-chill > requirements.txt
</code></pre></div></div>
<p>Shuffling around the project takes a little bit of work but the basic structure of the new Django application will contain the same custom logic adapter, preprocessor functions, and tests that we wrote last time under the <code class="language-plaintext highlighter-rouge">chatbot</code> namespace.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── manage.py
├── requirements.txt
└── schedule_bot
├── __init__.py
├── asgi.py
├── chatbot
│ ├── __init__.py
│ ├── chatbot.py
│ ├── logic
│ │ ├── __init__.py
│ │ └── schedule_adapter.py
│ ├── preprocessors.py
│ ├── tests
│ │ ├── __init__.py
│ │ ├── test_chatbot.py
│ │ └── test_logic_adapters.pyc
│ ├── urls.py
│ └── views.py
├── scheduler
│ ├── __init__.py
│ ├── admin.py
│ ├── apps.py
│ ├── migrations
│ │ └── __init__.py
│ ├── models.py
│ ├── tests.py
│ ├── urls.py
│ └── views.py
├── settings.py
├── urls.py
└── wsgi.py
</code></pre></div></div>
<p>Our <code class="language-plaintext highlighter-rouge">chatbot</code> app will be concerned with parsing and interpreting incoming API requests with the user’s input. We will have another app called <code class="language-plaintext highlighter-rouge">scheduler</code> that will be concerned with taking these instructions, checking availability, and writing the appointments to the database. We can get a pretty nice admin interface for free when we choose to build a backend server application with Django. This will provide a convenient interface for viewing appointments and updating availability preferences. These preferences might include the time of the day that appointments can be scheduled, black out days, and how to handle conflicts.</p>
<p>There are also two new files in our <code class="language-plaintext highlighter-rouge">chatbot</code> folder called <code class="language-plaintext highlighter-rouge">urls.py</code> and <code class="language-plaintext highlighter-rouge">views.py</code>. For anyone not familiar with Django, these two files will be used to define the bots API endpoint and behavior. We will return to these two files after we first define our robot’s basic configuration in the Django settings.</p>
<h2 id="configuring-our-chatbot-in-django-settings">Configuring Our Chatbot In Django Settings</h2>
<p>Our configuration is going to look very similar to what we already have in chatbot/tests/test_chatbot.py. The only difference from before is that we are defining the chatbot in a compatible json format. We will also need to register our <code class="language-plaintext highlighter-rouge">chatbot</code> application along with the <code class="language-plaintext highlighter-rouge">chatterbot.ext.django_chatterbot</code> external library.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># schedule_bot/settings.py
...
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'schedule_bot.chatbot',
'chatterbot.ext.django_chatterbot'
]
...
CHATTERBOT = {
'name': 'Schedule Bot',
'logic_adapters': [
'schedule_bot.chatbot.logic.schedule_adapter.Schedule',
{
'import_path': 'chatterbot.logic.BestMatch',
'default_response': "I'm sorry I don't understand. I like to schedule appointments.",
'maxiumum_similarity_threshold': 0.90
}
],
'preprocessors': [
'schedule_bot.chatbot.preprocessors.format_dates',
'schedule_bot.chatbot.preprocessors.capitalize_months'
]
}
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">CHATTERBOT</code> configuration settings can now be imported into our <code class="language-plaintext highlighter-rouge">chatbot.ChatBotApiView</code> to create an instance of our chatbot to interpret incoming API requests.</p>
<h2 id="completing-the-api-interface">Completing the API Interface</h2>
<p>We will need to setup an API Endpoint and a Chatbot View that can interpret the incoming API requests. Our API endpoint will be called ‘api/chatbot’ and it will be used to interpret JSON strings attached to the request body.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python">
<span class="c1"># schedule_bot/chatbot/urls.py
</span>
<span class="kn">from</span> <span class="nn">django.conf.urls</span> <span class="kn">import</span> <span class="n">url</span>
<span class="kn">from</span> <span class="nn">schedule_bot.chatbot.views</span> <span class="kn">import</span> <span class="n">ChatBotApiView</span>
<span class="n">urlpatterns</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">url</span><span class="p">(</span><span class="sa">r</span><span class="s">'^api/chatbot/'</span><span class="p">,</span> <span class="n">ChatBotApiView</span><span class="p">.</span><span class="n">as_view</span><span class="p">(),</span> <span class="n">name</span><span class="o">=</span><span class="s">'chatbot'</span><span class="p">)</span>
<span class="p">]</span></code></pre></figure>
<p>The actual <code class="language-plaintext highlighter-rouge">ChatBotApiView</code> is a Django class based view. For the GET request we will simply return the name of the chatbot defined in the <code class="language-plaintext highlighter-rouge">ChatBotApiView.chatbot</code> property. The POST request scans the input string for the requisite ‘text’ key and then parses the response with our chatbot’s <code class="language-plaintext highlighter-rouge">get_response</code> method. This response is serialized and then returned as a valid Json response.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">@</span><span class="n">method_decorator</span><span class="p">(</span><span class="n">csrf_exempt</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">'dispatch'</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">ChatBotApiView</span><span class="p">(</span><span class="n">View</span><span class="p">):</span>
<span class="n">chatbot</span> <span class="o">=</span> <span class="n">ChatBot</span><span class="p">(</span><span class="o">**</span><span class="n">settings</span><span class="p">.</span><span class="n">CHATTERBOT</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">post</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="nb">input</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="n">loads</span><span class="p">(</span><span class="n">request</span><span class="p">.</span><span class="n">body</span><span class="p">.</span><span class="n">decode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">))</span>
<span class="k">if</span> <span class="s">'text'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="nb">input</span><span class="p">:</span>
<span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span>
<span class="s">'text'</span><span class="p">:</span> <span class="p">[</span>
<span class="s">'The attribute "text" is required.'</span>
<span class="p">]</span>
<span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">400</span><span class="p">)</span>
<span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">chatbot</span><span class="p">.</span><span class="n">get_response</span><span class="p">(</span><span class="nb">input</span><span class="p">[</span><span class="s">'text'</span><span class="p">])</span>
<span class="n">response_data</span> <span class="o">=</span> <span class="n">response</span><span class="p">.</span><span class="n">serialize</span><span class="p">()</span>
<span class="k">return</span> <span class="n">JsonResponse</span><span class="p">(</span><span class="n">response_data</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">return</span> <span class="n">JsonResponse</span><span class="p">({</span>
<span class="s">'name'</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">chatbot</span><span class="p">.</span><span class="n">name</span>
<span class="p">})</span></code></pre></figure>
<p>At this point we should be able to send a POST request and receive a response that confirms the scheduled appointment.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl --request POST \
--url http://localhost:8000/api/chatbot/ \
--header 'Content-Type: application/json' \
--data '{"text": "Ok. I would like to make an appointment for november 7th at 8 AM"}'
=>
{
"id": null,
"text": "scheduling appointment for 11/07/21 at 08:00:00",
"search_text": "",
"conversation": "",
"persona": "bot:Schedule Bot",
"tags": [],
"in_response_to": "Ok. I would like to make an appointment for November 7th at 8 AM",
"search_in_response_to": "",
"created_at": "2021-01-11T12:40:26.301Z"
}
</code></pre></div></div>
<h2 id="next-steps">Next Steps</h2>
<p>Next time we will complete the work needed to actually schedule the appointment. This will require receiving the requested time, checking to see if that time is available, and relaying a response back to the chatbot App. At this juncture, we can begin to also think about the front end that we want to use with this service. This could include a React Application with an embedded chat <a href="https://github.com/Wolox/react-chat-widget">widget</a> or even an incoming sms message from <a href="https://www.youtube.com/watch?v=cZeCz_QOoXw">twilio</a>. I look forward to exploring both options once we have implemented the scheduling functionality for our chatbot.</p>Last time we managed to create a terminal based chatbot with Chatterbot that can understand basic scheduling instructions. Today, we will take that simple terminal application and drop it into a Django application. This will allow us to wrap our chatbot’s functionality within a full fledged web server that can receive RESTful requests. It will also allow us to create a separate service that can schedule real appointments.Writing a Chatbot in Django: Part 12020-12-27T00:00:00+00:002020-12-27T00:00:00+00:00/2020/12/27/writing-a-chatbot-in-django-part-1<p>While brainstorming Django app ideas, I thought that it would be cool to design a personal assistant chatbot that can schedule appointments based on my availability and meeting time preferences. This would function as a simple chatbot version of <a href="https://calendly.com">Calendly</a>. Ideally this application can even be integrated with a service like Twilio so that users could text message the bot. Over the next series of articles, we will explore writing such a chatbot in Python Django.</p>
<h2 id="rule-based-chatbots">Rule Based Chatbots</h2>
<p>One of my favorite parts of working with Django is the availability of Python’s amazing AI and ML libraries. After doing a little bit of research, I decided that a rule based chatbot would be sufficient for the purpose of scheduling appointments. There are no shortage of rule based chatbot guides. In their simplest form, a rule based bot will look for keywords to determine the ‘intent’ of the user’s request and pick an appropriate response. For a simple yet capable rule based bot, I recommend checking out <a href="https://blog.datasciencedojo.com/building-a-rule-based-chatbot-in-python/">Building a Rule Based Chatbot in Python</a>.</p>
<p>Unfortunately, there are far too many ways to request an appointment without having to enforce an awkward limited set of commands that the user can use. These types of bots will often appear as button based bot. At that point the bot starts to resemble a webform embedded in a text message client. This is convenient but it’s not very conversational. My goal is to strike the balance between a rule based bot and a bot with a little more conversational ability.</p>
<h2 id="chatterbot">Chatterbot</h2>
<p>Python has many high level chatbot frameworks. I decided to go with <a href="https://chatterbot.readthedocs.io/en/stable/#">Chatterbot</a> for several reasons:</p>
<ol>
<li>Trains in real time based on user input</li>
<li>Supports Django ORM integration</li>
<li>Modular and extendable data preprocessors and logic adapters</li>
</ol>
<h2 id="the-project">The Project</h2>
<p>For this tutorial I will start small and create a simple terminal application without Django. The project will be organized into a chatbot directory with the main <code class="language-plaintext highlighter-rouge">chatbot.py</code> program, a <code class="language-plaintext highlighter-rouge">preprocessors.py</code> module for custom data preprocessing functions, and a logic directory containing <code class="language-plaintext highlighter-rouge">schedule_adapter.py</code> that will be be used to interpret schedule requests.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── chatbot
│ ├── __init__.py
│ ├── chatbot.py
│ ├── logic
│ │ ├── __init__.py
│ │ └── schedule_adapter.py
│ └── preprocessors.py
├── database.db
├── db.sqlite3
├── requirements.txt
└── tests
├── __init__.py
├── test_chatbot.py
└── test_logic_adapters.pyc
</code></pre></div></div>
<p>As is the case with most python projects, I recommend building a virtual environment. This makes installing and managing dependencies easier by localizing everything within your project. Don’t forget to add myenv (or whatever you call your virtual env) to <code class="language-plaintext highlighter-rouge">.gitignore</code>!</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ python3 -m venv myenv
</code></pre></div></div>
<p>For this project I have chosen the following dependencies. If you like to freeze your dependencies into a requirements.txt file then I recommend using <code class="language-plaintext highlighter-rouge">pip-chill > requirements.txt</code> which will only output the dependencies of the packages that you installed and omit the child dependencies.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>chatterbot==1.1.0
chatterbot-corpus==1.2.0
delorean==1.0.0
en-core-web-sm==2.1.0
freezegun==1.0.0
ipdb==0.13.4
pip-chill==1.0.0
pylint==2.6.0
</code></pre></div></div>
<h2 id="writing-custom-logic-adapters">Writing Custom Logic Adapters</h2>
<p>As mentioned previously, Chatterbot supports modular logic adapters that can be used to add behaviors to your chatbot. Logic adapters are loaded into the chatbot when they are instantiated.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="c1"># chatbot.py
</span><span class="kn">from</span> <span class="nn">chatterbot</span> <span class="kn">import</span> <span class="n">ChatBot</span>
<span class="kn">from</span> <span class="nn">chatterbot.trainers</span> <span class="kn">import</span> <span class="n">ChatterBotCorpusTrainer</span>
<span class="c1"># Create a new instance of a ChatBot
</span><span class="n">bot</span> <span class="o">=</span> <span class="n">ChatBot</span><span class="p">(</span>
<span class="s">'Terminal'</span><span class="p">,</span>
<span class="n">storage_adapter</span><span class="o">=</span><span class="s">'chatterbot.storage.SQLStorageAdapter'</span><span class="p">,</span>
<span class="n">logic_adapters</span><span class="o">=</span><span class="p">[</span>
<span class="s">'logic.schedule_adapter.Schedule'</span><span class="p">,</span>
<span class="p">{</span>
<span class="s">'import_path'</span><span class="p">:</span> <span class="s">'chatterbot.logic.BestMatch'</span><span class="p">,</span>
<span class="s">'default_response'</span><span class="p">:</span> <span class="s">"I'm sorry I don't understand. I schedule appointments. Is there a date and time you have in mind?"</span><span class="p">,</span>
<span class="s">'maximum_similarity_threshold'</span><span class="p">:</span> <span class="mf">0.90</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="n">preprocessors</span><span class="o">=</span><span class="p">[</span>
<span class="s">'preprocessors.format_dates'</span>
<span class="p">],</span>
<span class="n">database_uri</span><span class="o">=</span><span class="s">'sqlite:///database.db'</span>
<span class="p">)</span></code></pre></figure>
<p>We are using our custom logic adapter called <code class="language-plaintext highlighter-rouge">logic.schedule_adapter.Schedule</code> along with a base adapter called <code class="language-plaintext highlighter-rouge">chatterbot.logic.BestMatch</code>. The included <code class="language-plaintext highlighter-rouge">BestMatch</code> adapter is set to return a default response when none of the conditions for the other logic adapters are satisfied. The primary role of this bot will be to detect, interpret, and parse schedule requests. For this iteration we will keep things fairly simple and ensure that we can correspond with our bot like below.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ I would to schedule an appointment for january 5th at 5 PM
=> scheduling appointment for 1/05/21 at 17:00:00
</code></pre></div></div>
<p>Corresponding in this manner requires that our Logic Adapter can detect the user’s intent to schedule an appointment. All LogicAdapter classes need to respond to a <code class="language-plaintext highlighter-rouge">can_process</code> method that is used to invoke the adapter if the method returns <code class="language-plaintext highlighter-rouge">True</code>. This can be done with a simple regex condition. If <code class="language-plaintext highlighter-rouge">can_process</code> returns <code class="language-plaintext highlighter-rouge">True</code> then the class will process the request with the <code class="language-plaintext highlighter-rouge">process</code> method.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">chatterbot.logic</span> <span class="kn">import</span> <span class="n">LogicAdapter</span>
<span class="kn">from</span> <span class="nn">delorean</span> <span class="kn">import</span> <span class="n">parse</span>
<span class="kn">from</span> <span class="nn">re</span> <span class="kn">import</span> <span class="n">search</span>
<span class="kn">import</span> <span class="nn">spacy</span>
<span class="k">class</span> <span class="nc">Schedule</span><span class="p">(</span><span class="n">LogicAdapter</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">chatbot</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="nb">super</span><span class="p">().</span><span class="n">__init__</span><span class="p">(</span><span class="n">chatbot</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">can_process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">statement</span><span class="p">):</span>
<span class="k">if</span> <span class="n">search</span><span class="p">(</span><span class="s">'(make|schedule).*appointment'</span><span class="p">,</span> <span class="n">statement</span><span class="p">.</span><span class="n">text</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">True</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">False</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">input_statement</span><span class="p">,</span> <span class="n">additional_response_selection_parameters</span><span class="p">):</span>
<span class="kn">from</span> <span class="nn">chatterbot.conversation</span> <span class="kn">import</span> <span class="n">Statement</span>
<span class="n">nlp</span> <span class="o">=</span> <span class="n">spacy</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="s">'en_core_web_sm'</span><span class="p">)</span>
<span class="n">doc</span> <span class="o">=</span> <span class="n">nlp</span><span class="p">(</span><span class="n">input_statement</span><span class="p">.</span><span class="n">text</span><span class="p">)</span>
<span class="n">entities</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">ent</span> <span class="ow">in</span> <span class="n">doc</span><span class="p">.</span><span class="n">ents</span><span class="p">:</span>
<span class="n">entities</span><span class="p">[</span><span class="n">ent</span><span class="p">.</span><span class="n">label_</span><span class="p">]</span> <span class="o">=</span> <span class="n">ent</span><span class="p">.</span><span class="n">text</span>
<span class="n">date</span> <span class="o">=</span> <span class="n">entities</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'DATE'</span><span class="p">)</span>
<span class="n">time</span> <span class="o">=</span> <span class="n">entities</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'TIME'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">date</span> <span class="ow">and</span> <span class="n">time</span><span class="p">:</span>
<span class="n">appointment</span> <span class="o">=</span> <span class="n">parse</span><span class="p">(</span><span class="n">date</span><span class="o">+</span><span class="s">" "</span><span class="o">+</span><span class="n">time</span><span class="p">,</span> <span class="n">dayfirst</span><span class="o">=</span><span class="bp">False</span><span class="p">).</span><span class="n">datetime</span><span class="p">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">"%m/%d/%y at %H:%M:%S"</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">date</span><span class="p">:</span>
<span class="n">appointment</span> <span class="o">=</span> <span class="n">parse</span><span class="p">(</span><span class="n">date</span><span class="p">,</span> <span class="n">dayfirst</span><span class="o">=</span><span class="bp">False</span><span class="p">).</span><span class="n">date</span><span class="p">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">"%m/%d/%y"</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">time</span><span class="p">:</span>
<span class="n">appointment</span> <span class="o">=</span> <span class="n">parse</span><span class="p">(</span><span class="n">time</span><span class="p">).</span><span class="n">datetime</span><span class="p">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">"%m/%d/%y at %H:%M:%S"</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">Statement</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="s">"please provide a preferred date and time for your appointment"</span><span class="p">,</span> <span class="n">confidence</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="n">response</span> <span class="o">=</span> <span class="s">"scheduling appointment for {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">appointment</span><span class="p">)</span>
<span class="k">return</span> <span class="n">Statement</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="n">response</span><span class="p">,</span> <span class="n">confidence</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">process</code> method assumes that data has been sanitized by the data preprocessor functions (more on this in the next section). We can use the natural language processing library <a href="https://spacy.io/">spacy</a> to identify the ‘entities’ (date and times) in the user’s request. At this stage, we are not concerned about making the appointments. We only want to prove that we can interpret the datetimes from the natural language inputs.</p>
<p>The ‘en_core_web_sm’ model loaded by spacy does a very good job at extracting the date and time entities. To further parse the extracted date and time we turn to a library that deals with time transformations. This library is appropriately called Delorean. It’s parse function is used to parse any date format or natural language representation to a standard <code class="language-plaintext highlighter-rouge">m/d/y h:m:s</code> format.</p>
<p>Once this is hooked up to a calendar with available times, the bot will be able to schedule the appointment if it’s available by confirming the next available time with an app that we will create in the next tutorial. In cases where a date and time is provided the bot will try to fill that time. If only the date is provided then the bot will pick the next available time for that day. If only the time is provided, then the bot will schedule the appointment for the day it was requested. In all cases, the bot will confirm the time.</p>
<p>For this tutorial we will just have the bot return a standard message notifying the user that the appointment is being scheduled for the proposed time.</p>
<h2 id="preprocessor-functions">Preprocessor Functions</h2>
<p>Spacy does a great job at detecting entities. However, there were a few gaps in it’s ability to detect lowercase months and dates entered in the american mm/dd/yyyy format. I could have added examples to the model directly but a much easier approach is to format these likely inputs into the format that spacy can interpret.</p>
<p>Chatterbot preprocessors are simply functions that can be added to the Chatbot’s preprocessor kwarg. These functions will be evaluated before passing the input to any logic adapters. In our case this is perfect because we can use a little bit of regex sorcery to sanitize our inputs to a date format and month format that Spacy can interpret.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="c1"># preprocessors.py
</span><span class="kn">from</span> <span class="nn">re</span> <span class="kn">import</span> <span class="n">search</span><span class="p">,</span> <span class="n">sub</span><span class="p">,</span> <span class="n">split</span>
<span class="k">def</span> <span class="nf">format_dates</span><span class="p">(</span><span class="n">statement</span><span class="p">):</span>
<span class="n">pattern</span> <span class="o">=</span> <span class="sa">r</span><span class="s">'(\d{2}|\d{1})(-|\/)(\d{2}|\d{1})(-|\/)\d{4}'</span>
<span class="n">bad_date_format</span> <span class="o">=</span> <span class="n">search</span><span class="p">(</span><span class="n">pattern</span><span class="p">,</span> <span class="n">statement</span><span class="p">.</span><span class="n">text</span><span class="p">)</span>
<span class="k">if</span> <span class="n">bad_date_format</span><span class="p">:</span>
<span class="n">date</span> <span class="o">=</span> <span class="n">bad_date_format</span><span class="p">.</span><span class="n">group</span><span class="p">()</span>
<span class="n">m</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">split</span><span class="p">(</span><span class="sa">r</span><span class="s">"\/|-"</span><span class="p">,</span> <span class="n">date</span><span class="p">)</span>
<span class="n">date</span> <span class="o">=</span> <span class="s">"-"</span><span class="p">.</span><span class="n">join</span><span class="p">([</span><span class="n">y</span><span class="p">,</span><span class="n">m</span><span class="p">,</span><span class="n">d</span><span class="p">])</span>
<span class="n">statement</span><span class="p">.</span><span class="n">text</span> <span class="o">=</span> <span class="n">sub</span><span class="p">(</span><span class="n">pattern</span><span class="p">,</span> <span class="n">date</span><span class="p">,</span> <span class="n">bad_date_format</span><span class="p">.</span><span class="n">string</span><span class="p">)</span>
<span class="k">return</span> <span class="n">statement</span></code></pre></figure>
<p>Formatting dates involves detecting any dates that are delimited with ‘/’ or ‘-‘ and are assembled in mm/dd/yyyy or m/d/yyyy format. These dates are then split by their delimiter and reordered in ISO format (the preferred date format of Spacy).</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">capitalize_months</span><span class="p">(</span><span class="n">statement</span><span class="p">):</span>
<span class="n">jan_apr</span> <span class="o">=</span> <span class="sa">r</span><span class="s">'\b(jan|january)\b|\b(feb|february)\b|\b(mar|march)\b|\b(apr|april)\b'</span>
<span class="n">may_aug</span> <span class="o">=</span> <span class="sa">r</span><span class="s">'\b(may)\b|\b(jun|june)\b|\b(jul|july)\b|\b(aug|august)\b'</span>
<span class="n">sept_dec</span> <span class="o">=</span> <span class="sa">r</span><span class="s">'\b(sept|september)\b|\b(oct|october)\b|\b(nov|november)\b|\b(dec|december)\b'</span>
<span class="n">pattern</span> <span class="o">=</span> <span class="n">jan_apr</span><span class="o">+</span><span class="n">may_aug</span><span class="o">+</span><span class="n">sept_dec</span>
<span class="n">month</span> <span class="o">=</span> <span class="n">search</span><span class="p">(</span><span class="n">pattern</span><span class="p">,</span> <span class="n">statement</span><span class="p">.</span><span class="n">text</span><span class="p">)</span>
<span class="k">if</span> <span class="n">month</span><span class="p">:</span>
<span class="n">capitalized_month</span> <span class="o">=</span> <span class="n">month</span><span class="p">.</span><span class="n">group</span><span class="p">().</span><span class="n">capitalize</span><span class="p">()</span>
<span class="n">statement</span><span class="p">.</span><span class="n">text</span> <span class="o">=</span> <span class="n">sub</span><span class="p">(</span><span class="n">pattern</span><span class="p">,</span> <span class="n">capitalized_month</span><span class="p">,</span> <span class="n">month</span><span class="p">.</span><span class="n">string</span><span class="p">)</span>
<span class="k">return</span> <span class="n">statement</span></code></pre></figure>
<p>We want to look out for both lowercase abbreviations and months. Neither format is detected as a date by Spacy. After detecting the pattern, the matching portion of the statement is capitalized and spliced back into the original statement.</p>
<h2 id="putting-it-all-together">Putting It All Together</h2>
<p>In the end we have a chatbot that is conversational and can handle a pretty wide variety of appointment requests.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">bot</span> <span class="o">=</span> <span class="n">ChatBot</span><span class="p">(</span>
<span class="s">'Terminal'</span><span class="p">,</span>
<span class="n">storage_adapter</span><span class="o">=</span><span class="s">'chatterbot.storage.SQLStorageAdapter'</span><span class="p">,</span>
<span class="n">logic_adapters</span><span class="o">=</span><span class="p">[</span>
<span class="s">'logic.schedule_adapter.Schedule'</span><span class="p">,</span>
<span class="p">{</span>
<span class="s">'import_path'</span><span class="p">:</span> <span class="s">'chatterbot.logic.BestMatch'</span><span class="p">,</span>
<span class="s">'default_response'</span><span class="p">:</span> <span class="s">"I'm sorry I don't understand. I schedule appointments. Is there a date and time you have in mind?"</span><span class="p">,</span>
<span class="s">'maximum_similarity_threshold'</span><span class="p">:</span> <span class="mf">0.90</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="n">preprocessors</span><span class="o">=</span><span class="p">[</span>
<span class="s">'preprocessors.format_dates'</span><span class="p">,</span>
<span class="s">'chatbot.preprocessors.capitalize_months'</span>
<span class="p">],</span>
<span class="n">database_uri</span><span class="o">=</span><span class="s">'sqlite:///database.db'</span>
<span class="p">)</span>
<span class="n">trainer</span> <span class="o">=</span> <span class="n">ChatterBotCorpusTrainer</span><span class="p">(</span><span class="n">bot</span><span class="p">)</span>
<span class="n">trainer</span><span class="p">.</span><span class="n">train</span><span class="p">(</span><span class="s">'chatterbot.corpus.english.greetings'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Type something to begin...'</span><span class="p">)</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">user_input</span> <span class="o">=</span> <span class="nb">input</span><span class="p">()</span>
<span class="n">bot_response</span> <span class="o">=</span> <span class="n">bot</span><span class="p">.</span><span class="n">get_response</span><span class="p">(</span><span class="n">user_input</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">bot_response</span><span class="p">)</span>
<span class="c1"># Press ctrl-c or ctrl-d on the keyboard to exit
</span> <span class="k">except</span> <span class="p">(</span><span class="nb">KeyboardInterrupt</span><span class="p">,</span> <span class="nb">EOFError</span><span class="p">,</span> <span class="nb">SystemExit</span><span class="p">):</span>
<span class="k">break</span></code></pre></figure>
<p>The bot is conversational (can perform basic greetings) with the addition of the <code class="language-plaintext highlighter-rouge">english.greetings</code> corpus included in the <code class="language-plaintext highlighter-rouge">ChatterBotCorpusTrainer</code> library.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope you are as excited as I am about the progress we made. At this point we have a conversational bot that can do a pretty good job at detecting a user’s intent to schedule an appointment, extract a datetime, and respond with a confirmation of the proposed meeting. Next time we will drop this application into a Django application and actually have our chatbot schedule appointments based on preferred times and availability set in the Django admin.</p>While brainstorming Django app ideas, I thought that it would be cool to design a personal assistant chatbot that can schedule appointments based on my availability and meeting time preferences. This would function as a simple chatbot version of Calendly. Ideally this application can even be integrated with a service like Twilio so that users could text message the bot. Over the next series of articles, we will explore writing such a chatbot in Python Django.Caching in Django with the Prefetch and Annotate Pattern2020-12-19T00:00:00+00:002020-12-19T00:00:00+00:00/2020/12/19/caching-in-django<p>At some point you will add a property to a model that triggers an n+1 query. This is when an attribute of an associated table is requested. For each time this happens n additional queries are added to the request. What we need to do is request all of the joined tables as a single request and then reference the fetched data with the joined tables needed for the model’s property. Fortunately, Django makes caching extremely simple and also provides a built in test matcher that makes detecting n+1 queries a snap.</p>
<p>Our story begins with the addition of two innocent properties to a blog post model. For our blog posts we would like to include a count of reviews along with the average review score on the <code class="language-plaintext highlighter-rouge">post_list</code> page for each post.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">@</span><span class="nb">property</span>
<span class="k">def</span> <span class="nf">average_rating</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">review_set</span><span class="p">.</span><span class="n">aggregate</span><span class="p">(</span>
<span class="n">average_rating</span><span class="o">=</span><span class="n">Avg</span><span class="p">(</span><span class="s">'rating'</span><span class="p">)</span>
<span class="p">)[</span><span class="s">'average_rating'</span><span class="p">]</span>
<span class="o">@</span><span class="nb">property</span>
<span class="k">def</span> <span class="nf">number_of_reviews</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">review_set</span><span class="p">.</span><span class="n">count</span><span class="p">()</span></code></pre></figure>
<p>Functionally everything checks out and both properties can easily be referenced in the post index template. However, there is a severe performance hit that happens when we calculate averages or counts on a related table like <code class="language-plaintext highlighter-rouge">review_set</code>. We can write a test that asserts that <code class="language-plaintext highlighter-rouge">post_list</code> only results in one hit to the database.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">PostViewTests</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">user</span> <span class="o">=</span> <span class="n">factories</span><span class="p">.</span><span class="n">UserFactory</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">client</span> <span class="o">=</span> <span class="n">Client</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">posts</span> <span class="o">=</span> <span class="n">factories</span><span class="p">.</span><span class="n">PostFactory</span><span class="p">.</span><span class="n">create_batch</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="n">author</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">user</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_fetches_blog_articles_in_single_query</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">with</span> <span class="bp">self</span><span class="p">.</span><span class="n">assertNumQueries</span><span class="p">(</span><span class="mi">1</span><span class="p">):</span>
<span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">reverse</span><span class="p">(</span><span class="s">'post_list'</span><span class="p">))</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">status_code</span><span class="p">,</span> <span class="mi">200</span><span class="p">)</span></code></pre></figure>
<p>This test fails with 11 fetched queries! This makes sense considering that we are constructing a listing of 5 posts. Both fields add 5 additional queries in addition to the original query because they reference data on the <code class="language-plaintext highlighter-rouge">Review</code> model. This will eventually lead to some unacceptable performance bottlenecks as the list of posts grows and additional properties are added to Post.</p>
<p>Django provides two tools that can help us solve this problem - <code class="language-plaintext highlighter-rouge">QuerySet.annotate</code> and <code class="language-plaintext highlighter-rouge">QuerySet.prefetch</code>. We will start by creating a <code class="language-plaintext highlighter-rouge">PostQuerySet</code> class and assign that as a manager to our Post model’s objects property.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">PostQuerySet</span><span class="p">(</span><span class="n">models</span><span class="p">.</span><span class="n">QuerySet</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">prefetch_detail</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">annotate_with_average_rating</span><span class="p">(</span>
<span class="p">).</span><span class="n">annotate_with_num_reviews</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">annotate_with_average_rating</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">annotate</span><span class="p">(</span>
<span class="n">_average_rating_cache</span><span class="o">=</span><span class="n">Avg</span><span class="p">(</span><span class="s">'review__rating'</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">def</span> <span class="nf">annotate_with_num_reviews</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">annotate</span><span class="p">(</span>
<span class="n">_num_reviews</span><span class="o">=</span><span class="n">Count</span><span class="p">(</span><span class="s">'review'</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">class</span> <span class="nc">Post</span><span class="p">(</span><span class="n">models</span><span class="p">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">objects</span> <span class="o">=</span> <span class="n">PostQuerySet</span><span class="p">.</span><span class="n">as_manager</span><span class="p">()</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">QuerySet.annotate</code> method is used to add a private property to each post with the calculated average review_rating and review count. Posts can be fetched with the detail records in the <code class="language-plaintext highlighter-rouge">prefetch_detail</code> method. This method can be called when references to a Post’s <code class="language-plaintext highlighter-rouge">average_rating</code> and <code class="language-plaintext highlighter-rouge">number_of_reviews</code> are needed. The initial query might run slightly longer than a simple selection on posts. However, this will pay off when additional hits to the database no longer occur with the addition of the annotated calculations.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">@</span><span class="nb">property</span>
<span class="k">def</span> <span class="nf">average_rating</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s">'_average_rating_cache'</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">_average_rating_cache</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">review_set</span><span class="p">.</span><span class="n">aggregate</span><span class="p">(</span>
<span class="n">average_rating</span><span class="o">=</span><span class="n">Avg</span><span class="p">(</span><span class="s">'rating'</span><span class="p">)</span>
<span class="p">)[</span><span class="s">'average_rating'</span><span class="p">]</span>
<span class="o">@</span><span class="nb">property</span>
<span class="k">def</span> <span class="nf">number_of_reviews</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s">'_num_reviews'</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">_num_reviews</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">review_set</span><span class="p">.</span><span class="n">count</span><span class="p">()</span></code></pre></figure>
<p>A simple guard condition can now be added to each property that checks for the presence of the annotated cached value before calling the calculation on <code class="language-plaintext highlighter-rouge">Post.review_set</code>. With this update and the use of <code class="language-plaintext highlighter-rouge">prefetch_detail</code> in the <code class="language-plaintext highlighter-rouge">post_list</code> view method, the query count for the test returns back to a single query count and passes!</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="c1"># prefetch the blog articles with the annotations
</span> <span class="c1"># mysite.blog.views
</span> <span class="k">def</span> <span class="nf">post_list</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="n">posts</span> <span class="o">=</span> <span class="n">Post</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="n">prefetch_detail</span><span class="p">()</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s">'blog/post_list.html'</span><span class="p">,</span> <span class="p">{</span><span class="s">'posts'</span><span class="p">:</span> <span class="n">posts</span><span class="p">})</span></code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
<p>Testing view methods by using <code class="language-plaintext highlighter-rouge">TestCase.assertNumQueries</code> is an easy way to detect n+1 performance issues in Django applications. Fixing n+1 queries can usually be solved with the addition of an annotation for the calculated or related data needed by the view requesting the data. Generally, a single reusable <code class="language-plaintext highlighter-rouge">prefetch_detail</code> QuerySet method can be used in cases where a template or serializer needs data that should be calculated or included in advance. It is also up to the model to handle these annotations and use them when they exist.</p>
<p>Making these simple changes to an application can really help scale your Django application and possibly save money on the computation power needed for your database server.</p>At some point you will add a property to a model that triggers an n+1 query. This is when an attribute of an associated table is requested. For each time this happens n additional queries are added to the request. What we need to do is request all of the joined tables as a single request and then reference the fetched data with the joined tables needed for the model’s property. Fortunately, Django makes caching extremely simple and also provides a built in test matcher that makes detecting n+1 queries a snap.From Ruby on Rails to Python Django: Initial Thoughts2020-12-12T00:00:00+00:002020-12-12T00:00:00+00:00/2020/12/12/from-ruby-on-rails-to-django<p>I recently made the transition to the Python Django web framework after working with Ruby on Rails for 5 years. Both frameworks are similar but the adjustment to Django was a little more interesting than I expected. What follows is a brief summary of my initial reactions to the Django framework as a Rails developer.</p>
<h2 id="comparing-project-file-structures">Comparing Project File Structures</h2>
<p>Rails is a framework built on the ethos of ‘opinion over configuration’. The project structure of of a typical Rails application is defined by the framework itself. Over the years, new features in Rails have increased the number of directories in a project. What has remained consistent is the separation of the model, view, and controller concerns.</p>
<p>The heart of a Rails Application lives inside of the /App directory with an additional /config directory.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── assets
├── channels
├── controllers
│ ├── application_controller.rb
│ ├── concerns
│ └── posts_controller.rb
├── helpers
│ ├── application_helper.rb
│ └── posts_helper.rb
├── javascript
├── jobs
├── mailers
├── models
│ ├── application_record.rb
│ ├── concerns
│ ├── post.rb
│ └── user.rb
└── views
├── layouts
└── posts
├── _form.html.erb
├── _post.html.erb
├── edit.html.erb
├── index.html.erb
├── index.json.jbuilder
├── new.html.erb
├── show.html.erb
└── show.json.jbuilder
</code></pre></div></div>
<p>As of Rails 6, there are additional directories for web socket connections (channels), helper functions typically used in view templates (helpers), javascript files compiled by webpacker (javascript), async jobs (jobs), and email dispatchers (mailers). The asset folder is the legacy location for javascript. This includes project multimedia assets and sass files compiled by the Rails Asset Pipeline.</p>
<p>Python Django projects are organized by ‘Apps’. The root directory includes a manage.py script which is used to run all of the Django CLI commands. The Project folder includes any number of ‘Apps’ along with the project settings (settings.py), project urls (urls.py), and the wsgi server file (wsgi.py). The settings file, among other things, requires you to declare all ‘Apps’ used in the project.</p>
<p>I enjoy organizing different parts of my application into apps. This makes it easier to develop reusable app components that can be shared among projects. Rails supports apps within apps with RailsEngines but the feature is often overlooked and is not as integral to the way most Rails developers architect projects.</p>
<p>Most of the interesting parts of this project are happening in the blog app. Django projects leave you responsible for organizing how your apps are structured. For example, you might define multiple blog model classes within models.py such as ‘Post’ and ‘Comment’. You can also organize your models in a directory similar to the way Rails enforces project directories. The difference is that you are in complete control in Django.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>## Typical Django Project Layout
├── db.sqlite3
├── manage.py
├── mysite
│ ├── __init__.py
│ ├── blog
│ │ ├── __init__.py
│ │ ├── admin.py
│ │ ├── apps.py
│ │ ├── forms.py
│ │ ├── migrations
│ │ ├── models.py
│ │ ├── static
│ │ │ └── css
│ │ │ └── blog.css
│ │ ├── templates
│ │ │ └── blog
│ │ │ ├── base.html
│ │ │ ├── post_detail.html
│ │ │ ├── post_edit.html
│ │ │ └── post_list.html
│ │ ├── tests.py
│ │ ├── urls.py
│ │ └── views.py
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
└── requirements.txt
</code></pre></div></div>
<p>One of the really cool features that ships with Django is a built an Admin app complete with user authentication! The admin interface can be configured extensively in the admin.py file. Full CRUD functionality for posts can be achieved with just a simple one line registration of Post model.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">admin</span><span class="p">.</span><span class="n">site</span><span class="p">.</span><span class="n">register</span><span class="p">(</span><span class="n">Post</span><span class="p">)</span></code></pre></figure>
<h2 id="views-not-controllers">Views (Not Controllers)</h2>
<p>Django views take the place of controllers which is a little confusing considering that Rails calls its templates views. Once you get past the confusion of the name the general pattern is straightforward. Url routes are defined in the parent urls.py directory and each App folder typically has its own set of urls that are imported into the project. The routes for a simple blog application look like this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">django.urls</span> <span class="kn">import</span> <span class="n">path</span>
<span class="kn">from</span> <span class="nn">.</span> <span class="kn">import</span> <span class="n">views</span>
<span class="n">urlpatterns</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">path</span><span class="p">(</span><span class="s">''</span><span class="p">,</span> <span class="n">views</span><span class="p">.</span><span class="n">post_list</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">'post_list'</span><span class="p">),</span>
<span class="n">path</span><span class="p">(</span><span class="s">'post/<int:pk>/'</span><span class="p">,</span> <span class="n">views</span><span class="p">.</span><span class="n">post_detail</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">'post_detail'</span><span class="p">),</span>
<span class="n">path</span><span class="p">(</span><span class="s">'post/new/'</span><span class="p">,</span> <span class="n">views</span><span class="p">.</span><span class="n">post_new</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">'post_new'</span><span class="p">),</span>
<span class="n">path</span><span class="p">(</span><span class="s">'post/<int:pk>/edit/'</span><span class="p">,</span> <span class="n">views</span><span class="p">.</span><span class="n">post_edit</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">'post_edit'</span><span class="p">),</span>
<span class="p">]</span></code></pre></figure>
<p>Each url invokes the corresponding View function imported into the urls.py file. The root path of ‘’ calls the post_list function in views.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">post_list</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="n">posts</span> <span class="o">=</span> <span class="n">Post</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="nb">filter</span><span class="p">(</span><span class="n">published_date__lte</span><span class="o">=</span><span class="n">timezone</span><span class="p">.</span><span class="n">now</span><span class="p">()).</span><span class="n">order_by</span><span class="p">(</span><span class="s">'published_date'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s">'blog/post_list.html'</span><span class="p">,</span> <span class="p">{</span><span class="s">'posts'</span><span class="p">:</span> <span class="n">posts</span><span class="p">})</span></code></pre></figure>
<p>Every view function receives a mandatory request argument and an optional url parameter argument. The function is expected to return a valid HTTP response. In this case it is a rendered post_list.html template. The fetched posts are included in a dictionary which can be accessed by the embedded python code in the template.</p>
<p>An index listing of objects like post_list is simple enough to express but I do miss some of the syntax sugar Rails provides when working with POST and PUT methods. Unlike Django, Rails splits all RESTful actions out to predefined ActionController methods called index, show, new, create, edit, update, and delete. In fact, all we need to do to create a similar list of endpoints is set the posts controller controller as a resource in the config/routes.rb file.</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">resources</span> <span class="ss">:posts</span><span class="p">,</span> <span class="ss">except: </span><span class="p">[</span><span class="ss">:delete</span><span class="p">]</span></code></pre></figure>
<p>The request for the edit post form and the subsequent POST request that is used to save the update is broken down into two separate methods in the posts controller:</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">PostsController</span> <span class="o"><</span> <span class="no">ApplicationController</span>
<span class="n">before_action</span> <span class="ss">:set_post</span><span class="p">,</span> <span class="ss">only: </span><span class="p">[</span><span class="ss">:show</span><span class="p">,</span> <span class="ss">:edit</span><span class="p">,</span> <span class="ss">:update</span><span class="p">,</span> <span class="ss">:destroy</span><span class="p">]</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">new</span>
<span class="vi">@post</span> <span class="o">=</span> <span class="no">Post</span><span class="p">.</span><span class="nf">new</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">create</span>
<span class="vi">@post</span> <span class="o">=</span> <span class="no">Post</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">post_params</span><span class="p">)</span>
<span class="k">if</span> <span class="vi">@post</span><span class="p">.</span><span class="nf">save</span>
<span class="n">redirect_to</span> <span class="vi">@post</span><span class="p">,</span> <span class="ss">notice: </span><span class="s1">'Post was successfully created.'</span>
<span class="k">else</span>
<span class="n">render</span> <span class="ss">:new</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">post_params</span>
<span class="n">params</span><span class="p">.</span><span class="nf">require</span><span class="p">(</span><span class="ss">:post</span><span class="p">).</span><span class="nf">permit</span><span class="p">(</span><span class="ss">:title</span><span class="p">,</span> <span class="ss">:body</span><span class="p">,</span> <span class="ss">:published_at</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>For the new post page, we just need to create an empty instance of <code class="language-plaintext highlighter-rouge">@post</code> with the Post.new method. When Rails receives a POST request for posts, the router will call the <code class="language-plaintext highlighter-rouge">PostsController.create</code> method which will create the new post. The <code class="language-plaintext highlighter-rouge">post_params</code> variable takes all of the submitted parameters from the request restricted only to the fields allowed by that form - title, body, and published_at.</p>
<p>In the create method, the post_params data is passed to a new post and then saved. A model returns truthy when it’s successfully saved to the DB. We can keep the method pretty short with a simple condition to redirect to the newly created post if it’s saved or simply render the new form if there is a validation error.</p>
<p>Django does all of this stuff in a more obvious way by using conditional statements for the type of request. But Rails embraces this pattern in a more elegant object oriented way that is DRY and does not require writing conditions for different Http Methods in a function. In addition to this, Django requires the manual setting of each post property rather than just giving all of the submitted parameters to a new Post object. What follows is maybe something a little bit more readable for beginners but definitely something that gets harder to read as the complexity of a given view function increases.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">post_new</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="k">if</span> <span class="n">request</span><span class="p">.</span><span class="n">method</span> <span class="o">==</span> <span class="s">"POST"</span><span class="p">:</span>
<span class="n">form</span> <span class="o">=</span> <span class="n">PostForm</span><span class="p">(</span><span class="n">request</span><span class="p">.</span><span class="n">POST</span><span class="p">)</span>
<span class="k">if</span> <span class="n">form</span><span class="p">.</span><span class="n">is_valid</span><span class="p">():</span>
<span class="n">post</span> <span class="o">=</span> <span class="n">form</span><span class="p">.</span><span class="n">save</span><span class="p">(</span><span class="n">commit</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">post</span><span class="p">.</span><span class="n">author</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="n">user</span>
<span class="n">post</span><span class="p">.</span><span class="n">published_date</span> <span class="o">=</span> <span class="n">timezone</span><span class="p">.</span><span class="n">now</span><span class="p">()</span>
<span class="n">post</span><span class="p">.</span><span class="n">save</span><span class="p">()</span>
<span class="k">return</span> <span class="n">redirect</span><span class="p">(</span><span class="s">'post_detail'</span><span class="p">,</span> <span class="n">pk</span><span class="o">=</span><span class="n">post</span><span class="p">.</span><span class="n">pk</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">form</span> <span class="o">=</span> <span class="n">PostForm</span><span class="p">()</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s">'blog/post_edit.html'</span><span class="p">,</span> <span class="p">{</span><span class="s">'form'</span><span class="p">:</span> <span class="n">form</span><span class="p">})</span></code></pre></figure>
<h2 id="templates">Templates</h2>
<p>The Rails template engine is similar to the Django template system. Like Rails, Django injects any instance variables from the view that renders the template into the template. Django includes a myriad of built in template helpers that make it easy to iterate through objects, format data, and even render complete forms.</p>
<p>Django forms are generic by default and can be injected into Django templates via a Django view.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="c1"># Django mysite/blog/form.py
</span><span class="k">class</span> <span class="nc">PostForm</span><span class="p">(</span><span class="n">forms</span><span class="p">.</span><span class="n">ModelForm</span><span class="p">):</span>
<span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Post</span>
<span class="n">fields</span> <span class="o">=</span> <span class="p">(</span><span class="s">'title'</span><span class="p">,</span> <span class="s">'text'</span><span class="p">,)</span>
<span class="c1"># mysite/blog/views.py
</span><span class="k">def</span> <span class="nf">post_edit</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">pk</span><span class="p">):</span>
<span class="n">post</span> <span class="o">=</span> <span class="n">get_object_or_404</span><span class="p">(</span><span class="n">Post</span><span class="p">,</span> <span class="n">pk</span><span class="o">=</span><span class="n">pk</span><span class="p">)</span>
<span class="k">if</span> <span class="n">request</span><span class="p">.</span><span class="n">method</span> <span class="o">==</span> <span class="s">"POST"</span><span class="p">:</span>
<span class="n">form</span> <span class="o">=</span> <span class="n">PostForm</span><span class="p">(</span><span class="n">request</span><span class="p">.</span><span class="n">POST</span><span class="p">,</span> <span class="n">instance</span><span class="o">=</span><span class="n">post</span><span class="p">)</span>
<span class="k">if</span> <span class="n">form</span><span class="p">.</span><span class="n">is_valid</span><span class="p">():</span>
<span class="n">post</span> <span class="o">=</span> <span class="n">form</span><span class="p">.</span><span class="n">save</span><span class="p">(</span><span class="n">commit</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">post</span><span class="p">.</span><span class="n">author</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="n">user</span>
<span class="n">post</span><span class="p">.</span><span class="n">published_date</span> <span class="o">=</span> <span class="n">timezone</span><span class="p">.</span><span class="n">now</span><span class="p">()</span>
<span class="n">post</span><span class="p">.</span><span class="n">save</span><span class="p">()</span>
<span class="k">return</span> <span class="n">redirect</span><span class="p">(</span><span class="s">'post_detail'</span><span class="p">,</span> <span class="n">pk</span><span class="o">=</span><span class="n">post</span><span class="p">.</span><span class="n">pk</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">form</span> <span class="o">=</span> <span class="n">PostForm</span><span class="p">(</span><span class="n">instance</span><span class="o">=</span><span class="n">post</span><span class="p">)</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s">'blog/post_edit.html'</span><span class="p">,</span> <span class="p">{</span><span class="s">'form'</span><span class="p">:</span> <span class="n">form</span><span class="p">}</span></code></pre></figure>
<p>Injecting class names for styling purposes is a little tricky and involves customization of the form.py file in ways that I think over complicates forms. Rails, in contrast, makes the building blocks of the form and it’s controls available as simple template helper functions. Integrating helpers with standard HTML code is very easy to do in Rails.</p>
<h2 id="models">Models</h2>
<p>The differences between Rails ActiveRecord and the built in Django ORM is probably worthy of a dedicated article. I found myself quickly adjusting to the Django way of writing models. In short you are good as soon as you realize that a model’s ‘objects’ are accessible via the objects property. The rest of the syntax only varies slightly.</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">Post</span><span class="p">.</span><span class="nf">find</span> <span class="mi">1</span>
<span class="no">Post</span><span class="p">.</span><span class="nf">find_by</span> <span class="ss">title: </span><span class="s1">'From Ruby on Rails to Django'</span>
<span class="no">Post</span><span class="p">.</span><span class="nf">where</span><span class="p">.</span><span class="nf">not</span> <span class="ss">published_at: </span><span class="kp">nil</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="c1"># Django ORM examples
</span><span class="n">Post</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="n">Post</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="s">'From Ruby on Rails to Django'</span><span class="p">)</span>
<span class="n">Post</span><span class="p">.</span><span class="n">objects</span><span class="p">.</span><span class="n">exclude</span><span class="p">(</span><span class="n">published_at</span><span class="o">=</span><span class="n">null</span><span class="p">)</span></code></pre></figure>
<p>One impressive feature of Django is the ability to just add properties to a model and then run the <code class="language-plaintext highlighter-rouge">python migrate.py makemigrations</code> cli command to generate a migration file for the new fields. Rails requires running a generator command with it’s own dsl with the attributes and types included as arguments: <code class="language-plaintext highlighter-rouge">rails g migration AddPublishedDateToPost published_at:date</code>.</p>
<p>Once the migration script runs, Rails models store all of the database fields in the /db/schema.rb file. I like having the ability to see the fields inside of the model itself. While the Rails version is more compact, I prefer the transparency Django models offer.</p>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># Rails app/models/post.rb</span>
<span class="k">class</span> <span class="nc">Post</span> <span class="o"><</span> <span class="no">ApplicationRecord</span>
<span class="k">def</span> <span class="nf">publish!</span>
<span class="n">published_date</span> <span class="o">=</span> <span class="no">Time</span><span class="p">.</span><span class="nf">now</span>
<span class="n">save</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="c1"># Django mysite/blog/model.py
</span><span class="k">class</span> <span class="nc">Post</span><span class="p">(</span><span class="n">models</span><span class="p">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">author</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="n">ForeignKey</span><span class="p">(</span><span class="n">settings</span><span class="p">.</span><span class="n">AUTH_USER_MODEL</span><span class="p">,</span> <span class="n">on_delete</span><span class="o">=</span><span class="n">models</span><span class="p">.</span><span class="n">CASCADE</span><span class="p">)</span>
<span class="n">title</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">200</span><span class="p">)</span>
<span class="n">text</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="n">TextField</span><span class="p">()</span>
<span class="n">created_date</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="n">DateTimeField</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="n">timezone</span><span class="p">.</span><span class="n">now</span><span class="p">)</span>
<span class="n">published_date</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="n">DateTimeField</span><span class="p">(</span><span class="n">blank</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">null</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">publish</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">published_date</span> <span class="o">=</span> <span class="n">timezone</span><span class="p">.</span><span class="n">now</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">save</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">title</span></code></pre></figure>
<h2 id="parting-thoughts">Parting Thoughts</h2>
<p>While there are may differences between the two frameworks, I do appreciate both frameworks for what they are. I think Rails will always feel more elegant and productive when you choose to work within its strict and well documented best practices. Django is literal representation code and is probably more approachable for developers looking for a little more flexibility. I look forward to breaking down more advanced use cases and continuing to share my thoughts about Django.</p>Joshua JarvisI recently made the transition to the Python Django web framework after working with Ruby on Rails for 5 years. Both frameworks are similar but the adjustment to Django was a little more interesting than I expected. What follows is a brief summary of my initial reactions to the Django framework as a Rails developer.