Testing Flask App

Testing in Python vs JavaScript?

28 April 2022

Python Flask Testing Pytest Unittest
Testing Flask App

Recently, I have been working on a web application built from the Flask framework. Alongside regular development work for this project I have tested the app with various test methods such as unit testing, functional testing, and end-to-end testing. This is the very first time I have written tests in Python. In this blog post I will include some notes around the set up, how to write tests, and some thoughts on testing in Python vs JavaScript.

General Testing

There are two popular Python testing frameworks - Pytest and Unittest. Pytest is easier to work with than the standard testing library Unittest, therefore I have chosen it for its simplicity so it can be pick up easily by other team members as well.

Set up

To get started, install pytest using pip or whatever method that suits your development environment. For me it was simple - we have containerised the environment so all I need to do is add pytest to the requirements.txt file and rebuild the docker image.

To keep test files organised I have created a tests folder under the root directory, in addition, I have also created the file tests/conftest.py which includes all the custom fixtures that I have defined. You can learn about fixtures in depth from the pytest docs, but on the high level fixtures are constants that can be used in any of the tests in the current directory or levels below relative to the conftest.py file. A fixture can be a dataset, a database object, a class, or just a string. There are predefined fixtures you may want to explore here. Here is a simple example of usage:

@pytest.fixture
def my_website_name():
    return "yldweng"

The output from the my_website_name() function then can be used in any test files by insert it into the function parameter list:

def test_website_url(my_website_name):
  assert website_url == 'https://' + my_website_name + '.gitlab.io'

Fixtures have an option parameter scope which you can control when to destroy or clean up the fixture. For example, if you have provided scope="session" then this fixture will only be created and destroyed once, which is very useful for things like databases and external services that are used multiple times in a single test session. You can find out more about scopes here.

Add test

Pytest finds test files with file names in the format test_*.py or *_test.py by default. To add a test, we need to define a function and pass any required fixtures to the function and include one or more logical comparisons by using the assert keyword. A simple example is the test_website_url function shown in the previous section.

In order to test a Flask app and sending requests without running a live server, we will need some fixtures looks like the following:

@pytest.fixture(scope='session')
def test_app():
  """
  https://flask.palletsprojects.com/en/2.0.x/testing/
  """
  os.environ['FLASK_TESTING'] = 'True'

  app = create_app()

  app.config.update({
    "TESTING": True,
    "WTF_CSRF_METHODS": [],
    'CSRF_CHECK_REFERER': False,
    # add any other config as required
  })

  # other setup can go here
  yield app

  # clean up / reset resources here

@pytest.fixture(scope='module')
def test_client(test_app):
  """
  Create a test client using the Flask application configured for testing
  The test client makes requests to the application without running a live server
  """
  with test_app.test_client() as testing_client:
    # Establish an application context
    with test_app.app_context():
      yield testing_client  # Use this in other test files

This test_client fixture is very useful and we can it to test requests:

def test_record_create_edit_delete(test_app, test_client, mock_record):
  """
  Create, edit and delete
  """
  response_create = test_client.post(
      '/record/create',
      follow_redirects=True,
      data=mock_record
  )
  assert response_create.status_code == 200

  # edit
  response_edit = test_client.post(
      '/record/mock_record_1/edit',
      follow_redirects=True,
      data=mock_record
  )
  assert response_edit.status_code == 200

  # edit non exist items
  response_edit = test_client.post(
      '/record/some_random_record/edit',
      follow_redirects=True,
      data=mock_record
  )
  assert response_edit.status_code == 404

  # delete
  response = test_client.get('/record/mock_record_1/delete', follow_redirects=True)
  assert response.status_code == 200
  assert b"Deleted mock_record_1" in response.data

  # delete collection that is not exist
  response = test_client.get('/schema/mock_record_1/delete', follow_redirects=True)
  assert response.status_code == 200

Things are so far so good when testing requests, but an extra step is required for testing functions that are called from views/commands, otherwise you will get this error - working outside of *** context. It could be request, session or application. These errors occurred because the function we are testing requires to interact with a particular context - where Flask stores data about HTTP requests and other important information. If the error complains about request and session then we need to pass these contexts to the test function, this will be the same for the application context.

You have already seen one example shown above - the test_client fixture uses the application context app_context. For functions requires request and session context, use

with test_app.test_request_context():

End-to-End Testing

For e2e tests I used the Selenium Python bindings to get access to the Selenium WebDriver and the standard library Unittest as the testing framework to run e2e tests (why not Pytest? I tried but there are a lot of glitches so I turned to the official examples which use Unittest). As a minimum requirement you need to install Selenium (preferably use pip) and any one of the drivers. Since our team are working on a containerised environment, I used the official Selenium docker image to spin up a remote Chrome WebDriver, and this allows us to connect the webdriver directly via a url.

After setting up everything I have created a dedicated directory e2e under the test directory so that we can run all e2e tests just from that folder.

A very basic example of an e2e test can be testing whether the page has the correct title. Let’s create a test_page_title.py file and add the following:

import unittest
from selenium import webdriver

class TestPageTitle(unittest.TestCase):
    __options = webdriver.ChromeOptions()

    @classmethod
    def setUpClass(cls):
        cls.__options.add_argument('--headless')
        cls.driver = webdriver.Remote(
          "http://selenium-chrome:4444", options=cls.__options)

    def test_title(self):
        pageUrl = "https://google.co.uk"
        driver = self.driver
        driver.maximize_window()
        driver.get(pageUrl)
        assert "Google" in driver.title

    @classmethod
    def tearDownClass(cls):
        cls.driver.delete_all_cookies()
        cls.driver.quit()


if __name__ == "__main__":
    unittest.main()

We can now use python -m unittest discover tests/e2e to run tests in the tests/e2e directory. unittest.TestCase is the basic block of unittest and you can add as many tests in the file as you like, though you could also use unittest.TestSuite() to aggregate multiple testCase into one collection.

setupClass and tearDownClass are class methods which are run once before and after all tests, respectively. There are also setUp and tearDown methods that are similar to the class methods but they are run once for every test in the current class. In this class we only defined a single test test_title that tests the title for Google UK.

If you have multiple test cases that require the same set up or tear down methods, then you may wish to consider to inheriting common set up file from different test cases.

For example we can create a setup.py file with the same class methods as shown above:

import unittest
from selenium import webdriver

class ChromeSetup(unittest.TestCase):
    """
    A class for setup options for Chrome and do cleanup afterward.
    Each classmethod is run once per test file.
    """
    __options = webdriver.ChromeOptions()

    # A class method called before tests in an individual class are run
    @classmethod
    def setUpClass(cls):
        # https://docs.python.org/3/library/functions.html#classmethod
        # If a class method is called for a derived class, the derived class object is passed as the implied first argument.

        cls.__options.add_argument('--headless')
        cls.driver = webdriver.Remote("http://selenium-chrome:4444", options=cls.__options)

    # A class method called after tests in an individual class have run.
    @classmethod
    def tearDownClass(cls):
        cls.driver.delete_all_cookies()
        cls.driver.quit()

and re-use them in the test file:

import unittest
from setup import ChromeSetup

class TestPageTitle(ChromeSetup, unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        super(TestPageTitle, cls).setUpClass()

    def test_title(self):
        pageUrl = "https://google.co.uk"
        driver = self.driver
        driver.maximize_window()
        driver.get(pageUrl)
        assert "Google" in driver.title

    @classmethod
    def tearDownClass(cls):
        super(TestPageTitle, cls).tearDownClass()


if __name__ == "__main__":
    unittest.main()

Here we pass cls as the first argument to methods in the setup.py file.

Python vs JavaScript?

For me the transition of writing tests in JavaScript to Python was quite smooth because the fundamentals were more or less the same. However, one thing that I like about testing Python is you only need to use one assert function for any comparison, whereas in JavaScript you often need to lookup and find the right assert.someFunction or expect.someFunction. The difficulty of setting up and getting tests running are the same in Python and JavaScript and mostly depends on your proficiency with the language and frameworks/libraries you are using. The better you understand how the application is running from top to bottom, the quicker you can write test suites.

I would like to summarise the process into few steps:

  • Do some research and knowing what test runners, frameworks and tools are available for the programming language, in particular, are there already packages that integrates the test framework and frameworks you are using for the application (e.g. pytest-flask for Flask framework and pytest). Test frameworks often comes with their own test runner to help you run the tests, but that is not always the case with other tools like selenium and Chai (assertion library for JS), then you will need to select a test runner (like unittest and pytest in Python) to get tests running.
  • Choose the test toolkit and follow the official get started tutorial, write some very basic tests and get familiar with how the framework does assertions. Make sure you also take a look at what configurations provided by the framework and what integrations are available.
  • Learn about how to organise test cases for your chosen test framework and ways to define global constants, or fixtures. Additional things to look for:
    • How to run or exclude specific test
    • Hooks for running functions before or after tests
    • How to handle asynchronous codes
  • Dive deep into documentation and find out how to mock different components of your application (especially if you are building a web app). Start with tests at the lower level (unit tests), and work upwards to test components, and the system as a whole.
  • Integrate the testing into CI/CD.
  • Happy testing and don’t forget to document important information for your team!