Using Python to Tackle Big Problems

We’ll cover a few principles which can help to make your life (and those of your collaborators and readers) easier:

• Keeping your code organised
• Testing your code
• Packaging your code for the world
• Writing good documentation

To do this we’ll touch on some things which are specific to the Python programming language, and to the “ecosystem” which is built around it. All of the principles, however, can carry over to other languages (though most of the time Python does it best!).

In Python everything is an “object”, if you’ve programmed in a language like C++, Objective C, or Java in the past this might be a familiar idea to you (if you’ve programmed in Javascript this will also be quite familiar, but oddly different¹).

A code object can be thought of approximately analogously with physical objects, as they have properties, and can perform various different actions. If we wanted to represent a car in code we might want to be able to represent some things like its model, color, engine_size, as its properties, and accelerate, brake, change_gear.

There are obviously ways that you can do this without turning to a new way of coding, perhaps something like this:

 1:  enginesize = 1600
 2:  model = "Renault Captur"
 3:  color = "Midnight Sierra"
 4: 
 5:  def accelerate(enginesize, model):
 6:     """
 7:     Simulate the acceleration of the car.
 8: 
 9:     Parameters
10:     ---------- 
11:     enginesize : int
12:        The engine size of the car in cubic centimetres.
13:     model : str
14:        The model of the car.
15: 
16:     Notes
17:     -----
18:     We need to know the weight of the car, and the power of the
19:     engine to calculate the acceleration, so we need to collect
20:     these details in the arguments of the function.
21:     """
22: 
23:     <<< DO SOME PHYSICS AND MATHS >>>
24: 
25: def change_gear(current_gear, new_gear):
26:    """
27:    Calculate the effect of changing gear on the speed of the car.
28: 
29:    Parameters
30:    ---------- 
31:    current_gear : int 
32:       The number of the current gear
33:    new_gear : int
34:       The number of the gear we're switching to.
35: 
36:    Return
37:    ------
38:    gear_ratio : float
39:       The ratio between two gears
40:    """
41: 
42:    <<< DO SOME MORE MATHS >>>

This is all very well, but suppose we want to simulate two cars: we’re going to end up with lots of variables with names like enginesize_clio, and color_zafira. Objects give us a tidier way of doing this, and make it easier to recycle code.

We can design an object in Python using a class, which you can think of as the blueprint to help Python build it.

 1: class Car:
 2:    """
 3:    This class represents a car. A motorised vehicle which can move over land.
 4:    """
 5: 
 6:    def __init__(self, model, enginesize, color):
 7:       """
 8:       Set-up a car object.
 9: 
10:       Parameters
11:       ---------- 
12:       enginesize : int
13: 	 The engine size of the car in cubic centimetres.
14:       model : str
15: 	 The model of the car.
16:       color : str
17: 	 The color of the car.
18:       """
19: 
20:       self.enginesize = enginesize
21:       self.model = model
22:       self.color = color
23: 
24:    def accelerate(self, time):
25:       """
26:       Simulate the acceleration of the car.
27: 
28:       Parameters
29:       ---------- 
30:       time : float
31: 	 The time at which to calculate the acceleration
32: 
33:       Notes
34:       -----
35:       We need to know the weight of the car, and the power of the
36:       engine to calculate the acceleration, so we need to collect
37:       these details in the arguments of the function.
38:       """
39: 
40:       # Before we needed to collect information in the function
41:       # arguments, here it's replaced by ~self~, because we can
42:       # access all of the properties of the object from it.
43: 
44:       acceleration = (self.enginepower / (2*self.mass*time))**0.5
45: 
46:       return acceleration
47: 

We’ve introduced some new things here, to make the class work: the weirdest looking of these is the function inside the class called __init__, which is the ’constructor’ for the class. It sets all of the variables up in the object. When we make an object, by running something like

fiesta = Car("Ford Fiesta", 1350, "red")

the init function is what’s called.

The object now contains various bits of information about the car, so we can use:

print(fiesta.color)

to print the color of the car.

Say we want to introduce another car into our program, we can just do that by defining it in another variable:

mondeo = Car("Ford Mondeo", 1900, "blue")

So the objects keep everything together neatly. We also keep all of the logic which applies to the car with the data (but don’t duplicate code), because we can run

a = fiesta.accelerate(100)

to find a property of the object which changes.

We’ve already seen a function which can calculate the mean of a list of data: try making an object which can store a dataset, and perform some simple statistical operations on the data (try standard deviation first).

This has been a very break-neck introduction to Python objects, and we’ve not really had time to look at other neat things we can do with them:

Inheritance

You can use one class to build another (for example we could build the Car class atop a Vehicle class).

Operator overloading

You can define arithmetic operations on your classes (for example, if we add a number to one of our Dataset objects, what should happen?) You can do this be defining the method __add_() in a class. For more on this see the Python documentation.

Clearly as we keep adding new tests we’re going to want a sensible way of managing them. This is the point at which we turn to a testing framework.

Let’s start by adding a new directory to our project to keep the test files in: we’ll put all our tests here in a file called test_dataset.py.

|
|-script.py
|-dataset
| |-__init__.py
| |-dataset.py
|-tests
| |-test_dataset.py

We now need to install a new python module called nose, which we can do by running

pip install nose

Nose makes things easy by looking for files, classes, and functions which match the “regular expression”

(?:^|[\\b_\\.-])[Tt]est

(basically, anything that starts with test.

We should update the last test to use nose:

1: from dataset import Dataset
2: from nose.tools import assert_equal
3: def test_mean():
4:    data = [1,2,3]
5:    result = 2
6: 
7:    testobj = Dataset(data)
8: 
9:    assert_equal(testobj.mean(), data)

This allows us to run nosetests in the tests directory, to run all of the tests in the files in that directory.

All of the tests we’ve looked at so far are “unit tests”, which individually test the smallest units of your code (what that smallest unit is might well be up for debate, but often it means functions or object methods). There are other types of test which it can be sensible to implement:

interation tests

These perform checks that all of the parts of your code interoperate the way you expect. Integration tests may test functions which depend on the functions which are unit-tested, or they may implement simple versions of the behaviour you expect your code to be used in in functions.

regression tests

These act a bit like short-term memory for your project, and compare the outputs of a new version of your code with one from before which was in some way “accepted”. These can be useful as a way of identifying the time that a change in the code’s behaviour was introduced, but can be bad at actually finding the underlying change. They might be useful if you work in an environment where releases of your software undergo peer-review.

It’s also a good idea to include a file called “README” or some obvious variant of it in the root directory of your project which contains some basic information about the project. That might be a description, and a link to the full documentation, or it might be the full documentation, depending how complex the project is.

Other common files are

CONTRIBUTING

Instructions on how to contribute to the project

LICENSE

Details of the license the code is released under. If this isn’t present people can’t reuse your code in most jurisdictions)

You’ll probably want to include some additional documentation on your project (maybe theory, or large usage examples). The standard tool for doing this in Python is called sphinx.

We can install it by running

pip install sphinx

And then set it up in the repository with

mkdir docs
cd docs
sphinx-quickstart

Because we’ve used numpy-style docstrings we should install the numpydoc extension too:

pip install numpydoc

Sphinx will set us up with some files to get us going, and the docs directory will now look something like this:

docs
| Makefile
|-build
|-source
| |-conf.py
| |-index.rst

You’ll have some extra directories which I left out, but right now we don’t need to worry about them.

The first thing we need to do is to activate the numpydoc extension, and we can do that by editing the configuration file, source/conf.py.

You’ll need to find the place where the variable extensions is defined. It should looks something like

extensions = ['sphinx.ext.autodoc',
 'sphinx.ext.doctest',
 'sphinx.ext.coverage',
 'sphinx.ext.viewcode']

and then add in 'numpydoc', so we get

extensions = ['sphinx.ext.autodoc',
 'sphinx.ext.doctest',
 'sphinx.ext.coverage',
 'sphinx.ext.viewcode',
 'numpydoc']

We can then edit the main index.rst page of the documentation to look something like:

.. dataset documentation master file, created by

Welcome to dataset's documentation!
=====================================
Contents:

.. toctree::
   :maxdepth: 2

Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
API
====
.. toctree::
   :maxdepth: 1
   dataset

Then we can make a new file, dataset.rst for the documentation for that class:

.. _dataset_dataset:

Dataset – :mod:`dataset.dataset`
================================

.. currentmodule:: dataset.dataset
.. automodule:: dataset.dataset

Write some documentation, whatever you like, really, about the
``dataset`` package here. The documentation of the API can be found
below.

Dataset class
*************

.. autoclass:: Dataset

We want to make your documentation easy (and pleasant) to read, and sphinx handles all of this too.

In the docs folder you should have a file called Makefile. If that’s there you can simply run

make html

To make html-format documentation, which you can upload to a web-server. (You can find it in the build directory under docs).

Sphinx can produce numerous other output formats as well, including epub, should you wish to peruse your documentation on your Kindle.

If you keep your code on an online service like github you can use a service called Read the Docs, which clones your repository, builds the documentation using sphinx, and then uploads it to a webserver, free. Alternatively you can set up your own workflow, which can involve ’hooks’ in your version control system (more on that later).

setuptools is also able to roll your package up in such a way that it can be distributed (and indeed uploaded to pypi, the Python package index, which pip pulls code from). We can do this by running

python setup.py sdist

Which generates a source distribution (which will be saved as a gzipped tarball in the dist directory), which can be unpacked and installed easily. By adding upload:

python setup.py sdist upload

and following the instructions it gives you, you can upload it to pypi (but please don’t spam it with test projects!)

It’s also possible to produce binary distributions with the bdist command, but this is a case of “There be dragons”, especially with Linux, so we’ll not cover this today.