Developing python packages

From Stat DS Computing Wiki


This wiki presents a motivational example of a lot of properties of python packages and how to use them. Please see packages motivation for motivation about why you should utilize the package framework while doing research. This material captures the basics presented by py-pkgs, (recently) developed by UBC professors. As such you’ll find that python packages online may have different structures. Still, py-pkgs provides a lot of helpful tools to automate python package development. The py-pkgs project was actually started to mirror the helpful package development tools in R. Note that this wiki page does provide additional commentary but the process proposed is very similar to py-pkgs’s approach.

Foundations (what tools will be used)

In the package development pipeline that is proposed by py-pkgs we have a few very power packages to help you develop an python package as painlessly as possible.

  • poetry help build a python package
  • cookiecutter helps utilize pre-made templates (for our package’s files)
  • pytest provides clean and standardized tools for testing
  • (numpy style code) provides rubrics for documentation

Building an python package

This process is slightly different than the one we demonstrated in the Developing_R_packages wiki (we set up the structure of the package and then link to github).

Step 1: Creating a minimal source package

A python (source) package just files in a directory, formatted in a specific way.

To create the bare bones python package, we will be using the packages poetry and cookiecutter, so if you haven’t already done so, install those two packages (we’ll be using conda) by typing: 

conda install -c conda-forge poetry cookiecutter

Note that this line is taken directly from py-pkgs/02-setup, and it is possible that some day these packages can be installed wit ha simple conda install poetry cookiecutter.

Now, to actually start setting up your python package we will first use cookiecutter to copy a package structure provided by py-pkgs by running the following in our terminal: 

cookiecutter https://github.com/UBC-MDS/cookiecutter-ubc-mds.git

This will prompt you to answer a set of questions including

  1. the author name (can include caps and spaces)
  2. github username
  3. package name (can include caps and spaces)
  4. package slug name (can included underscores but no spaces)
  5. a short package description (can include caps and spaces)
  6. version number (we recommend either 0.0.1 or 0.1.0)
  7. python version 8. open source license (they list the options)
  8. if the package should include github actions (we suggest no for now, and plan to have wikis help with this later)

Note: our package for this example will be named tartanpy (an a copy of the "final" product can be found on [1].

Note: the cookiecutter template from py-pkgs (aka the one above) is still under construction - so it might look slightly different in the future.

This will create a tartanpy folder with the following items:

  1. Directory src/ (where all the function source code goes)
  2. Directory docs/ (where all the pretty looking documentation goes - auto generated)
  3. Directory tests/ (where you’ll store your test, with a placeholder test file)
  4. pyproject.toml file (a description file about your package, which includes the things you entered when you ran the cookiecutter call)
  5. LICENSE (booler plate License - don’t really need to deal with this)
  6. A set of .md files for document things like general info for viewers (README.md), info for contributors and their conduct (CONTRIBUTING.md and CONDUCT.md) and a file to include information on changes you’ve made CHANGELOG.md).
initial directory items

pyproject.toml file meta-data

There are few ways for packages to store their meta-data, and py-pkgs’s path is through the pyproject.toml. Other options include things like setup.py files. If you made a mistake when answering questions via your cookiecutter prompts you can change them by opening up the pyproject.toml file. Our example package’s pyproject.toml file looks like the image below

initialized description file

In a pyproject.toml file you’ll notice that each section is headed by a [info] heading. Notice that three of the sections in the image above have tool.poetry headers. This means that the package poetry can update these sections. As we showcase the power of this package you may find that using poetry to update a lot of these fields is more desirable than manually doing it yourself.

Step 2: Setting up a Github repo

Version control (and discussed in Stat Computing) is a very useful tool to help streamline file management and provide piece of mind. Oftentimes, you will want to have a copy of the repository not just on your personal computer but online (to help with sharing, just in case your computer dies, it’s cool, etc.). As such these steps will walk you through how to set up a github repo for your new package and how to connect that repo to a new python package. This step will mirror thoughts in py-pkgs/03-how-to-package-a-python.

  1. In Github, do the following

    • go to the top-level directory (i.e., github.com/<your user name>)

    • click on “+” at top right, and select “New repository”

    • name the repository (e.g., “tartan_python”). Note that if you’d like to share your package with others via pypi you’ll need to make your package name unique!

    • provide a short description of the repository (don’t leave completely blank!)

    • keep the repository public (as students you have access to free private repos https://education.github.com/pack, but for purposes of this example we will keep the repo public)

    • do not initalize with a README, .gitignore, or license (It even says “Skip this step if you’re importing an exisiting repo”!!!)

    • click on “Create Repository”

  2. …then in your terminal, do the following

    • First, initialize your repository with 
      git init

      then add all cookiecutter files and create your initial commit with bash git add . git commit -m "initial package setup"

    • setup connection to github by adding a a remote, your main branch and pushing your first commit with 
      git remote add origin https://github.com/benjaminleroy/tartanpy.git
git branch -M main
git push -u origin main

      Note that github provides you with walk throughs that mirror what was recommended above and showcases 2 approaches if you want to use the command line (https and ssh) as well as how you might do it with github desktop. So helpful - thanks github!

Step 3: Writing a function

All of your python code needs to go in .py files in the src/tartanpy/ directory. Suppose our first set of functions deal with communicating to the user - then we might make a communicate.py file in src/tartanpy/ folder.
Inside this file, let’s image we add the following function 

def call_scotty_demo(your_name):

    name_length = len(your_name)

    # say Hiiiiii
    hi_string = "\n" +\
        "-" * (name_length + 8) + "\n" +\
        "| Hi " + your_name + "! | \n" +\
        "|   " + "-" * (name_length + 4) + "\n" +\
        "| /\n"

    scotty_string = ""+\
        " |/  |\\_/|          \n" +\
        " |   |q p|   /}      \n" +\
        " |   ( 0 )\"\"\"\\   \n" +\
        "  \\  |\"-\"`    |    \n" +\
        "     || /=\\\\  |     \n" +\
        "     \"'\"  '\"\"\"'"

    print(hi_string + scotty_string)

To get access to this function we can use poetry to locally install our package with the bash command below (that needs to be run in the top folder of our package) 

poetry install

Note that without poetry this could be done with a pip install -e . in the same directory and this would actually not require running this line of code every time we update the package. On the other hand - we’ll show the versatility of poetry in the rest of the article.

To utilize our function we can then open up a interactive python instance (e.g. ipython in your terminal) and then run 

In [1]: from tartanpy import communicate

In [2]: communicate.call_scotty_demo("Andrew Carnegie")

which will return: 

-----------------------
| Hi Andrew Carnegie! |
|   -------------------
| /
 |/  |\_/|
 |   |q p|   /}
 |   ( 0 )"""\
  \  |"-"`    |
     || /=\\  |
     "'"  '"""'

Note that poetry install can be called over and over if you keep updating your function files and will re-load the newest verion of all your functions.

Additional notes:

Understanding __init__.py

For python packages you’ll always have an __init__.py file in this directory. We currently have to import the communicate file from tartanpy in order to get access to our amazing function. To make this function come directly with the package, in the __init__.py file we can write 

from .communicate import call_scotty_demo

Note the dot in .communicate says “in this same directory as this current file”, and we don’t have to include the .py file link at the end. Additionally This approach only imports call_scotty_demo to the global package, we could use ... import * to import all functions (but this approach is smart if we had helper functions we didn’t really need to share with the end user).*

With the above change, we can then do (the poetry part still needs to be at the top directory): 

poetry install
ipython

and then in the ipython shell: 

In [1]: import tartanpy

In [2]: tartanpy.call_scotty_demo("Andrew Mellon")

which will return: 

---------------------
| Hi Andrew Mellon! |
|   -----------------
| /
 |/  |\_/|
 |   |q p|   /}
 |   ( 0 )"""\
  \  |"-"`    |
     || /=\\  |
     "'"  '"""'
Using external packages

[You'll want to read the next section as well for a full understanding of dealing with external packages.]

Suppose I wished to truncate really long names I’m greeting but also give the user a warning that I’m going to do so. I’d need to use the warnings package. To use this in our package I’d need to change our communicate.py file to 

import warnings

def call_scotty_demo(your_name):
    if len(your_name) > 30:
        warnings.warn("your name is beyond 30 characters, "+\
                        "has been truncated")
        your_name = your_name[:30]

    name_length = len(your_name)

    # say Hiiiiii
    hi_string = "\n" +\
        "-" * (name_length + 8) + "\n" +\
        "| Hi " + your_name + "! | \n" +\
        "|   " + "-" * (name_length + 4) + "\n" +\
        "| /\n"

    scotty_string = ""+\
        " |/  |\\_/|          \n" +\
        " |   |q p|   /}      \n" +\
        " |   ( 0 )\"\"\"\\   \n" +\
        "  \\  |\"-\"`    |    \n" +\
        "     || /=\\\\  |     \n" +\
        "     \"'\"  '\"\"\"'"

    print(hi_string + scotty_string)

Notice that we have to impot the warnings package at the top of our function file.

Now we have the additional funcitonality (after reloading with poetry) 

In [1]: import tartanpy

In [2]: tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")

which returns 

.../tartanpy/src/tartanpy/communicate.py:6: UserWarning: your name is beyond 30 characters, has been truncated
  warnings.warn("your name is beyond 30 characters, "+\

--------------------------------------
| Hi Oski Bear, your a wonderful, b! |
|   ----------------------------------
| /
 |/  |\_/|
 |   |q p|   /}
 |   ( 0 )"""\
  \  |"-"`    |
     || /=\\  |
     "'"  '"""'

Using external packages: declaring dependences & Using local functions in other files

  1. Declaring dependencies: The warnings module is actually a built in python package, but suppose we also needed to use numpy in our package. For example, suppose we created another .py file called utils.py, in it we had

    import numpy as np

def _top_and_bottom(name_length):
    length_vec = np.array([8,4]) + name_length
    return(length_vec)

    The first thing we’d need to do (so the code works for others) is to tell the package we require the installation of numpy for anyone using our package. We do so by running

    poetry add numpy

    This will update our pyproject.toml and add asssociated dependences (there are none for numpy), and will put that note under [tool.poetry.dependencies].

    Note: as of 8/20/21 this also required restricting my python version (in the pyproject.toml to python = "?=3.8,<3.11")

  2. Internal Functions: To use our interal function _top_and_bottom we might change the communicate.py file to:

    import warnings
from .utils import _top_and_bottom

def call_scotty_demo(your_name):
    # import warnings
    if len(your_name) > 30:
        warnings.warn("your name is beyond 30 characters, "+\
                        "has been truncated")
        your_name = your_name[:30]

    name_length = len(your_name)

    tb_info = _top_and_bottom(name_length)

    # say Hiiiiii
    hi_string = "\n" +\
        "-" * tb_info[0] + "\n" +\
        "| Hi " + your_name + "! | \n" +\
        "|   " + "-" * tb_info[1] + "\n" +\
        "| /\n"

    ...

    Notice that to give call_scotty_demo and all functions in the communicate.py file access to _top_and_bottom we had to import it using from .utils import _top_and_bottom. Additionally, it’s best practice to use add a _ at the beginning of a function name if you plan to keep it internal.

Step 4: Documenting your functions

Python documentation is simplier than R documentation as when you look at the documentation of a python function you just get show a docstring at the head of function. In this section we’ll discuss a bit about the extra support poetry can do so that you can generate a localized website for your documentation (that you could upload to [https:://readthedocs.org readthedocs] later).

For documentation to be easily publishable at a later date we recommend that your docstrings follow the numpydoc format (more details can be found at numpydoc.readthedocs). Using this guideline we can document call_scotty_demo as follows: 

def call_scotty_demo(your_name):
    """
    Greet Scotty, the Scottie Dog

    Scotty, the Scottie Dog greets your by name!

    Parameters
    ----------
    your_name: str
        string of your name (max length 30 characters)

    Returns
    -------
    None
        Though scotty appears on your screen and says Hi to you.

    Examples
    --------
    >>> import tartanpy
    >>> tartanpy.call_scotty_demo("Andrew Carnegie")

    -----------------------
    | Hi Andrew Carnegie! |
    |   -------------------
    | /
    |/  |\_/|
    |   |q p|   /}
    |   ( 0 )\"\"\"\\
     \\  |\"-\"`    |
        || /=\\\\  |
        \"'\"  '\"\"\"'
    """
    ...

The basic structure of the documention is decently straight forward with the title / description of your function first, followed by (1) Parameters header with the parameter name, the type, and the description in the format above (2) Returns header followed by similar structure as Parameters. Adding (3) examples with standard python lines can be helpful as well.

After reloading the package you could see that 

In [1]: import tartanpy

In [2]: ?tartanpy.call_scotty_demo
Signature: tartanpy.call_scotty_demo(your_name)
Docstring:
Greet Scotty, the Scottie Dog

Scotty, the Scottie Dog greets your by name!

Parameters
----------
your_name: str
    string of your name (max length 30 characters)

Returns
-------
None
    Though scotty appears on your screen and says Hi to you.

Examples
--------
>>> import tartanpy
>>> tartanpy.call_scotty_demo("Andrew Carnegie")
-----------------------
| Hi Andrew Carnegie! |
|   -------------------
| /
|/  |\_/|
|   |q p|   /}
|   ( 0 )"""\
 \  |"-"`    |
    || /=\\  |
    "'"  '"""'
File:      ..../tartanpy/src/tartanpy/communicate.py
Type:      function

Towards "readthedocs"-style html documentation

py-pkgs and poetry also provide a quick way to transform your documentation into a local html website so you can quickly look through your documentation or load it to readthedocs later, with information on how to here: py-pkgs/06-documentation.

Generally speaking you have to

  1. install some additional packages to the development side of your package with 
    poetry add --dev sphinx nbsphinx ipykernel
  2. and then go to your docs folder and make the website locally (the --dev parameter makes these packages not tracked for those just wanted to use your package, but required as you continue to develop the package locally). 
    cd docs
poetry run make html

    and this will create a local webpage (just open docs/index.html) that contains your documentation. If you find the docs website useful maybe bookmark the local file?

Step 5: Testing

Testing is critical to making sure your functions do what you expect. There are few ways to write unit tests in python (e.g. unittest, pytest and nose). We recommend using pytest as it requires the least amount of overhead and provides the most details write off the bat.

To use pytest we first to tell poetry that we’ll use pytest in our development with 

poetry add --dev pytest

What’s the philosophy behind pytest

  1. Assertions: pytest’s foundation is defined around checking basic assertion statements like 
    In [1]: assert 1 == 1, "a true statement shouldn't have errored..."
   ...: assert .3 - .2 == .1, "but sometimes floating points are bad..."

    that return: 

    ---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-3-0daf5f7e9b9c> in <module>
      1 assert 1 == 1, "a true statement shouldn't have errored..."
----> 2 assert .3 - .2 == .1, "but sometimes floating points are bad..."

AssertionError: but sometimes floating points are bad...

    as the second line actual fails (due to some floating point silliness). This simple structure can be combined with standard logical and and or as well as other tools like numpy logic (.e.g np.all()). Notice that following a logical statement with a string nicely associated a statement to be returned if the assertion is not true.

  2. Grouping Assertions: We can imagine wishing to combine a set of assertions, and we can do this easily with a function wrapper. For example 
    import tartanpy
import pytest # only for warning

def test_call_scotty_demo():
    """
    test the basics of call_scotty_demo
    """

    assert tartanpy.call_scotty_demo("Oski Bear") is None, \
        "the output of call_scotty_demo should be None"

    # test warning
    with pytest.warns(UserWarning) as record:
        tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")

    assert len(record) > 0, \
        "expect a warning for name_string longer than 30 for call_scotty_demo"

    assert .3 - .2 == .1, "but sometimes floating points are bad..."
  3. To actually leverage pytest we can put A set of grouped tests in single file. When building a package your functions will likelyl live in different .py files in the src/<packagename> folder. To mirror this structure we put of our tests associated with a single .py file into 1 test file. These files sit in the tests/ folder and must start with test. In these larger files (relative to group functions) need to make sure our functions are brought in, for example at the head we need to put import tartanpy as see in the code block above. To run these tests you can either do 
    pytest

    or (making sure you have all the correct packages) 

    poetry run pytest

    Note: we actually have some printouts in call_scotty_demo, which are not desirable, so we actually follow a suggestion from stackoverflow to suppress the printing (doesn’t mess with the testing), and is seen in the code block below: 

    import tartanpy
import pytest # only for warning

def test_call_scotty_demo():
    """
    test the basics of call_scotty_demo
    """
    with open(os.devnull, "w") as f, contextlib.redirect_stdout(f):
        assert tartanpy.call_scotty_demo("Oski Bear") is None, \
            "the output of call_scotty_demo should be None"

        # test warning
        with pytest.warns(UserWarning) as record:
            tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")

        assert len(record) > 0, \
            "expect a warning for name_string longer than 30 for call_scotty_demo"



        assert .3 - .2 == .1, "but sometimes floating points are bad..."

    If we remove the silly floating point error assession we get the following:

    output of pytest

    And if there are errors pytest tries to be very clear about where in your tests you had an error.

More comments on testing

  1. Hidden and/or not exported functions In our example we have a hidden function _top_and_bottom. Different thoughts exist on whether or not one should test hidden functions, but to do so we have to be a bit tricker since the function isn’t loaded by the package. We can still test it by using the full path to the function, that is: 
    import tartanpy
import numpy as np

def test__top_and_bottom():
    """
    test for hidden function _top_and_bottom
    """
    option4 = tartanpy.utils._top_and_bottom(len("test"))
    option17 = tartanpy.utils._top_and_bottom(len("test_a_longer_one"))

    assert np.all(option17 - option4 == 17-4), \
        "expect _top_and_bottom values to only differ by length of input"

    assert np.all(option17 - 17 == np.array([8,4])), \
        "expect that _top_and_bottom adds 8 and 4 to length of input"

    Additionally, since we started the name off with a _, we could also find that if we did 

    from tartanpy.utils import *

    which should import everything from the util.py file we won’t actually have access to it, whereas 

    from tartanpy.communicate import *

    would get access to call_scotty_demo even if it wasn’t in the __init__.py file.

  2. Numerical instablity We included an example where floating point differnces made .3-.2 not equal to .1. To avoid such errors we recommend using tools like
    • np.allclose (which allows user to provide relative and absolute tolerance between vectors)
    • or do something like np.abs(a-b)<1e-08 or np.abs(a-b)/np.abs(b) < 1e-08

Step 6. Sharing / using on other computers

You could publish your package to pypi (documented here: py-pkg/07-releasing-versioning) but if you’re still actively working on a developer version another easy way is to make sure you have a clean copy on github, have the user clone the repo and then run either (when in the top directory of the cloned repo) 

pip install .

or 

pip install -e .

where the first installs the package 1 time, and the second actually installs the package and updates the package if that user was to do more pulls / updates the the package.


Step 7. And beyond!

In the above example, we still didn’t cover a lot of things, including:

  • publishing the package
  • publishing the docs
  • including data
  • CI/CD (which will be provided in another wiki)

See py-pkgs for some good starting blocks for most of these.

Retrieved from "https://info.stat.cmu.edu/index.php?title=Developing_python_packages&oldid=809"