View source for Developing python packages


This wiki presents a motivational example of a lot of properties of python packages and how to use them. Please see [[packages motivation]] for motivation about why you should utilize the package framework while doing research. This material captures the basics presented by [https://py-pkgs.org/ py-pkgs], (recently) developed by UBC professors. As such you’ll find that python packages online may have different structures. Still, <code>py-pkgs</code> provides a lot of helpful tools to automate python package development. The <code>py-pkgs</code> project was actually started to mirror the helpful [[Developing_R_packages|package development tools in <code>R</code>]]. Note that this wiki page does provide additional commentary but the process proposed is very similar to <code>py-pkgs</code>’s approach.

== Foundations (what tools will be used) ==

In the package development pipeline that is proposed by <code>py-pkgs</code> we have a few very power packages to help you develop an <code>python</code> package as painlessly as possible.

* <code>poetry</code> help build a python package
* <code>cookiecutter</code> helps utilize pre-made templates (for our package’s files)
* <code>pytest</code> provides clean and standardized tools for testing
* (numpy style code) provides rubrics for documentation

== Building an python package ==

This process is slightly different than the one we demonstrated in the [[Developing_R_packages]] wiki (we set up the structure of the package and then link to github).

=== Step 1: Creating a minimal source package ===

A python (''source'') package just files in a directory, formatted in a specific way.

To create the bare bones python package, we will be using the packages <code>poetry</code> and <code>cookiecutter</code>, so if you haven’t already done so, install those two packages (we’ll be using <code>conda</code>) by typing:

<syntaxhighlight lang="bash">conda install -c conda-forge poetry cookiecutter</syntaxhighlight>
''Note that this line is taken directly from [https://py-pkgs.org/02-setup py-pkgs/02-setup], and it is possible that some day these packages can be installed wit ha simple <code>conda install poetry cookiecutter</code>.''

'''Now''', to actually start setting up your python package we will first use <code>cookiecutter</code> to copy a package structure provided by <code>py-pkgs</code> by running the following in our terminal:

<syntaxhighlight lang="bash">cookiecutter https://github.com/UBC-MDS/cookiecutter-ubc-mds.git</syntaxhighlight>
This will prompt you to answer a set of questions including 
<ol>
<li> the author name (can include caps and spaces) </li>
<li> github username </li>
<li> package name (can include caps and spaces) </li>
<li> package slug name (can included underscores but no spaces) </li>
<li> a short package description (can include caps and spaces) </li>
<li> version number (we recommend either 0.0.1 or 0.1.0) </li>
<li> python version 8. open source license (they list the options) </li>
<li> if the package should include github actions (we suggest no for now, and plan to have wikis help with this later) </li>
</ol>

''Note: our package for this example will be named <code>tartanpy</code>'' (an a copy of the "final" product can be found on [https://github.com/benjaminleroy/tartanpy|github].

''Note: the cookiecutter template from <code>py-pkg</code>s (aka the one above) is still under construction - so it might look slightly different in the future.''

This will create a <code>tartanpy</code> folder with the following items:

# Directory <code>src/</code> (where all the function source code goes)
# Directory <code>docs/</code> (where all the pretty looking documentation goes '''- auto generated''')
# Directory <code>tests/</code> (where you’ll store your test, with a placeholder test file)
# <code>pyproject.toml</code> file (a description file about your package, which includes the things you entered when you ran the <code>cookiecutter</code> call)
# <code>LICENSE</code> (booler plate License - don’t really need to deal with this)
# A set of <code>.md</code> files for document things like general info for viewers (<code>README.md</code>), info for contributors and their conduct (<code>CONTRIBUTING.md</code> and <code>CONDUCT.md</code>) and a file to include information on changes you’ve made <code>CHANGELOG.md</code>).

[[File:Tartanpy_folder.png|frameless|600px|alt=initial directory items|initial directory items]]

==== pyproject.toml file meta-data ====

There are few ways for packages to store their meta-data, and <code>py-pkgs</code>’s path is through the <code>pyproject.toml</code>. Other options include things like <code>setup.py</code> files. If you made a mistake when answering questions via your cookiecutter prompts you can change them by opening up the <code>pyproject.toml</code> file. Our example package’s <code>pyproject.toml</code> file looks like the image below

[[File:Tartanpy_pyproject_description.png|frameless|600px|alt=initialized description file|initialized description file]]

In a <code>pyproject.toml</code> file you’ll notice that each section is headed by a <code>[info]</code> heading. Notice that three of the sections in the image above have <code>tool.poetry</code> headers. This means that the package <code>poetry</code> can update these sections. As we showcase the power of this package you may find that using poetry to update a lot of these fields is more desirable than manually doing it yourself.

=== Step 2: Setting up a Github repo ===

Version control (and discussed in Stat Computing) is a very useful tool to help streamline file management and provide piece of mind. Oftentimes, you will want to have a copy of the repository not just on your personal computer but online (to help with sharing, just in case your computer dies, it’s cool, etc.). As such these steps will walk you through how to set up a github repo for your new package and how to connect that repo to a new python package. This step will mirror thoughts in [https://py-pkgs.org/03-how-to-package-a-python#set-up-remote-version-control py-pkgs/03-how-to-package-a-python].

<ol style="list-style-type: decimal;">
<li><p>'''In <code>Github</code>''', do the following</p>
<ul>
<li><p>go to the top-level directory (i.e., <code>github.com/<your user name></code>)</p></li>
<li><p>click on “+” at top right, and select “New repository”</p></li>
<li><p>name the repository (e.g., “tartan_python”). Note that if you’d like to share your package with others via pypi you’ll need to make your package name unique!</p></li>
<li><p>provide a short description of the repository (don’t leave completely blank!)</p></li>
<li><p>keep the repository public (as students you have access to free private repos https://education.github.com/pack, but for purposes of this example we will keep the repo public)</p></li>
<li><p>'''do not''' initalize with a README, .gitignore, or license (It even says “Skip this step if you’re importing an exisiting repo”!!!)</p></li>
<li><p>click on “Create Repository”</p></li></ul>
</li>
<li><p>'''…then in your terminal, do the following'''</p>
<ul>
<li>First, initialize your repository with

<syntaxhighlight lang="bash">git init</syntaxhighlight>
<p>then add all <code>cookiecutter</code> files and create your initial commit with <code>bash  git add .  git commit -m "initial package setup"</code></p>
</li>
<li>setup connection to github by adding a a remote, your main branch and pushing your first commit with

<syntaxhighlight lang="bash">git remote add origin https://github.com/benjaminleroy/tartanpy.git
git branch -M main
git push -u origin main</syntaxhighlight>
<p>''Note that github provides you with walk throughs that mirror what was recommended above and showcases 2 approaches if you want to use the command line (<code>https</code> and <code>ssh</code>) as well as how you might do it with github desktop. So helpful - thanks github!''</p></li></ul>

</li></ol>

=== Step 3: Writing a function ===

All of your python code needs to go in <code>.py</code> files in the <code>src/tartanpy/</code> directory. Suppose our first set of functions deal with communicating to the user - then we might make a <code>communicate.py</code> file in <code>src/tartanpy/</code> folder. <br> Inside this file, let’s image we add the following function

<syntaxhighlight lang="python">def call_scotty_demo(your_name):

    name_length = len(your_name)

    # say Hiiiiii
    hi_string = "\n" +\
        "-" * (name_length + 8) + "\n" +\
        "| Hi " + your_name + "! | \n" +\
        "|   " + "-" * (name_length + 4) + "\n" +\
        "| /\n"

    scotty_string = ""+\
        " |/  |\\_/|          \n" +\
        " |   |q p|   /}      \n" +\
        " |   ( 0 )\"\"\"\\   \n" +\
        "  \\  |\"-\"`    |    \n" +\
        "     || /=\\\\  |     \n" +\
        "     \"'\"  '\"\"\"'"

    print(hi_string + scotty_string)
</syntaxhighlight>
To get access to this function we can use <code>poetry</code> to locally install our package with the bash command below (that needs to be run in the top folder of our package)

<pre>poetry install</pre>
''Note that without poetry this could be done with a <code>pip install -e .</code> in the same directory and this would actually not require running this line of code every time we update the package. On the other hand - we’ll show the versatility of <code>poetry</code> in the rest of the article.''

To utilize our function we can then open up a interactive python instance (e.g. <code>ipython</code> in your terminal) and then run

<syntaxhighlight lang="python">In [1]: from tartanpy import communicate

In [2]: communicate.call_scotty_demo("Andrew Carnegie")</syntaxhighlight>
which will return:

<syntaxhighlight lang="bash">-----------------------
| Hi Andrew Carnegie! |
|   -------------------
| /
 |/  |\_/|
 |   |q p|   /}
 |   ( 0 )"""\
  \  |"-"`    |
     || /=\\  |
     "'"  '"""'</syntaxhighlight>
Note that <code>poetry install</code> can be called over and over if you keep updating your function files and will re-load the newest verion of all your functions.

==== Additional notes: ====

===== Understanding <code>__init__.py</code> =====

For <code>python</code> packages you’ll always have an <code>__init__.py</code> file in this directory. We currently have to import the <code>communicate</code> file from <code>tartanpy</code> in order to get access to our amazing function. To make this function come directly with the package, in the <code>__init__.py</code> file we can write

<syntaxhighlight lang="python">from .communicate import call_scotty_demo</syntaxhighlight>
''Note the dot in <code>.communicate</code> says “in this same directory as this current file”, and we don’t have to include the <code>.py</code> file link at the end.'' Additionally This approach '''only''' imports call_scotty_demo to the global package, we could use <code>... import *</code> to import all functions (but this approach is smart if we had helper functions we didn’t really need to share with the end user).*

With the above change, we can then do (the <code>poetry</code> part still needs to be at the top directory):

<syntaxhighlight lang="bash">poetry install
ipython</syntaxhighlight>
and then in the ipython shell:

<syntaxhighlight lang="python">In [1]: import tartanpy

In [2]: tartanpy.call_scotty_demo("Andrew Mellon")</syntaxhighlight>
which will return:

<syntaxhighlight lang="bash">---------------------
| Hi Andrew Mellon! |
|   -----------------
| /
 |/  |\_/|
 |   |q p|   /}
 |   ( 0 )"""\
  \  |"-"`    |
     || /=\\  |
     "'"  '"""'</syntaxhighlight>
===== Using external packages =====

[You'll want to read the next section as well for a full understanding of dealing with external packages.]

Suppose I wished to truncate really long names I’m greeting but also give the user a warning that I’m going to do so. I’d need to use the <code>warnings</code> package. To use this in our package I’d need to change our <code>communicate.py</code> file to

<syntaxhighlight lang="python">import warnings

def call_scotty_demo(your_name):
    if len(your_name) > 30:
        warnings.warn("your name is beyond 30 characters, "+\
                        "has been truncated")
        your_name = your_name[:30]

    name_length = len(your_name)

    # say Hiiiiii
    hi_string = "\n" +\
        "-" * (name_length + 8) + "\n" +\
        "| Hi " + your_name + "! | \n" +\
        "|   " + "-" * (name_length + 4) + "\n" +\
        "| /\n"

    scotty_string = ""+\
        " |/  |\\_/|          \n" +\
        " |   |q p|   /}      \n" +\
        " |   ( 0 )\"\"\"\\   \n" +\
        "  \\  |\"-\"`    |    \n" +\
        "     || /=\\\\  |     \n" +\
        "     \"'\"  '\"\"\"'"

    print(hi_string + scotty_string)</syntaxhighlight>
Notice that we have to impot the <code>warnings</code> package at the top of our function file.

Now we have the additional funcitonality (after reloading with <code>poetry</code>)

<syntaxhighlight lang="python">In [1]: import tartanpy

In [2]: tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")</syntaxhighlight>
which returns

<syntaxhighlight lang="python">.../tartanpy/src/tartanpy/communicate.py:6: UserWarning: your name is beyond 30 characters, has been truncated
  warnings.warn("your name is beyond 30 characters, "+\

--------------------------------------
| Hi Oski Bear, your a wonderful, b! |
|   ----------------------------------
| /
 |/  |\_/|
 |   |q p|   /}
 |   ( 0 )"""\
  \  |"-"`    |
     || /=\\  |
     "'"  '"""'</syntaxhighlight>
==== Using external packages: declaring dependences & Using local functions in other files ====

<ol style="list-style-type: decimal;">
<li><p>'''Declaring dependencies''': The <code>warnings</code> module is actually a built in python package, but suppose we also needed to use <code>numpy</code> in our package. For example, suppose we created another <code>.py</code> file called <code>utils.py</code>, in it we had</p>
<syntaxhighlight lang="python">import numpy as np

def _top_and_bottom(name_length):
    length_vec = np.array([8,4]) + name_length
    return(length_vec)</syntaxhighlight>
<p>The first thing we’d need to do (so the code works for others) is to tell the package we require the installation of <code>numpy</code> for anyone using our package. We do so by running</p>
<syntaxhighlight lang="bash">poetry add numpy</syntaxhighlight>
<p>This will update our <code>pyproject.toml</code> and add asssociated dependences (there are none for <code>numpy</code>), and will put that note under <code>[tool.poetry.dependencies]</code>.</p>
<p>''Note: as of 8/20/21 this also required restricting my python version (in the <code>pyproject.toml</code> to <code>python = "?=3.8,<3.11"</code>)</p></li>
<li><p>'''Internal Functions''': To use our interal function <code>_top_and_bottom</code> we might change the <code>communicate.py</code> file to:</p>
<syntaxhighlight lang="python">import warnings
from .utils import _top_and_bottom

def call_scotty_demo(your_name):
    # import warnings
    if len(your_name) > 30:
        warnings.warn("your name is beyond 30 characters, "+\
                        "has been truncated")
        your_name = your_name[:30]

    name_length = len(your_name)

    tb_info = _top_and_bottom(name_length)

    # say Hiiiiii
    hi_string = "\n" +\
        "-" * tb_info[0] + "\n" +\
        "| Hi " + your_name + "! | \n" +\
        "|   " + "-" * tb_info[1] + "\n" +\
        "| /\n"

    ...</syntaxhighlight>
<p>Notice that to give <code>call_scotty_demo</code> and all functions in the <code>communicate.py</code> file access to <code>_top_and_bottom</code> we had to import it using <code>from .utils import _top_and_bottom</code>. Additionally, it’s best practice to use add a <code>_</code> at the beginning of a function name if you plan to keep it internal.</p></li></ol>

=== Step 4: Documenting your functions ===

Python documentation is simplier than R documentation as when you look at the documentation of a python function you just get show a docstring at the head of function. In this section we’ll discuss a bit about the extra support poetry can do so that you can generate a localized website for your documentation (that you could upload to [https:://readthedocs.org readthedocs] later).

For documentation to be easily publishable at a later date we recommend that your docstrings follow the <code>numpydoc</code> format (more details can be found at [https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard numpydoc.readthedocs]). Using this guideline we can document <code>call_scotty_demo</code> as follows:

<syntaxhighlight lang="python">def call_scotty_demo(your_name):
    """
    Greet Scotty, the Scottie Dog

    Scotty, the Scottie Dog greets your by name!

    Parameters
    ----------
    your_name: str
        string of your name (max length 30 characters)

    Returns
    -------
    None
        Though scotty appears on your screen and says Hi to you.

    Examples
    --------
    >>> import tartanpy
    >>> tartanpy.call_scotty_demo("Andrew Carnegie")

    -----------------------
    | Hi Andrew Carnegie! |
    |   -------------------
    | /
    |/  |\_/|
    |   |q p|   /}
    |   ( 0 )\"\"\"\\
     \\  |\"-\"`    |
        || /=\\\\  |
        \"'\"  '\"\"\"'
    """
    ...</syntaxhighlight>
The basic structure of the documention is decently straight forward with the title / description of your function first, followed by (1) Parameters header with the parameter name, the type, and the description in the format above (2) Returns header followed by similar structure as Parameters. Adding (3) examples with standard python lines can be helpful as well.

After reloading the package you could see that

<syntaxhighlight lang="python">In [1]: import tartanpy

In [2]: ?tartanpy.call_scotty_demo
Signature: tartanpy.call_scotty_demo(your_name)
Docstring:
Greet Scotty, the Scottie Dog

Scotty, the Scottie Dog greets your by name!

Parameters
----------
your_name: str
    string of your name (max length 30 characters)

Returns
-------
None
    Though scotty appears on your screen and says Hi to you.

Examples
--------
>>> import tartanpy
>>> tartanpy.call_scotty_demo("Andrew Carnegie")
-----------------------
| Hi Andrew Carnegie! |
|   -------------------
| /
|/  |\_/|
|   |q p|   /}
|   ( 0 )"""\
 \  |"-"`    |
    || /=\\  |
    "'"  '"""'
File:      ..../tartanpy/src/tartanpy/communicate.py
Type:      function</syntaxhighlight>
==== Towards "readthedocs"-style html documentation ====

py-pkgs and poetry also provide a quick way to transform your documentation into a local html website so you can quickly look through your documentation or load it to readthedocs later, with information on how to here: [https://py-pkgs.org/06-documentation py-pkgs/06-documentation].

Generally speaking you have to

<ol>
<li> install some additional packages to the development side of your package with

<syntaxhighlight lang="bash">poetry add --dev sphinx nbsphinx ipykernel</syntaxhighlight> </li>

<li>and then go to your docs folder and make the website locally (the <code>--dev</code> parameter makes these packages not tracked for those just wanted to use your package, but required as you continue to develop the package locally).

<syntaxhighlight lang="bash">cd docs
poetry run make html</syntaxhighlight>
and this will create a local webpage (just open <code>docs/index.html</code>) that contains your documentation. If you find the docs website useful maybe bookmark the local file?

</li></ol>

=== Step 5: Testing ===

Testing is critical to making sure your functions do what you expect. There are few ways to write unit tests in python (e.g. <code>unittest</code>, <code>pytest</code> and <code>nose</code>). We recommend using <code>pytest</code> as it requires the least amount of overhead and provides the most details write off the bat.

To use <code>pytest</code> we first to tell <code>poetry</code> that we’ll use <code>pytest</code> in our development with

<syntaxhighlight lang="bash">poetry add --dev pytest</syntaxhighlight>
==== What’s the philosophy behind <code>pytest</code> ====

<ol>
<li>'''Assertions''': <code>pytest</code>’s foundation is defined around checking basic assertion statements like

<syntaxhighlight lang="python">In [1]: assert 1 == 1, "a true statement shouldn't have errored..."
   ...: assert .3 - .2 == .1, "but sometimes floating points are bad..."</syntaxhighlight>
that return:

<syntaxhighlight lang="python">---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-3-0daf5f7e9b9c> in <module>
      1 assert 1 == 1, "a true statement shouldn't have errored..."
----> 2 assert .3 - .2 == .1, "but sometimes floating points are bad..."

AssertionError: but sometimes floating points are bad...</syntaxhighlight>
as the second line actual fails (due to some floating point silliness). This simple structure can be combined with standard logical <code>and</code> and <code>or</code> as well as other tools like <code>numpy</code> logic (.e.g <code>np.all()</code>). Notice that following a logical statement with a string nicely associated a statement to be returned if the assertion is not true.
</li>
<li>'''Grouping Assertions''': We can imagine wishing to combine a set of assertions, and we can do this easily with a function wrapper. For example

<syntaxhighlight lang="python">import tartanpy
import pytest # only for warning

def test_call_scotty_demo():
    """
    test the basics of call_scotty_demo
    """

    assert tartanpy.call_scotty_demo("Oski Bear") is None, \
        "the output of call_scotty_demo should be None"

    # test warning
    with pytest.warns(UserWarning) as record:
        tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")

    assert len(record) > 0, \
        "expect a warning for name_string longer than 30 for call_scotty_demo"

    assert .3 - .2 == .1, "but sometimes floating points are bad..."</syntaxhighlight>
</li>
<li>To actually leverage <code>pytest</code> we can put '''A set of grouped tests''' in single file. When building a package your functions will likelyl live in different <code>.py</code> files in the <code>src/<packagename></code> folder. To mirror this structure we put of our tests associated with a single <code>.py</code> file into 1 test file. These files sit in the <code>tests/</code> folder and must start with <code>test</code>.

In these larger files (relative to group functions) need to make sure our functions are brought in, for example at the head we need to put <code>import tartanpy</code> as see in the code block above.

To run these tests you can either do

<syntaxhighlight lang="bash">pytest</syntaxhighlight>
or (making sure you have all the correct packages)

<syntaxhighlight lang="bash">poetry run pytest</syntaxhighlight>
''Note: we actually have some printouts in <code>call_scotty_demo</code>, which are not desirable, so we actually follow a suggestion from [https://stackoverflow.com/questions/8391411/how-to-block-calls-to-print stackoverflow] to suppress the printing (doesn’t mess with the testing), and is seen in the code block below:''

<syntaxhighlight lang="python">import tartanpy
import pytest # only for warning

def test_call_scotty_demo():
    """
    test the basics of call_scotty_demo
    """
    with open(os.devnull, "w") as f, contextlib.redirect_stdout(f):
        assert tartanpy.call_scotty_demo("Oski Bear") is None, \
            "the output of call_scotty_demo should be None"

        # test warning
        with pytest.warns(UserWarning) as record:
            tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")

        assert len(record) > 0, \
            "expect a warning for name_string longer than 30 for call_scotty_demo"



        assert .3 - .2 == .1, "but sometimes floating points are bad..."</syntaxhighlight>
If we remove the silly floating point error assession we get the following:

[[File:Pytest_example.png|frameless|600px|alt=output of pytest|output of pytest]]

And if there are errors <code>pytest</code> tries to be very clear about where in your tests you had an error.
</li></ol>

==== More comments on testing ====

<ol>
<li>'''Hidden and/or not exported functions'''

In our example we have a hidden function <code>_top_and_bottom</code>. Different thoughts exist on whether or not one should test hidden functions, but to do so we have to be a bit tricker since the function isn’t loaded by the package. We can still test it by using the full path to the function, that is:

<syntaxhighlight lang="python">import tartanpy
import numpy as np

def test__top_and_bottom():
    """
    test for hidden function _top_and_bottom
    """
    option4 = tartanpy.utils._top_and_bottom(len("test"))
    option17 = tartanpy.utils._top_and_bottom(len("test_a_longer_one"))

    assert np.all(option17 - option4 == 17-4), \
        "expect _top_and_bottom values to only differ by length of input"

    assert np.all(option17 - 17 == np.array([8,4])), \
        "expect that _top_and_bottom adds 8 and 4 to length of input"

</syntaxhighlight>
Additionally, since we started the name off with a <code>_</code>, we could also find that if we did

<syntaxhighlight lang="python">from tartanpy.utils import *</syntaxhighlight>
which should import everything from the <code>util.py</code> file we won’t actually have access to it, whereas

<syntaxhighlight lang="python">from tartanpy.communicate import *</syntaxhighlight>
would get access to <code>call_scotty_demo</code> even if it wasn’t in the <code>__init__.py</code> file.

</li>
<li>'''Numerical instablity'''

We included an example where floating point differnces made <code>.3-.2</code> not equal to <code>.1</code>. To avoid such errors we recommend using tools like

* <code>np.allclose</code> (which allows user to provide relative and absolute tolerance between vectors)
* or do something like <code>np.abs(a-b)<1e-08</code> or <code>np.abs(a-b)/np.abs(b) < 1e-08</code>
</li></ol>

=== Step 6. Sharing / using on other computers ===

You could publish your package to pypi (documented here: [https://py-pkgs.org/07-releasing-versioning py-pkg/07-releasing-versioning]) but if you’re still actively working on a developer version another easy way is to make sure you have a clean copy on github, have the user clone the repo and then run either (when in the top directory of the cloned repo)

<syntaxhighlight lang="bash">pip install .</syntaxhighlight>
or

<syntaxhighlight lang="bash">pip install -e .</syntaxhighlight>
where the first installs the package 1 time, and the second actually installs the package and updates the package if that user was to do more pulls / updates the the package.



=== Step 7. And beyond! ===

In the above example, we still didn’t cover a lot of things, including:

*publishing the package 
*publishing the docs 
*including data 
*CI/CD (which will be provided in another wiki) 

See [https://py-pkgs.org py-pkgs] for some good starting blocks for most of these.