# How to split large Python Functions across multiple files

## Are your Python functions bringing you joy? Lucas Roesler shares his tips on how to split your Python functions into multiple modules to help your project grow

January 29, 2019  
Python

Using multiple files to create sub-modules helps keep the code organized and makes reusing code between projects/functions much easier. Functions and variables defined within a module importable into other modules and allows you to scope your function and variable names without worrying about conflicts. From [the Python docs](https://docs.python.org/3/tutorial/modules.html#more-on-modules):

> Each module has its own private symbol table, which is used as the global symbol table by all functions defined in the module. Thus, the author of a module can use global variables in the module without worrying about accidental clashes with a user’s global variables.

Today, I will demonstrate how to do this in OpenFaaS by creating a Python 3 word count function that is split across multiple files.

You should have a basic knowledge of OpenFaaS and Python, but if you’re new you can get up to speed using the [workshop](https://github.com/openfaas/workshop). I want to focus on how to take a “one-liner” function and turn it into a larger multi-module project.

## Start the function

Next, to start a new Python 3 function, I like to use the flask template because I am going to load a static list of “stop words” into memory when the function starts. Using the flask templates allows the function to do this only once instead of on each invocation.

- Let’s create a new folder to work in

```
mkdir -p ~/dev/multi-module
cd ~/dev/multi-module
```

- Type in `docker login`

- Change `--prefix` to your Docker Hub account or your private registry

- Now pull in the `python3-flask` template and create a new function named `wordcount`

```
$ faas-cli template store pull python3-flask
$ faas-cli new wordcount --lang python3-flask --prefix=USERNAME
```

The project should now look like

```txt
.
├── stack.yaml
├── template
│   ├── python27-flask
│   │   ├ ...
│   ├── python3-flask
│   │   ├ ...
│   └── python3-flask-armhf
│       ├ ...
└── wordcount
    ├── __init__.py
    ├── handler.py
    └── requirements.txt
```

## Relative imports

Python allows you to reuse code from other files [by `import`ing it](https://www.python.org/dev/peps/pep-0328/). For functions, [relative imports](https://realpython.com/absolute-vs-relative-python-imports/#relative-imports) allow us to specify those resources relative to the file doing the import. This provides maximum flexibility because the import is then agnostic to how/where the package is installed, which is perfect for OpenFaaS functions.

We’re going to start by creating two more files:

```
touch wordcount/stopwords
touch wordcount/wordcount.py
```

The project should now look like this

```txt
.
├── stack.yaml
├── template
│   ├── ...
└── wordcount
    ├── __init__.py
    ├── handler.py
    ├── requirements.txt
    ├── stopwords
    └── wordcount.py
```

The `stopwords` is a plain text file of words that will be excluded in the wordcount. These are short common words that you would not want to include in a wordcount visualization such as:

```txt
a
an
him
her
```

This list of words will depend on your use case and local, you should add more to match your needs, for example [the 100 most common English words](https://www.espressoenglish.net/the-100-most-common-words-in-english/) or [the 100 most common French words](https://www.vistawide.com/french/top_100_french_words.htm).

All of the code for processing text and generating the counts is found in `wordcount.py`:

```
# modified wordcount.py
import unicodedata
import os
from typing import Dict, List

from operator import itemgetter
from collections import defaultdict

FILE = os.path.dirname(__file__)
STOPWORDS = set(
    map(str.lower,
        map(str.strip,
            open(os.path.join(FILE, 'stopwords'), 'r').readlines()
            )
        )
)

def process_text(text: bytes) -> Dict[str, int]:
    """Splits a long text into words a count of interesting words in the text.

process_text will eliminate any of the stopwords, punctuation, and normalize
    the text to merge cases and plurals into a single value.
    """
    words = text.decode("utf-8").split()
    # remove stopwords
    # remove 's
    words = [\
        word[:-2] if word.lower().endswith(("'s",))\
        else word\
        for word in words\
    ]
    # remove numbers
    words = [word for word in words if not word.isdigit()]
    words = [strip_punctuation(word) for word in words]
    words = [word for word in words if word.lower() not in STOPWORDS]

return process_tokens(words)

def process_tokens(words: List[str]) -> Dict[str, int]:
    """Normalize cases and remove plurals.
    """
    # ...

def strip_punctuation(text: str) -> str:
    # ...
```

Note that processing the `STOPWORDS` at the start of the file means it will be loaded into memory once the package is imported rather than on every request, which would create additional latency and I/O overhead.

Using a relative import, we can very easily use the `process_text` method to create a very simple `handler.py`:

```
# handler.py
import json
from .wordcount import process_text

def handle(req):
    """handle a request to the function
    Args:
        req (str): request body
    """

return json.dumps(process_text(req))
```

This style of relative imports will work for any file or sub-package you include inside of your function folder. Additionally, your IDE and linter will be able to resolve the imported code correctly!

## Deploy and test the function

You should have [OpenFaaS deployed](https://docs.openfaas.com/deployment/) and have run `faas-cli login` already.

```
$  faas-cli up -f wordcount.yml
# Docker build output ...
Deploying: wordcount.

Deployed. 202 Accepted.
URL: http://127.0.0.1:31112/function/wordcount

$ echo \
  'This is some example text that we want to see a frequency response for.  It has text like apple, apples, apple tree, etc' \
  | faas-cli -f wordcount.yml invoke wordcount

{"example": 1, "text": 2, "want": 1, "see": 1, "frequency": 1, "response": 1, "for": 1, "apple": 3, "tree": 1, "etc": 1}
```

## A note on Python 2 usage

Using the [`__future__` package](https://docs.python.org/2/library/__future__.html) you can get the same behavior in your Python 2 functions. Add `from __future__ import absolute_import` as the first import in `handler.py` and `wordcount.py` to ensure that the relative imports are resolved correctly.

Note, Python 2 is [End Of Life this year](https://legacy.python.org/dev/peps/pep-0373/) and will not receive any bugfix releases after 2020. If you are still transitioning to Python 3, use the `__future__` package to help smooth the transition.

## Wrapping up

When we use relative imports then we can easily split our code over several files for better organisation. We could take this further and import from sub-folders or sub-folders of sub-folders. This has the added benefit that the code is valid in both your local environment and the final docker container. Try the [completed code example in this repo.](https://github.com/LucasRoesler/openfaas-multifile-example)

Checkout the [OpenFaas Workshop](https://github.com/openfaas/workshop) for a step-by-step guide of writing and deploying a Python function detailing the other features of OpenFaas: asynchronous functions, timeouts, auto-scaling, and managing secret values.