Repeatedly writing the same Python scripts is tedious, and eventually, bad scripting practices can take over. Personal scripts quickly become team scripts, changes are not documented, information silos form, and the list goes on. As time passes and teams get relaxed, these bad practices can slowly take over until they suddenly find themselves in a dark and confusing place. Welcome to script hell.

Using modularization and packaging can break down large scripts into smaller manageable files. Modularization is effective, but it needs to be done the right way or you could go from script hell to package hell. This article will teach you how to make Python packages that keep you out of script hell and on the road to Python mastery.

Script Hell

Script hell is a place where chaos rules. In script hell, every change made to the code breaks two or three other parts. No change is trivial. Every addition or modification to the system requires that the tangles, twists, and knots be “understood” so that more tangles, twists, and knots can be added.

Over time the code becomes a labyrinth that is impossible to organize or manage effectively. Once in script hell teams that were moving very fast at the beginning of a project can find themselves moving at a snail’s pace.

What is Modularization and How Does it Help With Script Hell?

Modularization is the technique of splitting a large programming task into smaller, separate, and manageable subtasks. Like most modern programming languages, Python is a modular programming language. Python scripts are modularized through functions, modules, and packages.

Functions allow developers to reduce repetition in their code by executing the same block of code multiple times in one program. By giving the function a name, it can be reused by referencing it’s name instead of rewriting the code. For example, consider a simple function that greets a user after entering their name.

def greet(name):
    print("Hello, " + name + ". Good morning!")

When we run the code, nothing happens but if we reference the function name like so:

def greet(name):
    print("Hello, " + name + ". Good morning!")
greet('Michael')

The code returns:

Hello, Michael. Good morning!

Modules are “. py” files containing Python definitions and statements. You’re certainly already using modules without even realizing it!

A package is a set of modules (i.e. .py files) organized in folders and subfolders. Python accesses the modules in a package by referencing the package name.

In the above example, the various modules are organized in relevant folders like templates and website. By referencing the package name “templates”, Python can access the modules inside the package.

Packages are at the heart of code reuse, and are likely something you’re already using with PyPI (free and open-source Python packages) and “pip install”.

Creating and Distributing Packages

Python packages can either act as “import” packages or “distribution” packages. Both package types are important to escaping Python script hell. Import packages allow you to modularize code while distribution packages allow you to distribute the code you’ve modularized.

• Import packages (.py files) contain reusable modules that can be loaded into a Python interpreter to execute various commands.
• Distribution packages (zip files) have an archive that contains a library of reusable modules and metadata about the library.

Creating a Python package is simple enough that anyone with Python or coding experience can create one, but make sure to follow best practices when doing so. There are many ways to make Python packages, the most common being setuptools and then uploading it via twine, but ultimately it depends on the developer’s preferences.

How to Make Better Packages

Learning how to create a Python Package can be done in a single sitting, but if you’re not confident in your ability to create high-quality packages you’re better off using scripts.

While packages are extremely useful, a bad package can do much more damage than a bad script. One or two mediocre packages are negligible, but when you start including dozens of low-quality packages you can quickly end up in a place worse than script hell: package hell. Scripts are more likely to be higher quality due to them being reviewed before running. Users expect packages to “just work” and often will not bother doing quality checks.

Creating good packages takes years of practice and experience. Packages can be difficult to manage efficiently thanks to versioning and package dependencies. Investing the time to learn how to create good packages will keep you out of both scripts and package hell. A bad package may “work” in the beginning, but it will make life difficult for whoever needs to maintain the package in the future.

Better Packages through Better Coding

Learning how to create good Python packages and implementing modularization will take time, especially if your team was near or in script hell beforehand.

A good thing to internalize early on is the fact that not everything needs to be modularized. Creating packages and modularizing everything is understandably very tempting. However, having endless unorganized packages is guaranteed to earn you a ticket straight to package hell. Knowing what to package is difficult and takes years of experience to do correctly, thus the person deciding what to modularize should have strong institutional knowledge.

To avoid future headaches, it’s a good idea to identify which copy/paste functions you will be using and double-check their quality. Remember a good quality script function is infinitely better than a mediocre package. There’s no magic formula to determine the right number of packages to use, but be aware that too few and too many packages are both forms of package hell.

Better Packages Through Better Packaging

The first step to making better packages is to accept that no matter your skill level you will make mistakes and improve in the process. The following skills are not easy, but they are worth learning if you want to avoid slipping into script hell.

Package Versioning

When you make changes to a file create a new version with an updated version number. Versioning needs to be organized, have meaning, and be strictly followed. Python doesn’t fully support Semantic Versioning, but we recommend using three-part versioning in the same manner. Versioning is so important that we wrote an entire article on how to version Python packages.

Source Control

Once you publish a package you should always know exactly where the original code (.py file) is. Use source control to manage your packages rather than downloading, editing, changing their version, and republishing. ProGet can help with this by “repackaging” packages and maintaining an audit trail within the package.

Change Control

Remember that script hell is utter chaos. What’s the opposite of chaos? Order. It’s important to have a clear outline of who is responsible for changes and how they will be updating packages. This will take an upfront resource investment, but it’s well worth the number of headaches it will avoid.

Documentation

The key to maintaining long-term good packaging practices is keeping accurate and up-to-date documentation. It’s really easy to get carried away and make too many packages. If this happens AND you weren’t keeping good documentation on the packages, you will certainly have a long and confusing journey out of package hell. It’s also imperative to document changes made to packages because you may not be the only one maintaining them.

Automation

Once you have implemented these better packaging practices it’s time to think like a developer and automate! Establishing a Software Development Lifecycle (SDLC) that incorporates versioning, source control, change control, and documentation will have your team modulating like a well-oiled machine.

Be Wise When You Modularize

Python modularization can keep you out of script hell as long as you plan ahead and follow clear practices for maintaining and creating high-quality Python packages.

Creating high-quality Python packages is an important skill. However, there are many other Python skills, like versioning and package approval workflows, that are necessary too. Read our Effective Package Management in Python guide to learn all the skills necessary to turn your team into Python experts.