Menu
Free Pack
Download BuildMaster Free Trial

Python Management in the Enterprise for 2022

by Crista Perlton, on Apr 6, 2022 11:12:00 AM

Python was ranked the second most popular coding language in a 2021 GitLab survey, raising from fourth place in 2014.

What is causing this upward trend?

Any Python developer will tell you it’s obvious: Python is a versatile language, has a widely diverse open-source community, and is easy to start.

It’s ease of access, however, is a double-edged sword. It’s a good language for beginners, but it creates a huge knowledge gap between members on one team. How can Managers ensure a cross-trained group when standards aren't required for good scripting?

We’ve curated a series of articles to address the many problems a diverse team of DevOps Engineers, Data Scientists, or anyone in a tech-related field will face when using Python in their operations.

How Exactly is Python Used?

Just as the internet-of-things is expanding to ordinary objects, Python language is being used in multiple fields thanks to its diverse functionality. It can run stats on returns in fintech, it can be scripted to run jobs in DevOps, it can collect, scrape, and clean data for academic research – it can even help social media marketing teams analyze their KPIs.

Teams choose Python because it can create scripts, if not full applications. Strict standards don’t need to be applied to Python, unlike C# or Java, so skill level and language fluence are more open.

A user doesn’t have to be an expert to expertly use Python for their use-case.

The versatility of the language explains why so many non-developers use it in their daily operations. Python is great for data processing, statistical analysis, finance, machine learning, deep learning, network automation, artificial intelligence, and so on.

Python Graphic User Interfaces

A.K.A. a Python GUI, is a great option for those not comfortable writing Python code.

Although it is a beginner friendly language, it can still be difficult to learn for non-developers. Python needs to have accurate command-line arguments and any errors could lead to serious consequences.

Teams can avoid these learning gaps by introducing a GUI into their system. Teams with a Python GUI can safety and confidently:

  • run Python scripts independently
  • provide a job-specific interface to enforce limited input
  • limit who can do what with a Python script in that given situation
  • give immediate visual feedback (aka lets you know when you’ve forgotten a required form, for example)

Since they are meant to be intuitive and easy-to-use for anyone, it makes inaccessible technology accessible (especially for Linux).

Teams can create original GUIs through their own system or tools like Tkinter. We recommend creating Python scripts in Otter’s GUI-based development tool. Otter can run the scripts, but also help with the learning curve of your company’s unique job processes.

Python Packages

Developers and non-developers alike can rely on the open-source community to jump-start their projects thanks to the hundreds of thousands of Python packages available online.

Some popular Python packages include:

  • pandas: a fast, powerful, flexible and easy to use open source data analysis and manipulation tool. It’s known as a fast, efficient, and easy-to-use tool for data analysis and manipulation.
  • NumPy: a Python library that provides a multidimensional array object, an assortment of routines for fast operations on arrays, and much more.
  • TensorFlow: an end-to-end open source platform for machine learning. It has a comprehensive ecosystem of tools, libraries, and community resources that lets researchers and developers easily build and deploy ML powered applications.

Developers can rely on these packages to cut down on production costs and avoid redundant work (why reinvent the wheel when it’s available and free on PyPi.org).

Just ensure you have a Package Approval Process before letting a team have free-reign on open-source sites.

Python Package Managers

Python’s true power tool is the ecosystem of free and open-source libraries like Tensorflow, Netmiko, and Flask. These can be installed with a single command using a package manager.

Most Python users have a preferred manager. Some common ones are:

PyPI: The Package Index

Python’s own official third-party software repository. The Python Package Index (PyPI) is a repository of software that hosts an extensive collection of Python packages, development frameworks, tools, and libraries.

Pip: The Standard Package Manager

Pip is built into Python and can install packages from many different sources with PyPI.org as the primary and default package source used.

Virtual Environments & Virtualenv

In the Python world, a virtual environment is a folder containing packages and other dependencies that a Python project needs. The purpose of these environments is to keep projects separate and prevent dependency, version, and permission conflicts.

Problems with Python Packages

The downside to the huge available of open-source Python packages can be boiled down to three main issues:

#1: License Agreements

Causally downloading a package from PyPI.org and integrating it into your project without checking it’s meta could have serious consequences.

Malicious packages or unacceptable licenses must be avoided while browsing open-source sites. If you or your team uses PyPI.org or third-party packages, consider including Python packages into your licensing policy.

Integrating Python packages into an organization’s existing third-party software policy allows developers to download from PyPI.org without constantly checking metadata because it’s already been done for them.

Adding packages to an existing licensing policy is easy: just ask for permission. A team lead or department head can set standards via a manual approval process or automatic tool like ProGet and know they’re not at risk of legal trouble.

#2: Vulnerabilities

Did you hear about the malicious PyPI package that collects environment details and sends the info to an unknown webservice?

A great way to prevent these possibly dangerous packages making it through to production is by setting up a Package Approval Workflow and Connector Filters.

A package approval workflow is just like a code review but for open-source packages from PyPI. To use a package in a project, it’s reviewed by a trained "Approver.” Once reviewed and approved it’s made available to Python users, otherwise it’s blocked from organization use.

This ensures that a trained set of eyes get on every single PyPI package before it makes its way to Python coders or eventually production. The PyPI package is then put into a repository like ProGet so all Python users can easily access all the packages they need.

Connector filters can be introduced to any repository you have connecting to PyPI and can filter out pre-approved or even pre-denied packages. This helps streamline your package approval workflow and gets your developers the packages they need quickly.

#3: Managing Python Dependencies

Python packages often depend on other packages known as dependencies. These dependencies can have their own dependencies, resulting in a complicated dependency tree.

If you’re building an application with Python and two packages require different versions of the same package, then Python will have a version conflict and your project may not build.

Fortunately, there’s two easy ways to deal with complex dependencies:

  • Use Requirements.txt for Repeatable Builds
  • Use Package Consumers to Track Dependencies

Utilizing requirements.txt files, a package approval workflow, and ProGet’s Package Consumer will help ensure predictable builds and keep unwanted packages out of them.

Creating Python Packages

Many organizations choose to utilize third-party packages from PyPI, but many others create their own proprietary packages, never to be shared in the open-source community.

Making a Python package is like making a zip file with metadata. There are multiple tools available to make a package like setuptools and then uploading via twine, but ultimately it depends on the developer’s preferences.

We recommend using the following four best practices when creating Python packages. Since Python users are so diverse, setting these standards can help make distribution of packages more efficient.

  1. Use One Repository & Wiki Per Library
  2. Keep Metadata Simple
  3. Use SemVer to help with Versioning and Dependency Tracking
  4. Use Wheels for Built Distribution

Integrating CI/CD into Your Python Development

The two aren’t often discussed together, but CI/CD principals and best practices can 100% apply to Python development.

Every heard of a Python Pull and Pray? When you clone your application using git, install the dependencies with pip, and hope it all works out? This method causes a lot of problems: bloated deployment time; errors in the build; and no guarantee the app is the same server-to-server.

Enter CI/CD for Python: create and maintain a stable base, clone and commit frequently, and test rapidly.

It instantly results in faster deployments thanks to no install bloat. The process eliminates having to manually watch an execution, so Python users can take that saved time and to put it towards more developing.

It’s easy to set up a Python CI/CD pipeline and a CI/CD development pipeline is entirely customizable to an organization’s needs or a team’s preferences.

Optimizing Python for Your Projects

The huge potential of Python is reflected in the multiple industries it’s used in. Thanks to that, Python users are becoming more and more common, but that also means the knowledge gap is getting bigger and bigger.

In order to catch an entire team up to the organization’s speed, a manager could put hours into training, or they can set standards everyone can work in thanks to tools like ProGet and Otter.

Rely on Otter’s GUI-based server-configuration to create reoccurring jobs or use ProGet to filter out unwanted packages and improve security.

This article was only the tip of the iceberg, consider reading out more in-depth blog posts about Vulnerability Detection, SemVer & CI/CD for Python, Creating Custom GUIs, and more.

 

Topics:python

Related Posts

About Inedo

Inedo is a software product company bringing you the "tech behind the tech."

Makers of Windows-first, enterprise DevOps tools BuildMaster CI/CD, ProGet private package management, and Otter IaC. Maximize developer time, minimize release risk, and empower stakeholders to bring their vision to life faster, all with the people and technology you have right now.

Follow us on social media

Follow Inedo on YouTube Follow Inedo on Facebook Follow Inedo Twitter New call-to-action

Free e-books

Free PowerShell Book NuGet for the Enterprise Guide Jenkins CICD Guide Free CICD Book Free dotnet book free IaC book